[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664647#comment-16664647 ] ASF GitHub Bot commented on GRIFFIN-203: Github user gavlyukovskiy commented on the issue: https://github.com/apache/incubator-griffin/pull/446 Thank you everyone for feedback, I'm not UI guy so structure was messed little bit :) I changed it to be consistent with other pages and now it looks much better. > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-208) Job status is SUCCESS even if some stages have failed
[
https://issues.apache.org/jira/browse/GRIFFIN-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664575#comment-16664575
]
ASF GitHub Bot commented on GRIFFIN-208:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/448#discussion_r228391307
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/Application.scala ---
@@ -104,12 +106,18 @@ object Application extends Loggable {
case Success(_) =>
info("process end success")
case Failure(ex) =>
-error(s"process end error: ${ex.getMessage}")
+error(s"process end error: ${ex.getMessage}", ex)
shutdown
sys.exit(-5)
}
shutdown
+
+if (success) {
--- End diff --
removed exit(0), looks like org.apache.spark.deploy.yarn.ApplicationMaster
is not liking direct exit(0)
> Job status is SUCCESS even if some stages have failed
> -
>
> Key: GRIFFIN-208
> URL: https://issues.apache.org/jira/browse/GRIFFIN-208
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Major
>
> When some steps (MetricWrite or SparkSql, for example) fail, errors are just
> logged, but not reported as part of job status. Symptoms:
> {code:none}
> 18/10/22 17:17:58 ERROR transform.SparkSqlTransformStep: run spark sql [
> ] error: ...
> {code}
> YarnApplicationState: FINISHED
> FinalStatus Reported by AM: SUCCEEDED
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin issue #446: [GRIFFIN-203] "Plaintext mode" for measure cre...
Github user icesmartjuan commented on the issue: https://github.com/apache/incubator-griffin/pull/446 Awesome feature! There seems an issue when zoom up the page, the textare is outside and the ```submit``` button can't be clicked/triggered.  ---
[jira] [Commented] (GRIFFIN-208) Job status is SUCCESS even if some stages have failed
[
https://issues.apache.org/jira/browse/GRIFFIN-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664516#comment-16664516
]
ASF GitHub Bot commented on GRIFFIN-208:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/448#discussion_r228385053
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/Application.scala ---
@@ -104,12 +106,18 @@ object Application extends Loggable {
case Success(_) =>
info("process end success")
case Failure(ex) =>
-error(s"process end error: ${ex.getMessage}")
+error(s"process end error: ${ex.getMessage}", ex)
shutdown
sys.exit(-5)
}
shutdown
+
+if (success) {
--- End diff --
looks like this one breaks a lot of stuff
> Job status is SUCCESS even if some stages have failed
> -
>
> Key: GRIFFIN-208
> URL: https://issues.apache.org/jira/browse/GRIFFIN-208
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Major
>
> When some steps (MetricWrite or SparkSql, for example) fail, errors are just
> logged, but not reported as part of job status. Symptoms:
> {code:none}
> 18/10/22 17:17:58 ERROR transform.SparkSqlTransformStep: run spark sql [
> ] error: ...
> {code}
> YarnApplicationState: FINISHED
> FinalStatus Reported by AM: SUCCEEDED
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #448: [GRIFFIN-208] log exception details whe...
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/448#discussion_r228385053
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/Application.scala ---
@@ -104,12 +106,18 @@ object Application extends Loggable {
case Success(_) =>
info("process end success")
case Failure(ex) =>
-error(s"process end error: ${ex.getMessage}")
+error(s"process end error: ${ex.getMessage}", ex)
shutdown
sys.exit(-5)
}
shutdown
+
+if (success) {
--- End diff --
looks like this one breaks a lot of stuff
---
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664282#comment-16664282 ] ASF GitHub Bot commented on GRIFFIN-203: Github user chemikadze commented on the issue: https://github.com/apache/incubator-griffin/pull/446 Looks like there are still some rendering issues: https://user-images.githubusercontent.com/867294/47530131-9ac61b80-d85e-11e8-8bf1-aa825105ff52.png;> > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] incubator-griffin issue #446: [GRIFFIN-203] "Plaintext mode" for measure cre...
Github user chemikadze commented on the issue: https://github.com/apache/incubator-griffin/pull/446 Looks like there are still some rendering issues: https://user-images.githubusercontent.com/867294/47530131-9ac61b80-d85e-11e8-8bf1-aa825105ff52.png;> ---
[jira] [Commented] (GRIFFIN-208) Job status is SUCCESS even if some stages have failed
[
https://issues.apache.org/jira/browse/GRIFFIN-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664119#comment-16664119
]
ASF GitHub Bot commented on GRIFFIN-208:
GitHub user chemikadze opened a pull request:
https://github.com/apache/incubator-griffin/pull/448
[GRIFFIN-208] log exception details when it makes sense
Sometimes error messages are not descriptive enough and do not allow
to pinpoint exact issue. Exception stack traces should make it easier
to troubleshoot issues.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-208
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-griffin/pull/448.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #448
commit cb7e1225a2b6a76a0dcbd057a1e8c625ed8cff1d
Author: Nikolay Sokolov
Date: 2018-10-25T18:18:17Z
[GRIFFIN-208] log exception details when it makes sense
Sometimes error messages are not descriptive enough and do not allow
to pinpoint exact issue. Exception stack traces should make it easier
to troubleshoot issues.
> Job status is SUCCESS even if some stages have failed
> -
>
> Key: GRIFFIN-208
> URL: https://issues.apache.org/jira/browse/GRIFFIN-208
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Major
>
> When some steps (MetricWrite or SparkSql, for example) fail, errors are just
> logged, but not reported as part of job status. Symptoms:
> {code:none}
> 18/10/22 17:17:58 ERROR transform.SparkSqlTransformStep: run spark sql [
> ] error: ...
> {code}
> YarnApplicationState: FINISHED
> FinalStatus Reported by AM: SUCCEEDED
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #448: [GRIFFIN-208] log exception details whe...
GitHub user chemikadze opened a pull request: https://github.com/apache/incubator-griffin/pull/448 [GRIFFIN-208] log exception details when it makes sense Sometimes error messages are not descriptive enough and do not allow to pinpoint exact issue. Exception stack traces should make it easier to troubleshoot issues. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-208 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/448.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #448 commit cb7e1225a2b6a76a0dcbd057a1e8c625ed8cff1d Author: Nikolay Sokolov Date: 2018-10-25T18:18:17Z [GRIFFIN-208] log exception details when it makes sense Sometimes error messages are not descriptive enough and do not allow to pinpoint exact issue. Exception stack traces should make it easier to troubleshoot issues. ---
[jira] [Commented] (GRIFFIN-205) Accuracy measure check should provide matchedFraction to store
[
https://issues.apache.org/jira/browse/GRIFFIN-205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664081#comment-16664081
]
Nikolay Sokolov commented on GRIFFIN-205:
-
[~artem.shutak] as we've merged this change, can you please close the ticket?
> Accuracy measure check should provide matchedFraction to store
> --
>
> Key: GRIFFIN-205
> URL: https://issues.apache.org/jira/browse/GRIFFIN-205
> Project: Griffin (Incubating)
> Issue Type: Improvement
> Components: accuracy-batch, accuracy-real-time
>Affects Versions: 0.3.1-incubating
>Reporter: Artem Shutak
>Assignee: Artem Shutak
>Priority: Major
>
> Currently, {{accuracy}} measure results contains "total", "miss" and
> "matched" counts.
> As a result, It's hard to analyze accuracy fraction based on results stored
> in ElasticSearch, because ElasticSearch does not provide straight forward
> capability to get "field divided by field" query results.
> {{Accuracy}} measure results should also contain {{matchedFraction}} field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #442: Support "cache" field on rules in servi...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/442 ---
[GitHub] incubator-griffin pull request #442: Support "cache" field on rules in servi...
Github user guoyuepeng commented on a diff in the pull request: https://github.com/apache/incubator-griffin/pull/442#discussion_r228217972 --- Diff: service/src/main/java/org/apache/griffin/core/measure/entity/Rule.java --- @@ -81,6 +81,9 @@ Licensed to the Apache Software Foundation (ASF) under one @Column(name = "\"out\"") private String out; +@JsonInclude(JsonInclude.Include.NON_NULL) --- End diff -- I think the default value is false by current implementation. ---
[GitHub] incubator-griffin pull request #442: Support "cache" field on rules in servi...
Github user bhlx3lyx7 commented on a diff in the pull request: https://github.com/apache/incubator-griffin/pull/442#discussion_r228215711 --- Diff: service/src/main/java/org/apache/griffin/core/measure/entity/Rule.java --- @@ -81,6 +81,9 @@ Licensed to the Apache Software Foundation (ASF) under one @Column(name = "\"out\"") private String out; +@JsonInclude(JsonInclude.Include.NON_NULL) --- End diff -- This parameter `cache` in rule configuration indicates spark will cache the output data frame, to avoid redundant calculation of the same data frame if it is used twice or more. In "griffin-dsl" mode, the rule will be translated internally, and the `cache` parameter would be filled by the translator. That's why it only works in "spark-sql" and "df-ops" mode. In the current versions, UI doesn't support user defined "spark-sql" rules or "df-ops" rules, the `cache` parameter seems confusing for users either. But if we want to support user defined "spark-sql" rules, no matter through UI or service, it would be better to make the parameters configurable. It's ok to set it optional, I prefer to set it `false` by default. ---
[GitHub] incubator-griffin issue #446: [GRIFFIN-203] "Plaintext mode" for measure cre...
Github user guoyuepeng commented on the issue: https://github.com/apache/incubator-griffin/pull/446 Awesome! ---
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663842#comment-16663842 ] ASF GitHub Bot commented on GRIFFIN-203: Github user guoyuepeng commented on the issue: https://github.com/apache/incubator-griffin/pull/446 Awesome! > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] incubator-griffin pull request #445: Update deploy-guide.md
Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/445 ---
[GitHub] incubator-griffin pull request #443: Fix conventions in batch-sample
Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/443 ---
[jira] [Resolved] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
William Guo resolved GRIFFIN-207.
-
Resolution: Fixed
Fix Version/s: 0.3.1-incubating
Issue resolved by pull request 441
[https://github.com/apache/incubator-griffin/pull/441]
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
> Fix For: 0.3.1-incubating
>
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663802#comment-16663802
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-griffin/pull/441
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
> Fix For: 0.3.1-incubating
>
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #441: [GRIFFIN-207] LDAP login service improv...
Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/441 ---
[GitHub] incubator-griffin pull request #447: [GRIFFIN-194] single call to fetch all ...
GitHub user chemikadze opened a pull request: https://github.com/apache/incubator-griffin/pull/447 [GRIFFIN-194] single call to fetch all tables of all dbs Significantly improves opening speed for measure dialogs. Previously taking 10+ seconds on huge schemas, with this call it takes fraction of second to load. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-194 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/447.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #447 commit 68ab221ddba6526143f40ab372b3af8c328cf6a6 Author: Nikolay Sokolov Date: 2018-10-25T06:58:42Z [GRIFFIN-194] single call to fetch all tables of all dbs Significantly improves opening speed for measure dialogs. Previously taking 10+ seconds on huge schemas, with this call it takes fraction of second to load. ---
[jira] [Commented] (GRIFFIN-194) [service] Hive API improvement
[ https://issues.apache.org/jira/browse/GRIFFIN-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663346#comment-16663346 ] ASF GitHub Bot commented on GRIFFIN-194: GitHub user chemikadze opened a pull request: https://github.com/apache/incubator-griffin/pull/447 [GRIFFIN-194] single call to fetch all tables of all dbs Significantly improves opening speed for measure dialogs. Previously taking 10+ seconds on huge schemas, with this call it takes fraction of second to load. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-194 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/447.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #447 commit 68ab221ddba6526143f40ab372b3af8c328cf6a6 Author: Nikolay Sokolov Date: 2018-10-25T06:58:42Z [GRIFFIN-194] single call to fetch all tables of all dbs Significantly improves opening speed for measure dialogs. Previously taking 10+ seconds on huge schemas, with this call it takes fraction of second to load. > [service] Hive API improvement > -- > > Key: GRIFFIN-194 > URL: https://issues.apache.org/jira/browse/GRIFFIN-194 > Project: Griffin (Incubating) > Issue Type: Sub-task >Reporter: Nikolay Sokolov >Priority: Minor > > Purpose is mainly to support GRIFFIN-195 with single request to get table > list information, while avoiding transferring all table metadata and making > lots of metastore requests. > Hive API provides following relevant APIs right now: > * listing DBs > * getting all table names in DB > * listing all table _objects_ in all _dbs_ > What's seems to be missing, is API call for all table names in all DBs (as > middle ground between n+1 API requests and 1 API request with huge payload > and n*m+1 metastore requests on backend). > This api request should take no parameters, and return Map List> in response. > Proposed API endpoint: TBD -- This message was sent by Atlassian JIRA (v7.6.3#76005)
