[jira] [Commented] (GRIFFIN-213) Support pluggable datasource connectors
[
https://issues.apache.org/jira/browse/GRIFFIN-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694294#comment-16694294
]
ASF GitHub Bot commented on GRIFFIN-213:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/456#discussion_r235266418
--- Diff: griffin-doc/measure/measure-configuration-guide.md ---
@@ -188,7 +188,7 @@ Above lists DQ job configure parameters.
- **sinks**: Whitelisted sink types for this job. Note: no sinks will be
used, if empty or omitted.
### Data Connector
-- **type**: Data connector type, "AVRO", "HIVE", "TEXT-DIR" for batch
mode, "KAFKA" for streaming mode.
+- **type**: Data connector type: "AVRO", "HIVE", "TEXT-DIR", "CUSTOM" for
batch mode; "KAFKA", "CUSTOM" for streaming mode.
--- End diff --
ANY seems to have "auto-detect" meaning to it, while in fact it's just
plugin mechanism.
What about CLASS, CLASSNAME, PLUGIN or EXTERNAL?
> Support pluggable datasource connectors
> ---
>
> Key: GRIFFIN-213
> URL: https://issues.apache.org/jira/browse/GRIFFIN-213
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> As of Griffin 0.3, code modification is required, in order to add new data
> connectors.
> Proposal is to add new data connector type, CUSTOM, that would allow to
> specify class name of data connector implementation to use. Additional jars
> with custom connector implementations would be provided in spark
> configuration template.
> Class name would be specified in "class" config of data connector. For
> example:
> {code:json}
> "connectors": [
> {
> "type": "CUSTOM",
> "config": {
> "class": "org.example.griffin.JDBCConnector"
> // extra connector-specific parameters
> }
> }
> ]
> {code}
> Proposed contract for implementations is based on current convention:
> - for batch
> ** class should be a subclass of BatchDataConnector
> ** if should have method with signature:
> {code:java}
> public static BatchDataConnector apply(ctx: BatchDataConnectorContext)
> {code}
> - for streaming
> ** class should be a subclass of StreamingDataConnector
> ** it should have method with signature:
> {code:java}
> public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext)
> {code}
> Signatures of context objects:
> {code:scala}
> case class BatchDataConnectorContext(@transient sparkSession: SparkSession,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage)
> case class StreamingDataConnectorContext(@transient sparkSession:
> SparkSession,
> @transient ssc: StreamingContext,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage,
> streamingCacheClientOpt:
> Option[StreamingCacheClient])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-213) Support pluggable datasource connectors
[
https://issues.apache.org/jira/browse/GRIFFIN-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16693982#comment-16693982
]
ASF GitHub Bot commented on GRIFFIN-213:
Github user guoyuepeng commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/456#discussion_r235226675
--- Diff: griffin-doc/measure/measure-configuration-guide.md ---
@@ -188,7 +188,7 @@ Above lists DQ job configure parameters.
- **sinks**: Whitelisted sink types for this job. Note: no sinks will be
used, if empty or omitted.
### Data Connector
-- **type**: Data connector type, "AVRO", "HIVE", "TEXT-DIR" for batch
mode, "KAFKA" for streaming mode.
+- **type**: Data connector type: "AVRO", "HIVE", "TEXT-DIR", "CUSTOM" for
batch mode; "KAFKA", "CUSTOM" for streaming mode.
--- End diff --
CUSTOM looks good to me.
> Support pluggable datasource connectors
> ---
>
> Key: GRIFFIN-213
> URL: https://issues.apache.org/jira/browse/GRIFFIN-213
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> As of Griffin 0.3, code modification is required, in order to add new data
> connectors.
> Proposal is to add new data connector type, CUSTOM, that would allow to
> specify class name of data connector implementation to use. Additional jars
> with custom connector implementations would be provided in spark
> configuration template.
> Class name would be specified in "class" config of data connector. For
> example:
> {code:json}
> "connectors": [
> {
> "type": "CUSTOM",
> "config": {
> "class": "org.example.griffin.JDBCConnector"
> // extra connector-specific parameters
> }
> }
> ]
> {code}
> Proposed contract for implementations is based on current convention:
> - for batch
> ** class should be a subclass of BatchDataConnector
> ** if should have method with signature:
> {code:java}
> public static BatchDataConnector apply(ctx: BatchDataConnectorContext)
> {code}
> - for streaming
> ** class should be a subclass of StreamingDataConnector
> ** it should have method with signature:
> {code:java}
> public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext)
> {code}
> Signatures of context objects:
> {code:scala}
> case class BatchDataConnectorContext(@transient sparkSession: SparkSession,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage)
> case class StreamingDataConnectorContext(@transient sparkSession:
> SparkSession,
> @transient ssc: StreamingContext,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage,
> streamingCacheClientOpt:
> Option[StreamingCacheClient])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-212) [Measure] Add matchedFraction in streaming accuracy metrics as in batch mode
[ https://issues.apache.org/jira/browse/GRIFFIN-212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16692462#comment-16692462 ] ASF GitHub Bot commented on GRIFFIN-212: Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/455 > [Measure] Add matchedFraction in streaming accuracy metrics as in batch mode > > > Key: GRIFFIN-212 > URL: https://issues.apache.org/jira/browse/GRIFFIN-212 > Project: Griffin (Incubating) > Issue Type: Task >Reporter: Lionel Liu >Assignee: Lionel Liu >Priority: Major > Fix For: 0.3.1-incubating > > > In batch mode, we've added matchedFraction in accuracy metric, we should keep > the metric format in streaming mode align with batch mode. So we need to add > this field in streaming accuracy metric as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-212) [Measure] Add matchedFraction in streaming accuracy metrics as in batch mode
[ https://issues.apache.org/jira/browse/GRIFFIN-212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691428#comment-16691428 ] ASF GitHub Bot commented on GRIFFIN-212: Github user guoyuepeng commented on the issue: https://github.com/apache/incubator-griffin/pull/455 I don't know what is wrong between apache repo and github repo, I had merged/closed this PR, but it still exists here as OPEN. > [Measure] Add matchedFraction in streaming accuracy metrics as in batch mode > > > Key: GRIFFIN-212 > URL: https://issues.apache.org/jira/browse/GRIFFIN-212 > Project: Griffin (Incubating) > Issue Type: Task >Reporter: Lionel Liu >Assignee: Lionel Liu >Priority: Major > Fix For: 0.3.1-incubating > > > In batch mode, we've added matchedFraction in accuracy metric, we should keep > the metric format in streaming mode align with batch mode. So we need to add > this field in streaming accuracy metric as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-213) Support pluggable datasource connectors
[
https://issues.apache.org/jira/browse/GRIFFIN-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16691153#comment-16691153
]
ASF GitHub Bot commented on GRIFFIN-213:
Github user toyboxman commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/456#discussion_r234475362
--- Diff: griffin-doc/measure/measure-configuration-guide.md ---
@@ -188,7 +188,7 @@ Above lists DQ job configure parameters.
- **sinks**: Whitelisted sink types for this job. Note: no sinks will be
used, if empty or omitted.
### Data Connector
-- **type**: Data connector type, "AVRO", "HIVE", "TEXT-DIR" for batch
mode, "KAFKA" for streaming mode.
+- **type**: Data connector type: "AVRO", "HIVE", "TEXT-DIR", "CUSTOM" for
batch mode; "KAFKA", "CUSTOM" for streaming mode.
--- End diff --
do you think about 'ANY' as replacement for 'CUSTOM'
**type**: Data connector type, "AVRO", "HIVE", "TEXT-DIR" for batch mode,
"KAFKA" for streaming mode, "ANY" for boths.
> Support pluggable datasource connectors
> ---
>
> Key: GRIFFIN-213
> URL: https://issues.apache.org/jira/browse/GRIFFIN-213
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> As of Griffin 0.3, code modification is required, in order to add new data
> connectors.
> Proposal is to add new data connector type, CUSTOM, that would allow to
> specify class name of data connector implementation to use. Additional jars
> with custom connector implementations would be provided in spark
> configuration template.
> Class name would be specified in "class" config of data connector. For
> example:
> {code:json}
> "connectors": [
> {
> "type": "CUSTOM",
> "config": {
> "class": "org.example.griffin.JDBCConnector"
> // extra connector-specific parameters
> }
> }
> ]
> {code}
> Proposed contract for implementations is based on current convention:
> - for batch
> ** class should be a subclass of BatchDataConnector
> ** if should have method with signature:
> {code:java}
> public static BatchDataConnector apply(ctx: BatchDataConnectorContext)
> {code}
> - for streaming
> ** class should be a subclass of StreamingDataConnector
> ** it should have method with signature:
> {code:java}
> public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext)
> {code}
> Signatures of context objects:
> {code:scala}
> case class BatchDataConnectorContext(@transient sparkSession: SparkSession,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage)
> case class StreamingDataConnectorContext(@transient sparkSession:
> SparkSession,
> @transient ssc: StreamingContext,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage,
> streamingCacheClientOpt:
> Option[StreamingCacheClient])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-213) Support pluggable datasource connectors
[
https://issues.apache.org/jira/browse/GRIFFIN-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690744#comment-16690744
]
ASF GitHub Bot commented on GRIFFIN-213:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/456#discussion_r234426015
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/datasource/connector/DataConnectorFactory.scala
---
@@ -84,6 +87,26 @@ object DataConnectorFactory extends Loggable {
}
}
+ private def getCustomConnector(session: SparkSession,
+ context: StreamingContext,
+ param: DataConnectorParam,
+ storage: TimestampStorage,
+ maybeClient:
Option[StreamingCacheClient]): DataConnector = {
+val className = param.getConfig("class").asInstanceOf[String]
+val cls = Class.forName(className)
+if (classOf[BatchDataConnector].isAssignableFrom(cls)) {
+ val ctx = BatchDataConnectorContext(session, param, storage)
+ val meth = cls.getDeclaredMethod("apply",
classOf[BatchDataConnectorContext])
+ meth.invoke(null, ctx).asInstanceOf[BatchDataConnector]
+} else if (classOf[StreamingDataConnector].isAssignableFrom(cls)) {
+ val ctx = StreamingDataConnectorContext(session, context, param,
storage, maybeClient)
+ val meth = cls.getDeclaredMethod("apply",
classOf[StreamingDataConnectorContext])
+ meth.invoke(null, ctx).asInstanceOf[StreamingDataConnector]
+} else {
+ throw new ClassCastException("")
--- End diff --
Oh, thanks for reminding! Planned to do that, but got distracted.
> Support pluggable datasource connectors
> ---
>
> Key: GRIFFIN-213
> URL: https://issues.apache.org/jira/browse/GRIFFIN-213
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> As of Griffin 0.3, code modification is required, in order to add new data
> connectors.
> Proposal is to add new data connector type, CUSTOM, that would allow to
> specify class name of data connector implementation to use. Additional jars
> with custom connector implementations would be provided in spark
> configuration template.
> Class name would be specified in "class" config of data connector. For
> example:
> {code:json}
> "connectors": [
> {
> "type": "CUSTOM",
> "config": {
> "class": "org.example.griffin.JDBCConnector"
> // extra connector-specific parameters
> }
> }
> ]
> {code}
> Proposed contract for implementations is based on current convention:
> - for batch
> ** class should be a subclass of BatchDataConnector
> ** if should have method with signature:
> {code:java}
> public static BatchDataConnector apply(ctx: BatchDataConnectorContext)
> {code}
> - for streaming
> ** class should be a subclass of StreamingDataConnector
> ** it should have method with signature:
> {code:java}
> public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext)
> {code}
> Signatures of context objects:
> {code:scala}
> case class BatchDataConnectorContext(@transient sparkSession: SparkSession,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage)
> case class StreamingDataConnectorContext(@transient sparkSession:
> SparkSession,
> @transient ssc: StreamingContext,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage,
> streamingCacheClientOpt:
> Option[StreamingCacheClient])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-213) Support pluggable datasource connectors
[
https://issues.apache.org/jira/browse/GRIFFIN-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690737#comment-16690737
]
ASF GitHub Bot commented on GRIFFIN-213:
Github user gavlyukovskiy commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/456#discussion_r234425707
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/datasource/connector/DataConnectorFactory.scala
---
@@ -84,6 +87,26 @@ object DataConnectorFactory extends Loggable {
}
}
+ private def getCustomConnector(session: SparkSession,
+ context: StreamingContext,
+ param: DataConnectorParam,
+ storage: TimestampStorage,
+ maybeClient:
Option[StreamingCacheClient]): DataConnector = {
+val className = param.getConfig("class").asInstanceOf[String]
+val cls = Class.forName(className)
+if (classOf[BatchDataConnector].isAssignableFrom(cls)) {
+ val ctx = BatchDataConnectorContext(session, param, storage)
+ val meth = cls.getDeclaredMethod("apply",
classOf[BatchDataConnectorContext])
+ meth.invoke(null, ctx).asInstanceOf[BatchDataConnector]
+} else if (classOf[StreamingDataConnector].isAssignableFrom(cls)) {
+ val ctx = StreamingDataConnectorContext(session, context, param,
storage, maybeClient)
+ val meth = cls.getDeclaredMethod("apply",
classOf[StreamingDataConnectorContext])
+ meth.invoke(null, ctx).asInstanceOf[StreamingDataConnector]
+} else {
+ throw new ClassCastException("")
--- End diff --
It would be nice to have here message that custom connector class must
extend `BatchDataConnector` or `StreamingDataConnector`
> Support pluggable datasource connectors
> ---
>
> Key: GRIFFIN-213
> URL: https://issues.apache.org/jira/browse/GRIFFIN-213
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> As of Griffin 0.3, code modification is required, in order to add new data
> connectors.
> Proposal is to add new data connector type, CUSTOM, that would allow to
> specify class name of data connector implementation to use. Additional jars
> with custom connector implementations would be provided in spark
> configuration template.
> Class name would be specified in "class" config of data connector. For
> example:
> {code:json}
> "connectors": [
> {
> "type": "CUSTOM",
> "config": {
> "class": "org.example.griffin.JDBCConnector"
> // extra connector-specific parameters
> }
> }
> ]
> {code}
> Proposed contract for implementations is based on current convention:
> - for batch
> ** class should be a subclass of BatchDataConnector
> ** if should have method with signature:
> {code:java}
> public static BatchDataConnector apply(ctx: BatchDataConnectorContext)
> {code}
> - for streaming
> ** class should be a subclass of StreamingDataConnector
> ** it should have method with signature:
> {code:java}
> public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext)
> {code}
> Signatures of context objects:
> {code:scala}
> case class BatchDataConnectorContext(@transient sparkSession: SparkSession,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage)
> case class StreamingDataConnectorContext(@transient sparkSession:
> SparkSession,
> @transient ssc: StreamingContext,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage,
> streamingCacheClientOpt:
> Option[StreamingCacheClient])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-213) Support pluggable datasource connectors
[
https://issues.apache.org/jira/browse/GRIFFIN-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16690723#comment-16690723
]
ASF GitHub Bot commented on GRIFFIN-213:
GitHub user chemikadze opened a pull request:
https://github.com/apache/incubator-griffin/pull/456
[GRIFFIN-213] Custom connector support
Provide ability to extend batch and streaming data integrations
with custom user-provided connectors. Introduces new data connector
type, `CUSTOM`, parameterized with `class` property. Also adds support
for custom data connector enum on service side.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-213
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-griffin/pull/456.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #456
commit d487347a363f172cfc9e26225d5687cc3f95ab73
Author: Nikolay Sokolov
Date: 2018-11-17T22:37:36Z
[GRIFFIN-213] Custom connector support
Provide ability to extend batch and streaming data integrations
with custom user-provided connectors. Introduces new data connector
type, `CUSTOM`, parameterized with `class` property. Also adds support
for custom data connector enum on service side.
> Support pluggable datasource connectors
> ---
>
> Key: GRIFFIN-213
> URL: https://issues.apache.org/jira/browse/GRIFFIN-213
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> As of Griffin 0.3, code modification is required, in order to add new data
> connectors.
> Proposal is to add new data connector type, CUSTOM, that would allow to
> specify class name of data connector implementation to use. Additional jars
> with custom connector implementations would be provided in spark
> configuration template.
> Class name would be specified in "class" config of data connector. For
> example:
> {code:json}
> "connectors": [
> {
> "type": "CUSTOM",
> "config": {
> "class": "org.example.griffin.JDBCConnector"
> // extra connector-specific parameters
> }
> }
> ]
> {code}
> Proposed contract for implementations is based on current convention:
> - for batch
> ** class should be a subclass of BatchDataConnector
> ** if should have method with signature:
> {code:java}
> public static BatchDataConnector apply(ctx: BatchDataConnectorContext)
> {code}
> - for streaming
> ** class should be a subclass of StreamingDataConnector
> ** it should have method with signature:
> {code:java}
> public static StreamingDataConnector apply(ctx: StreamingDataConnectorContext)
> {code}
> Signatures of context objects:
> {code:scala}
> case class BatchDataConnectorContext(@transient sparkSession: SparkSession,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage)
> case class StreamingDataConnectorContext(@transient sparkSession:
> SparkSession,
> @transient ssc: StreamingContext,
> dcParam: DataConnectorParam,
> timestampStorage: TimestampStorage,
> streamingCacheClientOpt:
> Option[StreamingCacheClient])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-212) [Measure] Add matchedFraction in streaming accuracy metrics as in batch mode
[ https://issues.apache.org/jira/browse/GRIFFIN-212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689075#comment-16689075 ] ASF GitHub Bot commented on GRIFFIN-212: Github user toyboxman commented on the issue: https://github.com/apache/incubator-griffin/pull/455 I'm fine on this change > [Measure] Add matchedFraction in streaming accuracy metrics as in batch mode > > > Key: GRIFFIN-212 > URL: https://issues.apache.org/jira/browse/GRIFFIN-212 > Project: Griffin (Incubating) > Issue Type: Task >Reporter: Lionel Liu >Assignee: Lionel Liu >Priority: Major > > In batch mode, we've added matchedFraction in accuracy metric, we should keep > the metric format in streaming mode align with batch mode. So we need to add > this field in streaming accuracy metric as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-212) [Measure] Add matchedFraction in streaming accuracy metrics as in batch mode
[ https://issues.apache.org/jira/browse/GRIFFIN-212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16689047#comment-16689047 ] ASF GitHub Bot commented on GRIFFIN-212: GitHub user bhlx3lyx7 opened a pull request: https://github.com/apache/incubator-griffin/pull/455 [GRIFFIN-212] Add matchedFraction in streaming accuracy metrics You can merge this pull request into a Git repository by running: $ git pull https://github.com/bhlx3lyx7/incubator-griffin util-update-0.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/455.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #455 commit c57e64d07b728c91cfd3e4ce9adb154597ef7788 Author: Lionel Liu Date: 2018-11-16T05:28:36Z [GRIFFIN-212] Add matchedFraction in streaming accuracy metrics > [Measure] Add matchedFraction in streaming accuracy metrics as in batch mode > > > Key: GRIFFIN-212 > URL: https://issues.apache.org/jira/browse/GRIFFIN-212 > Project: Griffin (Incubating) > Issue Type: Task >Reporter: Lionel Liu >Assignee: Lionel Liu >Priority: Major > > In batch mode, we've added matchedFraction in accuracy metric, we should keep > the metric format in streaming mode align with batch mode. So we need to add > this field in streaming accuracy metric as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-211) [Service] JobInstance appUrl error
[
https://issues.apache.org/jira/browse/GRIFFIN-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688920#comment-16688920
]
ASF GitHub Bot commented on GRIFFIN-211:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-griffin/pull/454
> [Service] JobInstance appUrl error
> --
>
> Key: GRIFFIN-211
> URL: https://issues.apache.org/jira/browse/GRIFFIN-211
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: xiangrong,chen
>Assignee: xiangrong,chen
>Priority: Major
> Attachments: WeChat Image_20181116093345.png
>
>
> when I try to get jobInstance by calling API
> "http://localhost:8080/api/v1/jobs/instances?jobId=8=0=10;,
> response is:
> {
> "id": 212,
> "sessionId": 53,
> "state": "SUCCESS",
> "type": "BATCH",
> "appId": "application_1542252279758_0053",
> "appUri": "{color:#33}http://griffin:8088{color}
> {color:#FF}/cluster/app/{color} application_1542252279758_0053",
> "predicateGroup": "PG",
> "predicateName": "accu-job1_predicate_154233028",
> "timestamp": 154233028,
> "expireTimestamp": 1542934800028
> },
> I find there are 2 blanks on both sides of '/cluster/app/', and it make this
> url can not work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-211) [Service] JobInstance appUrl error
[
https://issues.apache.org/jira/browse/GRIFFIN-211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688894#comment-16688894
]
ASF GitHub Bot commented on GRIFFIN-211:
GitHub user justACT opened a pull request:
https://github.com/apache/incubator-griffin/pull/454
[Service] JobInstace setJobInstanceIdAndUri, update instance.setAppU…
GRIFFIN-211 [Service] JobInstance appUrl error
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/justACT/incubator-griffin newmaster
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-griffin/pull/454.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #454
commit cb973c548bc9721f35c8ffbce85c3134c0de7cd5
Author: justACT
Date: 2018-11-16T01:40:25Z
[Service] JobInstace setJobInstanceIdAndUri, update instance.setAppUri,
delete blanks
> [Service] JobInstance appUrl error
> --
>
> Key: GRIFFIN-211
> URL: https://issues.apache.org/jira/browse/GRIFFIN-211
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: xiangrong,chen
>Assignee: xiangrong,chen
>Priority: Major
> Attachments: WeChat Image_20181116093345.png
>
>
> when I try to get jobInstance by calling API
> "http://localhost:8080/api/v1/jobs/instances?jobId=8=0=10;,
> response is:
> {
> "id": 212,
> "sessionId": 53,
> "state": "SUCCESS",
> "type": "BATCH",
> "appId": "application_1542252279758_0053",
> "appUri": "{color:#33}http://griffin:8088{color}
> {color:#FF}/cluster/app/{color} application_1542252279758_0053",
> "predicateGroup": "PG",
> "predicateName": "accu-job1_predicate_154233028",
> "timestamp": 154233028,
> "expireTimestamp": 1542934800028
> },
> I find there are 2 blanks on both sides of '/cluster/app/', and it make this
> url can not work.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-209) [Measure] In paramUtil, the util function getParamStringMap doesn't work as expected
[ https://issues.apache.org/jira/browse/GRIFFIN-209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16686584#comment-16686584 ] ASF GitHub Bot commented on GRIFFIN-209: Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/452 > [Measure] In paramUtil, the util function getParamStringMap doesn't work as > expected > > > Key: GRIFFIN-209 > URL: https://issues.apache.org/jira/browse/GRIFFIN-209 > Project: Griffin (Incubating) > Issue Type: Bug >Reporter: Lionel Liu >Assignee: Lionel Liu >Priority: Major > Fix For: 0.3.1-incubating > > > Need to update paramUtil, to make getParamStringMap and the other util > functions work as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-209) [Measure] In paramUtil, the util function getParamStringMap doesn't work as expected
[ https://issues.apache.org/jira/browse/GRIFFIN-209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685959#comment-16685959 ] ASF GitHub Bot commented on GRIFFIN-209: Github user toyboxman commented on the issue: https://github.com/apache/incubator-griffin/pull/452 LGTM > [Measure] In paramUtil, the util function getParamStringMap doesn't work as > expected > > > Key: GRIFFIN-209 > URL: https://issues.apache.org/jira/browse/GRIFFIN-209 > Project: Griffin (Incubating) > Issue Type: Bug >Reporter: Lionel Liu >Assignee: Lionel Liu >Priority: Major > Fix For: 0.3.1-incubating > > > Need to update paramUtil, to make getParamStringMap and the other util > functions work as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-209) [Measure] In paramUtil, the util function getParamStringMap doesn't work as expected
[ https://issues.apache.org/jira/browse/GRIFFIN-209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685255#comment-16685255 ] ASF GitHub Bot commented on GRIFFIN-209: GitHub user bhlx3lyx7 opened a pull request: https://github.com/apache/incubator-griffin/pull/452 [GRIFFIN-209] update param util, add UT for param util You can merge this pull request into a Git repository by running: $ git pull https://github.com/bhlx3lyx7/incubator-griffin param-util Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/452.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #452 commit f60106a957289b2e09fd50ae5453755c366376ea Author: Lionel Liu Date: 2018-11-13T13:58:06Z [GRIFFIN-209] update param util, add UT for param util > [Measure] In paramUtil, the util function getParamStringMap doesn't work as > expected > > > Key: GRIFFIN-209 > URL: https://issues.apache.org/jira/browse/GRIFFIN-209 > Project: Griffin (Incubating) > Issue Type: Bug >Reporter: Lionel Liu >Assignee: Lionel Liu >Priority: Major > Fix For: 0.3.1-incubating > > > Need to update paramUtil, to make getParamStringMap and the other util > functions work as expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-200) Lifecycle hooks support
[
https://issues.apache.org/jira/browse/GRIFFIN-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16681534#comment-16681534
]
ASF GitHub Bot commented on GRIFFIN-200:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-griffin/pull/444
> Lifecycle hooks support
> ---
>
> Key: GRIFFIN-200
> URL: https://issues.apache.org/jira/browse/GRIFFIN-200
> Project: Griffin (Incubating)
> Issue Type: New Feature
>Reporter: Nikolay Sokolov
>Assignee: Eugene
>Priority: Minor
>
> In some environments, users might want to perform certain actions
> before/after job is created, before/after job is activated, before/after job
> is deleted, and so on.
> To fullfill that need, some hook plugin mechanism can be provided, similar to
> what Hive is doing. User would place respective jar files into Service module
> classpath at deployment time, and would specify class names using some
> annotation or using property listing class names (particular mechanism is yet
> to be determined).
> Proposed signature:
> {code:none}
> public interface GriffinHook {
> void onEvent(GriffinHookEvent event) throws Exception;
> }
> public interface GriffinHookEvent { ... }
> public interface JobEvent implements GriffinHookEvent { ... }
> public class BeforeJobCreated implements JobEvent { ... }
> public class AfterJobCreated implements JobEvent { ... }
> public class BeforeJobDeleted implements JobEvent { ... }
> public class AfterJobDeleted implements JobEvent { ... }
> public interface JobInstanceEvent implements GriffinHookEvent { ... }
> public class BeforeJobInstanceStart implements JobInstanceEvent { ... }
> public class AfterJobInstanceEnd implements JobInstanceEvent { ... }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-208) Job status is SUCCESS even if some stages have failed
[
https://issues.apache.org/jira/browse/GRIFFIN-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668266#comment-16668266
]
ASF GitHub Bot commented on GRIFFIN-208:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-griffin/pull/448
> Job status is SUCCESS even if some stages have failed
> -
>
> Key: GRIFFIN-208
> URL: https://issues.apache.org/jira/browse/GRIFFIN-208
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Major
>
> When some steps (MetricWrite or SparkSql, for example) fail, errors are just
> logged, but not reported as part of job status. Symptoms:
> {code:none}
> 18/10/22 17:17:58 ERROR transform.SparkSqlTransformStep: run spark sql [
> ] error: ...
> {code}
> YarnApplicationState: FINISHED
> FinalStatus Reported by AM: SUCCEEDED
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1655#comment-1655 ] ASF GitHub Bot commented on GRIFFIN-203: Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/446 > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1644#comment-1644 ] ASF GitHub Bot commented on GRIFFIN-203: Github user icesmartjuan commented on the issue: https://github.com/apache/incubator-griffin/pull/446 Great work, it works fine > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-208) Job status is SUCCESS even if some stages have failed
[
https://issues.apache.org/jira/browse/GRIFFIN-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664963#comment-16664963
]
ASF GitHub Bot commented on GRIFFIN-208:
Github user bhlx3lyx7 commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/448#discussion_r228469933
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/Application.scala ---
@@ -104,12 +106,16 @@ object Application extends Loggable {
case Success(_) =>
info("process end success")
case Failure(ex) =>
-error(s"process end error: ${ex.getMessage}")
+error(s"process end error: ${ex.getMessage}", ex)
shutdown
sys.exit(-5)
}
shutdown
+
+if (!success) {
+ sys.exit(-5)
+}
--- End diff --
If any rule step fails, explicitly break down the application can let the
spark application shut down with a FAILED state. It makes sense.
Actually, we can think about how to get the job state. Through livy or
yarn, we can only get the lifecycle state of spark job, we still need to check
logs to address the detailed error, and it would be much more difficult for
common users. We can define some job states in calculation, like `starting`,
`loading data source`, `pre-processing`, `rule step N success`, and let spark
job report its state by configured notify method like http request to service
side or something else. In such a way, service can manage states of job
instances.
I think job state management is the final solution, explicitly exit the
application could work as a temporary solution at current.
> Job status is SUCCESS even if some stages have failed
> -
>
> Key: GRIFFIN-208
> URL: https://issues.apache.org/jira/browse/GRIFFIN-208
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Major
>
> When some steps (MetricWrite or SparkSql, for example) fail, errors are just
> logged, but not reported as part of job status. Symptoms:
> {code:none}
> 18/10/22 17:17:58 ERROR transform.SparkSqlTransformStep: run spark sql [
> ] error: ...
> {code}
> YarnApplicationState: FINISHED
> FinalStatus Reported by AM: SUCCEEDED
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-194) [service] Hive API improvement
[ https://issues.apache.org/jira/browse/GRIFFIN-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664923#comment-16664923 ] ASF GitHub Bot commented on GRIFFIN-194: Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/447 > [service] Hive API improvement > -- > > Key: GRIFFIN-194 > URL: https://issues.apache.org/jira/browse/GRIFFIN-194 > Project: Griffin (Incubating) > Issue Type: Sub-task >Reporter: Nikolay Sokolov >Priority: Minor > > Purpose is mainly to support GRIFFIN-195 with single request to get table > list information, while avoiding transferring all table metadata and making > lots of metastore requests. > Hive API provides following relevant APIs right now: > * listing DBs > * getting all table names in DB > * listing all table _objects_ in all _dbs_ > What's seems to be missing, is API call for all table names in all DBs (as > middle ground between n+1 API requests and 1 API request with huge payload > and n*m+1 metastore requests on backend). > This api request should take no parameters, and return Map List> in response. > Proposed API endpoint: TBD -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-194) [service] Hive API improvement
[ https://issues.apache.org/jira/browse/GRIFFIN-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664848#comment-16664848 ] ASF GitHub Bot commented on GRIFFIN-194: Github user chemikadze commented on the issue: https://github.com/apache/incubator-griffin/pull/447 @guoyuepeng can you please clarify your comments, please? Not very clear what changes are required, if any. > [service] Hive API improvement > -- > > Key: GRIFFIN-194 > URL: https://issues.apache.org/jira/browse/GRIFFIN-194 > Project: Griffin (Incubating) > Issue Type: Sub-task >Reporter: Nikolay Sokolov >Priority: Minor > > Purpose is mainly to support GRIFFIN-195 with single request to get table > list information, while avoiding transferring all table metadata and making > lots of metastore requests. > Hive API provides following relevant APIs right now: > * listing DBs > * getting all table names in DB > * listing all table _objects_ in all _dbs_ > What's seems to be missing, is API call for all table names in all DBs (as > middle ground between n+1 API requests and 1 API request with huge payload > and n*m+1 metastore requests on backend). > This api request should take no parameters, and return Map List> in response. > Proposed API endpoint: TBD -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-194) [service] Hive API improvement
[
https://issues.apache.org/jira/browse/GRIFFIN-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664734#comment-16664734
]
ASF GitHub Bot commented on GRIFFIN-194:
Github user guoyuepeng commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/447#discussion_r228420528
--- Diff: ui/angular/src/app/measure/create-measure/ac/ac.component.ts ---
@@ -644,48 +644,37 @@ export class AcComponent implements OnInit,
AfterViewChecked {
}
ngOnInit() {
-let allDatabases = this.serviceService.config.uri.dblist;
-let getTableNames = this.serviceService.config.uri.tablenames;
+let getTableNames = this.serviceService.config.uri.dbtablenames;
--- End diff --
add one more endpoint.
> [service] Hive API improvement
> --
>
> Key: GRIFFIN-194
> URL: https://issues.apache.org/jira/browse/GRIFFIN-194
> Project: Griffin (Incubating)
> Issue Type: Sub-task
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> Purpose is mainly to support GRIFFIN-195 with single request to get table
> list information, while avoiding transferring all table metadata and making
> lots of metastore requests.
> Hive API provides following relevant APIs right now:
> * listing DBs
> * getting all table names in DB
> * listing all table _objects_ in all _dbs_
> What's seems to be missing, is API call for all table names in all DBs (as
> middle ground between n+1 API requests and 1 API request with huge payload
> and n*m+1 metastore requests on backend).
> This api request should take no parameters, and return Map List> in response.
> Proposed API endpoint: TBD
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-194) [service] Hive API improvement
[
https://issues.apache.org/jira/browse/GRIFFIN-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664726#comment-16664726
]
ASF GitHub Bot commented on GRIFFIN-194:
Github user guoyuepeng commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/447#discussion_r228419404
--- Diff:
service/src/main/java/org/apache/griffin/core/metastore/hive/HiveMetaStoreServiceImpl.java
---
@@ -99,8 +100,17 @@ public HiveMetaStoreServiceImpl() {
public List getAllTable(String db) {
return getTables(db);
}
-
-
+
+@Override
+@Cacheable(unless = "#result==null || #result.isEmpty()")
--- End diff --
ok, add one more service for all table names.
> [service] Hive API improvement
> --
>
> Key: GRIFFIN-194
> URL: https://issues.apache.org/jira/browse/GRIFFIN-194
> Project: Griffin (Incubating)
> Issue Type: Sub-task
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> Purpose is mainly to support GRIFFIN-195 with single request to get table
> list information, while avoiding transferring all table metadata and making
> lots of metastore requests.
> Hive API provides following relevant APIs right now:
> * listing DBs
> * getting all table names in DB
> * listing all table _objects_ in all _dbs_
> What's seems to be missing, is API call for all table names in all DBs (as
> middle ground between n+1 API requests and 1 API request with huge payload
> and n*m+1 metastore requests on backend).
> This api request should take no parameters, and return Map List> in response.
> Proposed API endpoint: TBD
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664647#comment-16664647 ] ASF GitHub Bot commented on GRIFFIN-203: Github user gavlyukovskiy commented on the issue: https://github.com/apache/incubator-griffin/pull/446 Thank you everyone for feedback, I'm not UI guy so structure was messed little bit :) I changed it to be consistent with other pages and now it looks much better. > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-208) Job status is SUCCESS even if some stages have failed
[
https://issues.apache.org/jira/browse/GRIFFIN-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664575#comment-16664575
]
ASF GitHub Bot commented on GRIFFIN-208:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/448#discussion_r228391307
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/Application.scala ---
@@ -104,12 +106,18 @@ object Application extends Loggable {
case Success(_) =>
info("process end success")
case Failure(ex) =>
-error(s"process end error: ${ex.getMessage}")
+error(s"process end error: ${ex.getMessage}", ex)
shutdown
sys.exit(-5)
}
shutdown
+
+if (success) {
--- End diff --
removed exit(0), looks like org.apache.spark.deploy.yarn.ApplicationMaster
is not liking direct exit(0)
> Job status is SUCCESS even if some stages have failed
> -
>
> Key: GRIFFIN-208
> URL: https://issues.apache.org/jira/browse/GRIFFIN-208
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Major
>
> When some steps (MetricWrite or SparkSql, for example) fail, errors are just
> logged, but not reported as part of job status. Symptoms:
> {code:none}
> 18/10/22 17:17:58 ERROR transform.SparkSqlTransformStep: run spark sql [
> ] error: ...
> {code}
> YarnApplicationState: FINISHED
> FinalStatus Reported by AM: SUCCEEDED
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-208) Job status is SUCCESS even if some stages have failed
[
https://issues.apache.org/jira/browse/GRIFFIN-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664516#comment-16664516
]
ASF GitHub Bot commented on GRIFFIN-208:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/448#discussion_r228385053
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/Application.scala ---
@@ -104,12 +106,18 @@ object Application extends Loggable {
case Success(_) =>
info("process end success")
case Failure(ex) =>
-error(s"process end error: ${ex.getMessage}")
+error(s"process end error: ${ex.getMessage}", ex)
shutdown
sys.exit(-5)
}
shutdown
+
+if (success) {
--- End diff --
looks like this one breaks a lot of stuff
> Job status is SUCCESS even if some stages have failed
> -
>
> Key: GRIFFIN-208
> URL: https://issues.apache.org/jira/browse/GRIFFIN-208
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Major
>
> When some steps (MetricWrite or SparkSql, for example) fail, errors are just
> logged, but not reported as part of job status. Symptoms:
> {code:none}
> 18/10/22 17:17:58 ERROR transform.SparkSqlTransformStep: run spark sql [
> ] error: ...
> {code}
> YarnApplicationState: FINISHED
> FinalStatus Reported by AM: SUCCEEDED
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664282#comment-16664282 ] ASF GitHub Bot commented on GRIFFIN-203: Github user chemikadze commented on the issue: https://github.com/apache/incubator-griffin/pull/446 Looks like there are still some rendering issues: https://user-images.githubusercontent.com/867294/47530131-9ac61b80-d85e-11e8-8bf1-aa825105ff52.png;> > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-208) Job status is SUCCESS even if some stages have failed
[
https://issues.apache.org/jira/browse/GRIFFIN-208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16664119#comment-16664119
]
ASF GitHub Bot commented on GRIFFIN-208:
GitHub user chemikadze opened a pull request:
https://github.com/apache/incubator-griffin/pull/448
[GRIFFIN-208] log exception details when it makes sense
Sometimes error messages are not descriptive enough and do not allow
to pinpoint exact issue. Exception stack traces should make it easier
to troubleshoot issues.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-208
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-griffin/pull/448.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #448
commit cb7e1225a2b6a76a0dcbd057a1e8c625ed8cff1d
Author: Nikolay Sokolov
Date: 2018-10-25T18:18:17Z
[GRIFFIN-208] log exception details when it makes sense
Sometimes error messages are not descriptive enough and do not allow
to pinpoint exact issue. Exception stack traces should make it easier
to troubleshoot issues.
> Job status is SUCCESS even if some stages have failed
> -
>
> Key: GRIFFIN-208
> URL: https://issues.apache.org/jira/browse/GRIFFIN-208
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Major
>
> When some steps (MetricWrite or SparkSql, for example) fail, errors are just
> logged, but not reported as part of job status. Symptoms:
> {code:none}
> 18/10/22 17:17:58 ERROR transform.SparkSqlTransformStep: run spark sql [
> ] error: ...
> {code}
> YarnApplicationState: FINISHED
> FinalStatus Reported by AM: SUCCEEDED
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663842#comment-16663842 ] ASF GitHub Bot commented on GRIFFIN-203: Github user guoyuepeng commented on the issue: https://github.com/apache/incubator-griffin/pull/446 Awesome! > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663802#comment-16663802
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-griffin/pull/441
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
> Fix For: 0.3.1-incubating
>
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-194) [service] Hive API improvement
[ https://issues.apache.org/jira/browse/GRIFFIN-194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663346#comment-16663346 ] ASF GitHub Bot commented on GRIFFIN-194: GitHub user chemikadze opened a pull request: https://github.com/apache/incubator-griffin/pull/447 [GRIFFIN-194] single call to fetch all tables of all dbs Significantly improves opening speed for measure dialogs. Previously taking 10+ seconds on huge schemas, with this call it takes fraction of second to load. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-194 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/447.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #447 commit 68ab221ddba6526143f40ab372b3af8c328cf6a6 Author: Nikolay Sokolov Date: 2018-10-25T06:58:42Z [GRIFFIN-194] single call to fetch all tables of all dbs Significantly improves opening speed for measure dialogs. Previously taking 10+ seconds on huge schemas, with this call it takes fraction of second to load. > [service] Hive API improvement > -- > > Key: GRIFFIN-194 > URL: https://issues.apache.org/jira/browse/GRIFFIN-194 > Project: Griffin (Incubating) > Issue Type: Sub-task >Reporter: Nikolay Sokolov >Priority: Minor > > Purpose is mainly to support GRIFFIN-195 with single request to get table > list information, while avoiding transferring all table metadata and making > lots of metastore requests. > Hive API provides following relevant APIs right now: > * listing DBs > * getting all table names in DB > * listing all table _objects_ in all _dbs_ > What's seems to be missing, is API call for all table names in all DBs (as > middle ground between n+1 API requests and 1 API request with huge payload > and n*m+1 metastore requests on backend). > This api request should take no parameters, and return Map List> in response. > Proposed API endpoint: TBD -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[
https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663272#comment-16663272
]
ASF GitHub Bot commented on GRIFFIN-203:
Github user gavlyukovskiy commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/446#discussion_r228039612
--- Diff: ui/angular/src/app/measure/create-measure/raw/raw.component.ts ---
@@ -0,0 +1,146 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+import {Component, OnInit} from "@angular/core";
+import {ServiceService} from "../../../service/service.service";
+import {TREE_ACTIONS, ITreeOptions} from "angular-tree-component";
+import {ToasterService} from "angular2-toaster";
+import * as $ from "jquery";
+import {HttpClient} from "@angular/common/http";
+import {ActivatedRoute, Router} from "@angular/router";
+import {AfterViewChecked, ElementRef} from "@angular/core";
+import {MeasureFormatService, Format} from
"../../../service/measure-format.service";
+
+@Component({
+ selector: "app-raw",
+ templateUrl: "./raw.component.html",
+ providers: [ServiceService, MeasureFormatService],
+ styleUrls: ["./raw.component.css"]
+})
+export class RawComponent implements AfterViewChecked {
+
+ constructor(
+private elementRef: ElementRef,
+private toasterService: ToasterService,
+private measureFormatService: MeasureFormatService,
+private http: HttpClient,
+private router: Router,
+public serviceService: ServiceService
+ ) {
+ }
+
+ data = "";
+ valid = false;
+ Format: typeof Format = Format;
+ format: Format;
+ createResult: any;
+ public visible = false;
+ public visibleAnimate = false;
+
+ public hide(): void {
+this.visibleAnimate = false;
+setTimeout(() => (this.visible = false), 300);
+$("#save").removeAttr("disabled");
+ }
+
+ public onContainerClicked(event: MouseEvent): void {
+if ((event.target).classList.contains("modal")) {
+ this.hide();
+}
+ }
+
+ onResize(event) {
+this.resizeWindow();
+ }
+
+ resizeWindow() {
+var stepSelection = ".formStep";
+$(stepSelection).css({
+ height: window.innerHeight - $(stepSelection).offset().top
+});
+$("fieldset").height(
+ $(stepSelection).height() -
+ $(stepSelection + ">.stepDesc").height() -
+ $(".btn-container").height() -
+ 130
+);
+$(".y-scrollable").css({
+ height: $("fieldset").height()
+});
+ }
+
+ submit(form) {
+if (!form.valid) {
+ this.toasterService.pop(
+"error",
+"Error!",
+"please complete the form in this step before proceeding"
+ );
+ return false;
+}
+this.visible = true;
+setTimeout(() => (this.visibleAnimate = true), 100);
+ }
+
+ save() {
+let measure2Save = this.measureFormatService.parse(this.data,
this.format);
+console.log(measure2Save);
+let addModels = this.serviceService.config.uri.addModels;
+$("#save").attr("disabled", "true");
+this.http.post(addModels, measure2Save).subscribe(
+ data => {
+this.createResult = data;
+this.hide();
+this.router.navigate(["/measures"]);
+ },
+ err => {
+let response = JSON.parse(err.error);
+if (response.code === '40901') {
+ this.toasterService.pop("error", "Error!", "Measure name already
exists!");
+} else {
+ this.toasterService.pop("error", "Error!", response.message);
+}
+console.log("Error when creating measure");
+ }
+);
+ }
+
+ onInputChange() {
--- End diff --
As I can see it's done on every change, but is not noticeable on my laptop.
Not sure if it will affect others, but
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663269#comment-16663269 ] ASF GitHub Bot commented on GRIFFIN-203: Github user chemikadze commented on the issue: https://github.com/apache/incubator-griffin/pull/446 LGTM > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[
https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663264#comment-16663264
]
ASF GitHub Bot commented on GRIFFIN-203:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/446#discussion_r228037817
--- Diff: ui/angular/src/app/measure/create-measure/raw/raw.component.ts ---
@@ -0,0 +1,146 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+import {Component, OnInit} from "@angular/core";
+import {ServiceService} from "../../../service/service.service";
+import {TREE_ACTIONS, ITreeOptions} from "angular-tree-component";
+import {ToasterService} from "angular2-toaster";
+import * as $ from "jquery";
+import {HttpClient} from "@angular/common/http";
+import {ActivatedRoute, Router} from "@angular/router";
+import {AfterViewChecked, ElementRef} from "@angular/core";
+import {MeasureFormatService, Format} from
"../../../service/measure-format.service";
+
+@Component({
+ selector: "app-raw",
+ templateUrl: "./raw.component.html",
+ providers: [ServiceService, MeasureFormatService],
+ styleUrls: ["./raw.component.css"]
+})
+export class RawComponent implements AfterViewChecked {
+
+ constructor(
+private elementRef: ElementRef,
+private toasterService: ToasterService,
+private measureFormatService: MeasureFormatService,
+private http: HttpClient,
+private router: Router,
+public serviceService: ServiceService
+ ) {
+ }
+
+ data = "";
+ valid = false;
+ Format: typeof Format = Format;
+ format: Format;
+ createResult: any;
+ public visible = false;
+ public visibleAnimate = false;
+
+ public hide(): void {
+this.visibleAnimate = false;
+setTimeout(() => (this.visible = false), 300);
+$("#save").removeAttr("disabled");
+ }
+
+ public onContainerClicked(event: MouseEvent): void {
+if ((event.target).classList.contains("modal")) {
+ this.hide();
+}
+ }
+
+ onResize(event) {
+this.resizeWindow();
+ }
+
+ resizeWindow() {
+var stepSelection = ".formStep";
+$(stepSelection).css({
+ height: window.innerHeight - $(stepSelection).offset().top
+});
+$("fieldset").height(
+ $(stepSelection).height() -
+ $(stepSelection + ">.stepDesc").height() -
+ $(".btn-container").height() -
+ 130
+);
+$(".y-scrollable").css({
+ height: $("fieldset").height()
+});
+ }
+
+ submit(form) {
+if (!form.valid) {
+ this.toasterService.pop(
+"error",
+"Error!",
+"please complete the form in this step before proceeding"
+ );
+ return false;
+}
+this.visible = true;
+setTimeout(() => (this.visibleAnimate = true), 100);
+ }
+
+ save() {
+let measure2Save = this.measureFormatService.parse(this.data,
this.format);
+console.log(measure2Save);
+let addModels = this.serviceService.config.uri.addModels;
+$("#save").attr("disabled", "true");
+this.http.post(addModels, measure2Save).subscribe(
+ data => {
+this.createResult = data;
+this.hide();
+this.router.navigate(["/measures"]);
+ },
+ err => {
+let response = JSON.parse(err.error);
+if (response.code === '40901') {
+ this.toasterService.pop("error", "Error!", "Measure name already
exists!");
+} else {
+ this.toasterService.pop("error", "Error!", response.message);
+}
+console.log("Error when creating measure");
+ }
+);
+ }
+
+ onInputChange() {
--- End diff --
how frequently is it triggered? on every key press, or there is some
throttle?
is parsing delay noticeable while
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[
https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663260#comment-16663260
]
ASF GitHub Bot commented on GRIFFIN-203:
Github user gavlyukovskiy commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/446#discussion_r228037399
--- Diff: ui/angular/src/app/measure/create-measure/raw/raw.component.ts ---
@@ -0,0 +1,180 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+import {Component, OnInit} from "@angular/core";
+import {ServiceService} from "../../../service/service.service";
+import {TREE_ACTIONS, ITreeOptions} from "angular-tree-component";
+import {ToasterService} from "angular2-toaster";
+import * as $ from "jquery";
+import {HttpClient} from "@angular/common/http";
+import {ActivatedRoute, Router} from "@angular/router";
+import {AfterViewChecked, ElementRef} from "@angular/core";
+import {MeasureFormatService, Format} from
"../../../service/measure-format.service";
+
+@Component({
+ selector: "app-raw",
+ templateUrl: "./raw.component.html",
+ providers: [ServiceService, MeasureFormatService],
+ styleUrls: ["./raw.component.css"]
+})
+export class RawComponent implements AfterViewChecked, OnInit {
+
+ constructor(
+private elementRef: ElementRef,
+private toasterService: ToasterService,
+private measureFormatService: MeasureFormatService,
+private http: HttpClient,
+private router: Router,
+public serviceService: ServiceService
+ ) {
+ }
+
+ data = "";
+ valid = false;
+ Format: typeof Format = Format;
+ format: Format;
+ createResult: any;
+ public visible = false;
+ public visibleAnimate = false;
+
+ public hide(): void {
+this.visibleAnimate = false;
+setTimeout(() => (this.visible = false), 300);
+$("#save").removeAttr("disabled");
+ }
+
+ public onContainerClicked(event: MouseEvent): void {
+if ((event.target).classList.contains("modal")) {
+ this.hide();
+}
+ }
+
+ onResize(event) {
+this.resizeWindow();
+ }
+
+ resizeWindow() {
+var stepSelection = ".formStep";
+$(stepSelection).css({
+ height: window.innerHeight - $(stepSelection).offset().top
+});
+$("fieldset").height(
+ $(stepSelection).height() -
+ $(stepSelection + ">.stepDesc").height() -
+ $(".btn-container").height() -
+ 130
+);
+$(".y-scrollable").css({
+ height: $("fieldset").height()
+});
+ }
+
+ submit(form) {
+if (!form.valid) {
+ this.toasterService.pop(
+"error",
+"Error!",
+"please complete the form in this step before proceeding"
+ );
+ return false;
+}
+this.visible = true;
+setTimeout(() => (this.visibleAnimate = true), 100);
+ }
+
+ save() {
+let measure2Save = this.measureFormatService.parse(this.data,
this.format);
+console.log(measure2Save);
+let addModels = this.serviceService.config.uri.addModels;
+$("#save").attr("disabled", "true");
+this.http.post(addModels, measure2Save).subscribe(
+ data => {
+this.createResult = data;
+this.hide();
+this.router.navigate(["/measures"]);
+ },
+ err => {
+let response = JSON.parse(err.error);
+if (response.code === '40901') {
+ this.toasterService.pop("error", "Error!", "Measure name already
exists!");
+} else {
+ this.toasterService.pop("error", "Error!", "Measure is not
valid");
+}
+console.log("Error when creating measure");
+ }
+);
+ }
+
+ onInputChange() {
+let format = this.measureFormatService.determineFormat(this.data);
+if (format) {
+ this.format =
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[
https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663261#comment-16663261
]
ASF GitHub Bot commented on GRIFFIN-203:
Github user gavlyukovskiy commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/446#discussion_r228037408
--- Diff: ui/angular/src/app/measure/create-measure/raw/raw.component.ts ---
@@ -0,0 +1,180 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+import {Component, OnInit} from "@angular/core";
+import {ServiceService} from "../../../service/service.service";
+import {TREE_ACTIONS, ITreeOptions} from "angular-tree-component";
+import {ToasterService} from "angular2-toaster";
+import * as $ from "jquery";
+import {HttpClient} from "@angular/common/http";
+import {ActivatedRoute, Router} from "@angular/router";
+import {AfterViewChecked, ElementRef} from "@angular/core";
+import {MeasureFormatService, Format} from
"../../../service/measure-format.service";
+
+@Component({
+ selector: "app-raw",
+ templateUrl: "./raw.component.html",
+ providers: [ServiceService, MeasureFormatService],
+ styleUrls: ["./raw.component.css"]
+})
+export class RawComponent implements AfterViewChecked, OnInit {
+
+ constructor(
+private elementRef: ElementRef,
+private toasterService: ToasterService,
+private measureFormatService: MeasureFormatService,
+private http: HttpClient,
+private router: Router,
+public serviceService: ServiceService
+ ) {
+ }
+
+ data = "";
+ valid = false;
+ Format: typeof Format = Format;
+ format: Format;
+ createResult: any;
+ public visible = false;
+ public visibleAnimate = false;
+
+ public hide(): void {
+this.visibleAnimate = false;
+setTimeout(() => (this.visible = false), 300);
+$("#save").removeAttr("disabled");
+ }
+
+ public onContainerClicked(event: MouseEvent): void {
+if ((event.target).classList.contains("modal")) {
+ this.hide();
+}
+ }
+
+ onResize(event) {
+this.resizeWindow();
+ }
+
+ resizeWindow() {
+var stepSelection = ".formStep";
+$(stepSelection).css({
+ height: window.innerHeight - $(stepSelection).offset().top
+});
+$("fieldset").height(
+ $(stepSelection).height() -
+ $(stepSelection + ">.stepDesc").height() -
+ $(".btn-container").height() -
+ 130
+);
+$(".y-scrollable").css({
+ height: $("fieldset").height()
+});
+ }
+
+ submit(form) {
+if (!form.valid) {
+ this.toasterService.pop(
+"error",
+"Error!",
+"please complete the form in this step before proceeding"
+ );
+ return false;
+}
+this.visible = true;
+setTimeout(() => (this.visibleAnimate = true), 100);
+ }
+
+ save() {
+let measure2Save = this.measureFormatService.parse(this.data,
this.format);
+console.log(measure2Save);
+let addModels = this.serviceService.config.uri.addModels;
+$("#save").attr("disabled", "true");
+this.http.post(addModels, measure2Save).subscribe(
+ data => {
+this.createResult = data;
+this.hide();
+this.router.navigate(["/measures"]);
+ },
+ err => {
+let response = JSON.parse(err.error);
+if (response.code === '40901') {
+ this.toasterService.pop("error", "Error!", "Measure name already
exists!");
+} else {
+ this.toasterService.pop("error", "Error!", "Measure is not
valid");
--- End diff --
fixed
> "Plaintext mode" for measure creation
> -
>
> Key: GRIFFIN-203
> URL: https://issues.apache.org/jira/browse/GRIFFIN-203
> Project: Griffin
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663256#comment-16663256 ] ASF GitHub Bot commented on GRIFFIN-203: Github user chemikadze commented on a diff in the pull request: https://github.com/apache/incubator-griffin/pull/446#discussion_r228036689 --- Diff: ui/angular/src/app/measure/create-measure/raw/raw.component.html --- @@ -0,0 +1,100 @@ + + + +Create Measure + + + + + + + + +1 + + Configuration + + + + + +Please setup the measure required information + + + + +Required Information + + + + + + Raw measure: + + + +JSON +YAML + + + + + +Measure is not valid + + + + + + After submitted, please go to "Measures" + to check the measure status + + + + + + + + Submit + + + + + + --- End diff -- Oh ok. Thanks for checking! > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663255#comment-16663255 ] ASF GitHub Bot commented on GRIFFIN-203: Github user gavlyukovskiy commented on a diff in the pull request: https://github.com/apache/incubator-griffin/pull/446#discussion_r228036531 --- Diff: ui/angular/src/app/measure/create-measure/raw/raw.component.html --- @@ -0,0 +1,100 @@ + + + +Create Measure + + + + + + + + +1 + + Configuration + + + + + +Please setup the measure required information + + + + +Required Information + + + + + + Raw measure: + + + +JSON +YAML + + + + + +Measure is not valid + + + + + + After submitted, please go to "Measures" + to check the measure status + + + + + + + + Submit + + + + + + --- End diff -- I just didn't captured whole window, I've just checked and it's centered. > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[
https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663249#comment-16663249
]
ASF GitHub Bot commented on GRIFFIN-203:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/446#discussion_r228035192
--- Diff: ui/angular/src/app/measure/create-measure/raw/raw.component.ts ---
@@ -0,0 +1,180 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+import {Component, OnInit} from "@angular/core";
+import {ServiceService} from "../../../service/service.service";
+import {TREE_ACTIONS, ITreeOptions} from "angular-tree-component";
+import {ToasterService} from "angular2-toaster";
+import * as $ from "jquery";
+import {HttpClient} from "@angular/common/http";
+import {ActivatedRoute, Router} from "@angular/router";
+import {AfterViewChecked, ElementRef} from "@angular/core";
+import {MeasureFormatService, Format} from
"../../../service/measure-format.service";
+
+@Component({
+ selector: "app-raw",
+ templateUrl: "./raw.component.html",
+ providers: [ServiceService, MeasureFormatService],
+ styleUrls: ["./raw.component.css"]
+})
+export class RawComponent implements AfterViewChecked, OnInit {
+
+ constructor(
+private elementRef: ElementRef,
+private toasterService: ToasterService,
+private measureFormatService: MeasureFormatService,
+private http: HttpClient,
+private router: Router,
+public serviceService: ServiceService
+ ) {
+ }
+
+ data = "";
+ valid = false;
+ Format: typeof Format = Format;
+ format: Format;
+ createResult: any;
+ public visible = false;
+ public visibleAnimate = false;
+
+ public hide(): void {
+this.visibleAnimate = false;
+setTimeout(() => (this.visible = false), 300);
+$("#save").removeAttr("disabled");
+ }
+
+ public onContainerClicked(event: MouseEvent): void {
+if ((event.target).classList.contains("modal")) {
+ this.hide();
+}
+ }
+
+ onResize(event) {
+this.resizeWindow();
+ }
+
+ resizeWindow() {
+var stepSelection = ".formStep";
+$(stepSelection).css({
+ height: window.innerHeight - $(stepSelection).offset().top
+});
+$("fieldset").height(
+ $(stepSelection).height() -
+ $(stepSelection + ">.stepDesc").height() -
+ $(".btn-container").height() -
+ 130
+);
+$(".y-scrollable").css({
+ height: $("fieldset").height()
+});
+ }
+
+ submit(form) {
+if (!form.valid) {
+ this.toasterService.pop(
+"error",
+"Error!",
+"please complete the form in this step before proceeding"
+ );
+ return false;
+}
+this.visible = true;
+setTimeout(() => (this.visibleAnimate = true), 100);
+ }
+
+ save() {
+let measure2Save = this.measureFormatService.parse(this.data,
this.format);
+console.log(measure2Save);
+let addModels = this.serviceService.config.uri.addModels;
+$("#save").attr("disabled", "true");
+this.http.post(addModels, measure2Save).subscribe(
+ data => {
+this.createResult = data;
+this.hide();
+this.router.navigate(["/measures"]);
+ },
+ err => {
+let response = JSON.parse(err.error);
+if (response.code === '40901') {
+ this.toasterService.pop("error", "Error!", "Measure name already
exists!");
+} else {
+ this.toasterService.pop("error", "Error!", "Measure is not
valid");
+}
+console.log("Error when creating measure");
+ }
+);
+ }
+
+ onInputChange() {
+let format = this.measureFormatService.determineFormat(this.data);
+if (format) {
+ this.format = format;
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[
https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663250#comment-16663250
]
ASF GitHub Bot commented on GRIFFIN-203:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/446#discussion_r228035101
--- Diff: ui/angular/src/app/measure/create-measure/raw/raw.component.ts ---
@@ -0,0 +1,180 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+import {Component, OnInit} from "@angular/core";
+import {ServiceService} from "../../../service/service.service";
+import {TREE_ACTIONS, ITreeOptions} from "angular-tree-component";
+import {ToasterService} from "angular2-toaster";
+import * as $ from "jquery";
+import {HttpClient} from "@angular/common/http";
+import {ActivatedRoute, Router} from "@angular/router";
+import {AfterViewChecked, ElementRef} from "@angular/core";
+import {MeasureFormatService, Format} from
"../../../service/measure-format.service";
+
+@Component({
+ selector: "app-raw",
+ templateUrl: "./raw.component.html",
+ providers: [ServiceService, MeasureFormatService],
+ styleUrls: ["./raw.component.css"]
+})
+export class RawComponent implements AfterViewChecked, OnInit {
+
+ constructor(
+private elementRef: ElementRef,
+private toasterService: ToasterService,
+private measureFormatService: MeasureFormatService,
+private http: HttpClient,
+private router: Router,
+public serviceService: ServiceService
+ ) {
+ }
+
+ data = "";
+ valid = false;
+ Format: typeof Format = Format;
+ format: Format;
+ createResult: any;
+ public visible = false;
+ public visibleAnimate = false;
+
+ public hide(): void {
+this.visibleAnimate = false;
+setTimeout(() => (this.visible = false), 300);
+$("#save").removeAttr("disabled");
+ }
+
+ public onContainerClicked(event: MouseEvent): void {
+if ((event.target).classList.contains("modal")) {
+ this.hide();
+}
+ }
+
+ onResize(event) {
+this.resizeWindow();
+ }
+
+ resizeWindow() {
+var stepSelection = ".formStep";
+$(stepSelection).css({
+ height: window.innerHeight - $(stepSelection).offset().top
+});
+$("fieldset").height(
+ $(stepSelection).height() -
+ $(stepSelection + ">.stepDesc").height() -
+ $(".btn-container").height() -
+ 130
+);
+$(".y-scrollable").css({
+ height: $("fieldset").height()
+});
+ }
+
+ submit(form) {
+if (!form.valid) {
+ this.toasterService.pop(
+"error",
+"Error!",
+"please complete the form in this step before proceeding"
+ );
+ return false;
+}
+this.visible = true;
+setTimeout(() => (this.visibleAnimate = true), 100);
+ }
+
+ save() {
+let measure2Save = this.measureFormatService.parse(this.data,
this.format);
+console.log(measure2Save);
+let addModels = this.serviceService.config.uri.addModels;
+$("#save").attr("disabled", "true");
+this.http.post(addModels, measure2Save).subscribe(
+ data => {
+this.createResult = data;
+this.hide();
+this.router.navigate(["/measures"]);
+ },
+ err => {
+let response = JSON.parse(err.error);
+if (response.code === '40901') {
+ this.toasterService.pop("error", "Error!", "Measure name already
exists!");
+} else {
+ this.toasterService.pop("error", "Error!", "Measure is not
valid");
--- End diff --
AFAIK response typically contains valid error message that can be shown
there was similar issue in other place:
https://github.com/apache/incubator-griffin/pull/423/files
> "Plaintext mode" for measure creation
>
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663248#comment-16663248 ] ASF GitHub Bot commented on GRIFFIN-203: Github user chemikadze commented on a diff in the pull request: https://github.com/apache/incubator-griffin/pull/446#discussion_r228035823 --- Diff: ui/angular/src/app/measure/create-measure/raw/raw.component.html --- @@ -0,0 +1,100 @@ + + + +Create Measure + + + + + + + + +1 + + Configuration + + + + + +Please setup the measure required information + + + + +Required Information + + + + + + Raw measure: + + + +JSON +YAML + + + + + +Measure is not valid + + + + + + After submitted, please go to "Measures" + to check the measure status + + + + + + + + Submit + + + + + + --- End diff -- this dialog looked not centered on screen record -- is that true or it's just me? can you make it appear on center of the screen? > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-203) "Plaintext mode" for measure creation
[ https://issues.apache.org/jira/browse/GRIFFIN-203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663238#comment-16663238 ] ASF GitHub Bot commented on GRIFFIN-203: GitHub user gavlyukovskiy opened a pull request: https://github.com/apache/incubator-griffin/pull/446 [GRIFFIN-203] "Plaintext mode" for measure creation Created json/yaml measure creation flow:  Added ability to view measure as yaml:  You can merge this pull request into a Git repository by running: $ git pull https://github.com/gavlyukovskiy/incubator-griffin raw-measures Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/446.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #446 commit e07a7e169b18f67fd5661ab085524c3bf9206c8b Author: Arthur Gavlyukovskiy Date: 2018-10-25T04:53:01Z [GRIFFIN-203] "Plaintext mode" for measure creation Created json/yaml measure creation flow Added ability to view measure as yaml > "Plaintext mode" for measure creation > - > > Key: GRIFFIN-203 > URL: https://issues.apache.org/jira/browse/GRIFFIN-203 > Project: Griffin (Incubating) > Issue Type: New Feature >Reporter: Nikolay Sokolov >Priority: Major > > Creating custom rules from API might be cumbersome -- body should be prepared > outside of UI, and then submitted using HTTP call. To make user's life > easier, it would be useful to allow measure creation by editing JSON directly > on UI. Viewing side of this feature would be GRIFFIN-202. > Also, experience-wise, JSON might not be the best option for complex > spark-sql rules. Possible solution to that would be allowing to write YAML > representation instead of JSON, and then either submitting YAML body or > converting from YAML to JSON on UI side before submission. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16663110#comment-16663110
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user toyboxman commented on the issue:
https://github.com/apache/incubator-griffin/pull/441
LGTM
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-200) Lifecycle hooks support
[
https://issues.apache.org/jira/browse/GRIFFIN-200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661835#comment-16661835
]
ASF GitHub Bot commented on GRIFFIN-200:
GitHub user toyboxman opened a pull request:
https://github.com/apache/incubator-griffin/pull/444
Define griffin plain-vanilla hook.
the purpose of hook is for integration with components outside. Griffin
would offer information about internal task status.
Task: GRIFFIN-200
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/toyboxman/incubator-griffin hook
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-griffin/pull/444.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #444
commit 2fbd30a64c563febe59240b67aae4653e996a522
Author: Eugene
Date: 2018-10-22T14:12:38Z
Define griffin plain-vanilla hook.
the purpose of hook is for integration with components outside. Griffin
would offer information about internal task status.
Task: GRIFFIN-200
> Lifecycle hooks support
> ---
>
> Key: GRIFFIN-200
> URL: https://issues.apache.org/jira/browse/GRIFFIN-200
> Project: Griffin (Incubating)
> Issue Type: New Feature
>Reporter: Nikolay Sokolov
>Assignee: William Guo
>Priority: Minor
>
> In some environments, users might want to perform certain actions
> before/after job is created, before/after job is activated, before/after job
> is deleted, and so on.
> To fullfill that need, some hook plugin mechanism can be provided, similar to
> what Hive is doing. User would place respective jar files into Service module
> classpath at deployment time, and would specify class names using some
> annotation or using property listing class names (particular mechanism is yet
> to be determined).
> Proposed signature:
> {code:none}
> public interface GriffinHook {
> void onEvent(GriffinHookEvent event) throws Exception;
> }
> public interface GriffinHookEvent { ... }
> public interface JobEvent implements GriffinHookEvent { ... }
> public class BeforeJobCreated implements JobEvent { ... }
> public class AfterJobCreated implements JobEvent { ... }
> public class BeforeJobDeleted implements JobEvent { ... }
> public class AfterJobDeleted implements JobEvent { ... }
> public interface JobInstanceEvent implements GriffinHookEvent { ... }
> public class BeforeJobInstanceStart implements JobInstanceEvent { ... }
> public class AfterJobInstanceEnd implements JobInstanceEvent { ... }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660037#comment-16660037
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r227214831
--- Diff:
service/src/main/java/org/apache/griffin/core/login/ldap/SelfSignedSocketFactory.java
---
@@ -0,0 +1,100 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+
+package org.apache.griffin.core.login.ldap;
+
+import javax.net.SocketFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocketFactory;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.Socket;
+import java.net.UnknownHostException;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+
+/**
+ * SocketFactory ignoring insecure (self-signed, expired) certificates.
+ *
+ * Maintains internal {@code SSLSocketFactory} configured with {@code
NoopTrustManager}.
+ * All SocketFactory methods are proxied to internal SSLSocketFactory
instance.
+ * Accepts all client and server certificates, from any issuers.
+ */
+public class SelfSignedSocketFactory extends SocketFactory {
+private SSLSocketFactory sf;
+
+private SelfSignedSocketFactory() throws Exception {
+SSLContext ctx = SSLContext.getInstance("TLS");
+ctx.init(null, new TrustManager[]{new NoopTrustManager()}, null);
+sf = ctx.getSocketFactory();
+}
+
+/**
+ * Part of SocketFactory contract, used by javax.net internals to
create new instance.
+ */
+public static SocketFactory getDefault() {
+try {
+return new SelfSignedSocketFactory();
+} catch (Exception e) {
+throw new RuntimeException(e);
--- End diff --
@whhe fixed, thanks
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659926#comment-16659926
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user whhe commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r227194519
--- Diff:
service/src/main/java/org/apache/griffin/core/login/ldap/SelfSignedSocketFactory.java
---
@@ -0,0 +1,100 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+
+package org.apache.griffin.core.login.ldap;
+
+import javax.net.SocketFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocketFactory;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.Socket;
+import java.net.UnknownHostException;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+
+/**
+ * SocketFactory ignoring insecure (self-signed, expired) certificates.
+ *
+ * Maintains internal {@code SSLSocketFactory} configured with {@code
NoopTrustManager}.
+ * All SocketFactory methods are proxied to internal SSLSocketFactory
instance.
+ * Accepts all client and server certificates, from any issuers.
+ */
+public class SelfSignedSocketFactory extends SocketFactory {
+private SSLSocketFactory sf;
+
+private SelfSignedSocketFactory() throws Exception {
+SSLContext ctx = SSLContext.getInstance("TLS");
+ctx.init(null, new TrustManager[]{new NoopTrustManager()}, null);
+sf = ctx.getSocketFactory();
+}
+
+/**
+ * Part of SocketFactory contract, used by javax.net internals to
create new instance.
+ */
+public static SocketFactory getDefault() {
+try {
+return new SelfSignedSocketFactory();
+} catch (Exception e) {
+throw new RuntimeException(e);
--- End diff --
It is not recommended to throw RuntimeException directly in Griffin, maybe
you can use ServiceException from GriffinException class instead.
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659533#comment-16659533
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/441
@toyboxman i've updated the code according to your comments, thanks!
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658594#comment-16658594
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user toyboxman commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r226904191
--- Diff:
service/src/main/java/org/apache/griffin/core/login/ldap/SelfSignedSocketFactory.java
---
@@ -0,0 +1,68 @@
+package org.apache.griffin.core.login.ldap;
--- End diff --
you should add apache license claim here
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658593#comment-16658593
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user toyboxman commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r226904188
--- Diff:
service/src/main/java/org/apache/griffin/core/login/ldap/SelfSignedSocketFactory.java
---
@@ -0,0 +1,68 @@
+package org.apache.griffin.core.login.ldap;
+
+import javax.net.SocketFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocketFactory;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.Socket;
+import java.net.UnknownHostException;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+
+public class SelfSignedSocketFactory extends SocketFactory {
--- End diff --
as a new file, you have to add java doc to describe
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658591#comment-16658591
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user toyboxman commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r226903885
--- Diff:
service/src/main/java/org/apache/griffin/core/login/ldap/SelfSignedSocketFactory.java
---
@@ -0,0 +1,68 @@
+package org.apache.griffin.core.login.ldap;
--- End diff --
you should add license information here
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658590#comment-16658590
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user toyboxman commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r226903865
--- Diff:
service/src/main/java/org/apache/griffin/core/login/ldap/SelfSignedSocketFactory.java
---
@@ -0,0 +1,68 @@
+package org.apache.griffin.core.login.ldap;
+
+import javax.net.SocketFactory;
+import javax.net.ssl.SSLContext;
+import javax.net.ssl.SSLSocketFactory;
+import javax.net.ssl.TrustManager;
+import javax.net.ssl.X509TrustManager;
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.Socket;
+import java.net.UnknownHostException;
+import java.security.cert.CertificateException;
+import java.security.cert.X509Certificate;
+
+public class SelfSignedSocketFactory extends SocketFactory {
--- End diff --
as a public class, you have to add java doc to describe
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16658262#comment-16658262
]
ASF GitHub Bot commented on GRIFFIN-207:
Github user guoyuepeng commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/441#discussion_r226866713
--- Diff:
service/src/main/java/org/apache/griffin/core/login/LoginServiceLdapImpl.java
---
@@ -48,68 +53,137 @@ Licensed to the Apache Software Foundation (ASF) under
one
private String searchBase;
private String searchPattern;
private SearchControls searchControls;
+private boolean sslSkipVerify;
+private String bindDN;
+private String bindPassword;
public LoginServiceLdapImpl(String url, String email, String
searchBase,
-String searchPattern) {
+String searchPattern, boolean
sslSkipVerify,
+String bindDN, String bindPassword) {
this.url = url;
this.email = email;
this.searchBase = searchBase;
this.searchPattern = searchPattern;
+this.sslSkipVerify = sslSkipVerify;
+this.bindDN = bindDN;
+this.bindPassword = bindPassword;
SearchControls searchControls = new SearchControls();
searchControls.setSearchScope(SearchControls.SUBTREE_SCOPE);
this.searchControls = searchControls;
}
@Override
public ResponseEntity> login(Map
map) {
-String ntAccount = map.get("username");
+String username = map.get("username");
String password = map.get("password");
-String searchFilter = searchPattern.replace("{0}", ntAccount);
+
+// use separate bind credentials, if bindDN is provided
+String bindAccount = StringUtils.isEmpty(this.bindDN) ? username :
this.bindDN;
+String bindPassword = StringUtils.isEmpty(this.bindDN) ? password
: this.bindPassword;
+String searchFilter = searchPattern.replace("{0}", username);
+LdapContext ctx = null;
try {
-LdapContext ctx = getContextInstance(ntAccount, password);
+ctx = getContextInstance(toPrincipal(bindAccount),
bindPassword);
+
NamingEnumeration results =
ctx.search(searchBase,
searchFilter, searchControls);
-String fullName = getFullName(results, ntAccount);
+SearchResult userObject = getSingleUser(results);
+
+// verify password if different bind user is used
+if (!StringUtils.equals(username, bindAccount)) {
+String userDN = getAttributeValue(userObject,
"distinguishedName", toPrincipal(username));
+checkPassword(userDN, password);
+}
+
Map message = new HashMap<>();
-message.put("ntAccount", ntAccount);
-message.put("fullName", fullName);
+message.put("ntAccount", username);
+message.put("fullName", getFullName(userObject, username));
message.put("status", 0);
return new ResponseEntity<>(message, HttpStatus.OK);
-} catch (NamingException e) {
-LOGGER.warn("User {} failed to login with LDAP auth. {}",
ntAccount,
+} catch (AuthenticationException e) {
+LOGGER.warn("User {} failed to login with LDAP auth. {}",
username,
e.getMessage());
+} catch (NamingException e) {
+LOGGER.warn(String.format("User %s failed to login with LDAP
auth.", username), e);
+} finally {
--- End diff --
LGTM
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform
[jira] [Commented] (GRIFFIN-207) LDAP auth is not supporting group filters and non-CN login names
[
https://issues.apache.org/jira/browse/GRIFFIN-207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657663#comment-16657663
]
ASF GitHub Bot commented on GRIFFIN-207:
GitHub user chemikadze opened a pull request:
https://github.com/apache/incubator-griffin/pull/441
[GRIFFIN-207] LDAP login service improvements
- allow non-CN usernames
- allow disabling certificate checks
- allow limiting set of login users
- improve logging
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-207
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-griffin/pull/441.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #441
commit 8e1b56bfd0a458daacdc086190740ccf471d3e2e
Author: Nikolay Sokolov
Date: 2018-10-20T02:04:20Z
[GRIFFIN-207] LDAP login service improvements
- allow non-CN usernames
- allow disabling certificate checks
- allow limiting set of login users
- improve logging
> LDAP auth is not supporting group filters and non-CN login names
>
>
> Key: GRIFFIN-207
> URL: https://issues.apache.org/jira/browse/GRIFFIN-207
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Major
>
> Currently LDAP auth performs bind to principal with name
> "${username}${ldap.email}", and searches through user objects
> ldap.searchPattern. Result of search then only used to retrieve fullName of
> the user.
> There are two problems here:
> * login username can not be different than CN, as it is used to perform LDAP
> bind
> * it is not possible to restrict access to specific groups
> Typical approach used in other software products is to use separate bind
> account, which would search through LDAP objects using search pattern, and
> then use found object's DN to perform password check.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-202) Allow viewing of raw measure details from UI
[ https://issues.apache.org/jira/browse/GRIFFIN-202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649934#comment-16649934 ] ASF GitHub Bot commented on GRIFFIN-202: Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/437 > Allow viewing of raw measure details from UI > > > Key: GRIFFIN-202 > URL: https://issues.apache.org/jira/browse/GRIFFIN-202 > Project: Griffin (Incubating) > Issue Type: Improvement >Reporter: Nikolay Sokolov >Assignee: Nikolay Sokolov >Priority: Minor > Fix For: 0.3.1-incubating > > > Sometimes it's desirable to see raw JSON of specific measure. This is > especially useful when custom measurements are created from API. > Two implementations are possible (one or even both can be implemented): > * show link with URL of measure details API on UI > * add section under "Mapping Rules", showing formatted plain text of API > response on UI -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649675#comment-16649675
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-griffin/pull/435
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
> Fix For: 0.3.1-incubating
>
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649672#comment-16649672
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user guoyuepeng commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
data partition timezone is come from data connector.
https://github.com/apache/incubator-griffin/blob/master/service/src/main/java/org/apache/griffin/core/measure/entity/DataConnector.java#L158
This only relate to scheduler timezone, after tell user this timezone in UI,
I am ok for this now.
Will merge it.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649670#comment-16649670
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user guoyuepeng commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
let me check
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649669#comment-16649669
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
@guoyuepeng @ahutsunshine is there anything left preventing the merge?
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649666#comment-16649666
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ahutsunshine commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
> cron.time.zone
you are right.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649660#comment-16649660
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
@ahutsunshine I've found only usage of cron.time.zone is only during
schedule creation:
https://github.com/apache/incubator-griffin/blob/master/service/src/main/java/org/apache/griffin/core/job/JobServiceImpl.java#L423
At the same time I see partition time zone is coming from data.time.zone
set in measure JSON:
https://github.com/apache/incubator-griffin/blob/master/service/src/main/java/org/apache/griffin/core/job/JobInstance.java#L221
Am I missing something?
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649652#comment-16649652
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ahutsunshine commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
> Added time zone information to batch UI:
> https://user-images.githubusercontent.com/867294/46914000-4c567a00-cf4c-11e8-85cb-be6428e91895.png;>
> Added time zone information to batch UI:
> https://user-images.githubusercontent.com/867294/46914000-4c567a00-cf4c-11e8-85cb-be6428e91895.png;>
Actually, this timezone is used for data partition not scheduling time. For
example, Converting hive partition format, such as dt=#MMdd# AND hour=#HH#,
to specific date, such as dt=20181015 AND hour=01. So I think we'd better
revise the hints 'Cron expression is using brower's time zone'.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649289#comment-16649289
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
Added time zone information to batch UI:
https://user-images.githubusercontent.com/867294/46914000-4c567a00-cf4c-11e8-85cb-be6428e91895.png;>
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649274#comment-16649274
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224981272
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
> IC, this timezone is used for data partition.
As far as I see from code and behavior, this time zone is used only to
interpret cron expression.
Data partitions are referring to timezones configured on datasets
themselves.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649102#comment-16649102
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224970940
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
@guoyuepeng @ashutakGG Moved to utility function. Angular is not my
specialty, I was really surprised how much complexity Angular community has
about it.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-205) Accuracy measure check should provide matchedFraction to store
[
https://issues.apache.org/jira/browse/GRIFFIN-205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648874#comment-16648874
]
ASF GitHub Bot commented on GRIFFIN-205:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-griffin/pull/434
> Accuracy measure check should provide matchedFraction to store
> --
>
> Key: GRIFFIN-205
> URL: https://issues.apache.org/jira/browse/GRIFFIN-205
> Project: Griffin (Incubating)
> Issue Type: Improvement
> Components: accuracy-batch, accuracy-real-time
>Affects Versions: 1.0.0-incubating
>Reporter: Artem Shutak
>Assignee: Artem Shutak
>Priority: Major
>
> Currently, {{accuracy}} measure results contains "total", "miss" and
> "matched" counts.
> As a result, It's hard to analyze accuracy fraction based on results stored
> in ElasticSearch, because ElasticSearch does not provide straight forward
> capability to get "field divided by field" query results.
> {{Accuracy}} measure results should also contain {{matchedFraction}} field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648866#comment-16648866
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user guoyuepeng commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224955984
--- Diff: service/src/main/java/org/apache/griffin/core/util/TimeUtil.java
---
@@ -148,4 +149,11 @@ private static String refreshEscapeHashTag(String str)
{
return str.replaceAll(escapeHashTagPattern, hashTag);
}
+public static TimeZone getTimeZone(String timezone) {
+if (StringUtils.isEmpty(timezone)) {
+return TimeZone.getDefault();
+}
+return TimeZone.getTimeZone(timezone);
+}
+
}
--- End diff --
yes, it is utility function.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648864#comment-16648864
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user guoyuepeng commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224948342
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
I have little concern about this.
1. if we want to get the timezone based on current time in browser, we need
to extract and rename the function as gettimezoneBasedonTime
2. why client want to control the schedule timezone, the timezone should be
owned by cluster, client should only need to control schedule time, right?
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648789#comment-16648789
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224952112
--- Diff: service/src/main/java/org/apache/griffin/core/util/TimeUtil.java
---
@@ -148,4 +149,11 @@ private static String refreshEscapeHashTag(String str)
{
return str.replaceAll(escapeHashTagPattern, hashTag);
}
+public static TimeZone getTimeZone(String timezone) {
+if (StringUtils.isEmpty(timezone)) {
+return TimeZone.getDefault();
+}
+return TimeZone.getTimeZone(timezone);
+}
+
}
--- End diff --
@ahutsunshine Well, it's about time, it's utility function, static makes it
easier to test. Felt like it belongs here.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648788#comment-16648788
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ahutsunshine commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224951998
--- Diff: service/src/main/java/org/apache/griffin/core/util/TimeUtil.java
---
@@ -148,4 +149,11 @@ private static String refreshEscapeHashTag(String str)
{
return str.replaceAll(escapeHashTagPattern, hashTag);
}
+public static TimeZone getTimeZone(String timezone) {
+if (StringUtils.isEmpty(timezone)) {
+return TimeZone.getDefault();
+}
+return TimeZone.getTimeZone(timezone);
+}
+
}
--- End diff --
@chemikadze
I suggest this code be kept in its original place. Currently just modify ui
code to fix bug.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-204) postman collection is not up to date, some APIs don't work well, like create measure
[
https://issues.apache.org/jira/browse/GRIFFIN-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648729#comment-16648729
]
ASF GitHub Bot commented on GRIFFIN-204:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-griffin/pull/433
> postman collection is not up to date, some APIs don't work well, like create
> measure
>
>
> Key: GRIFFIN-204
> URL: https://issues.apache.org/jira/browse/GRIFFIN-204
> Project: Griffin (Incubating)
> Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Eugene
>Assignee: Eugene
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> apply
> [https://github.com/apache/incubator-griffin/blob/master/griffin-doc/service/postman/griffin.json,]
> and launch create measure api, then get an error. I find the json is not
> matched to latest implementation
> {
> "timestamp": 1539073909006,
> "status": 400,
> "error": "Bad Request",
> "exception":
> "org.springframework.http.converter.HttpMessageNotReadableException",
> "message": "Could not read document: Can not construct instance of
> org.apache.griffin.core.measure.entity.DqType from String value 'accuracy':
> value not one of declared Enum instance names: [COMPLETENESS, TIMELINESS,
> PROFILING, CONSISTENCY, ACCURACY, UNIQUENESS]\n at [Source:
> java.io.PushbackInputStream@565f5e70; line: 4, column: 15] (through reference
> chain: org.apache.griffin.core.measure.entity.GriffinMeasure[\"dq.type\"]);
> nested exception is
> com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not construct
> instance of org.apache.griffin.core.measure.entity.DqType from String value
> 'accuracy': value not one of declared Enum instance names: [COMPLETENESS,
> TIMELINESS, PROFILING, CONSISTENCY, ACCURACY, UNIQUENESS]\n at [Source:
> java.io.PushbackInputStream@565f5e70; line: 4, column: 15] (through reference
> chain: org.apache.griffin.core.measure.entity.GriffinMeasure[\"dq.type\"])",
> "path": "/api/v1/measures"
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648705#comment-16648705
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user guoyuepeng commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224947629
--- Diff: service/src/main/java/org/apache/griffin/core/util/TimeUtil.java
---
@@ -148,4 +149,11 @@ private static String refreshEscapeHashTag(String str)
{
return str.replaceAll(escapeHashTagPattern, hashTag);
}
+public static TimeZone getTimeZone(String timezone) {
+if (StringUtils.isEmpty(timezone)) {
+return TimeZone.getDefault();
+}
+return TimeZone.getTimeZone(timezone);
+}
+
}
--- End diff --
LGTM, refactor to better place.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648606#comment-16648606
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
@ahutsunshine I was thinking on handling UTCxxx format on backend to fix
existing jobs, but from other hand, this feels like unnecessary breaking
change. I've provided UI fix for now.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647548#comment-16647548
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ahutsunshine commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
@chemikadze
You are right. In backend, the method of getTimeZone always returns GMT
timezone for UTC-1 or UTC+01:00 or other similar formats, as the given
timezone ID cannot be understood. Currently, I have no better idea to parse
different timezone formats in backend. I prefer to solve the problem in front
end using a full name such as "America/Los_Angeles", or a standard ID such as
"GMT-1:00". Do you have better idea for backend ?
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647108#comment-16647108
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224622499
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
Looks like there big bunch of different techniques, half of them is based
on regex and way browser formats date, another half requires third-party
libraries. Apparently, `getTimeZoneOffset()` is "accepted answer" over here:
https://stackoverflow.com/questions/1091372/getting-the-clients-timezone-in-javascript
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647096#comment-16647096
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224620459
--- Diff:
ui/angular/src/app/measure/create-measure/configuration/configuration.component.ts
---
@@ -56,47 +56,47 @@ export class ConfigurationComponent implements OnInit {
};
timetypes = ["day", "hour", "minute"];
timetype: string;
- timezones = [
-"UTC-12(IDL)",
-"UTC-11(MIT)",
-"UTC-10(HST)",
-"UTC-9:30(MSIT)",
-"UTC-9(AKST)",
-"UTC-8(PST)",
-"UTC-7(MST)",
-"UTC-6(CST)",
-"UTC-5(EST)",
-"UTC-4(AST)",
-"UTC-3:30(NST)",
-"UTC-3(SAT)",
-"UTC-2(BRT)",
-"UTC-1(CVT)",
-"UTC(WET,GMT)",
-"UTC+1(CET)",
-"UTC+2(EET)",
-"UTC+3(MSK)",
-"UTC+3:30(IRT)",
-"UTC+4(META)",
-"UTC+4:30(AFT)",
-"UTC+5(METB)",
-"UTC+5:30(IDT)",
-"UTC+5:45(NPT)",
-"UTC+6(BHT)",
-"UTC+6:30(MRT)",
-"UTC+7(IST)",
-"UTC+8(EAT)",
-"UTC+8:30(KRT)",
-"UTC+9(FET)",
-"UTC+9:30(ACST)",
-"UTC+10(AEST)",
-"UTC+10:30(FAST)",
-"UTC+11(VTT)",
-"UTC+11:30(NFT)",
-"UTC+12(PSTB)",
-"UTC+12:45(CIT)",
-"UTC+13(PSTC)",
-"UTC+14(PSTD)"
- ];
+ timezones: Map = new Map([
+["UTC-12(IDL)", "GMT-12"],
+["UTC-11(MIT)", "GMT-11"],
+["UTC-10(HST)", "GMT-10"],
+["UTC-9:30(MSIT)", "GMT-9:30"],
+["UTC-9(AKST)", "GMT-9"],
+["UTC-8(PST)", "GMT-8"],
+["UTC-7(MST)", "GMT-7"],
+["UTC-6(CST)", "GMT-6"],
+["UTC-5(EST)", "GMT-5"],
+["UTC-4(AST)", "GMT-4"],
+["UTC-3:30(NST)", "GMT-3:30"],
+["UTC-3(SAT)", "GMT-3"],
+["UTC-2(BRT)", "GMT-2"],
+["UTC-1(CVT)", "GMT-1"],
+["UTC(WET,GMT)", "GMT"],
+["UTC+1(CET)", "GMT+1"],
+["UTC+2(EET)", "GMT+2"],
+["UTC+3(MSK)", "GMT+3"],
+["UTC+3:30(IRT)", "GMT+3:30"],
+["UTC+4(META)", "GMT+4"],
+["UTC+4:30(AFT)", "GMT+4:30"],
+["UTC+5(METB)", "GMT+5"],
+["UTC+5:30(IDT)", "GMT+5:30"],
+["UTC+5:45(NPT)", "GMT+5:45"],
+["UTC+6(BHT)", "GMT+6"],
+["UTC+6:30(MRT)", "GMT+6:30"],
+["UTC+7(IST)", "GMT+7"],
+["UTC+8(EAT)", "GMT+8"],
+["UTC+8:30(KRT)", "GMT+8:30"],
+["UTC+9(FET)", "GMT+9"],
+["UTC+9:30(ACST)", "GMT+9:30"],
+["UTC+10(AEST)", "GMT+10"],
+["UTC+10:30(FAST)", "GMT+10:30"],
+["UTC+11(VTT)", "GMT+11"],
+["UTC+11:30(NFT)", "GMT+11:30"],
+["UTC+12(PSTB)", "GMT+12"],
+["UTC+12:45(CIT)", "GMT+12:45"],
+["UTC+13(PSTC)", "GMT+13"],
+["UTC+14(PSTD)", "GMT+15"],
--- End diff --
good catch, thanks
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646880#comment-16646880
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user gavlyukovskiy commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224560857
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
+tests.put("", TimeZone.getDefault().getID());
+// standard cases
+tests.put("GMT", "GMT");
+tests.put("GMT+1", "GMT+01:00");
+tests.put("GMT+1:00", "GMT+01:00");
+tests.put("GMT+01:00", "GMT+01:00");
+tests.put("GMT-1", "GMT-01:00");
+tests.put("GMT-1:00", "GMT-01:00");
+tests.put("GMT-01:00", "GMT-01:00");
+// values pushed by UI for jobs
+tests.put("GMT1", "GMT");
+tests.put("GMT1:00", "GMT");
+tests.put("GMT01:00", "GMT");
+// values generated by UI for datasets in a past
+tests.put("UTC1", "GMT");
+tests.put("UTC1:00", "GMT");
+tests.put("UTC01:00", "GMT");
+tests.put("UTC-1", "GMT");
+tests.put("UTC-1:00", "GMT");
+tests.put("UTC-01:00", "GMT");
+for (HashMap.Entry e: tests.entrySet()) {
--- End diff --
```
tests.forEach((key, value) -> {...})
```
Looks even better :)
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646548#comment-16646548
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224482419
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
Is there some best practices regarding such logic in angular?
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646543#comment-16646543
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224481345
--- Diff:
ui/angular/src/app/measure/create-measure/configuration/configuration.component.ts
---
@@ -56,47 +56,47 @@ export class ConfigurationComponent implements OnInit {
};
timetypes = ["day", "hour", "minute"];
timetype: string;
- timezones = [
-"UTC-12(IDL)",
-"UTC-11(MIT)",
-"UTC-10(HST)",
-"UTC-9:30(MSIT)",
-"UTC-9(AKST)",
-"UTC-8(PST)",
-"UTC-7(MST)",
-"UTC-6(CST)",
-"UTC-5(EST)",
-"UTC-4(AST)",
-"UTC-3:30(NST)",
-"UTC-3(SAT)",
-"UTC-2(BRT)",
-"UTC-1(CVT)",
-"UTC(WET,GMT)",
-"UTC+1(CET)",
-"UTC+2(EET)",
-"UTC+3(MSK)",
-"UTC+3:30(IRT)",
-"UTC+4(META)",
-"UTC+4:30(AFT)",
-"UTC+5(METB)",
-"UTC+5:30(IDT)",
-"UTC+5:45(NPT)",
-"UTC+6(BHT)",
-"UTC+6:30(MRT)",
-"UTC+7(IST)",
-"UTC+8(EAT)",
-"UTC+8:30(KRT)",
-"UTC+9(FET)",
-"UTC+9:30(ACST)",
-"UTC+10(AEST)",
-"UTC+10:30(FAST)",
-"UTC+11(VTT)",
-"UTC+11:30(NFT)",
-"UTC+12(PSTB)",
-"UTC+12:45(CIT)",
-"UTC+13(PSTC)",
-"UTC+14(PSTD)"
- ];
+ timezones: Map = new Map([
+["UTC-12(IDL)", "GMT-12"],
+["UTC-11(MIT)", "GMT-11"],
+["UTC-10(HST)", "GMT-10"],
+["UTC-9:30(MSIT)", "GMT-9:30"],
+["UTC-9(AKST)", "GMT-9"],
+["UTC-8(PST)", "GMT-8"],
+["UTC-7(MST)", "GMT-7"],
+["UTC-6(CST)", "GMT-6"],
+["UTC-5(EST)", "GMT-5"],
+["UTC-4(AST)", "GMT-4"],
+["UTC-3:30(NST)", "GMT-3:30"],
+["UTC-3(SAT)", "GMT-3"],
+["UTC-2(BRT)", "GMT-2"],
+["UTC-1(CVT)", "GMT-1"],
+["UTC(WET,GMT)", "GMT"],
+["UTC+1(CET)", "GMT+1"],
+["UTC+2(EET)", "GMT+2"],
+["UTC+3(MSK)", "GMT+3"],
+["UTC+3:30(IRT)", "GMT+3:30"],
+["UTC+4(META)", "GMT+4"],
+["UTC+4:30(AFT)", "GMT+4:30"],
+["UTC+5(METB)", "GMT+5"],
+["UTC+5:30(IDT)", "GMT+5:30"],
+["UTC+5:45(NPT)", "GMT+5:45"],
+["UTC+6(BHT)", "GMT+6"],
+["UTC+6:30(MRT)", "GMT+6:30"],
+["UTC+7(IST)", "GMT+7"],
+["UTC+8(EAT)", "GMT+8"],
+["UTC+8:30(KRT)", "GMT+8:30"],
+["UTC+9(FET)", "GMT+9"],
+["UTC+9:30(ACST)", "GMT+9:30"],
+["UTC+10(AEST)", "GMT+10"],
+["UTC+10:30(FAST)", "GMT+10:30"],
+["UTC+11(VTT)", "GMT+11"],
+["UTC+11:30(NFT)", "GMT+11:30"],
+["UTC+12(PSTB)", "GMT+12"],
+["UTC+12:45(CIT)", "GMT+12:45"],
+["UTC+13(PSTC)", "GMT+13"],
+["UTC+14(PSTD)", "GMT+15"],
--- End diff --
14 -> 15? Is it done intentionally?
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646542#comment-16646542
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224461713
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
+tests.put("", TimeZone.getDefault().getID());
+// standard cases
+tests.put("GMT", "GMT");
+tests.put("GMT+1", "GMT+01:00");
+tests.put("GMT+1:00", "GMT+01:00");
+tests.put("GMT+01:00", "GMT+01:00");
+tests.put("GMT-1", "GMT-01:00");
+tests.put("GMT-1:00", "GMT-01:00");
+tests.put("GMT-01:00", "GMT-01:00");
+// values pushed by UI for jobs
+tests.put("GMT1", "GMT");
+tests.put("GMT1:00", "GMT");
+tests.put("GMT01:00", "GMT");
+// values generated by UI for datasets in a past
+tests.put("UTC1", "GMT");
+tests.put("UTC1:00", "GMT");
+tests.put("UTC01:00", "GMT");
+tests.put("UTC-1", "GMT");
+tests.put("UTC-1:00", "GMT");
+tests.put("UTC-01:00", "GMT");
+for (HashMap.Entry e: tests.entrySet()) {
--- End diff --
Consider
```
tests.entrySet().forEach(e -> {...}
```
as an option.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646540#comment-16646540
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224459867
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
--- End diff --
Please, use interface name instead of implementation.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646541#comment-16646541
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224479994
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
Looks like a hack (this and next lines). Probably worth to add a comment.
Or even extract common function as long as it happens twice in a code.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646539#comment-16646539
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224461843
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
+tests.put("", TimeZone.getDefault().getID());
+// standard cases
+tests.put("GMT", "GMT");
+tests.put("GMT+1", "GMT+01:00");
+tests.put("GMT+1:00", "GMT+01:00");
+tests.put("GMT+01:00", "GMT+01:00");
+tests.put("GMT-1", "GMT-01:00");
+tests.put("GMT-1:00", "GMT-01:00");
+tests.put("GMT-01:00", "GMT-01:00");
+// values pushed by UI for jobs
+tests.put("GMT1", "GMT");
+tests.put("GMT1:00", "GMT");
+tests.put("GMT01:00", "GMT");
+// values generated by UI for datasets in a past
+tests.put("UTC1", "GMT");
+tests.put("UTC1:00", "GMT");
+tests.put("UTC01:00", "GMT");
+tests.put("UTC-1", "GMT");
+tests.put("UTC-1:00", "GMT");
+tests.put("UTC-01:00", "GMT");
+for (HashMap.Entry e: tests.entrySet()) {
+String input = e.getKey();
+String actual = TimeUtil.getTimeZone(e.getKey()).getID();
--- End diff --
Reuse `input` here.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646021#comment-16646021
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
Went with UI route for a time being, to avoid introducing any incompatible
behavior from backend side.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645832#comment-16645832
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
@ahutsunshine I've traced time zone logic to your commit, are you aware of
problem described in a ticket? Challenge is that UI format is not in sync with
parsing logic. Basically there are two options:
- fix behavior on backend to understand format
- alternatively we can change UI to use right format
Fixing it on server side might be good (by making existing jobs and
examples created with wrong format to be working), from other hand it might
break compatibility, as jobs with malformed zones will start to run in new time
zone. What's your take on it?
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645644#comment-16645644
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
Looks like positive time zones without signs are not parsed properly, so
this is not a complete fix.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645389#comment-16645389
]
ASF GitHub Bot commented on GRIFFIN-206:
GitHub user chemikadze opened a pull request:
https://github.com/apache/incubator-griffin/pull/435
[GRIFFIN-206] fix job timezone when creating from UI
Demonstration:
```
> new Date()
Wed Oct 10 2018 11:30:48 GMT-0700 (Pacific Daylight Time)
> new Date().getTimezoneOffset()/60
7
```
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-206
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-griffin/pull/435.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #435
commit 03059a27a1f7fa8a7f839681f716fe1e66e0
Author: Nikolay Sokolov
Date: 2018-10-10T18:28:51Z
[GRIFFIN-206] fix job timezone when creating from UI
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-205) Accuracy measure check should provide matchedFraction to store
[
https://issues.apache.org/jira/browse/GRIFFIN-205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645166#comment-16645166
]
ASF GitHub Bot commented on GRIFFIN-205:
GitHub user ashutakGG opened a pull request:
https://github.com/apache/incubator-griffin/pull/434
Griffin-205: accuracy matched fraction
https://issues.apache.org/jira/browse/GRIFFIN-205
This pull request covers only batch dq type.
We need to decide is it worth to add "matched fraction" to streaming type.
Accuracy transformation tests added.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ashutakGG/incubator-griffin
GRIFFIN-205-accuracy-matchedFraction
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-griffin/pull/434.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #434
commit a5c90ed258281408c196008e97b682e6337d5a5c
Author: ashutak
Date: 2018-10-10T13:59:55Z
GRIFFIN-205: Added basic tests for accuracy dqType.
commit 9df3188def1c6076bf0618d73b5f1e73e93f522f
Author: ashutak
Date: 2018-10-10T15:38:34Z
GRIFFIN-205: Added matchedFraction for Batch dq type
> Accuracy measure check should provide matchedFraction to store
> --
>
> Key: GRIFFIN-205
> URL: https://issues.apache.org/jira/browse/GRIFFIN-205
> Project: Griffin (Incubating)
> Issue Type: Improvement
> Components: accuracy-batch, accuracy-real-time
>Affects Versions: 1.0.0-incubating
>Reporter: Artem Shutak
>Assignee: Artem Shutak
>Priority: Major
>
> Currently, {{accuracy}} measure results contains "total", "miss" and
> "matched" counts.
> As a result, It's hard to analyze accuracy fraction based on results stored
> in ElasticSearch, because ElasticSearch does not provide straight forward
> capability to get "field divided by field" query results.
> {{Accuracy}} measure results should also contain {{matchedFraction}} field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-204) postman collection is not up to date, some APIs don't work well, like create measure
[
https://issues.apache.org/jira/browse/GRIFFIN-204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643456#comment-16643456
]
ASF GitHub Bot commented on GRIFFIN-204:
GitHub user toyboxman opened a pull request:
https://github.com/apache/incubator-griffin/pull/433
Fix a bug in postman script
postman collection is not up to date, some APIs don't work well.
script has been refreshed in this patch by postman 2.1 standard.
Jira task: GRIFFIN-204
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/toyboxman/incubator-griffin doc/postman
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-griffin/pull/433.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #433
commit 46bc51526836353cf58e06947d23542415d2e64c
Author: Eugene
Date: 2018-10-09T13:13:35Z
Fix a bug in postman script
postman collection is not up to date, some APIs don't work well.
script has been refreshed in this patch by postman 2.1 standard.
Jira task: GRIFFIN-204
> postman collection is not up to date, some APIs don't work well, like create
> measure
>
>
> Key: GRIFFIN-204
> URL: https://issues.apache.org/jira/browse/GRIFFIN-204
> Project: Griffin (Incubating)
> Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Eugene
>Assignee: Eugene
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> apply
> [https://github.com/apache/incubator-griffin/blob/master/griffin-doc/service/postman/griffin.json,]
> and launch create measure api, then get an error. I find the json is not
> matched to latest implementation
> {
> "timestamp": 1539073909006,
> "status": 400,
> "error": "Bad Request",
> "exception":
> "org.springframework.http.converter.HttpMessageNotReadableException",
> "message": "Could not read document: Can not construct instance of
> org.apache.griffin.core.measure.entity.DqType from String value 'accuracy':
> value not one of declared Enum instance names: [COMPLETENESS, TIMELINESS,
> PROFILING, CONSISTENCY, ACCURACY, UNIQUENESS]\n at [Source:
> java.io.PushbackInputStream@565f5e70; line: 4, column: 15] (through reference
> chain: org.apache.griffin.core.measure.entity.GriffinMeasure[\"dq.type\"]);
> nested exception is
> com.fasterxml.jackson.databind.exc.InvalidFormatException: Can not construct
> instance of org.apache.griffin.core.measure.entity.DqType from String value
> 'accuracy': value not one of declared Enum instance names: [COMPLETENESS,
> TIMELINESS, PROFILING, CONSISTENCY, ACCURACY, UNIQUENESS]\n at [Source:
> java.io.PushbackInputStream@565f5e70; line: 4, column: 15] (through reference
> chain: org.apache.griffin.core.measure.entity.GriffinMeasure[\"dq.type\"])",
> "path": "/api/v1/measures"
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-195) [UI] Don't list all table objects from UI
[ https://issues.apache.org/jira/browse/GRIFFIN-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641353#comment-16641353 ] ASF GitHub Bot commented on GRIFFIN-195: Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/431 > [UI] Don't list all table objects from UI > - > > Key: GRIFFIN-195 > URL: https://issues.apache.org/jira/browse/GRIFFIN-195 > Project: Griffin (Incubating) > Issue Type: Sub-task >Reporter: Nikolay Sokolov >Priority: Major > > Listing all table objects when rendering list of tables to profile takes lots > of time, and adds significant latency even if response is cached, when number > of tables is big and/or schema contains lots of columns. > Two solutions are possible: > 1) Without GRIFFIN-194, use DB list API and then list tables API is called > for each DB, in order to collect all tables. When table is clicked on, > request to get table details can be made. > 2) With GRIFFIN-194, DB list followed by number of table list operations can > be replaced by "list all table names in all databases" operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-195) [UI] Don't list all table objects from UI
[ https://issues.apache.org/jira/browse/GRIFFIN-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641352#comment-16641352 ] ASF GitHub Bot commented on GRIFFIN-195: Github user bhlx3lyx7 commented on the issue: https://github.com/apache/incubator-griffin/pull/431 Cool, seems good to me, I can merge it now. > [UI] Don't list all table objects from UI > - > > Key: GRIFFIN-195 > URL: https://issues.apache.org/jira/browse/GRIFFIN-195 > Project: Griffin (Incubating) > Issue Type: Sub-task >Reporter: Nikolay Sokolov >Priority: Major > > Listing all table objects when rendering list of tables to profile takes lots > of time, and adds significant latency even if response is cached, when number > of tables is big and/or schema contains lots of columns. > Two solutions are possible: > 1) Without GRIFFIN-194, use DB list API and then list tables API is called > for each DB, in order to collect all tables. When table is clicked on, > request to get table details can be made. > 2) With GRIFFIN-194, DB list followed by number of table list operations can > be replaced by "list all table names in all databases" operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-195) [UI] Don't list all table objects from UI
[ https://issues.apache.org/jira/browse/GRIFFIN-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641329#comment-16641329 ] ASF GitHub Bot commented on GRIFFIN-195: Github user chemikadze commented on the issue: https://github.com/apache/incubator-griffin/pull/431 @bhlx3lyx7 pushed the same for accuracy > [UI] Don't list all table objects from UI > - > > Key: GRIFFIN-195 > URL: https://issues.apache.org/jira/browse/GRIFFIN-195 > Project: Griffin (Incubating) > Issue Type: Sub-task >Reporter: Nikolay Sokolov >Priority: Major > > Listing all table objects when rendering list of tables to profile takes lots > of time, and adds significant latency even if response is cached, when number > of tables is big and/or schema contains lots of columns. > Two solutions are possible: > 1) Without GRIFFIN-194, use DB list API and then list tables API is called > for each DB, in order to collect all tables. When table is clicked on, > request to get table details can be made. > 2) With GRIFFIN-194, DB list followed by number of table list operations can > be replaced by "list all table names in all databases" operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-199) [UI] Non-descriptive message when job creation failed
[
https://issues.apache.org/jira/browse/GRIFFIN-199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641310#comment-16641310
]
ASF GitHub Bot commented on GRIFFIN-199:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/423
Closes GRIFFIN-199
> [UI] Non-descriptive message when job creation failed
> -
>
> Key: GRIFFIN-199
> URL: https://issues.apache.org/jira/browse/GRIFFIN-199
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> When job creation fails, non-descriptive pop-up is shown with message "Error:
> Error when creating job!".
> At the same time, API error contains "code", "error" and "message" fields,
> which could be useful part of this message:
> {code:none}
> {
> "timestamp":1537912602882,
> "status":400,
> "error":"Bad Request",
> "code":"40010",
> "message":"Property 'cron.expression' is invalid",
> "path":"/api/v1/jobs"
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-195) [UI] Don't list all table objects from UI
[ https://issues.apache.org/jira/browse/GRIFFIN-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641309#comment-16641309 ] ASF GitHub Bot commented on GRIFFIN-195: Github user chemikadze commented on the issue: https://github.com/apache/incubator-griffin/pull/431 @bhlx3lyx7 my thoughts exactly, will push in a few minutes > [UI] Don't list all table objects from UI > - > > Key: GRIFFIN-195 > URL: https://issues.apache.org/jira/browse/GRIFFIN-195 > Project: Griffin (Incubating) > Issue Type: Sub-task >Reporter: Nikolay Sokolov >Priority: Major > > Listing all table objects when rendering list of tables to profile takes lots > of time, and adds significant latency even if response is cached, when number > of tables is big and/or schema contains lots of columns. > Two solutions are possible: > 1) Without GRIFFIN-194, use DB list API and then list tables API is called > for each DB, in order to collect all tables. When table is clicked on, > request to get table details can be made. > 2) With GRIFFIN-194, DB list followed by number of table list operations can > be replaced by "list all table names in all databases" operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-195) [UI] Don't list all table objects from UI
[ https://issues.apache.org/jira/browse/GRIFFIN-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641299#comment-16641299 ] ASF GitHub Bot commented on GRIFFIN-195: Github user bhlx3lyx7 commented on the issue: https://github.com/apache/incubator-griffin/pull/431 I've tested this PR, the filtering feature works good, but only in profiling creation step. Would you add it in accuracy creation steps as well? > [UI] Don't list all table objects from UI > - > > Key: GRIFFIN-195 > URL: https://issues.apache.org/jira/browse/GRIFFIN-195 > Project: Griffin (Incubating) > Issue Type: Sub-task >Reporter: Nikolay Sokolov >Priority: Major > > Listing all table objects when rendering list of tables to profile takes lots > of time, and adds significant latency even if response is cached, when number > of tables is big and/or schema contains lots of columns. > Two solutions are possible: > 1) Without GRIFFIN-194, use DB list API and then list tables API is called > for each DB, in order to collect all tables. When table is clicked on, > request to get table details can be made. > 2) With GRIFFIN-194, DB list followed by number of table list operations can > be replaced by "list all table names in all databases" operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-130) [Site] Modify griffin site template
[ https://issues.apache.org/jira/browse/GRIFFIN-130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16641262#comment-16641262 ] ASF GitHub Bot commented on GRIFFIN-130: Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin-site/pull/2 > [Site] Modify griffin site template > --- > > Key: GRIFFIN-130 > URL: https://issues.apache.org/jira/browse/GRIFFIN-130 > Project: Griffin (Incubating) > Issue Type: Task >Reporter: Lionel Liu >Assignee: Juan Li >Priority: Major > Original Estimate: 24h > Remaining Estimate: 24h > > Change the template of griffin site -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-195) [UI] Don't list all table objects from UI
[ https://issues.apache.org/jira/browse/GRIFFIN-195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16640994#comment-16640994 ] ASF GitHub Bot commented on GRIFFIN-195: GitHub user chemikadze opened a pull request: https://github.com/apache/incubator-griffin/pull/431 [GRIFFIN-195] Don't list all tables from UI Initially, load only list of databases and list of table names. Table information is loaded only if user clicks on specific table. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chemikadze/incubator-griffin GRIFFIN-193 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/431.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #431 commit bca5bc78a316c435742df1b862f7c8f4b09e1458 Author: Nikolay Sokolov Date: 2018-10-07T00:39:07Z [GRIFFIN-195] Don't list all tables from UI Initially, load only list of databases and list of table names. Table information is loaded only if user clicks on specific table. commit 366427e747d6fa2aa2149efc9b79dab5559814d0 Author: Nikolay Sokolov Date: 2018-10-07T08:14:40Z Trigger tree rendering only when all dbs loaded > [UI] Don't list all table objects from UI > - > > Key: GRIFFIN-195 > URL: https://issues.apache.org/jira/browse/GRIFFIN-195 > Project: Griffin (Incubating) > Issue Type: Sub-task >Reporter: Nikolay Sokolov >Priority: Major > > Listing all table objects when rendering list of tables to profile takes lots > of time, and adds significant latency even if response is cached, when number > of tables is big and/or schema contains lots of columns. > Two solutions are possible: > 1) Without GRIFFIN-194, use DB list API and then list tables API is called > for each DB, in order to collect all tables. When table is clicked on, > request to get table details can be made. > 2) With GRIFFIN-194, DB list followed by number of table list operations can > be replaced by "list all table names in all databases" operation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (GRIFFIN-197) Job is left in UNKNOWN state by Service if Yarn RM is restarted
[
https://issues.apache.org/jira/browse/GRIFFIN-197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633278#comment-16633278
]
ASF GitHub Bot commented on GRIFFIN-197:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-griffin/pull/421
> Job is left in UNKNOWN state by Service if Yarn RM is restarted
> ---
>
> Key: GRIFFIN-197
> URL: https://issues.apache.org/jira/browse/GRIFFIN-197
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> From one hand, according to Livy behavior, missing app can be treaded as DEAD
> state.
> From other hand, logged client errors are polluting the log with unnecessary
> stack traces, but not showing error description, returned by Yarn.
> Sample stack trace on Service side:
> {code:none}
> 2018-09-21 14:30:58.016 WARN 14699 --- [nio-8080-exec-4]
> o.a.g.c.j.JobServiceImpl : sessionId(300)
> appId(application_1534940268145_0318) 404 Not Found.
> 2018-09-21 14:30:58.016 WARN 14699 --- [nio-8080-exec-4]
> o.a.g.c.j.JobServiceImpl : Spark session 300 may be overdue!
> Now we use yarn to update state.
> 2018-09-21 14:30:58.020 ERROR 14699 --- [nio-8080-exec-4]
> o.a.g.c.u.YarnNetUtil: update exception happens by yarn.
> {}
> org.springframework.web.client.HttpClientErrorException: 404 Not Found
> at
> org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:287)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.apache.griffin.core.util.YarnNetUtil.update(YarnNetUtil.java:53)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobServiceImpl.setStateByYarn(JobServiceImpl.java:569)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobServiceImpl.setStateByYarn(JobServiceImpl.java:530)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobServiceImpl.syncInstancesOfJob(JobServiceImpl.java:514)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobServiceImpl.updateState(JobServiceImpl.java:274)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobServiceImpl.findInstancesOfJob(JobServiceImpl.java:267)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobController.findInstancesOfJob(JobController.java:94)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
> ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> [spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-197) Job is left in UNKNOWN state by Service if Yarn RM is restarted
[
https://issues.apache.org/jira/browse/GRIFFIN-197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16625350#comment-16625350
]
ASF GitHub Bot commented on GRIFFIN-197:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/421#discussion_r219724832
--- Diff:
service/src/main/java/org/apache/griffin/core/util/YarnNetUtil.java ---
@@ -56,6 +62,14 @@ public static boolean update(String url, JobInstanceBean
instance) {
instance.setState(LivySessionStates.toLivyState(state));
}
return true;
+} catch (HttpClientErrorException e) {
+LOGGER.warn("client error {} from yarn: {}",
+e.getMessage(), e.getResponseBodyAsString());
+if (e.getStatusCode() == HttpStatus.NOT_FOUND) {
+// in sync with Livy behavior, see
com.cloudera.livy.utils.SparkYarnApp
+instance.setState(DEAD);
--- End diff --
Only 404 is handled here, which should not be result of network issue.
It looks like any kind of error reported by Yarn client (after internal
retries) results in DEADing job on Livy side:
https://github.com/cloudera/livy/blob/master/server/src/main/scala/com/cloudera/livy/utils/SparkYarnApp.scala#L307
I'll need to double check whether not found applications are ever getting
retried, to make sure behavior is same as on Livy side. If not -- then that's
what Livy would do.
> Job is left in UNKNOWN state by Service if Yarn RM is restarted
> ---
>
> Key: GRIFFIN-197
> URL: https://issues.apache.org/jira/browse/GRIFFIN-197
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Priority: Minor
>
> From one hand, according to Livy behavior, missing app can be treaded as DEAD
> state.
> From other hand, logged client errors are polluting the log with unnecessary
> stack traces, but not showing error description, returned by Yarn.
> Sample stack trace on Service side:
> {code:none}
> 2018-09-21 14:30:58.016 WARN 14699 --- [nio-8080-exec-4]
> o.a.g.c.j.JobServiceImpl : sessionId(300)
> appId(application_1534940268145_0318) 404 Not Found.
> 2018-09-21 14:30:58.016 WARN 14699 --- [nio-8080-exec-4]
> o.a.g.c.j.JobServiceImpl : Spark session 300 may be overdue!
> Now we use yarn to update state.
> 2018-09-21 14:30:58.020 ERROR 14699 --- [nio-8080-exec-4]
> o.a.g.c.u.YarnNetUtil: update exception happens by yarn.
> {}
> org.springframework.web.client.HttpClientErrorException: 404 Not Found
> at
> org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.handleResponse(RestTemplate.java:700)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:653)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.execute(RestTemplate.java:613)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.springframework.web.client.RestTemplate.getForObject(RestTemplate.java:287)
> ~[spring-web-4.3.6.RELEASE.jar!/:4.3.6.RELEASE]
> at
> org.apache.griffin.core.util.YarnNetUtil.update(YarnNetUtil.java:53)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobServiceImpl.setStateByYarn(JobServiceImpl.java:569)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobServiceImpl.setStateByYarn(JobServiceImpl.java:530)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobServiceImpl.syncInstancesOfJob(JobServiceImpl.java:514)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobServiceImpl.updateState(JobServiceImpl.java:274)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobServiceImpl.findInstancesOfJob(JobServiceImpl.java:267)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at
> org.apache.griffin.core.job.JobController.findInstancesOfJob(JobController.java:94)
> [classes!/:0.3.1-incubating-SNAPSHOT]
> at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source)
> ~[?:?]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_181]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_181]
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
>
[jira] [Commented] (GRIFFIN-198) HiveMetaStoreServiceImpl caches incorrect state of metastore
[ https://issues.apache.org/jira/browse/GRIFFIN-198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16624679#comment-16624679 ] ASF GitHub Bot commented on GRIFFIN-198: Github user asfgit closed the pull request at: https://github.com/apache/incubator-griffin/pull/419 > HiveMetaStoreServiceImpl caches incorrect state of metastore > > > Key: GRIFFIN-198 > URL: https://issues.apache.org/jira/browse/GRIFFIN-198 > Project: Griffin (Incubating) > Issue Type: Bug >Reporter: Nikolay Sokolov >Priority: Major > > Currently, HiveMetaStoreServiceImpl is caching: > * nulls and empty lists in some methods, when metastore request failed for > some reason > * empty list of tables as response to getAllTable(String) and getAllTable() > calls > Both problems result in confusing output on large amount of tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
