[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647108#comment-16647108
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224622499
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
Looks like there big bunch of different techniques, half of them is based
on regex and way browser formats date, another half requires third-party
libraries. Apparently, `getTimeZoneOffset()` is "accepted answer" over here:
https://stackoverflow.com/questions/1091372/getting-the-clients-timezone-in-javascript
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647096#comment-16647096
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224620459
--- Diff:
ui/angular/src/app/measure/create-measure/configuration/configuration.component.ts
---
@@ -56,47 +56,47 @@ export class ConfigurationComponent implements OnInit {
};
timetypes = ["day", "hour", "minute"];
timetype: string;
- timezones = [
-"UTC-12(IDL)",
-"UTC-11(MIT)",
-"UTC-10(HST)",
-"UTC-9:30(MSIT)",
-"UTC-9(AKST)",
-"UTC-8(PST)",
-"UTC-7(MST)",
-"UTC-6(CST)",
-"UTC-5(EST)",
-"UTC-4(AST)",
-"UTC-3:30(NST)",
-"UTC-3(SAT)",
-"UTC-2(BRT)",
-"UTC-1(CVT)",
-"UTC(WET,GMT)",
-"UTC+1(CET)",
-"UTC+2(EET)",
-"UTC+3(MSK)",
-"UTC+3:30(IRT)",
-"UTC+4(META)",
-"UTC+4:30(AFT)",
-"UTC+5(METB)",
-"UTC+5:30(IDT)",
-"UTC+5:45(NPT)",
-"UTC+6(BHT)",
-"UTC+6:30(MRT)",
-"UTC+7(IST)",
-"UTC+8(EAT)",
-"UTC+8:30(KRT)",
-"UTC+9(FET)",
-"UTC+9:30(ACST)",
-"UTC+10(AEST)",
-"UTC+10:30(FAST)",
-"UTC+11(VTT)",
-"UTC+11:30(NFT)",
-"UTC+12(PSTB)",
-"UTC+12:45(CIT)",
-"UTC+13(PSTC)",
-"UTC+14(PSTD)"
- ];
+ timezones: Map = new Map([
+["UTC-12(IDL)", "GMT-12"],
+["UTC-11(MIT)", "GMT-11"],
+["UTC-10(HST)", "GMT-10"],
+["UTC-9:30(MSIT)", "GMT-9:30"],
+["UTC-9(AKST)", "GMT-9"],
+["UTC-8(PST)", "GMT-8"],
+["UTC-7(MST)", "GMT-7"],
+["UTC-6(CST)", "GMT-6"],
+["UTC-5(EST)", "GMT-5"],
+["UTC-4(AST)", "GMT-4"],
+["UTC-3:30(NST)", "GMT-3:30"],
+["UTC-3(SAT)", "GMT-3"],
+["UTC-2(BRT)", "GMT-2"],
+["UTC-1(CVT)", "GMT-1"],
+["UTC(WET,GMT)", "GMT"],
+["UTC+1(CET)", "GMT+1"],
+["UTC+2(EET)", "GMT+2"],
+["UTC+3(MSK)", "GMT+3"],
+["UTC+3:30(IRT)", "GMT+3:30"],
+["UTC+4(META)", "GMT+4"],
+["UTC+4:30(AFT)", "GMT+4:30"],
+["UTC+5(METB)", "GMT+5"],
+["UTC+5:30(IDT)", "GMT+5:30"],
+["UTC+5:45(NPT)", "GMT+5:45"],
+["UTC+6(BHT)", "GMT+6"],
+["UTC+6:30(MRT)", "GMT+6:30"],
+["UTC+7(IST)", "GMT+7"],
+["UTC+8(EAT)", "GMT+8"],
+["UTC+8:30(KRT)", "GMT+8:30"],
+["UTC+9(FET)", "GMT+9"],
+["UTC+9:30(ACST)", "GMT+9:30"],
+["UTC+10(AEST)", "GMT+10"],
+["UTC+10:30(FAST)", "GMT+10:30"],
+["UTC+11(VTT)", "GMT+11"],
+["UTC+11:30(NFT)", "GMT+11:30"],
+["UTC+12(PSTB)", "GMT+12"],
+["UTC+12:45(CIT)", "GMT+12:45"],
+["UTC+13(PSTC)", "GMT+13"],
+["UTC+14(PSTD)", "GMT+15"],
--- End diff --
good catch, thanks
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #435: [GRIFFIN-206] fix job timezone when cre...
Github user chemikadze commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224620459
--- Diff:
ui/angular/src/app/measure/create-measure/configuration/configuration.component.ts
---
@@ -56,47 +56,47 @@ export class ConfigurationComponent implements OnInit {
};
timetypes = ["day", "hour", "minute"];
timetype: string;
- timezones = [
-"UTC-12(IDL)",
-"UTC-11(MIT)",
-"UTC-10(HST)",
-"UTC-9:30(MSIT)",
-"UTC-9(AKST)",
-"UTC-8(PST)",
-"UTC-7(MST)",
-"UTC-6(CST)",
-"UTC-5(EST)",
-"UTC-4(AST)",
-"UTC-3:30(NST)",
-"UTC-3(SAT)",
-"UTC-2(BRT)",
-"UTC-1(CVT)",
-"UTC(WET,GMT)",
-"UTC+1(CET)",
-"UTC+2(EET)",
-"UTC+3(MSK)",
-"UTC+3:30(IRT)",
-"UTC+4(META)",
-"UTC+4:30(AFT)",
-"UTC+5(METB)",
-"UTC+5:30(IDT)",
-"UTC+5:45(NPT)",
-"UTC+6(BHT)",
-"UTC+6:30(MRT)",
-"UTC+7(IST)",
-"UTC+8(EAT)",
-"UTC+8:30(KRT)",
-"UTC+9(FET)",
-"UTC+9:30(ACST)",
-"UTC+10(AEST)",
-"UTC+10:30(FAST)",
-"UTC+11(VTT)",
-"UTC+11:30(NFT)",
-"UTC+12(PSTB)",
-"UTC+12:45(CIT)",
-"UTC+13(PSTC)",
-"UTC+14(PSTD)"
- ];
+ timezones: Map = new Map([
+["UTC-12(IDL)", "GMT-12"],
+["UTC-11(MIT)", "GMT-11"],
+["UTC-10(HST)", "GMT-10"],
+["UTC-9:30(MSIT)", "GMT-9:30"],
+["UTC-9(AKST)", "GMT-9"],
+["UTC-8(PST)", "GMT-8"],
+["UTC-7(MST)", "GMT-7"],
+["UTC-6(CST)", "GMT-6"],
+["UTC-5(EST)", "GMT-5"],
+["UTC-4(AST)", "GMT-4"],
+["UTC-3:30(NST)", "GMT-3:30"],
+["UTC-3(SAT)", "GMT-3"],
+["UTC-2(BRT)", "GMT-2"],
+["UTC-1(CVT)", "GMT-1"],
+["UTC(WET,GMT)", "GMT"],
+["UTC+1(CET)", "GMT+1"],
+["UTC+2(EET)", "GMT+2"],
+["UTC+3(MSK)", "GMT+3"],
+["UTC+3:30(IRT)", "GMT+3:30"],
+["UTC+4(META)", "GMT+4"],
+["UTC+4:30(AFT)", "GMT+4:30"],
+["UTC+5(METB)", "GMT+5"],
+["UTC+5:30(IDT)", "GMT+5:30"],
+["UTC+5:45(NPT)", "GMT+5:45"],
+["UTC+6(BHT)", "GMT+6"],
+["UTC+6:30(MRT)", "GMT+6:30"],
+["UTC+7(IST)", "GMT+7"],
+["UTC+8(EAT)", "GMT+8"],
+["UTC+8:30(KRT)", "GMT+8:30"],
+["UTC+9(FET)", "GMT+9"],
+["UTC+9:30(ACST)", "GMT+9:30"],
+["UTC+10(AEST)", "GMT+10"],
+["UTC+10:30(FAST)", "GMT+10:30"],
+["UTC+11(VTT)", "GMT+11"],
+["UTC+11:30(NFT)", "GMT+11:30"],
+["UTC+12(PSTB)", "GMT+12"],
+["UTC+12:45(CIT)", "GMT+12:45"],
+["UTC+13(PSTC)", "GMT+13"],
+["UTC+14(PSTD)", "GMT+15"],
--- End diff --
good catch, thanks
---
[GitHub] incubator-griffin pull request #435: [GRIFFIN-206] fix job timezone when cre...
Github user gavlyukovskiy commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224560857
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
+tests.put("", TimeZone.getDefault().getID());
+// standard cases
+tests.put("GMT", "GMT");
+tests.put("GMT+1", "GMT+01:00");
+tests.put("GMT+1:00", "GMT+01:00");
+tests.put("GMT+01:00", "GMT+01:00");
+tests.put("GMT-1", "GMT-01:00");
+tests.put("GMT-1:00", "GMT-01:00");
+tests.put("GMT-01:00", "GMT-01:00");
+// values pushed by UI for jobs
+tests.put("GMT1", "GMT");
+tests.put("GMT1:00", "GMT");
+tests.put("GMT01:00", "GMT");
+// values generated by UI for datasets in a past
+tests.put("UTC1", "GMT");
+tests.put("UTC1:00", "GMT");
+tests.put("UTC01:00", "GMT");
+tests.put("UTC-1", "GMT");
+tests.put("UTC-1:00", "GMT");
+tests.put("UTC-01:00", "GMT");
+for (HashMap.Entry e: tests.entrySet()) {
--- End diff --
```
tests.forEach((key, value) -> {...})
```
Looks even better :)
---
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646880#comment-16646880
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user gavlyukovskiy commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224560857
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
+tests.put("", TimeZone.getDefault().getID());
+// standard cases
+tests.put("GMT", "GMT");
+tests.put("GMT+1", "GMT+01:00");
+tests.put("GMT+1:00", "GMT+01:00");
+tests.put("GMT+01:00", "GMT+01:00");
+tests.put("GMT-1", "GMT-01:00");
+tests.put("GMT-1:00", "GMT-01:00");
+tests.put("GMT-01:00", "GMT-01:00");
+// values pushed by UI for jobs
+tests.put("GMT1", "GMT");
+tests.put("GMT1:00", "GMT");
+tests.put("GMT01:00", "GMT");
+// values generated by UI for datasets in a past
+tests.put("UTC1", "GMT");
+tests.put("UTC1:00", "GMT");
+tests.put("UTC01:00", "GMT");
+tests.put("UTC-1", "GMT");
+tests.put("UTC-1:00", "GMT");
+tests.put("UTC-01:00", "GMT");
+for (HashMap.Entry e: tests.entrySet()) {
--- End diff --
```
tests.forEach((key, value) -> {...})
```
Looks even better :)
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
GRIFFIN-205 split on 2 tickets
Hello all, In process of working on https://issues.apache.org/jira/browse/GRIFFIN-205 I'v implemented a patch for "batch" dq type. Now I see at least 2 options: - Apply the patch as is and create separate ticket for "streaming" dq type. As an advantage here we will improve "batch" dq type as soon as possible (and it covers our needs for now). A disadvantage is an inconsistent functionality provided by "batch" and "streaming" types. - Implement the same logic for streaming and only then merge it to master. Need to keep in mind, that for batch case I've added automated tests which cover basic cases. Looks like for "streaming" it will be much more difficult. It requires more changes in code (due to some abstractions leaks in streaming processing) and as a result it requires quite good testing, but we don't use this mode in our cases (yet). My question is how strong is Apache Griffin's features consistency regarding "batch" and "streaming" dq types? I would go with the first option if you don't mind. Objections? Thanks, Artem.
RE: Metrics not persisted when writing a query in SPARK-SQL insteadof Griffin DSL
That’s cool!
Thanks
Lionel, Liu
From: Vikram Jain
Sent: 2018年10月11日 22:52
To: Lionel Liu
Cc: dev@griffin.incubator.apache.org
Subject: Re: Metrics not persisted when writing a query in SPARK-SQL insteadof
Griffin DSL
Thank you Lionel for your help.
We figured it out just before your mail arrived :)
Regards,
Vikram
On 11-Oct-2018, at 8:20 PM, Lionel Liu wrote:
Hi Vikram,
In your JSON body, I notice that in the "rules" field, there's no "out" field,
which means griffin measure application will only calculate without output. You
might just changed the "dsl.type" from "griffin-dsl" to "spark-sql", actually,
for a "griffin-dsl" rule with "dq.type" as "profiling", we create a output for
it in transform phase:
https://github.com/apache/incubator-griffin/blob/griffin-0.3.0-incubating-rc1/measure/src/main/scala/org/apache/griffin/measure/step/builder/dsl/transform/ProfilingExpr2DQSteps.scala#L97,
but for a "spark-sql" rule, we don't parse it, so we don't know how it would
work, you need to manually configure the output field to enable it.
You can refer to this document to configure the output field:
https://github.com/apache/incubator-griffin/blob/master/griffin-doc/measure/measure-configuration-guide.md#rule
Or just simply refer to the demo json for spark-sql profiling rules:
https://github.com/apache/incubator-griffin/blob/griffin-0.3.0-incubating-rc1/measure/src/test/resources/_profiling-batch-sparksql.json
Hope this could help you.
--
Regards,
Lionel, Liu
At 2018-10-11 17:30:29, "Vikram Jain" wrote:
>Hello,
>
>I was trying to create a measure and write the rule in Spark-SQL directly
>instead of Griffin-DSL. I use Postman to create the measure. The measure is
>created successfully, the job is created and executed successfully.
>
>However, the output metrics of execution of jobs are not persisted in
>ElasticSearch. The entry is created in Elastic but the "metricValues" array is
>NULL.
>
>The same SQL query works fine directly on Spark-Shell.
>
>I am not using Docker and building the environment (Griffin 3.0) on my local
>machine. All the measures created using UI are executing well. And measures
>created using Postman with griffin-dsl rule are also working well.
>
>Below is the body of json which I am passing to add measure API call from
>Postman. Please help me understand what is going wrong.
>
>
>{
> "name": "custom_profiling_measure_2",
> "measure.type": "griffin",
> "dq.type": "PROFILING",
> "rule.description": {
> "details": [
> {
> "name": "id",
> "infos": "Total Count"
> }
> ]
> },
> "process.type": "BATCH",
> "owner": "test",
> "description": "custom_profiling_measure_2",
> "data.sources": [
> {
> "name": "source",
> "connectors": [
> {
> "name": "source123",
> "type": "HIVE",
> "version": "1.2",
> "data.unit": "1day",
> "data.time.zone": "",
> "config": {
> "database": "default",
> "table.name": "demo_src",
> "where": ""
> }
> }
> ]
> }
> ],
> "evaluate.rule": {
> "out.dataframe.name": "profiling_2",
> "rules": [
> {
> "dsl.type": "spark-sql",
> "dq.type": "PROFILING",
> "rule": "SELECT count(id) AS cnt, max(age) AS Max_Age from demo_src",
> "out.dataframe.name": "id_count_2"
> }
> ]
> }
>}
>
>
>
>
>
>Regards,
>
>Vikram
>
[GitHub] incubator-griffin pull request #436: always use apache griffin together as o...
Github user ashutakGG commented on a diff in the pull request: https://github.com/apache/incubator-griffin/pull/436#discussion_r224487868 --- Diff: griffin-doc/dev/dev-env-build.md --- @@ -85,7 +85,7 @@ mvn clean install ``` ### For service module and ui module -1. Login to docker container, and stop running griffin service. +1. Login to docker container, and stop running apache griffin service. --- End diff -- Probably it's better to use `Apache Griffin` with capital letters everywhere. ---
[GitHub] incubator-griffin pull request #434: Griffin-205: accuracy matched fraction
Github user guoyuepeng commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/434#discussion_r224486025
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/configuration/dqdefinition/DQConfig.scala
---
@@ -74,9 +74,9 @@ case class DQConfig(@JsonProperty("name") private val
name: String,
*/
@JsonInclude(Include.NON_NULL)
case class DataSourceParam( @JsonProperty("name") private val name: String,
-@JsonProperty("baseline") private val
baseline: Boolean,
--- End diff --
LGTM
---
[GitHub] incubator-griffin pull request #436: always use apache griffin together as o...
GitHub user guoyuepeng opened a pull request: https://github.com/apache/incubator-griffin/pull/436 always use apache griffin together as our mark You can merge this pull request into a Git repository by running: $ git pull https://github.com/guoyuepeng/incubator-griffin use_apache_griffin_as_trademark Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/436.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #436 commit bba61d6be4b4c77a30560e08362aee01de4d17a9 Author: William Guo Date: 2018-10-11T14:58:06Z always use apache griffin together as our mark ---
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646548#comment-16646548
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224482419
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
Is there some best practices regarding such logic in angular?
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #435: [GRIFFIN-206] fix job timezone when cre...
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224482419
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
Is there some best practices regarding such logic in angular?
---
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646543#comment-16646543
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224481345
--- Diff:
ui/angular/src/app/measure/create-measure/configuration/configuration.component.ts
---
@@ -56,47 +56,47 @@ export class ConfigurationComponent implements OnInit {
};
timetypes = ["day", "hour", "minute"];
timetype: string;
- timezones = [
-"UTC-12(IDL)",
-"UTC-11(MIT)",
-"UTC-10(HST)",
-"UTC-9:30(MSIT)",
-"UTC-9(AKST)",
-"UTC-8(PST)",
-"UTC-7(MST)",
-"UTC-6(CST)",
-"UTC-5(EST)",
-"UTC-4(AST)",
-"UTC-3:30(NST)",
-"UTC-3(SAT)",
-"UTC-2(BRT)",
-"UTC-1(CVT)",
-"UTC(WET,GMT)",
-"UTC+1(CET)",
-"UTC+2(EET)",
-"UTC+3(MSK)",
-"UTC+3:30(IRT)",
-"UTC+4(META)",
-"UTC+4:30(AFT)",
-"UTC+5(METB)",
-"UTC+5:30(IDT)",
-"UTC+5:45(NPT)",
-"UTC+6(BHT)",
-"UTC+6:30(MRT)",
-"UTC+7(IST)",
-"UTC+8(EAT)",
-"UTC+8:30(KRT)",
-"UTC+9(FET)",
-"UTC+9:30(ACST)",
-"UTC+10(AEST)",
-"UTC+10:30(FAST)",
-"UTC+11(VTT)",
-"UTC+11:30(NFT)",
-"UTC+12(PSTB)",
-"UTC+12:45(CIT)",
-"UTC+13(PSTC)",
-"UTC+14(PSTD)"
- ];
+ timezones: Map = new Map([
+["UTC-12(IDL)", "GMT-12"],
+["UTC-11(MIT)", "GMT-11"],
+["UTC-10(HST)", "GMT-10"],
+["UTC-9:30(MSIT)", "GMT-9:30"],
+["UTC-9(AKST)", "GMT-9"],
+["UTC-8(PST)", "GMT-8"],
+["UTC-7(MST)", "GMT-7"],
+["UTC-6(CST)", "GMT-6"],
+["UTC-5(EST)", "GMT-5"],
+["UTC-4(AST)", "GMT-4"],
+["UTC-3:30(NST)", "GMT-3:30"],
+["UTC-3(SAT)", "GMT-3"],
+["UTC-2(BRT)", "GMT-2"],
+["UTC-1(CVT)", "GMT-1"],
+["UTC(WET,GMT)", "GMT"],
+["UTC+1(CET)", "GMT+1"],
+["UTC+2(EET)", "GMT+2"],
+["UTC+3(MSK)", "GMT+3"],
+["UTC+3:30(IRT)", "GMT+3:30"],
+["UTC+4(META)", "GMT+4"],
+["UTC+4:30(AFT)", "GMT+4:30"],
+["UTC+5(METB)", "GMT+5"],
+["UTC+5:30(IDT)", "GMT+5:30"],
+["UTC+5:45(NPT)", "GMT+5:45"],
+["UTC+6(BHT)", "GMT+6"],
+["UTC+6:30(MRT)", "GMT+6:30"],
+["UTC+7(IST)", "GMT+7"],
+["UTC+8(EAT)", "GMT+8"],
+["UTC+8:30(KRT)", "GMT+8:30"],
+["UTC+9(FET)", "GMT+9"],
+["UTC+9:30(ACST)", "GMT+9:30"],
+["UTC+10(AEST)", "GMT+10"],
+["UTC+10:30(FAST)", "GMT+10:30"],
+["UTC+11(VTT)", "GMT+11"],
+["UTC+11:30(NFT)", "GMT+11:30"],
+["UTC+12(PSTB)", "GMT+12"],
+["UTC+12:45(CIT)", "GMT+12:45"],
+["UTC+13(PSTC)", "GMT+13"],
+["UTC+14(PSTD)", "GMT+15"],
--- End diff --
14 -> 15? Is it done intentionally?
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646542#comment-16646542
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224461713
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
+tests.put("", TimeZone.getDefault().getID());
+// standard cases
+tests.put("GMT", "GMT");
+tests.put("GMT+1", "GMT+01:00");
+tests.put("GMT+1:00", "GMT+01:00");
+tests.put("GMT+01:00", "GMT+01:00");
+tests.put("GMT-1", "GMT-01:00");
+tests.put("GMT-1:00", "GMT-01:00");
+tests.put("GMT-01:00", "GMT-01:00");
+// values pushed by UI for jobs
+tests.put("GMT1", "GMT");
+tests.put("GMT1:00", "GMT");
+tests.put("GMT01:00", "GMT");
+// values generated by UI for datasets in a past
+tests.put("UTC1", "GMT");
+tests.put("UTC1:00", "GMT");
+tests.put("UTC01:00", "GMT");
+tests.put("UTC-1", "GMT");
+tests.put("UTC-1:00", "GMT");
+tests.put("UTC-01:00", "GMT");
+for (HashMap.Entry e: tests.entrySet()) {
--- End diff --
Consider
```
tests.entrySet().forEach(e -> {...}
```
as an option.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646540#comment-16646540
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224459867
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
--- End diff --
Please, use interface name instead of implementation.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646541#comment-16646541
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224479994
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
Looks like a hack (this and next lines). Probably worth to add a comment.
Or even extract common function as long as it happens twice in a code.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646539#comment-16646539
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224461843
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
+tests.put("", TimeZone.getDefault().getID());
+// standard cases
+tests.put("GMT", "GMT");
+tests.put("GMT+1", "GMT+01:00");
+tests.put("GMT+1:00", "GMT+01:00");
+tests.put("GMT+01:00", "GMT+01:00");
+tests.put("GMT-1", "GMT-01:00");
+tests.put("GMT-1:00", "GMT-01:00");
+tests.put("GMT-01:00", "GMT-01:00");
+// values pushed by UI for jobs
+tests.put("GMT1", "GMT");
+tests.put("GMT1:00", "GMT");
+tests.put("GMT01:00", "GMT");
+// values generated by UI for datasets in a past
+tests.put("UTC1", "GMT");
+tests.put("UTC1:00", "GMT");
+tests.put("UTC01:00", "GMT");
+tests.put("UTC-1", "GMT");
+tests.put("UTC-1:00", "GMT");
+tests.put("UTC-01:00", "GMT");
+for (HashMap.Entry e: tests.entrySet()) {
+String input = e.getKey();
+String actual = TimeUtil.getTimeZone(e.getKey()).getID();
--- End diff --
Reuse `input` here.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin pull request #435: [GRIFFIN-206] fix job timezone when cre...
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224461843
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
+tests.put("", TimeZone.getDefault().getID());
+// standard cases
+tests.put("GMT", "GMT");
+tests.put("GMT+1", "GMT+01:00");
+tests.put("GMT+1:00", "GMT+01:00");
+tests.put("GMT+01:00", "GMT+01:00");
+tests.put("GMT-1", "GMT-01:00");
+tests.put("GMT-1:00", "GMT-01:00");
+tests.put("GMT-01:00", "GMT-01:00");
+// values pushed by UI for jobs
+tests.put("GMT1", "GMT");
+tests.put("GMT1:00", "GMT");
+tests.put("GMT01:00", "GMT");
+// values generated by UI for datasets in a past
+tests.put("UTC1", "GMT");
+tests.put("UTC1:00", "GMT");
+tests.put("UTC01:00", "GMT");
+tests.put("UTC-1", "GMT");
+tests.put("UTC-1:00", "GMT");
+tests.put("UTC-01:00", "GMT");
+for (HashMap.Entry e: tests.entrySet()) {
+String input = e.getKey();
+String actual = TimeUtil.getTimeZone(e.getKey()).getID();
--- End diff --
Reuse `input` here.
---
[GitHub] incubator-griffin pull request #435: [GRIFFIN-206] fix job timezone when cre...
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224461713
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
+tests.put("", TimeZone.getDefault().getID());
+// standard cases
+tests.put("GMT", "GMT");
+tests.put("GMT+1", "GMT+01:00");
+tests.put("GMT+1:00", "GMT+01:00");
+tests.put("GMT+01:00", "GMT+01:00");
+tests.put("GMT-1", "GMT-01:00");
+tests.put("GMT-1:00", "GMT-01:00");
+tests.put("GMT-01:00", "GMT-01:00");
+// values pushed by UI for jobs
+tests.put("GMT1", "GMT");
+tests.put("GMT1:00", "GMT");
+tests.put("GMT01:00", "GMT");
+// values generated by UI for datasets in a past
+tests.put("UTC1", "GMT");
+tests.put("UTC1:00", "GMT");
+tests.put("UTC01:00", "GMT");
+tests.put("UTC-1", "GMT");
+tests.put("UTC-1:00", "GMT");
+tests.put("UTC-01:00", "GMT");
+for (HashMap.Entry e: tests.entrySet()) {
--- End diff --
Consider
```
tests.entrySet().forEach(e -> {...}
```
as an option.
---
[GitHub] incubator-griffin pull request #435: [GRIFFIN-206] fix job timezone when cre...
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224459867
--- Diff:
service/src/test/java/org/apache/griffin/core/util/TimeUtilTest.java ---
@@ -110,4 +111,36 @@ public void testFormatWithIllegalException() {
TimeUtil.format(format, time, TimeZone.getTimeZone(timeZone));
}
+@Test
+public void testGetTimeZone() {
+HashMap tests = new HashMap<>();
--- End diff --
Please, use interface name instead of implementation.
---
[GitHub] incubator-griffin pull request #435: [GRIFFIN-206] fix job timezone when cre...
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224481345
--- Diff:
ui/angular/src/app/measure/create-measure/configuration/configuration.component.ts
---
@@ -56,47 +56,47 @@ export class ConfigurationComponent implements OnInit {
};
timetypes = ["day", "hour", "minute"];
timetype: string;
- timezones = [
-"UTC-12(IDL)",
-"UTC-11(MIT)",
-"UTC-10(HST)",
-"UTC-9:30(MSIT)",
-"UTC-9(AKST)",
-"UTC-8(PST)",
-"UTC-7(MST)",
-"UTC-6(CST)",
-"UTC-5(EST)",
-"UTC-4(AST)",
-"UTC-3:30(NST)",
-"UTC-3(SAT)",
-"UTC-2(BRT)",
-"UTC-1(CVT)",
-"UTC(WET,GMT)",
-"UTC+1(CET)",
-"UTC+2(EET)",
-"UTC+3(MSK)",
-"UTC+3:30(IRT)",
-"UTC+4(META)",
-"UTC+4:30(AFT)",
-"UTC+5(METB)",
-"UTC+5:30(IDT)",
-"UTC+5:45(NPT)",
-"UTC+6(BHT)",
-"UTC+6:30(MRT)",
-"UTC+7(IST)",
-"UTC+8(EAT)",
-"UTC+8:30(KRT)",
-"UTC+9(FET)",
-"UTC+9:30(ACST)",
-"UTC+10(AEST)",
-"UTC+10:30(FAST)",
-"UTC+11(VTT)",
-"UTC+11:30(NFT)",
-"UTC+12(PSTB)",
-"UTC+12:45(CIT)",
-"UTC+13(PSTC)",
-"UTC+14(PSTD)"
- ];
+ timezones: Map = new Map([
+["UTC-12(IDL)", "GMT-12"],
+["UTC-11(MIT)", "GMT-11"],
+["UTC-10(HST)", "GMT-10"],
+["UTC-9:30(MSIT)", "GMT-9:30"],
+["UTC-9(AKST)", "GMT-9"],
+["UTC-8(PST)", "GMT-8"],
+["UTC-7(MST)", "GMT-7"],
+["UTC-6(CST)", "GMT-6"],
+["UTC-5(EST)", "GMT-5"],
+["UTC-4(AST)", "GMT-4"],
+["UTC-3:30(NST)", "GMT-3:30"],
+["UTC-3(SAT)", "GMT-3"],
+["UTC-2(BRT)", "GMT-2"],
+["UTC-1(CVT)", "GMT-1"],
+["UTC(WET,GMT)", "GMT"],
+["UTC+1(CET)", "GMT+1"],
+["UTC+2(EET)", "GMT+2"],
+["UTC+3(MSK)", "GMT+3"],
+["UTC+3:30(IRT)", "GMT+3:30"],
+["UTC+4(META)", "GMT+4"],
+["UTC+4:30(AFT)", "GMT+4:30"],
+["UTC+5(METB)", "GMT+5"],
+["UTC+5:30(IDT)", "GMT+5:30"],
+["UTC+5:45(NPT)", "GMT+5:45"],
+["UTC+6(BHT)", "GMT+6"],
+["UTC+6:30(MRT)", "GMT+6:30"],
+["UTC+7(IST)", "GMT+7"],
+["UTC+8(EAT)", "GMT+8"],
+["UTC+8:30(KRT)", "GMT+8:30"],
+["UTC+9(FET)", "GMT+9"],
+["UTC+9:30(ACST)", "GMT+9:30"],
+["UTC+10(AEST)", "GMT+10"],
+["UTC+10:30(FAST)", "GMT+10:30"],
+["UTC+11(VTT)", "GMT+11"],
+["UTC+11:30(NFT)", "GMT+11:30"],
+["UTC+12(PSTB)", "GMT+12"],
+["UTC+12:45(CIT)", "GMT+12:45"],
+["UTC+13(PSTC)", "GMT+13"],
+["UTC+14(PSTD)", "GMT+15"],
--- End diff --
14 -> 15? Is it done intentionally?
---
[GitHub] incubator-griffin pull request #435: [GRIFFIN-206] fix job timezone when cre...
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/435#discussion_r224479994
--- Diff: ui/angular/src/app/job/create-job/batch/batch.component.ts ---
@@ -134,8 +134,8 @@ export class BatchComponent implements OnInit,
AfterViewChecked {
return false;
}
this.measureid = this.getMeasureId();
-let time = new Date().getTimezoneOffset() / 60;
-let timezone = "GMT" + time + ":00";
+let time = -(new Date().getTimezoneOffset() / 60);
--- End diff --
Looks like a hack (this and next lines). Probably worth to add a comment.
Or even extract common function as long as it happens twice in a code.
---
Re:Metrics not persisted when writing a query in SPARK-SQL instead of Griffin DSL
Hi Vikram,
In your JSON body, I notice that in the "rules" field, there's no "out" field,
which means griffin measure application will only calculate without output. You
might just changed the "dsl.type" from "griffin-dsl" to "spark-sql", actually,
for a "griffin-dsl" rule with "dq.type" as "profiling", we create a output for
it in transform phase:
https://github.com/apache/incubator-griffin/blob/griffin-0.3.0-incubating-rc1/measure/src/main/scala/org/apache/griffin/measure/step/builder/dsl/transform/ProfilingExpr2DQSteps.scala#L97,
but for a "spark-sql" rule, we don't parse it, so we don't know how it would
work, you need to manually configure the output field to enable it.
You can refer to this document to configure the output field:
https://github.com/apache/incubator-griffin/blob/master/griffin-doc/measure/measure-configuration-guide.md#rule
Or just simply refer to the demo json for spark-sql profiling rules:
https://github.com/apache/incubator-griffin/blob/griffin-0.3.0-incubating-rc1/measure/src/test/resources/_profiling-batch-sparksql.json
Hope this could help you.
--
Regards,
Lionel, Liu
At 2018-10-11 17:30:29, "Vikram Jain" wrote:
>Hello,
>
>I was trying to create a measure and write the rule in Spark-SQL directly
>instead of Griffin-DSL. I use Postman to create the measure. The measure is
>created successfully, the job is created and executed successfully.
>
>However, the output metrics of execution of jobs are not persisted in
>ElasticSearch. The entry is created in Elastic but the "metricValues" array is
>NULL.
>
>The same SQL query works fine directly on Spark-Shell.
>
>I am not using Docker and building the environment (Griffin 3.0) on my local
>machine. All the measures created using UI are executing well. And measures
>created using Postman with griffin-dsl rule are also working well.
>
>Below is the body of json which I am passing to add measure API call from
>Postman. Please help me understand what is going wrong.
>
>
>{
> "name": "custom_profiling_measure_2",
> "measure.type": "griffin",
> "dq.type": "PROFILING",
> "rule.description": {
> "details": [
> {
> "name": "id",
> "infos": "Total Count"
> }
> ]
> },
> "process.type": "BATCH",
> "owner": "test",
> "description": "custom_profiling_measure_2",
> "data.sources": [
> {
> "name": "source",
> "connectors": [
> {
> "name": "source123",
> "type": "HIVE",
> "version": "1.2",
> "data.unit": "1day",
> "data.time.zone": "",
> "config": {
> "database": "default",
> "table.name": "demo_src",
> "where": ""
> }
> }
> ]
> }
> ],
> "evaluate.rule": {
> "out.dataframe.name": "profiling_2",
> "rules": [
> {
> "dsl.type": "spark-sql",
> "dq.type": "PROFILING",
> "rule": "SELECT count(id) AS cnt, max(age) AS Max_Age from demo_src",
> "out.dataframe.name": "id_count_2"
> }
> ]
> }
>}
>
>
>
>
>
>Regards,
>
>Vikram
>
[GitHub] incubator-griffin pull request #434: Griffin-205: accuracy matched fraction
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/434#discussion_r224438517
--- Diff:
measure/src/test/scala/org/apache/griffin/measure/transformations/AccuracyTransformationsIntegrationTest.scala
---
@@ -0,0 +1,169 @@
+/*
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+*/
+package org.apache.griffin.measure.transformations
+
+import com.holdenkarau.spark.testing.DataFrameSuiteBase
+import org.apache.spark.sql.DataFrame
+import org.scalatest._
+
+import org.apache.griffin.measure.configuration.dqdefinition._
+import org.apache.griffin.measure.configuration.enums.BatchProcessType
+import org.apache.griffin.measure.context.{ContextId, DQContext}
+import org.apache.griffin.measure.datasource.DataSourceFactory
+import org.apache.griffin.measure.job.builder.DQJobBuilder
+
+case class AccuracyResult(total: Long, miss: Long, matched: Long,
matchedFraction: Double)
+
+class AccuracyTransformationsIntegrationTest extends FlatSpec with
Matchers with DataFrameSuiteBase {
+ import spark.implicits._
+
+ private val EMPTY_PERSON_TABLE = "empty_person"
+ private val PERSON_TABLE = "person"
+
+ override def beforeAll(): Unit = {
+super.beforeAll()
+
+createPersonTable()
+createEmptyPersonTable()
+
+spark.conf.set("spark.sql.crossJoin.enabled", "true")
+ }
+
+ "accuracy" should "basically work" in {
+checkAccuracy(
+ sourceName = PERSON_TABLE,
+ targetName = PERSON_TABLE,
+ expectedResult = AccuracyResult(total = 2, miss = 0, matched = 2,
matchedFraction = 1.0))
+ }
+
+ "accuracy" should "work with empty target" in {
+checkAccuracy(
+ sourceName = PERSON_TABLE,
+ targetName = EMPTY_PERSON_TABLE,
+ expectedResult = AccuracyResult(total = 2, miss = 2, matched = 0,
matchedFraction = 0.0)
+)
+ }
+
+ "accuracy" should "work with empty source" in {
+checkAccuracy(
+ sourceName = EMPTY_PERSON_TABLE,
+ targetName = PERSON_TABLE,
+ expectedResult = AccuracyResult(total = 0, miss = 0, matched = 0,
matchedFraction = 1.0))
+ }
+
+ "accuracy" should "work with empty source and target" in {
+checkAccuracy(
+ sourceName = EMPTY_PERSON_TABLE,
+ targetName = EMPTY_PERSON_TABLE,
+ expectedResult = AccuracyResult(total = 0, miss = 0, matched = 0,
matchedFraction = 1.0))
+ }
+
+ private def checkAccuracy(sourceName: String, targetName: String,
expectedResult: AccuracyResult) = {
+val dqContext: DQContext = getDqContext(
+ dataSourcesParam = List(
+DataSourceParam(
+ name = "source",
+ connectors = List(dataConnectorParam(tableName = sourceName))
+),
+DataSourceParam(
+ name = "target",
+ connectors = List(dataConnectorParam(tableName = targetName))
+)
+ ))
+
+val accuracyRule = RuleParam(
+ dslType = "griffin-dsl",
+ dqType = "ACCURACY",
+ outDfName = "person_accuracy",
+ rule = "source.name = target.name"
+)
+
+val res = getRuleResults(dqContext, accuracyRule)
+ .as[AccuracyResult]
+ .collect()
+
+res.length shouldBe 1
+
+res(0) shouldEqual expectedResult
+ }
+
+ private def getRuleResults(dqContext: DQContext, rule: RuleParam):
DataFrame = {
+val dqJob = DQJobBuilder.buildDQJob(
+ dqContext,
+ evaluateRuleParam = EvaluateRuleParam(List(rule))
+)
+
+dqJob.execute(dqContext)
+
+spark.sql(s"select * from ${rule.getOutDfName()}")
+ }
+
+ private def createPersonTable(): Unit = {
+val personCsvPath =
getClass.getResource("/hive/person_table.csv").getFile
+
+spark.sql(
+ s"CREATE TABLE IF NOT EXISTS ${PERSON_TABLE} " +
+"( " +
+" name String," +
+" age int
[GitHub] incubator-griffin pull request #434: Griffin-205: accuracy matched fraction
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/434#discussion_r224432819
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/configuration/dqdefinition/DQConfig.scala
---
@@ -148,12 +148,12 @@ case class EvaluateRuleParam( @JsonProperty("rules")
private val rules: List[Rul
@JsonInclude(Include.NON_NULL)
case class RuleParam(@JsonProperty("dsl.type") private val dslType: String,
@JsonProperty("dq.type") private val dqType: String,
- @JsonProperty("in.dataframe.name") private val
inDfName: String,
- @JsonProperty("out.dataframe.name") private val
outDfName: String,
- @JsonProperty("rule") private val rule: String,
- @JsonProperty("details") private val details:
Map[String, Any],
- @JsonProperty("cache") private val cache: Boolean,
- @JsonProperty("out") private val outputs:
List[RuleOutputParam]
+ @JsonProperty("in.dataframe.name") private val
inDfName: String = null,
+ @JsonProperty("out.dataframe.name") private val
outDfName: String = null,
+ @JsonProperty("rule") private val rule: String = null,
+ @JsonProperty("details") private val details:
Map[String, Any] = null,
--- End diff --
Yes, it exactly is. Actually I expected such question :)
There is a bug in `jackson` support for scala ["Case classes with option
types don't follow serialization
configuration"](https://github.com/FasterXML/jackson-module-scala/issues/32).
Long story short, It does not support any default values and puts `null`s
everywhere. So, if we want to use this classes outside of jackson at the same
way as with jackson, `null`s is only chose we have.
---
[GitHub] incubator-griffin pull request #434: Griffin-205: accuracy matched fraction
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/434#discussion_r224431040
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/configuration/dqdefinition/DQConfig.scala
---
@@ -74,9 +74,9 @@ case class DQConfig(@JsonProperty("name") private val
name: String,
*/
@JsonInclude(Include.NON_NULL)
case class DataSourceParam( @JsonProperty("name") private val name: String,
-@JsonProperty("baseline") private val
baseline: Boolean,
--- End diff --
No, it's not. But keeping fields with default values as last elements in
case classes is kind of "best practice" in scala. For example, scala
compilation fails for the following snippet of code
```
case class Foo(a: Int, b: Int = 22, c: Double)
val foo = Foo(1, 3.3)
```
For this particular class we can avoid reordering. But I see more reasons
to do it right and don't see any reasons to keep it as is (except we are scared
to change any pice of code).
---
[GitHub] incubator-griffin pull request #434: Griffin-205: accuracy matched fraction
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/434#discussion_r224425167
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/step/builder/dsl/transform/AccuracyExpr2DQSteps.scala
---
@@ -125,14 +126,26 @@ case class AccuracyExpr2DQSteps(context: DQContext,
// 4. accuracy metric
val accuracyTableName = ruleParam.getOutDfName()
val matchedColName = details.getStringOrKey(_matched)
+ val matchedFractionColName = details.getStringOrKey(_matchedFraction)
val accuracyMetricSql = procType match {
case BatchProcessType =>
s"""
- |SELECT `${totalCountTableName}`.`${totalColName}` AS
`${totalColName}`,
- |coalesce(`${missCountTableName}`.`${missColName}`, 0) AS
`${missColName}`,
- |(`${totalCountTableName}`.`${totalColName}` -
coalesce(`${missCountTableName}`.`${missColName}`, 0)) AS `${matchedColName}`
- |FROM `${totalCountTableName}` LEFT JOIN
`${missCountTableName}`
- """.stripMargin
+ SELECT `${totalColName}`,
+`${missColName}`,
+`${matchedColName}`,
+coalesce(`${matchedColName}` / `${totalColName}`, 1.0)
AS `${matchedFractionColName}`
+
+ FROM (
+ SELECT `${totalColName}`,
+ `${missColName}`,
+ (`${totalColName}` - `${missColName}`) AS
`${matchedColName}`
--- End diff --
Whole point of nesting levels here to increase readability. Now, I have an
idea where it will "good enough" to read even with 1 level of nesting.
---
[GitHub] incubator-griffin pull request #434: Griffin-205: accuracy matched fraction
Github user ashutakGG commented on a diff in the pull request:
https://github.com/apache/incubator-griffin/pull/434#discussion_r224424475
--- Diff:
measure/src/main/scala/org/apache/griffin/measure/step/builder/dsl/transform/AccuracyExpr2DQSteps.scala
---
@@ -125,14 +126,26 @@ case class AccuracyExpr2DQSteps(context: DQContext,
// 4. accuracy metric
val accuracyTableName = ruleParam.getOutDfName()
val matchedColName = details.getStringOrKey(_matched)
+ val matchedFractionColName = details.getStringOrKey(_matchedFraction)
val accuracyMetricSql = procType match {
case BatchProcessType =>
s"""
- |SELECT `${totalCountTableName}`.`${totalColName}` AS
`${totalColName}`,
- |coalesce(`${missCountTableName}`.`${missColName}`, 0) AS
`${missColName}`,
- |(`${totalCountTableName}`.`${totalColName}` -
coalesce(`${missCountTableName}`.`${missColName}`, 0)) AS `${matchedColName}`
- |FROM `${totalCountTableName}` LEFT JOIN
`${missCountTableName}`
- """.stripMargin
+ SELECT `${totalColName}`,
+`${missColName}`,
+`${matchedColName}`,
+coalesce(`${matchedColName}` / `${totalColName}`, 1.0)
AS `${matchedFractionColName}`
+
+ FROM (
+ SELECT `${totalColName}`,
--- End diff --
Good point. For a sec I even thought you are right (an this is a potential
bug). But the following query works as expected.
`spark.sql("SELECT age FROM (SELECT count(age) as age FROM person)").show()`
It show 2 as a result.
So, looks like high level select does not have an access to tables used for
inner select.
But probably I will use this approach to increase readability.
---
Metrics not persisted when writing a query in SPARK-SQL instead of Griffin DSL
Hello,
I was trying to create a measure and write the rule in Spark-SQL directly
instead of Griffin-DSL. I use Postman to create the measure. The measure is
created successfully, the job is created and executed successfully.
However, the output metrics of execution of jobs are not persisted in
ElasticSearch. The entry is created in Elastic but the "metricValues" array is
NULL.
The same SQL query works fine directly on Spark-Shell.
I am not using Docker and building the environment (Griffin 3.0) on my local
machine. All the measures created using UI are executing well. And measures
created using Postman with griffin-dsl rule are also working well.
Below is the body of json which I am passing to add measure API call from
Postman. Please help me understand what is going wrong.
{
"name": "custom_profiling_measure_2",
"measure.type": "griffin",
"dq.type": "PROFILING",
"rule.description": {
"details": [
{
"name": "id",
"infos": "Total Count"
}
]
},
"process.type": "BATCH",
"owner": "test",
"description": "custom_profiling_measure_2",
"data.sources": [
{
"name": "source",
"connectors": [
{
"name": "source123",
"type": "HIVE",
"version": "1.2",
"data.unit": "1day",
"data.time.zone": "",
"config": {
"database": "default",
"table.name": "demo_src",
"where": ""
}
}
]
}
],
"evaluate.rule": {
"out.dataframe.name": "profiling_2",
"rules": [
{
"dsl.type": "spark-sql",
"dq.type": "PROFILING",
"rule": "SELECT count(id) AS cnt, max(age) AS Max_Age from demo_src",
"out.dataframe.name": "id_count_2"
}
]
}
}
Regards,
Vikram
Re: [DISCUSS] Graduate Apache Griffin (incubating) as a TLP
+1 agree to gradate as TLP. the community has been very actively contributing this software as a kind of unique solution for data quality. it has benefits to users and members are gained a lot of experiences on Apache way. thanks. From: john@apache Sent: Wednesday, October 10, 2018 11:45 PM To: dev@griffin.incubator.apache.org Subject: Re: [DISCUSS] Graduate Apache Griffin (incubating) as a TLP +1 Agree that it is the time to graduate. Griffin has been evolving and growing in the past 2 two years, with the support of a positive and mature community. Thanks, John On Tue, Oct 9, 2018 at 11:51 PM William Guo wrote: > Hi All, > > With the 0.3.0-incubating release officially out, the Apache Griffin > community and its mentors believe it is time to consider graduation to the > TLP. > > Apache Griffin entered incubation in December of 2016, since then, the > Griffin > community learned a lot about how to do things in Apache ways. Now we have > a healthy and engaged community, ready to help with all questions from the > Griffin community. We delivered five releases, now we can do self-driving > releases in good cadence. The PPMC has demonstrated a good understanding of > growing the community by electing 9 individuals as committers and PPMC > members. > The PPMC addressed the maturity issues one by one followed by > Apache Project Maturity Model, currently all the License and IP issues are > resolved. > This demonstrated our understanding of ASF's IP policies. > > All in all, I believe this project is qualified as a true TLP and we should > recognize this fact by formally awarding it such a status. This thread > means to open up the very same discussion that we had among the mentors and > Griffin community to the rest of the IPMC. It is a DISCUSS thread so feel > free > to ask questions. > > To get you all going, here are a few data points which may help: > > Project status: > http://incubator.apache.org/projects/griffin.html > > Project website: > http://griffin.incubator.apache.org/ > > Project documentation: > http://griffin.incubator.apache.org/docs/quickstart.html > http://griffin.incubator.apache.org/docs/download.html > > Maturity assessment: > https://cwiki.apache.org/confluence/display/GRIFFIN/ASF+Maturity+Evaluation > > DRAFT of the board resolution is at the bottom of this email > > Proposed PMC size: 18 members > > Total number of committers: 18 members > > PMC affiliation (* indicated chair): > > - eBay * (7) > - Meituan (1) > - VMWare (1) > - Ontario Institute for Cancer Research (1) > - NetEase (1) > - ASF (1) > - Pingan Bank (1) > - Satori Software (1) > - Grid Dynamics (1) > - JD.com (1) > - IBM (1) > - Huawei (1) > > > 452 commits on develop > 431 PR”s on GitHub > 30 contributors across all branches > > 200 issues created > 195 issues resolved > > dev list averaged ~20 msgs/month over last 12 months > > > committer affiliations: > > active > > * eBay > * meituan > * VMWare > * Grid Dynamics > > occasional > > * Ontario Institute for Cancer Research > * NetEase > * ASF > * Pingan Bank > * Satori Software > * JD.com > * IBM > * Huawei > > > Thanks, > William > > > **Notes:** > > * I'm proposing myself as initial PMC chair -- Please comment this or > propose other persons as well > > * This draft includes all existing PPMC members and mentors into the new > PMC. > > - For all: please indicate if you want to keep being part of the > PMC or if you prefer to be removed. > > > -- > > ## Resolution to create a TLP from graduating Incubator podling > > X. Establish the Apache Griffin Project > > WHEREAS, the Board of Directors deems it to be in the best > interests of the Foundation and consistent with the > Foundation's purpose to establish a Project Management > Committee charged with the creation and maintenance of > open-source software, for distribution at no charge to > the public, related to a data quality solution for big data, > including both streaming and batch mode. It offers an unified > process to measure data quality from different perspectives. > > NOW, THEREFORE, BE IT RESOLVED, that a Project Management > Committee (PMC), to be known as the "Apache Griffin Project", > be and hereby is established pursuant to Bylaws of the > Foundation; and be it further > > RESOLVED, that the Apache Griffin Project be and hereby is > responsible for the creation and maintenance of software > related to a data quality solution for big data, > including both streaming and batch mode. It offers an unified > process to measure data quality from different perspectives; > and be it further > > RESOLVED, that the office of "Vice President, Apache Griffin" be > and hereby is created, the person holding such office to > serve at the direction of the Board of Directors as the chair > of the Apache Griffin Project, and to have primary responsibility > for management of the projects within the scope of > responsibility of the Apache Griffin
[jira] [Commented] (GRIFFIN-190) Blank Health and DQ Metrics Screen
[
https://issues.apache.org/jira/browse/GRIFFIN-190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646042#comment-16646042
]
Nikolay Sokolov commented on GRIFFIN-190:
-
[~cwoytasik] looks like it've failed over here:
[https://github.com/apache/incubator-griffin/blob/griffin-0.2.0-incubating-rc4/measure/src/main/scala/org/apache/griffin/measure/Application.scala#L99]
If you are compiling it yourself, you can add extra line
{{ex.printStackTrace()}} before exit call, so it would be more clear what the
error is. Process init is mostly doing spark context setup there, and very
likely config is not complete or spark itself is not configured properly.
Looking at spark context init code, I have gut feel that you are having NPE
around [this
line|https://github.com/apache/incubator-griffin/blob/griffin-0.2.0-incubating-rc4/measure/src/main/scala/org/apache/griffin/measure/process/BatchDqProcess.scala#L57],
and adding "config" key your env.json could help:
{code:none}
"spark": {"log.level": "WARN", "config": {}}
{code}
[~Lionel_3L] should we log full stack trace there, instead of just message?
> Blank Health and DQ Metrics Screen
> --
>
> Key: GRIFFIN-190
> URL: https://issues.apache.org/jira/browse/GRIFFIN-190
> Project: Griffin (Incubating)
> Issue Type: Bug
>Affects Versions: 0.2.0-incubating
>Reporter: Cory Woytasik
>Priority: Major
> Attachments: PLDataLineageLoad061818.csv
>
>
> Griffin is up and running. We have both an accuracy measure and a profiling
> measure that is set to run every minute via jobs. When we click the chart
> icon next to the job we receive a "no content" message. When we click on the
> Health link or DQ Metrics link they think for a second and then display a
> blank screen. We are thinking this might be ES related, but aren't
> completely sure. Need some help. We assume it's a path or property setup
> issue. Here are the versions we are running:
> Hive - 3.1.0
> Elasticsearch - 5.3.1
> griffin - 0.2.0
> hadoop - 3.1.1
> livy - 0.3.0
> spark - 2.3.1
> Using postgres too
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
Re: Jira permissions
Was configuration done specifically to Artem.Shutak or overall? I tried to reassign GRIFFIN-190 on issue reporter for test purpose, but his name did not pop up. On Wed, Oct 10, 2018 at 12:00 AM William Guo wrote: > hi Nick Sokolov, > > It seems griffin ticket can be assigned to a contributor now. > Could you check it again? > > Thanks, > William > > > On Wed, Oct 10, 2018 at 9:05 AM William Guo wrote: > >> Let me look into this. >> >> On Wed, Oct 10, 2018 at 1:55 AM Nick Sokolov >> wrote: >> >>> Hi all, >>> >>> I've discovered, that it's not possible to reassign jira ticket to other >>> contributors (non-committers), and non-committers can not assign tickets >>> (even if they have created it and want to assign it to themselves). Is >>> that >>> expected behavior? >>> >>> I remember that in Hive jira non-committers have these permissions, and >>> ticket can be assigned to anyone within Apache jira. Should project >>> settings be adjusted? >>> >>
[jira] [Assigned] (GRIFFIN-205) Accuracy measure check should provide matchedFraction to store
[
https://issues.apache.org/jira/browse/GRIFFIN-205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Sokolov reassigned GRIFFIN-205:
---
Assignee: (was: Artem Shutak)
> Accuracy measure check should provide matchedFraction to store
> --
>
> Key: GRIFFIN-205
> URL: https://issues.apache.org/jira/browse/GRIFFIN-205
> Project: Griffin (Incubating)
> Issue Type: Improvement
> Components: accuracy-batch, accuracy-real-time
>Affects Versions: 1.0.0-incubating
>Reporter: Artem Shutak
>Priority: Major
>
> Currently, {{accuracy}} measure results contains "total", "miss" and
> "matched" counts.
> As a result, It's hard to analyze accuracy fraction based on results stored
> in ElasticSearch, because ElasticSearch does not provide straight forward
> capability to get "field divided by field" query results.
> {{Accuracy}} measure results should also contain {{matchedFraction}} field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (GRIFFIN-205) Accuracy measure check should provide matchedFraction to store
[
https://issues.apache.org/jira/browse/GRIFFIN-205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikolay Sokolov reassigned GRIFFIN-205:
---
Assignee: Artem Shutak
> Accuracy measure check should provide matchedFraction to store
> --
>
> Key: GRIFFIN-205
> URL: https://issues.apache.org/jira/browse/GRIFFIN-205
> Project: Griffin (Incubating)
> Issue Type: Improvement
> Components: accuracy-batch, accuracy-real-time
>Affects Versions: 1.0.0-incubating
>Reporter: Artem Shutak
>Assignee: Artem Shutak
>Priority: Major
>
> Currently, {{accuracy}} measure results contains "total", "miss" and
> "matched" counts.
> As a result, It's hard to analyze accuracy fraction based on results stored
> in ElasticSearch, because ElasticSearch does not provide straight forward
> capability to get "field divided by field" query results.
> {{Accuracy}} measure results should also contain {{matchedFraction}} field.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-206) Wrong sign of time zone offset when created from UI
[
https://issues.apache.org/jira/browse/GRIFFIN-206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16646021#comment-16646021
]
ASF GitHub Bot commented on GRIFFIN-206:
Github user chemikadze commented on the issue:
https://github.com/apache/incubator-griffin/pull/435
Went with UI route for a time being, to avoid introducing any incompatible
behavior from backend side.
> Wrong sign of time zone offset when created from UI
> ---
>
> Key: GRIFFIN-206
> URL: https://issues.apache.org/jira/browse/GRIFFIN-206
> Project: Griffin (Incubating)
> Issue Type: Bug
>Reporter: Nikolay Sokolov
>Assignee: Nikolay Sokolov
>Priority: Major
>
> When job is created from UI, time zone offset is created by this piece of
> code:
> {code:none}
> let time = new Date().getTimezoneOffset() / 60;
> let timezone = "GMT" + time + ":00";
> {code}
> Problem is, that for GMT-7 it yields positive number, resulting in calculated
> timezone to be GMT+7.
> Turns out that that some formats are not parsed properly, at least UTCXX:XX
> format used by UI is not processed correctly, and GMTXX:XX (without signs):
> {code:none}
> scala> java.util.TimeZone.getTimeZone("GMT6")
> res9: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("GMT+6")
> res10: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT+06:00",offset=2160,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC+6")
> res11: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> scala> java.util.TimeZone.getTimeZone("UTC-6(CST)")
> res12: java.util.TimeZone =
> sun.util.calendar.ZoneInfo[id="GMT",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin issue #435: [GRIFFIN-206] fix job timezone when creating f...
Github user chemikadze commented on the issue: https://github.com/apache/incubator-griffin/pull/435 Went with UI route for a time being, to avoid introducing any incompatible behavior from backend side. ---
