[jira] [Commented] (GRIFFIN-164) Make 'Regular expression detection count' available in UI
[
https://issues.apache.org/jira/browse/GRIFFIN-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567501#comment-16567501
]
ASF GitHub Bot commented on GRIFFIN-164:
Github user spencer-hivert-ck commented on the issue:
https://github.com/apache/incubator-griffin/pull/381
Thanks for taking a look at my PR @icesmartjuan!
I'm unable to re-produce any of the errors you've mentioned above.
Regarding #2, I've added default values to the fields in step 3, so that error
should not occur. Regarding #3, I didn't change the accuracy or job code, so
that error may be unrelated.
I brought up a fresh version of the service/ui and was unable to re-produce
the failures described above. Are you able to provide more detailed steps
and/or photos so I can attempt to re-produce these errors myself!
Thank you.
> Make 'Regular expression detection count' available in UI
> -
>
> Key: GRIFFIN-164
> URL: https://issues.apache.org/jira/browse/GRIFFIN-164
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Affects Versions: 0.1.6-incubating
>Reporter: Enrico D'Urso
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> Hi,
> I have been playing for one month now with Griffin.
> Given my experience, some companies (included the one am working for as a
> consultant) prefer doing stuff using UI.
> Personally, I find very useful the following feature:
>
> * Regular expression detection count
> which is, I have a column which should contain just numbers so I want to
> check if my ETL process, wrongly, has populated my table with non-numeric
> values.
> I have been able to run such a job creating my self the right config.json, in
> particular, using spark-sql as dialect:
> {code:java}
> select count(*) from src where account_id rlike [^0-9]
> {code}
> I saw that in pr.component.ts there is a commented line of code:
> {code:java}
> // {"id":10,"itemName":"Regular Expression Detection Count","category":
> "Advanced Statistics"}
> {code}
> which I think is what I am talking about.
> Also, I can read:
> {code:java}
> // case 'Regular Expression Detection Count': // return
> 'count(source.`'+col.name+'`) where source.`'+col.name+'` LIKE ';
> {code}
> which should be the griffin-dsl dialect, even if, probably, the regex should
> be added just after LIKE.
> Then, once that the above griffin-dsl statement is available in the backend,
> ProfilingRulePlanTrans class
> should map that into 'rlike' Spark-sql clause.
> Am not sure where (and if) ProfilingRulePlanTrans should be modified as
> preGroupbyClause should contains everything, but I do not have enough
> knowledge about it.
>
> Please judge yourself the priority of such a feature, which knowing well the
> code, should not be too hard to make.
> Thanks,
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin issue #381: GRIFFIN-164 GRIFFIN-186 GRIFFIN-187: Profiling...
Github user spencer-hivert-ck commented on the issue: https://github.com/apache/incubator-griffin/pull/381 Thanks for taking a look at my PR @icesmartjuan! I'm unable to re-produce any of the errors you've mentioned above. Regarding #2, I've added default values to the fields in step 3, so that error should not occur. Regarding #3, I didn't change the accuracy or job code, so that error may be unrelated. I brought up a fresh version of the service/ui and was unable to re-produce the failures described above. Are you able to provide more detailed steps and/or photos so I can attempt to re-produce these errors myself! Thank you. ---
[GitHub] incubator-griffin issue #380: Fix java import style violations in codes by c...
Github user toyboxman commented on the issue: https://github.com/apache/incubator-griffin/pull/380 @guoyuepeng @bhlx3lyx7 please consider to keep same code style and we can do those work step by step ---
[jira] [Commented] (GRIFFIN-164) Make 'Regular expression detection count' available in UI
[
https://issues.apache.org/jira/browse/GRIFFIN-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566630#comment-16566630
]
ASF GitHub Bot commented on GRIFFIN-164:
Github user icesmartjuan commented on the issue:
https://github.com/apache/incubator-griffin/pull/381
Test from UI based on your current measure and service :
1. create Accuracy and Publish measure successfully
2. fail to create Data Profiling measure
pr.component.ts: line 390 fails the Profiling and Publish measure creation,
as this.step3.size is undefined type
3. failed to create job with above created Accuracy measure
{"timestamp":1533207681903,"status":400,"error":"Bad
Request","code":"40011","message":"We don't support such measure
type.","path":"/api/v1/jobs"}
```this.newJob
{job.name: "job1", job.type: "batch", measure.id: 1, cron.expression: "0
0/4 * * * ?", cron.time.zone: "GMT-8:00", …}cron.expression: "0 0/4 * * *
?"cron.time.zone: "GMT-8:00"data.segments: Array(2)0: as.baseline:
truedata.connector.name: "source1533207565052"segment.range: begin:
"-1day"length: "1day"__proto__: Object__proto__: Objectconstructor: ƒ
Object()hasOwnProperty: ƒ hasOwnProperty()isPrototypeOf: ƒ
isPrototypeOf()propertyIsEnumerable: ƒ propertyIsEnumerable()toLocaleString: ƒ
toLocaleString()toString: ƒ ()valueOf: ƒ valueOf()__defineGetter__: ƒ
__defineGetter__()__defineSetter__: ƒ __defineSetter__()__lookupGetter__: ƒ
__lookupGetter__()__lookupSetter__: ƒ __lookupSetter__()get __proto__: ƒ
__proto__()set __proto__: ƒ __proto__()1: data.connector.name:
"target1533207568926"segment.range: begin: "-1day"length: "1day"__proto__:
Object__proto__: Objectlength: 2__proto__: Array(0)job.name: "job1"job.type:
"batch"measure.id: 1__proto__: Object```
> Make 'Regular expression detection count' available in UI
> -
>
> Key: GRIFFIN-164
> URL: https://issues.apache.org/jira/browse/GRIFFIN-164
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Affects Versions: 0.1.6-incubating
>Reporter: Enrico D'Urso
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> Hi,
> I have been playing for one month now with Griffin.
> Given my experience, some companies (included the one am working for as a
> consultant) prefer doing stuff using UI.
> Personally, I find very useful the following feature:
>
> * Regular expression detection count
> which is, I have a column which should contain just numbers so I want to
> check if my ETL process, wrongly, has populated my table with non-numeric
> values.
> I have been able to run such a job creating my self the right config.json, in
> particular, using spark-sql as dialect:
> {code:java}
> select count(*) from src where account_id rlike [^0-9]
> {code}
> I saw that in pr.component.ts there is a commented line of code:
> {code:java}
> // {"id":10,"itemName":"Regular Expression Detection Count","category":
> "Advanced Statistics"}
> {code}
> which I think is what I am talking about.
> Also, I can read:
> {code:java}
> // case 'Regular Expression Detection Count': // return
> 'count(source.`'+col.name+'`) where source.`'+col.name+'` LIKE ';
> {code}
> which should be the griffin-dsl dialect, even if, probably, the regex should
> be added just after LIKE.
> Then, once that the above griffin-dsl statement is available in the backend,
> ProfilingRulePlanTrans class
> should map that into 'rlike' Spark-sql clause.
> Am not sure where (and if) ProfilingRulePlanTrans should be modified as
> preGroupbyClause should contains everything, but I do not have enough
> knowledge about it.
>
> Please judge yourself the priority of such a feature, which knowing well the
> code, should not be too hard to make.
> Thanks,
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] incubator-griffin issue #381: GRIFFIN-164 GRIFFIN-186 GRIFFIN-187: Profiling...
Github user icesmartjuan commented on the issue:
https://github.com/apache/incubator-griffin/pull/381
Test from UI based on your current measure and service :
1. create Accuracy and Publish measure successfully
2. fail to create Data Profiling measure
pr.component.ts: line 390 fails the Profiling and Publish measure creation,
as this.step3.size is undefined type
3. failed to create job with above created Accuracy measure
{"timestamp":1533207681903,"status":400,"error":"Bad
Request","code":"40011","message":"We don't support such measure
type.","path":"/api/v1/jobs"}
```this.newJob
{job.name: "job1", job.type: "batch", measure.id: 1, cron.expression: "0
0/4 * * * ?", cron.time.zone: "GMT-8:00", â¦}cron.expression: "0 0/4 * * *
?"cron.time.zone: "GMT-8:00"data.segments: Array(2)0: as.baseline:
truedata.connector.name: "source1533207565052"segment.range: begin:
"-1day"length: "1day"__proto__: Object__proto__: Objectconstructor: Æ
Object()hasOwnProperty: Æ hasOwnProperty()isPrototypeOf: Æ
isPrototypeOf()propertyIsEnumerable: Æ propertyIsEnumerable()toLocaleString:
Æ toLocaleString()toString: Æ ()valueOf: Æ valueOf()__defineGetter__: Æ
__defineGetter__()__defineSetter__: Æ __defineSetter__()__lookupGetter__: Æ
__lookupGetter__()__lookupSetter__: Æ __lookupSetter__()get __proto__: Æ
__proto__()set __proto__: Æ __proto__()1: data.connector.name:
"target1533207568926"segment.range: begin: "-1day"length: "1day"__proto__:
Object__proto__: Objectlength: 2__proto__: Array(0)job.name: "job1"job.type:
"batch"measure.id: 1__proto__: Object```
---
[GitHub] incubator-griffin issue #380: Fix java import style violations in codes by c...
Github user toyboxman commented on the issue: https://github.com/apache/incubator-griffin/pull/380 by Griffin's code style rules extends google/sun, we need to reformat import statement. here I have gone thru service packages and cleaned/adjusted all import statements ---
[GitHub] incubator-griffin pull request #382: Modify measure module to support update...
GitHub user bhlx3lyx7 opened a pull request: https://github.com/apache/incubator-griffin/pull/382 Modify measure module to support updated env and config json files format Some important modification in env.json - "info.cache" -> "griffin.checkpoint" - "persist" -> "sinks" - "log" -> "console" - "http" -> "elasticsearch" - remove "cleaner" Some important modiffication in dq.json - add "baseline" in data source - add "dataframe.name" in data connector - in data source,"cache" -> "checkpoint" - in rule and pre-proc rule, add "in.dataframe.name" and "out.dataframe.name", remove "name" - in rule, add "out" param array, move "metric", "record" param inside "out" array - add "sinks" string array as filter of sinks in env.json You can merge this pull request into a Git repository by running: $ git pull https://github.com/bhlx3lyx7/incubator-griffin json-update Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-griffin/pull/382.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #382 commit ffb818aa58ef9c330c0bb88731bac8f2975298d8 Author: Lionel Liu Date: 2017-10-11T07:38:32Z ignore ui tmp commit dba1602366ef9084a392d7ca61e6e719827ee208 Author: Lionel Liu Date: 2017-10-11T07:48:00Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit e980e6c48a4b4ad616c3209ccc69268250d87f30 Author: Lionel Liu Date: 2017-10-12T04:29:48Z doc commit cf849f478592bf7d1c102fc33c0a3aaacd0a3ab5 Author: Lionel Liu Date: 2017-10-12T04:49:37Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 5cd047826016a8ba277c98a6cbc7978a3d2bc659 Author: Lionel Liu Date: 2017-10-13T06:46:44Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 7edf6ff9bf8d460f3b68c1a068ac597d908d0eae Author: Lionel Liu Date: 2017-10-16T02:21:33Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 673b717765b5a1e3fbc566cff190c4e9b615159e Author: Lionel Liu Date: 2017-10-16T05:25:11Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit daecbfc69aefe484d2c9cc72f43571caafe8187e Author: Lionel Liu Date: 2017-10-16T08:46:53Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit e71a1c56ae074ae1de9472edcdbf11c5e3a2d4c1 Author: Lionel Liu Date: 2017-10-17T04:57:10Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 46c2a4fccc6a83d12557b07cb72c740d19d5b529 Author: Lionel Liu Date: 2017-10-25T03:23:53Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 6c3cea1f96d8dbc694b76eec2ef52be8ed1b4ae6 Author: Lionel Liu Date: 2017-10-30T06:02:22Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 243ec9a4006de7fbe14c39be7eea9cd74b3d13fc Author: Lionel Liu Date: 2017-10-30T06:24:23Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 0c82eeb22c133b360f48cc29baf8678fb32070c3 Author: Lionel Liu Date: 2017-10-30T06:55:18Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 106849794fc4b84adb89bced411711d6a69731ac Author: Lionel Liu Date: 2017-10-30T08:17:46Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit f307409ebd3f1a0a806204ccf2d5fd56b2309284 Author: Lionel Liu Date: 2017-10-30T08:43:53Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 9af0f041e918e8d481a09cf407b57b109a8f07b7 Author: Lionel Liu Date: 2017-10-31T10:11:41Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 8bc6c41f87b167d9fed3ac81fbd51d730d37a087 Author: Lionel Liu Date: 2017-11-02T06:36:34Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 712b2b58d323553a2d3b1304145ed9970b459582 Author: Lionel Liu Date: 2017-11-02T06:41:35Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 02c48c600f70487b601d1b585d7ce617e035bdfe Author: Lionel Liu Date: 2017-11-02T09:40:27Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 10d35083ad192639b9a6f2e53562c0d4827c14e7 Author: Lionel Liu Date: 2017-11-02T10:10:32Z Merge branch 'master' of https://github.com/apache/incubator-griffin into apache-master commit 1114ed916c1b8e567482b70d16745a6bbe0e3d73 Author: Lionel Liu Date:
