[jira] [Commented] (GRIFFIN-164) Make 'Regular expression detection count' available in UI
[
https://issues.apache.org/jira/browse/GRIFFIN-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567895#comment-16567895
]
ASF GitHub Bot commented on GRIFFIN-164:
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-griffin/pull/381
> Make 'Regular expression detection count' available in UI
> -
>
> Key: GRIFFIN-164
> URL: https://issues.apache.org/jira/browse/GRIFFIN-164
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Affects Versions: 0.1.6-incubating
>Reporter: Enrico D'Urso
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> Hi,
> I have been playing for one month now with Griffin.
> Given my experience, some companies (included the one am working for as a
> consultant) prefer doing stuff using UI.
> Personally, I find very useful the following feature:
>
> * Regular expression detection count
> which is, I have a column which should contain just numbers so I want to
> check if my ETL process, wrongly, has populated my table with non-numeric
> values.
> I have been able to run such a job creating my self the right config.json, in
> particular, using spark-sql as dialect:
> {code:java}
> select count(*) from src where account_id rlike [^0-9]
> {code}
> I saw that in pr.component.ts there is a commented line of code:
> {code:java}
> // {"id":10,"itemName":"Regular Expression Detection Count","category":
> "Advanced Statistics"}
> {code}
> which I think is what I am talking about.
> Also, I can read:
> {code:java}
> // case 'Regular Expression Detection Count': // return
> 'count(source.`'+col.name+'`) where source.`'+col.name+'` LIKE ';
> {code}
> which should be the griffin-dsl dialect, even if, probably, the regex should
> be added just after LIKE.
> Then, once that the above griffin-dsl statement is available in the backend,
> ProfilingRulePlanTrans class
> should map that into 'rlike' Spark-sql clause.
> Am not sure where (and if) ProfilingRulePlanTrans should be modified as
> preGroupbyClause should contains everything, but I do not have enough
> knowledge about it.
>
> Please judge yourself the priority of such a feature, which knowing well the
> code, should not be too hard to make.
> Thanks,
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-164) Make 'Regular expression detection count' available in UI
[
https://issues.apache.org/jira/browse/GRIFFIN-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567884#comment-16567884
]
ASF GitHub Bot commented on GRIFFIN-164:
Github user icesmartjuan commented on the issue:
https://github.com/apache/incubator-griffin/pull/381
tested with newly commit, create Profiling measure successfully, create
jobs with the Profiling measure sucessfully
> Make 'Regular expression detection count' available in UI
> -
>
> Key: GRIFFIN-164
> URL: https://issues.apache.org/jira/browse/GRIFFIN-164
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Affects Versions: 0.1.6-incubating
>Reporter: Enrico D'Urso
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> Hi,
> I have been playing for one month now with Griffin.
> Given my experience, some companies (included the one am working for as a
> consultant) prefer doing stuff using UI.
> Personally, I find very useful the following feature:
>
> * Regular expression detection count
> which is, I have a column which should contain just numbers so I want to
> check if my ETL process, wrongly, has populated my table with non-numeric
> values.
> I have been able to run such a job creating my self the right config.json, in
> particular, using spark-sql as dialect:
> {code:java}
> select count(*) from src where account_id rlike [^0-9]
> {code}
> I saw that in pr.component.ts there is a commented line of code:
> {code:java}
> // {"id":10,"itemName":"Regular Expression Detection Count","category":
> "Advanced Statistics"}
> {code}
> which I think is what I am talking about.
> Also, I can read:
> {code:java}
> // case 'Regular Expression Detection Count': // return
> 'count(source.`'+col.name+'`) where source.`'+col.name+'` LIKE ';
> {code}
> which should be the griffin-dsl dialect, even if, probably, the regex should
> be added just after LIKE.
> Then, once that the above griffin-dsl statement is available in the backend,
> ProfilingRulePlanTrans class
> should map that into 'rlike' Spark-sql clause.
> Am not sure where (and if) ProfilingRulePlanTrans should be modified as
> preGroupbyClause should contains everything, but I do not have enough
> knowledge about it.
>
> Please judge yourself the priority of such a feature, which knowing well the
> code, should not be too hard to make.
> Thanks,
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-164) Make 'Regular expression detection count' available in UI
[
https://issues.apache.org/jira/browse/GRIFFIN-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567501#comment-16567501
]
ASF GitHub Bot commented on GRIFFIN-164:
Github user spencer-hivert-ck commented on the issue:
https://github.com/apache/incubator-griffin/pull/381
Thanks for taking a look at my PR @icesmartjuan!
I'm unable to re-produce any of the errors you've mentioned above.
Regarding #2, I've added default values to the fields in step 3, so that error
should not occur. Regarding #3, I didn't change the accuracy or job code, so
that error may be unrelated.
I brought up a fresh version of the service/ui and was unable to re-produce
the failures described above. Are you able to provide more detailed steps
and/or photos so I can attempt to re-produce these errors myself!
Thank you.
> Make 'Regular expression detection count' available in UI
> -
>
> Key: GRIFFIN-164
> URL: https://issues.apache.org/jira/browse/GRIFFIN-164
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Affects Versions: 0.1.6-incubating
>Reporter: Enrico D'Urso
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> Hi,
> I have been playing for one month now with Griffin.
> Given my experience, some companies (included the one am working for as a
> consultant) prefer doing stuff using UI.
> Personally, I find very useful the following feature:
>
> * Regular expression detection count
> which is, I have a column which should contain just numbers so I want to
> check if my ETL process, wrongly, has populated my table with non-numeric
> values.
> I have been able to run such a job creating my self the right config.json, in
> particular, using spark-sql as dialect:
> {code:java}
> select count(*) from src where account_id rlike [^0-9]
> {code}
> I saw that in pr.component.ts there is a commented line of code:
> {code:java}
> // {"id":10,"itemName":"Regular Expression Detection Count","category":
> "Advanced Statistics"}
> {code}
> which I think is what I am talking about.
> Also, I can read:
> {code:java}
> // case 'Regular Expression Detection Count': // return
> 'count(source.`'+col.name+'`) where source.`'+col.name+'` LIKE ';
> {code}
> which should be the griffin-dsl dialect, even if, probably, the regex should
> be added just after LIKE.
> Then, once that the above griffin-dsl statement is available in the backend,
> ProfilingRulePlanTrans class
> should map that into 'rlike' Spark-sql clause.
> Am not sure where (and if) ProfilingRulePlanTrans should be modified as
> preGroupbyClause should contains everything, but I do not have enough
> knowledge about it.
>
> Please judge yourself the priority of such a feature, which knowing well the
> code, should not be too hard to make.
> Thanks,
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-164) Make 'Regular expression detection count' available in UI
[
https://issues.apache.org/jira/browse/GRIFFIN-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566630#comment-16566630
]
ASF GitHub Bot commented on GRIFFIN-164:
Github user icesmartjuan commented on the issue:
https://github.com/apache/incubator-griffin/pull/381
Test from UI based on your current measure and service :
1. create Accuracy and Publish measure successfully
2. fail to create Data Profiling measure
pr.component.ts: line 390 fails the Profiling and Publish measure creation,
as this.step3.size is undefined type
3. failed to create job with above created Accuracy measure
{"timestamp":1533207681903,"status":400,"error":"Bad
Request","code":"40011","message":"We don't support such measure
type.","path":"/api/v1/jobs"}
```this.newJob
{job.name: "job1", job.type: "batch", measure.id: 1, cron.expression: "0
0/4 * * * ?", cron.time.zone: "GMT-8:00", …}cron.expression: "0 0/4 * * *
?"cron.time.zone: "GMT-8:00"data.segments: Array(2)0: as.baseline:
truedata.connector.name: "source1533207565052"segment.range: begin:
"-1day"length: "1day"__proto__: Object__proto__: Objectconstructor: ƒ
Object()hasOwnProperty: ƒ hasOwnProperty()isPrototypeOf: ƒ
isPrototypeOf()propertyIsEnumerable: ƒ propertyIsEnumerable()toLocaleString: ƒ
toLocaleString()toString: ƒ ()valueOf: ƒ valueOf()__defineGetter__: ƒ
__defineGetter__()__defineSetter__: ƒ __defineSetter__()__lookupGetter__: ƒ
__lookupGetter__()__lookupSetter__: ƒ __lookupSetter__()get __proto__: ƒ
__proto__()set __proto__: ƒ __proto__()1: data.connector.name:
"target1533207568926"segment.range: begin: "-1day"length: "1day"__proto__:
Object__proto__: Objectlength: 2__proto__: Array(0)job.name: "job1"job.type:
"batch"measure.id: 1__proto__: Object```
> Make 'Regular expression detection count' available in UI
> -
>
> Key: GRIFFIN-164
> URL: https://issues.apache.org/jira/browse/GRIFFIN-164
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Affects Versions: 0.1.6-incubating
>Reporter: Enrico D'Urso
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> Hi,
> I have been playing for one month now with Griffin.
> Given my experience, some companies (included the one am working for as a
> consultant) prefer doing stuff using UI.
> Personally, I find very useful the following feature:
>
> * Regular expression detection count
> which is, I have a column which should contain just numbers so I want to
> check if my ETL process, wrongly, has populated my table with non-numeric
> values.
> I have been able to run such a job creating my self the right config.json, in
> particular, using spark-sql as dialect:
> {code:java}
> select count(*) from src where account_id rlike [^0-9]
> {code}
> I saw that in pr.component.ts there is a commented line of code:
> {code:java}
> // {"id":10,"itemName":"Regular Expression Detection Count","category":
> "Advanced Statistics"}
> {code}
> which I think is what I am talking about.
> Also, I can read:
> {code:java}
> // case 'Regular Expression Detection Count': // return
> 'count(source.`'+col.name+'`) where source.`'+col.name+'` LIKE ';
> {code}
> which should be the griffin-dsl dialect, even if, probably, the regex should
> be added just after LIKE.
> Then, once that the above griffin-dsl statement is available in the backend,
> ProfilingRulePlanTrans class
> should map that into 'rlike' Spark-sql clause.
> Am not sure where (and if) ProfilingRulePlanTrans should be modified as
> preGroupbyClause should contains everything, but I do not have enough
> knowledge about it.
>
> Please judge yourself the priority of such a feature, which knowing well the
> code, should not be too hard to make.
> Thanks,
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-164) Make 'Regular expression detection count' available in UI
[
https://issues.apache.org/jira/browse/GRIFFIN-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16566375#comment-16566375
]
ASF GitHub Bot commented on GRIFFIN-164:
Github user bhlx3lyx7 commented on the issue:
https://github.com/apache/incubator-griffin/pull/381
I've reviewed the code for GRIFFIN-164, which seems good.
> Make 'Regular expression detection count' available in UI
> -
>
> Key: GRIFFIN-164
> URL: https://issues.apache.org/jira/browse/GRIFFIN-164
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Affects Versions: 0.1.6-incubating
>Reporter: Enrico D'Urso
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> Hi,
> I have been playing for one month now with Griffin.
> Given my experience, some companies (included the one am working for as a
> consultant) prefer doing stuff using UI.
> Personally, I find very useful the following feature:
>
> * Regular expression detection count
> which is, I have a column which should contain just numbers so I want to
> check if my ETL process, wrongly, has populated my table with non-numeric
> values.
> I have been able to run such a job creating my self the right config.json, in
> particular, using spark-sql as dialect:
> {code:java}
> select count(*) from src where account_id rlike [^0-9]
> {code}
> I saw that in pr.component.ts there is a commented line of code:
> {code:java}
> // {"id":10,"itemName":"Regular Expression Detection Count","category":
> "Advanced Statistics"}
> {code}
> which I think is what I am talking about.
> Also, I can read:
> {code:java}
> // case 'Regular Expression Detection Count': // return
> 'count(source.`'+col.name+'`) where source.`'+col.name+'` LIKE ';
> {code}
> which should be the griffin-dsl dialect, even if, probably, the regex should
> be added just after LIKE.
> Then, once that the above griffin-dsl statement is available in the backend,
> ProfilingRulePlanTrans class
> should map that into 'rlike' Spark-sql clause.
> Am not sure where (and if) ProfilingRulePlanTrans should be modified as
> preGroupbyClause should contains everything, but I do not have enough
> knowledge about it.
>
> Please judge yourself the priority of such a feature, which knowing well the
> code, should not be too hard to make.
> Thanks,
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-164) Make 'Regular expression detection count' available in UI
[
https://issues.apache.org/jira/browse/GRIFFIN-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565878#comment-16565878
]
ASF GitHub Bot commented on GRIFFIN-164:
GitHub user spencer-hivert-ck opened a pull request:
https://github.com/apache/incubator-griffin/pull/381
GRIFFIN-164 GRIFFIN-186 GRIFFIN-187: Profiling Re-factor + Regex/Empty
String Support
We've been working away on Griffin here at Credit Karma, and we'd love to
contribute back!
This PR tackles three separate tasks:
- [GRIFFIN-164](https://issues.apache.org/jira/browse/GRIFFIN-164): Regex
Support
- [GRIFFIN-186](https://issues.apache.org/jira/browse/GRIFFIN-186): Create
Profiling Measure Re-Factor
- [GRIFFIN-187](https://issues.apache.org/jira/browse/GRIFFIN-187): Empty
String Support
The details for each of these tasks can be found in the JIRA tickets linked
above!
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/spencer-hivert-ck/incubator-griffin
shivert/profiling-refactor-and-regex-support
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-griffin/pull/381.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #381
commit c4a344d7b771946ad41d9c89aa381b3925105464
Author: Spencer Hivert
Date: 2018-08-01T19:32:18Z
GRIFFIN-164 GRIFFIN-186 GRIFFIN-187: Profiling Re-factor + Regex/Empty
String Support
> Make 'Regular expression detection count' available in UI
> -
>
> Key: GRIFFIN-164
> URL: https://issues.apache.org/jira/browse/GRIFFIN-164
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Affects Versions: 0.1.6-incubating
>Reporter: Enrico D'Urso
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> Hi,
> I have been playing for one month now with Griffin.
> Given my experience, some companies (included the one am working for as a
> consultant) prefer doing stuff using UI.
> Personally, I find very useful the following feature:
>
> * Regular expression detection count
> which is, I have a column which should contain just numbers so I want to
> check if my ETL process, wrongly, has populated my table with non-numeric
> values.
> I have been able to run such a job creating my self the right config.json, in
> particular, using spark-sql as dialect:
> {code:java}
> select count(*) from src where account_id rlike [^0-9]
> {code}
> I saw that in pr.component.ts there is a commented line of code:
> {code:java}
> // {"id":10,"itemName":"Regular Expression Detection Count","category":
> "Advanced Statistics"}
> {code}
> which I think is what I am talking about.
> Also, I can read:
> {code:java}
> // case 'Regular Expression Detection Count': // return
> 'count(source.`'+col.name+'`) where source.`'+col.name+'` LIKE ';
> {code}
> which should be the griffin-dsl dialect, even if, probably, the regex should
> be added just after LIKE.
> Then, once that the above griffin-dsl statement is available in the backend,
> ProfilingRulePlanTrans class
> should map that into 'rlike' Spark-sql clause.
> Am not sure where (and if) ProfilingRulePlanTrans should be modified as
> preGroupbyClause should contains everything, but I do not have enough
> knowledge about it.
>
> Please judge yourself the priority of such a feature, which knowing well the
> code, should not be too hard to make.
> Thanks,
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (GRIFFIN-164) Make 'Regular expression detection count' available in UI
[
https://issues.apache.org/jira/browse/GRIFFIN-164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16565721#comment-16565721
]
Spencer Hivert commented on GRIFFIN-164:
I agree, this feature would be quite useful!
I'm working on this right now!
> Make 'Regular expression detection count' available in UI
> -
>
> Key: GRIFFIN-164
> URL: https://issues.apache.org/jira/browse/GRIFFIN-164
> Project: Griffin (Incubating)
> Issue Type: Improvement
>Affects Versions: 0.1.6-incubating
>Reporter: Enrico D'Urso
>Priority: Minor
> Fix For: 1.0.0-incubating
>
>
> Hi,
> I have been playing for one month now with Griffin.
> Given my experience, some companies (included the one am working for as a
> consultant) prefer doing stuff using UI.
> Personally, I find very useful the following feature:
>
> * Regular expression detection count
> which is, I have a column which should contain just numbers so I want to
> check if my ETL process, wrongly, has populated my table with non-numeric
> values.
> I have been able to run such a job creating my self the right config.json, in
> particular, using spark-sql as dialect:
> {code:java}
> select count(*) from src where account_id rlike [^0-9]
> {code}
> I saw that in pr.component.ts there is a commented line of code:
> {code:java}
> // {"id":10,"itemName":"Regular Expression Detection Count","category":
> "Advanced Statistics"}
> {code}
> which I think is what I am talking about.
> Also, I can read:
> {code:java}
> // case 'Regular Expression Detection Count': // return
> 'count(source.`'+col.name+'`) where source.`'+col.name+'` LIKE ';
> {code}
> which should be the griffin-dsl dialect, even if, probably, the regex should
> be added just after LIKE.
> Then, once that the above griffin-dsl statement is available in the backend,
> ProfilingRulePlanTrans class
> should map that into 'rlike' Spark-sql clause.
> Am not sure where (and if) ProfilingRulePlanTrans should be modified as
> preGroupbyClause should contains everything, but I do not have enough
> knowledge about it.
>
> Please judge yourself the priority of such a feature, which knowing well the
> code, should not be too hard to make.
> Thanks,
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
