[
https://issues.apache.org/jira/browse/GRIFFIN-333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160517#comment-17160517
]
Obaidul Karim commented on GRIFFIN-333:
---------------------------------------
Hi [~tushar.patil],
Thanks for the comment and suggestion.
If I want to measure the correctness of data by matching between source and
target, profiling is not a good option. It will need additional steps outside
of Griffin to match it again. It will reduce usability of Griffin.
Adding aggregation options will improve usability of Griffin as it will reduce
data transfer which leads to reduce cost as well.
So, why don't we add this option, especially when it could be done with minimum
changes in the code :)
> JDBC Connector: Ability to Use "group by" caluse
> ------------------------------------------------
>
> Key: GRIFFIN-333
> URL: https://issues.apache.org/jira/browse/GRIFFIN-333
> Project: Griffin
> Issue Type: Improvement
> Components: accuracy-batch
> Affects Versions: 0.6.0
> Reporter: Obaidul Karim
> Priority: Major
> Labels: column, groupby, jdbc
>
> *Background:*
> Refer to [https://issues.apache.org/jira/projects/GRIFFIN/issues/GRIFFIN-332].
> If we have the ability to select specific columns, it will open the door to
> use sql base aggregation, further reducing the volume of data from JDBC
> sources.
>
> *Proposed Improvement:*
> So, I propose the feature to allow JDBC connector to able to use sql based
> aggregations using clause `groupby`
> *Example:*
> Let's say we have source and target tables that have data like below.
> src:
> {code:java}
> ------------------------
> |employee_id |country|
> ------------------------
> |1 | NZ |
> |2 | DE |
> |3 | DE |
> |4 | NZ |
> |5 | DE |
> ....
> ....
> ------------------------
> {code}
> tgt:
> {code:java}
> ------------------------
> |total_employee|country|
> ------------------------
> |10 | NZ |
> |11 | DE |
> ------------------------
> {code}
> Then we can perform `accuracy` check [ `"rule":"src.total_employee =
> tgt.total_employee and src.country = tgt.country "` ] directly like below
> using `columns` and `groupby` clauses for source table:
> {code:java}
> {
> "name":"src",
> "connector":{
> "type":"jdbc",
> "config":{
> "database":"mydatabase",
> "tablename":"mytable",
> "columns":"count(*) total_employee, country",
> "groupby":"country",
> "url":"jdbc:sqlserver://myhost:1433;databaseName=mydatabase",
> "user":"user",
> "password":"password",
> "driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
> "where":""
> }
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
