[
https://issues.apache.org/jira/browse/GRIFFIN-259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wan kun updated GRIFFIN-259:
----------------------------
External issue URL: https://github.com/apache/griffin/pull/504
> Running measure transformStep with multiple threads
> ---------------------------------------------------
>
> Key: GRIFFIN-259
> URL: https://issues.apache.org/jira/browse/GRIFFIN-259
> Project: Griffin
> Issue Type: Improvement
> Reporter: wan kun
> Priority: Major
>
> There are sequence of dq steps in each DQJob,and run those steps one by one
> (with foldLeft function).
> We can use multiple threads to run some of those steps which have no
> dependency.
> For example:
> In a DQBatchJob, a accuracyExpr will have for steps *__missRecords
> ,__missCount , __totalCount, accu .*
> *__missCount_* _and ***__totalCount* step can run at the same time .
>
> In SeqDQStep ,it just need contains some root steps without dependency steps.
> If each step knows it's dependency steps, and when they are ready, we can run
> the step itself .
>
>
> {code:java}
> Running step :
> accu
> | |---__missCount
> | | |---__missRecords
> | |---__totalCount
> Running step :
> __missCount
> | |---__missRecords
> Running step :
> __missRecords
> Running step :
> __totalCount
>
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
