wankunde opened a new pull request #504: build and run transform steps with multiple threads URL: https://github.com/apache/griffin/pull/504 There are sequence of dq steps in each DQJob,and run those steps one by one (with foldLeft function). We can use multiple threads to run some of those steps which have no dependency. For example: In a DQBatchJob, a accuracyExpr will have for steps **_missRecords ,_missCount , __totalCount, accu** . _missCount and **_totalCount step can run at the same time . In SeqDQStep ,it just need contains some root steps without dependency steps. If each step knows it's dependency steps, and when they are ready, we can run the step itself . Running step : accu | |---__missCount | | |---__missRecords | |---__totalCount Running step : __missCount | |---__missRecords Running step : __missRecords Running step : __totalCount
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
