Abacn commented on PR #26179: URL: https://github.com/apache/beam/pull/26179#issuecomment-1500723768
I now believe this is somewhat red herring. Direct runner simply is not efficient: cron run 473: actual number of records 35990 smaller than expected: 100000. cron run 475: actual number of records 26030 smaller than expected: 100000. cron run 478: actual number of records 42390 smaller than expected: 100000. cron run 479: actual number of records 39350 smaller than expected: 100000. cron run 481: actual number of records 23190 smaller than expected: 100000. with this change: actual number of records 51990 smaller than expected: 100000. Defer the connections into startBundle seems help a little bit so I intend to preserve the change, but also decrease the number of record and see. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
