Abacn commented on issue #23904: URL: https://github.com/apache/beam/issues/23904#issuecomment-2013060164
Bump to using 50 workers the test passed. It tokes 2 h to run. Throughput is like this: input/output PCollection of GBK:  ------ However, using 5 worker the test isn't a matter of not finish in time, the pipeline just stucks after some time: input/output PCollection of GBK:  and there were worker crash happened throughout the pipeline ran: number of worker:  memory usage:  In summary, what happens is - If the num of worker is not large, each worker appear to accumulate more works, and causing OOM eventually, and the pipeline get stuck (persumably repeat retry - oom - retry - oom) - If there are sufficient number of workers, no worker crash, the data can be processed in time, though slowly -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
