ahmedabu98 commented on issue #32746: URL: https://github.com/apache/beam/issues/32746#issuecomment-2407222989
Fortunately when this error happens, the bundle is retried so there is no data-loss. The pipeline also adapts well by auto-scaling to more workers: <img width="500" alt="image" src="https://github.com/user-attachments/assets/2c1787f5-9c5a-41dd-8822-f100cfb3b6f2";> My guess is a higher triggering frequency means the worker has to store more records in memory when writing to files. We could also be doing something inefficient with memory -- will investigate further -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
