Abacn commented on PR #31253: URL: https://github.com/apache/beam/pull/31253#issuecomment-2118608486
> > Most importantly, currently the downscale decision won't be made until 3+3=6 min of pipeline run, which already cause workitem failing > > What does each 3 mean? is there a way to get around it? This is Dataflow autoscaler strategy thing The first 3 min is that the first throttled signal from the backend appears to be 3 min after pipeline running. Example log: ``` F11 is throttled (fraction of time throttled = 0.2472). Recommend 75.28 threads instead of 100 ``` then, there is downscale signal every 30 s. The second 3 min is due to downscale signal must be stable for 3 min then autoscaler will take action. - Example recommendation < 3 min: ``` why: Desire to downscale because overall work duration is 17h59m48.01785516525s and desired parallelism is 74 of which 100% is allocated to this pool, but there was a large decrease for only 2m30.000081865s, less than 3m ``` - Example recommendation at 3 min: ``` why: Downscaling because overall work duration is 18h15m50.3475206785s and desired parallelism is 65 of which 100% is allocated to this pool and there was a large decrease for more than 3m ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
