Hello I have been directed to this mail list by posting to Stack Overflow. If this is inappropriate use of mail list please let me know.https://stackoverflow.com/questions/54100870/autoscale-stuck-at-1-never-see-getsplitbacklogbytes-execute
I am creating a custom IO connector in Java that runs is streaming mode with 2.8.0 SDK. It reads from a messaging queue for further pipeline processing, similar to JMS IO. All works well except the auto-scaling does not seem to work. The odd part for me is that I have implemented a getSplitBacklogBytes() method, but as far as the logs show, it is never executed. I am testing with a modified wordcount example to test, executed with: --runner=DataflowRunner --autoscalingAlgorithm=THROUGHPUT_BASED --maxNumWorkers=4 The Pipeline options and logs seem to show that auto-scaling is being set up correctly. PipelineOptions: Current Settings: autoscalingAlgorithm: THROUGHPUT_BASED maxNumWorkers: 4 numWorkers: 0 Logs show: 20:50:54:977 .. [INFO] .. - 2019-01-08T20:50:40.937Z: Autoscaling is enabled for job 2019-01-08_12_50_40-3227727519433807139. The number of workers will be between 1 and 4. 20:50:54:991 .. [INFO] .. - 2019-01-08T20:50:42.943Z: Autoscaling: Enabled for job 2019-01-08_12_50_40-3227727519433807139 between 1 and 4 worker processes. 20:50:55:052 .. [INFO] .. - 2019-01-08T20:50:49.186Z: Generating 4 persistent disks which will allow autoscaling from 1 to 4 workers. 20:50:55:053 .. [INFO] .. - 2019-01-08T20:50:49.493Z: Starting 1 workers... Is there anything else that needs to be done to enable this feature, or anything I might have done to inadvertently disable this feature? Source code lives here: https://github.com/SolaceLabs/solace-beam-integration/tree/development
