TheNeuralBit commented on issue #22422: URL: https://github.com/apache/beam/issues/22422#issuecomment-1193019507
This is a gross one. The problem is the space in the argument `--experiments=use_runner_v2, shuffle_mode=appliance` shuffle_mode=appliance is being interpreted as a separate argument because there's a space and it's not quoted. Then after we [parse the known arguments](https://github.com/apache/beam/blob/79db4d25cc83d5d4890556b28072d801badd430d/sdks/python/apache_beam/options/pipeline_options.py#L317) we're left with the unknown_args: ``` ['--iterations=10', '--number_of_counter_operations=0', '--number_of_counters=0', 'shuffle_mode=appliance'] ``` The [logic](https://github.com/apache/beam/blob/79db4d25cc83d5d4890556b28072d801badd430d/sdks/python/apache_beam/options/pipeline_options.py#L326) interprets shuffle_mode=appliance as the argument for an option with name `number_of_counters=0`. We can fix by changing `use_runner_v2, shuffle_mode=appliance`, `use_runner_v2,shuffle_mode=appliance`, or nmore generally by adding quoting logic in LoadTestsBuilder.groovy. We should also adjust the retain_unknown_options logic. It's bad that it assumes the arguments are consecutive. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
