Abacn commented on PR #29474: URL: https://github.com/apache/beam/pull/29474#issuecomment-1817118696
I remember currently we have similar warning, but just not quite accurate (many false positive). For example, this load test: `2023-11-17_07_58_43-10220172113544842485`, It has a WARNING worker log: ``` Discarding unparseable args: ['--autoscalingAlgorithm=NONE', '--dataflowJobId=2023-11-17_07_58_43-10220172113544842485', '--direct_runner_use_stacked_bundle', '--fanout=1', '--gcpTempLocation=gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-1-1117155301.1700236723.066487', '--influx_db_name=beam_test_metrics', '--influx_hostname=http://10.128.0.96:8086', '--influx_measurement=python_batch_gbk_1', '--input_options={"algorithm": "lcg", "key_size": 1, "num_records": 200000000, "value_size": 9}', '--iterations=1', '--jobId=2023-11-17_07_58_43-10220172113544842485', '--maxNumWorkers=0', '--metrics_dataset=load_test', '--metrics_table=python_dataflow_batch_gbk_1', '--numWorkers=5', '--pipelineUrl=gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-1-1117155301.1700236723.066487/pipeline.pb', '--pipeline_type_check', '--publish_to_big_query', '--tempLocation=gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-1-1117155301.1700236723.066487', '--timeout_ms=0'] ``` the full pipeline option also output in another INFO log: ``` Pipeline_options: {'runner': 'DataflowRunner', 'project': 'apache-beam-testing', 'job_name': 'load-tests-python-dataflow-batch-gbk-1-1117155301', 'staging_location': 'gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-1-1117155301.1700236723.066487', 'temp_location': 'gs://temp-storage-for-perf-tests/loadtests/load-tests-python-dataflow-batch-gbk-1-1117155301.1700236723.066487', 'region': 'us-central1', 'gcp_oauth_scopes': ['https://www.googleapis.com/auth/bigquery', 'https://www.googleapis.com/auth/cloud-platform', 'https://www.googleapis.com/auth/devstorage.full_control', 'https://www.googleapis.com/auth/userinfo.email', 'https://www.googleapis.com/auth/datastore', 'https://www.googleapis.com/auth/spanner.admin', 'https://www.googleapis.com/auth/spanner.data', 'https://www.googleapis.com/auth/bigquery', 'https://www.googleapis.com/auth/cloud-platform', 'https://www.googleapis.com/auth/devstorage.full_control', 'https://www.googleapis.com/auth/userinfo.email', 'https://www.googleapis.com/auth/datastore', 'https://www.googleapis.com/auth/spanner.admin', 'https://www.googleapis.com/auth/spanner.data'] 'num_workers': 5, 'autoscaling_algorithm': 'NONE', 'experiments': ['beam_fn_api', 'use_unified_worker', 'use_runner_v2', 'use_portable_job_submission', 'use_multiple_sdk_containers', 'enable_always_on_exception_sampling'], 'beam_plugins': ['apache_beam.io.filesystem.FileSystem', 'apache_beam.io.hadoopfilesystem.HadoopFileSystem', 'apache_beam.io.localfilesystem.LocalFileSystem', 'apache_beam.io.gcp.gcsfilesystem.GCSFileSystem', 'apache_beam.io.aws.s3filesystem.S3FileSystem', 'apache_beam.io.azure.blobstoragefilesystem.BlobStorageFileSystem'], 'sdk_location': '/runner/_work/beam/beam/sdks/python/build/apache-beam.tar.gz', 'sdk_worker_parallelism': '1', 'environment_cache_millis': '0'} ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
