Abacn commented on PR #27595:
URL: https://github.com/apache/beam/pull/27595#issuecomment-1646005465
The tests run successfully in local flink cluster. Reproduce step:
1. setup a flink cluster (flink-1.15) locally:
`./flink-1.15.0/bin/start-cluster.sh`
2. In beam repo, install beam Python: `cd ~/beam/sdks/python && pip install
.[gcp]`
3. Start job server: `docker run --publish 8099:8099 --publish 8098:8098
--publish 8097:8097
gcr.io/apache-beam-testing/beam_portability/beam_flink1.15_job_server:latest
--flink-master=host.docker.internal:8081`
4. start beam load test: `python -m
apache_beam.testing.load_tests.combine_test
--test-pipeline-options="--runner=PortableRunner --job_endpoint=localhost:8099
--environment_type=DOCKER
--environment_config=gcr.io/apache-beam-testing/beam_portability/beam_python3.8_sdk:latest
--publish_to_big_query=false
--input_options='{\"num_records\":2500000,\"key_size\":10,\"value_size\":90}'
--top_count=20"`
Result:
```
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Missing InfluxDB
options. Metrics will not be published to InfluxDB
INFO:root:Using provided Python SDK container image:
gcr.io/apache-beam-testing/beam_portability/beam_python3.8_sdk:latest
INFO:root:Python SDK container image set to
"gcr.io/apache-beam-testing/beam_portability/beam_python3.8_sdk:latest" for
Docker environment
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function pack_combiners at 0x1193031f0> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function lift_combiners at 0x119303280> ====================
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function sort_stages at 0x1193039d0> ====================
WARNING:apache_beam.options.pipeline_options:Discarding unparseable args:
['--top_count=20']
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
STOPPED
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
STARTING
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
RUNNING
INFO:apache_beam.runners.portability.portable_runner:Job state changed to
DONE
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Load test
results for test: b9216a86d14440788a70f75ab9c92d58 and timestamp:
1689959067.058372:
INFO:apache_beam.testing.load_tests.load_test_metrics_utils:Metric:
default_runtime Value: 264
```
(running on Mac M1, amd64 python sdk container image is slow. If build
container image locally (`./gradlew :sdks:python:container:py38:docker` then do
not feed environment_config pipeline option), it takes 30 s.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
