Hello .
I would like to ask question about the error I am facing with worker
pool of sdk container.
I get below error when I run the pipeline.
----------------------------------------------------------------------------------------
python-sdk_1 | 2019/12/31 02:57:26 Starting worker pool 1: python -m
apache_beam.runners.worker.worker_pool_main --service_port=50000
--container_executable=/opt/apache/beam/boot
python-sdk_1 | INFO:root:Started worker pool servicer at port:
localhost:50000 with executable: /opt/apache/beam/boot
python-sdk_1 | WARNING:root:Starting worker with command
['/opt/apache/beam/boot', '--id=1-1',
'--logging_endpoint=localhost:35615',
'--artifact_endpoint=localhost:42723',
'--provision_endpoint=localhost:45759',
'--control_endpoint=localhost:43185']
python-sdk_1 | 2019/12/31 02:57:45 Initializing python harness:
/opt/apache/beam/boot --id=1-1 --logging_endpoint=localhost:35615
--artifact_endpoint=localhost:42723
--provision_endpoint=localhost:45759
--control_endpoint=localhost:43185
python-sdk_1 | 2019/12/31 02:59:45 Failed to obtain provisioning
information: failed to dial server at localhost:45759
python-sdk_1 | caused by:
python-sdk_1 | context deadline exceeded
----------------------------------------------------------------------------------------
In flink taskmanager's log , it keeps waiting for response from sdk container.
----------------------------------------------------------------------------------------
taskmanager_1 | 2019-12-31 02:57:45,445 INFO
org.apache.flink.configuration.GlobalConfiguration -
Loading configuration property: query.server.port, 6125
taskmanager_1 | 2019-12-31 02:58:26,678 INFO
org.apache.beam.runners.fnexecution.environment.ExternalEnvironmentFactory
- Still waiting for startup of environment from python-sdk:50000 for
worker id 1-1
----------------------------------------------------------------------------------------
Looking at flink jobmanager's log, error is logged after starting map
transform.
So looks like, request from taskmanager is reached to sdk conatiner
but not processed correctly.
Sounds like I am missing some setting for sdk container..
----------------------------------------------------------------------------------------
jobmanager_1 | 2019-12-31 02:57:44,987 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
DataSource (Impulse) (1/1) (4e6f68fd31bafa066b740943bc3ea736) switched
from RUNNING to FINISHED.
jobmanager_1 | 2019-12-31 02:57:44,989 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph -
MapPartition (MapPartition at [2]Create/{FlatMap(<lambda at
core.py:2468>), Map(decode)}) (1/1) (067ed452ebd15c1175ecde0ae40e8ac7)
switched from DEPLOYING to RUNNING.
----------------------------------------------------------------------------------------
Command line for building sdk container is
----------------------------------------------------------------------------------------
./gradlew :sdks:python:container:py37:dockerPush
-Pdocker-repository-root=${GCR_HOSTNAME}/${PROJECT_ID}
-Pdocker-tag=release-2.16.0
----------------------------------------------------------------------------------------
My docker compose
https://github.com/yuwtennis/beam-deployment/blob/master/flink-session-cluster/docker/docker-compose.yml
My pipeline code
https://github.com/yuwtennis/beam-deployment/blob/master/flink-session-cluster/docker/samples/src/sample.py
Would there be any settings I need to use for starting up sdk container ?
Best Regards,
Yu Watanabe
--
Yu Watanabe
[email protected]
