[
https://issues.apache.org/jira/browse/BEAM-8196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927832#comment-16927832
]
Udi Meiri commented on BEAM-8196:
---------------------------------
Current theory is that test_big_query_standard_sql_kms_key_native is failing
(though this involves a lot of guesswork so I'm 30% sure).
In any case, I found this error in the worker-startup logs:
{code}
I 2019-09-11T13:07:04.210788Z Executing: /usr/local/bin/python -m
dataflow_worker.start -Djob_id=2019-09-11_05_19_06-12909785153999113879
-Dproject_id=apache-beam-testing -Dreporting_enabled=True
-Droot_url=https://dataflow.googleapis.com
-Dservice_path=https://dataflow.googleapis.com/
-Dtemp_gcs_directory=gs://unused
-Dworker_id=beamapp-jenkins-091112185-09110519-wcy3-harness-k5j8
-Ddataflow.worker.logging.location=/var/log/dataflow/python-dataflow-0-json.log
-Dlocal_staging_directory=/var/opt/google/dataflow
-Dsdk_pipeline_options={"display_data":[{"key":"runner","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"TestDataflowRunner"},{"key":"project","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"apache-beam-testing"},{"key":"job_name","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"beamapp-jenkins-0911121857-115043"},{"key":"staging_location","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0911121857-115043.1568204337.115228"},{"key":"temp_location","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0911121857-115043.1568204337.115228"},{"key":"region","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"us-central1"},{"key":"dataflow_kms_key","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test"},{"key":"num_workers","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"INTEGER","value":1},{"key":"dataflow_worker_jar","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.16.0-SNAPSHOT.jar"},{"key":"experiments","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"['use_fastavro']"},{"key":"requirements_file","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"postcommit_requirements.txt"},{"key":"beam_plugins","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"['apache_beam.io.filesystem.FileSystem',
'apache_beam.io.hadoopfilesystem.HadoopFileSystem',
'apache_beam.io.localfilesystem.LocalFileSystem',
'apache_beam.io.gcp.gcsfilesystem.GCSFileSystem',
'apache_beam.io.filesystem_test.TestingFileSystem',
'apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer',
'apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer',
'apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer',
'apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer']"},{"key":"sdk_location","namespace":"apache_beam.options.pipeline_options.PipelineOptions","type":"STRING","value":"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/build/apache-beam.tar.gz"}],"options":{"autoscalingAlgorithm":"NONE","beam_plugins":["apache_beam.io.filesystem.FileSystem","apache_beam.io.hadoopfilesystem.HadoopFileSystem","apache_beam.io.localfilesystem.LocalFileSystem","apache_beam.io.gcp.gcsfilesystem.GCSFileSystem","apache_beam.io.filesystem_test.TestingFileSystem","apache_beam.runners.interactive.display.pipeline_graph_renderer.PipelineGraphRenderer","apache_beam.runners.interactive.display.pipeline_graph_renderer.MuteRenderer","apache_beam.runners.interactive.display.pipeline_graph_renderer.TextRenderer","apache_beam.runners.interactive.display.pipeline_graph_renderer.PydotRenderer"],"dataflowJobId":"2019-09-11_05_19_06-12909785153999113879","dataflow_endpoint":"https://dataflow.googleapis.com","dataflow_kms_key":"projects/apache-beam-testing/locations/global/keyRings/beam-it/cryptoKeys/test","dataflow_worker_jar":"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/runners/google-cloud-dataflow-java/worker/build/libs/beam-runners-google-cloud-dataflow-java-fn-api-worker-2.16.0-SNAPSHOT.jar","direct_num_workers":1,"direct_runner_bundle_repeat":0,"direct_runner_use_stacked_bundle":true,"dry_run":false,"enable_streaming_engine":false,"environment_cache_millis":0,"experiments":["use_fastavro"],"fake_flag":false,"gcpTempLocation":"gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0911121857-115043.1568204337.115228","job_name":"beamapp-jenkins-0911121857-115043","m_flag":false,"male":false,"maxNumWorkers":0,"mock_flag":false,"no_auth":false,"numWorkers":1,"num_workers":1,"pipelineUrl":"gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0911121857-115043.1568204337.115228/pipeline.pb","pipeline_type_check":true,"profile_cpu":false,"profile_memory":false,"profile_sample_rate":1,"project":"apache-beam-testing","region":"us-central1","requirements_file":"postcommit_requirements.txt","runner":"TestDataflowRunner","runtime_type_check":false,"save_main_session":false,"sdk_location":"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/build/apache-beam.tar.gz","sdk_worker_parallelism":0,"staging_location":"gs://temp-storage-for-end-to-end-tests/staging-it/beamapp-jenkins-0911121857-115043.1568204337.115228","streaming":false,"style":"scrambled","temp_location":"gs://temp-storage-for-end-to-end-tests/temp-it/beamapp-jenkins-0911121857-115043.1568204337.115228","type_check_strictness":"DEFAULT_TO_ANY","update":false}}
I 2019-09-11T13:07:06.335089Z Traceback (most recent call last):
I 2019-09-11T13:07:06.335179Z File "/usr/local/lib/python3.7/runpy.py", line
193, in _run_module_as_main
I 2019-09-11T13:07:06.335194Z "__main__", mod_spec)
I 2019-09-11T13:07:06.335204Z File "/usr/local/lib/python3.7/runpy.py", line
85, in _run_code
I 2019-09-11T13:07:06.335215Z exec(code, run_globals)
I 2019-09-11T13:07:06.335225Z File
"/usr/local/lib/python3.7/site-packages/dataflow_worker/start.py", line 29, in
<module>
I 2019-09-11T13:07:06.335241Z from apache_beam.runners.worker import logger
I 2019-09-11T13:07:06.335261Z File
"/usr/local/lib/python3.7/site-packages/apache_beam/__init__.py", line 97, in
<module>
I 2019-09-11T13:07:06.335273Z from apache_beam import coders
I 2019-09-11T13:07:06.335284Z File
"/usr/local/lib/python3.7/site-packages/apache_beam/coders/__init__.py", line
19, in <module>
I 2019-09-11T13:07:06.335296Z from apache_beam.coders.coders import *
I 2019-09-11T13:07:06.335305Z File
"/usr/local/lib/python3.7/site-packages/apache_beam/coders/coders.py", line 33,
in <module>
I 2019-09-11T13:07:06.335333Z from apache_beam.coders import coder_impl
I 2019-09-11T13:07:06.335345Z File "apache_beam/utils/windowed_value.pxd",
line 28, in init apache_beam.coders.coder_impl
I 2019-09-11T13:07:06.335356Z File "apache_beam/utils/windowed_value.py",
line 34, in init apache_beam.utils.windowed_value
I 2019-09-11T13:07:06.335369Z File
"/usr/local/lib/python3.7/site-packages/apache_beam/utils/timestamp.py", line
34, in <module>
I 2019-09-11T13:07:06.335381Z from apache_beam.portability import
common_urns
I 2019-09-11T13:07:06.335391Z File
"/usr/local/lib/python3.7/site-packages/apache_beam/portability/common_urns.py",
line 24, in <module>
I 2019-09-11T13:07:06.335404Z from apache_beam.portability.api import
beam_runner_api_pb2
I 2019-09-11T13:07:06.335414Z File
"/usr/local/lib/python3.7/site-packages/apache_beam/portability/api/beam_runner_api_pb2.py",
line 16, in <module>
I 2019-09-11T13:07:06.335426Z import endpoints_pb2 as endpoints__pb2
I 2019-09-11T13:07:06.335435Z ModuleNotFoundError: No module named
'endpoints_pb2'
I 2019-09-11T13:07:06.472722Z /usr/local/bin/python failed with exit status 1
{code}
> Python 3.5 post commit timed out at 100 minutes
> -----------------------------------------------
>
> Key: BEAM-8196
> URL: https://issues.apache.org/jira/browse/BEAM-8196
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
> Reporter: Ahmet Altay
> Assignee: Udi Meiri
> Priority: Critical
>
> https://builds.apache.org/job/beam_PostCommit_Python35/435/
> This post commit took 100 minutes and timedout. Should we increase the
> timeout? We can also look into why this postcommit was slow. A later post
> commit (https://builds.apache.org/job/beam_PostCommit_Python35/437/)
> completed in 66 minutes.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
