See
<https://builds.apache.org/job/beam_PostCommit_Python36/2126/display/redirect?page=changes>
Changes:
[rohde.samuel] Add dependency comment in streaming cache
[ehudm] [BEAM-1894] Remove obsolete EagerRunner test
------------------------------------------
[...truncated 10.51 MB...]
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing
<https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>:
response: <{'vary': 'Origin, X-Origin, Referer', 'content-type':
'application/json; charset=UTF-8', 'date': 'Wed, 01 Apr 2020 18:59:18 GMT',
'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff',
'transfer-encoding': 'chunked', 'status': '403', 'content-length': '478',
'-content-encoding': 'gzip'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not
have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does
not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T18:59:24.186Z:
JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py",
line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py",
line 218, in execute
self._split_task)
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py",
line 226, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py",
line 263, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigquery.py",
line 669, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigquery.py",
line 728, in _export_files
include_header=False)
File "/usr/local/lib/python3.6/site-packages/apache_beam/utils/retry.py",
line 236, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 715, in perform_extract_job
response = self.client.jobs.Insert(request)
File
"/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing
<https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>:
response: <{'vary': 'Origin, X-Origin, Referer', 'content-type':
'application/json; charset=UTF-8', 'date': 'Wed, 01 Apr 2020 18:59:22 GMT',
'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff',
'transfer-encoding': 'chunked', 'status': '403', 'content-length': '478',
'-content-encoding': 'gzip'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not
have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does
not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T18:59:27.667Z:
JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py",
line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py",
line 218, in execute
self._split_task)
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py",
line 226, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py",
line 263, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigquery.py",
line 669, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigquery.py",
line 728, in _export_files
include_header=False)
File "/usr/local/lib/python3.6/site-packages/apache_beam/utils/retry.py",
line 236, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 715, in perform_extract_job
response = self.client.jobs.Insert(request)
File
"/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing
<https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>:
response: <{'vary': 'Origin, X-Origin, Referer', 'content-type':
'application/json; charset=UTF-8', 'date': 'Wed, 01 Apr 2020 18:59:25 GMT',
'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff',
'transfer-encoding': 'chunked', 'status': '403', 'content-length': '478',
'-content-encoding': 'gzip'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not
have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does
not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T18:59:31.280Z:
JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py",
line 647, in do_work
work_executor.execute()
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py",
line 218, in execute
self._split_task)
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py",
line 226, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py",
line 263, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigquery.py",
line 669, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigquery.py",
line 728, in _export_files
include_header=False)
File "/usr/local/lib/python3.6/site-packages/apache_beam/utils/retry.py",
line 236, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 715, in perform_extract_job
response = self.client.jobs.Insert(request)
File
"/usr/local/lib/python3.6/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.6/site-packages/apitools/base/py/base_api.py",
line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpForbiddenError: HttpError accessing
<https://bigquery.googleapis.com/bigquery/v2/projects/clouddataflow-readonly/jobs?alt=json>:
response: <{'vary': 'Origin, X-Origin, Referer', 'content-type':
'application/json; charset=UTF-8', 'date': 'Wed, 01 Apr 2020 18:59:29 GMT',
'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff',
'transfer-encoding': 'chunked', 'status': '403', 'content-length': '478',
'-content-encoding': 'gzip'}>, content <{
"error": {
"code": 403,
"message": "Access Denied: Project clouddataflow-readonly: User does not
have bigquery.jobs.create permission in project clouddataflow-readonly.",
"errors": [
{
"message": "Access Denied: Project clouddataflow-readonly: User does
not have bigquery.jobs.create permission in project clouddataflow-readonly.",
"domain": "global",
"reason": "accessDenied"
}
],
"status": "PERMISSION_DENIED"
}
}
>
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T18:59:31.715Z:
JOB_MESSAGE_BASIC: Finished operation
read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+months
with tornadoes+monthly count/GroupByKey+monthly count/Combine/Partial+monthly
count/GroupByKey/Reify+monthly count/GroupByKey/Write
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T18:59:31.791Z:
JOB_MESSAGE_DEBUG: Executing failure step failure22
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T18:59:31.823Z:
JOB_MESSAGE_ERROR: Workflow failed. Causes:
S02:read/Read+read/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+months
with tornadoes+monthly count/GroupByKey+monthly count/Combine/Partial+monthly
count/GroupByKey/Reify+monthly count/GroupByKey/Write failed., Internal Issue
(ba6b0bf1ba6516f5): 63963027:24514
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T18:59:31.954Z:
JOB_MESSAGE_DETAILED: Cleaning up.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T18:59:32.019Z:
JOB_MESSAGE_DEBUG: Starting worker pool teardown.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T18:59:32.053Z:
JOB_MESSAGE_BASIC: Stopping worker pool...
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T19:01:47.232Z:
JOB_MESSAGE_DETAILED: Autoscaling: Resized worker pool from 1 to 0.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T19:01:47.284Z:
JOB_MESSAGE_BASIC: Worker pool stopped.
apache_beam.runners.dataflow.dataflow_runner: INFO: 2020-04-01T19:01:47.329Z:
JOB_MESSAGE_DEBUG: Tearing down pending resources...
apache_beam.runners.dataflow.dataflow_runner: INFO: Job
2020-04-01_11_52_52-17178797724974179964 is in state JOB_STATE_FAILED
apache_beam.io.gcp.tests.utils: INFO: Clean up a BigQuery table with project:
apache-beam-testing, dataset: BigQueryTornadoesIT, table:
monthly_tornadoes_1585767149007.
google.auth.transport._http_client: DEBUG: Making request: GET
http://169.254.169.254
google.auth.transport._http_client: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/project/project-id
urllib3.util.retry: DEBUG: Converted retries value: 3 -> Retry(total=3,
connect=None, read=None, redirect=None, status=None)
google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true
urllib3.connectionpool: DEBUG: Starting new HTTP connection (1):
metadata.google.internal:80
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET
/computeMetadata/v1/instance/service-accounts/default/?recursive=true HTTP/1.1"
200 144
google.auth.transport.requests: DEBUG: Making request: GET
http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/[email protected]/token
urllib3.connectionpool: DEBUG: http://metadata.google.internal:80 "GET
/computeMetadata/v1/instance/service-accounts/[email protected]/token
HTTP/1.1" 200 192
urllib3.connectionpool: DEBUG: Starting new HTTPS connection (1):
bigquery.googleapis.com:443
urllib3.connectionpool: DEBUG: https://bigquery.googleapis.com:443 "DELETE
/bigquery/v2/projects/apache-beam-testing/datasets/BigQueryTornadoesIT/tables/monthly_tornadoes_1585767149007
HTTP/1.1" 404 None
--------------------- >> end captured logging << ---------------------
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_11_52_52-11138813513496711022?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_06_52-4551659823270885636?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_17_34-12607334891195767590?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_26_39-11721991656416340339?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_35_36-15849772090826769481?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_44_15-5690873115935729668?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_11_52_47-7478127911751832627?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_17_03-9441698879957066624?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_26_06-11732372312678135114?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_43_40-12980848998198170102?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_11_52_50-10845991994651489496?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_05_17-5773063670986289776?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_14_39-1078234394781753542?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_22_53-7004890905678871913?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_31_15-9706199082070021774?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_40_08-16430595636508667713?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_11_52_47-12526079005124822987?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_13_38-7193319510290490206?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_21_45-7041784639467406538?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_29_38-9997231722637973826?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_39_00-7805506943861357769?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_11_52_54-8370343399165891577?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_02_48-2123052712955532272?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_10_51-16220168099624578607?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_19_29-10313648285153955563?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_28_20-3670695645583699386?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_37_13-13369628821933729815?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_45_59-18083186752987814426?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_11_52_52-17178797724974179964?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_02_15-12621427799400079977?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_12_00-11606688037702343123?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_20_37-2996293538302341126?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_28_41-10985695310067344167?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_36_32-10033238959322873418?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_44_58-8846686890225773174?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_53_41-15013069486868132300?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_11_52_48-12756832722891854432?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_04_06-13320147555251839463?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_17_00-18083053361129565547?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_25_45-15424207669420523258?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_35_20-16083288834625384710?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_43_31-18323035647827640204?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_11_52_50-16178589898449828737?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_01_22-5778591070866273659?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_11_31-16674151678259395667?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_20_04-9119276360872077185?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_29_04-525283905688978108?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_37_56-15150341122605172720?project=apache-beam-testing
Worker logs:
https://console.cloud.google.com/dataflow/jobs/us-central1/2020-04-01_12_46_13-4262362517476812455?project=apache-beam-testing
----------------------------------------------------------------------
XML: nosetests-postCommitIT-df-py36.xml
----------------------------------------------------------------------
XML:
<https://builds.apache.org/job/beam_PostCommit_Python36/ws/src/sdks/python/nosetests.xml>
----------------------------------------------------------------------
Ran 59 tests in 4174.933s
FAILED (SKIP=9, errors=2)
> Task :sdks:python:test-suites:dataflow:py36:postCommitIT FAILED
FAILURE: Build failed with an exception.
* Where:
Build file
'<https://builds.apache.org/job/beam_PostCommit_Python36/ws/src/sdks/python/test-suites/dataflow/py36/build.gradle'>
line: 56
* What went wrong:
Execution failed for task ':sdks:python:test-suites:dataflow:py36:postCommitIT'.
> Process 'command 'sh'' finished with non-zero exit value 1
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 6.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/5.2.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 1h 15m 4s
86 actionable tasks: 84 executed, 2 from cache
Publishing build scan...
https://gradle.com/s/bl6dy5hw6x7yo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
