See
<https://ci-beam.apache.org/job/beam_PerformanceTests_BiqQueryIO_Read_Python/116/display/redirect?page=changes>
Changes:
[srohde] Add the ib.recordings API
[srohde] Fix macos IB recordings test
[srohde] Delete RM reference from environment when evicted
[srohde] Move pipeline_var into RM constructor
[noreply] Revert "[BEAM-10861] Adds URNs and payloads to PubSub transforms to
[aromanenko.dev] [BEAM-2546] Update CHANGES.md and add some checks
[Valentyn Tymofieiev] [BEAM-9372][BEAM-7372] Remove Py2 and Py35 test suites.
[Valentyn Tymofieiev] [BEAM-9372][BEAM-8371] Sunset Python 2 and Python 3.5
support in Apache
[Valentyn Tymofieiev] [BEAM-9372][BEAM-7372] Clean release script and correct
naming pattern
[srohde] Fix BEAM-10956
[noreply] [BEAM-10586] Remove Python 2.7 and Python 3.5 support in Dataflow
[noreply] Merge pull request #12912 from [BEAM-10938] Adds support for writing a
[sambvfx] [BEAM-8660] Use PortableOptions.artifact_endpoint if provided over
[sambvfx] Add artifact_endpoint to TestJobServicePlan
[sambvfx] Add simple test for artifact_endpoint
[noreply] [BEAM-9547] Roll forward #12858 (#12920)
[noreply] Merge pull request #12782 from [BEAM-10950] Overriding Dataflow Native
[noreply] [BEAM-10844] Add experiment option prebuild_sdk_container to prebuild
[Robin Qiu] Moving to 2.26.0-SNAPSHOT on master branch.
[noreply] [BEAM-9616] Consolidate Element and DoFn json impl (#12925)
[noreply] [BEAM-9154] Ensure Chicago Taxi Example is disabled on Jenkins
(#12929)
[noreply] Add to_pcollection example to wordcount_dataframe (#12923)
------------------------------------------
[...truncated 62.78 KB...]
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py",
line 807, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py",
line 879, in _export_files
include_header=False)
File "/usr/local/lib/python3.7/site-packages/apache_beam/utils/retry.py",
line 236, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 787, in perform_extract_job
response = self.client.jobs.Insert(request)
File
"/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpBadRequestError: HttpError accessing
<https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/jobs?alt=json>:
response: <{'vary': 'Origin, X-Origin, Referer', 'content-type':
'application/json; charset=UTF-8', 'date': 'Thu, 24 Sep 2020 16:06:06 GMT',
'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff',
'transfer-encoding': 'chunked', 'status': '400', 'content-length': '267',
'-content-encoding': 'gzip'}>, content <{
"error": {
"code": 400,
"message": "Required parameter is missing",
"errors": [
{
"message": "Required parameter is missing",
"domain": "global",
"reason": "required"
}
],
"status": "INVALID_ARGUMENT"
}
}
>
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:06:10.718Z:
JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dataflow_****/batch****.py",
line 649, in do_work
work_executor.execute()
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
226, in execute
self._split_task)
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
234, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
271, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py",
line 807, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py",
line 879, in _export_files
include_header=False)
File "/usr/local/lib/python3.7/site-packages/apache_beam/utils/retry.py",
line 236, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 787, in perform_extract_job
response = self.client.jobs.Insert(request)
File
"/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpBadRequestError: HttpError accessing
<https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/jobs?alt=json>:
response: <{'vary': 'Origin, X-Origin, Referer', 'content-type':
'application/json; charset=UTF-8', 'date': 'Thu, 24 Sep 2020 16:06:08 GMT',
'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff',
'transfer-encoding': 'chunked', 'status': '400', 'content-length': '267',
'-content-encoding': 'gzip'}>, content <{
"error": {
"code": 400,
"message": "Required parameter is missing",
"errors": [
{
"message": "Required parameter is missing",
"domain": "global",
"reason": "required"
}
],
"status": "INVALID_ARGUMENT"
}
}
>
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:06:11.557Z:
JOB_MESSAGE_BASIC: Finished operation
assert_that/Create/Read+assert_that/Group/pair_with_0+assert_that/Group/GroupByKey/Reify+assert_that/Group/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:06:13.865Z:
JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dataflow_****/batch****.py",
line 649, in do_work
work_executor.execute()
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
226, in execute
self._split_task)
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
234, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
271, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py",
line 807, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py",
line 879, in _export_files
include_header=False)
File "/usr/local/lib/python3.7/site-packages/apache_beam/utils/retry.py",
line 236, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 787, in perform_extract_job
response = self.client.jobs.Insert(request)
File
"/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpBadRequestError: HttpError accessing
<https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/jobs?alt=json>:
response: <{'vary': 'Origin, X-Origin, Referer', 'content-type':
'application/json; charset=UTF-8', 'date': 'Thu, 24 Sep 2020 16:06:11 GMT',
'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff',
'transfer-encoding': 'chunked', 'status': '400', 'content-length': '267',
'-content-encoding': 'gzip'}>, content <{
"error": {
"code": 400,
"message": "Required parameter is missing",
"errors": [
{
"message": "Required parameter is missing",
"domain": "global",
"reason": "required"
}
],
"status": "INVALID_ARGUMENT"
}
}
>
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:06:16.030Z:
JOB_MESSAGE_ERROR: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dataflow_****/batch****.py",
line 649, in do_work
work_executor.execute()
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
226, in execute
self._split_task)
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
234, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
271, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py",
line 807, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py",
line 879, in _export_files
include_header=False)
File "/usr/local/lib/python3.7/site-packages/apache_beam/utils/retry.py",
line 236, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 787, in perform_extract_job
response = self.client.jobs.Insert(request)
File
"/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpBadRequestError: HttpError accessing
<https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/jobs?alt=json>:
response: <{'vary': 'Origin, X-Origin, Referer', 'content-type':
'application/json; charset=UTF-8', 'date': 'Thu, 24 Sep 2020 16:06:13 GMT',
'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff',
'transfer-encoding': 'chunked', 'status': '400', 'content-length': '267',
'-content-encoding': 'gzip'}>, content <{
"error": {
"code": 400,
"message": "Required parameter is missing",
"errors": [
{
"message": "Required parameter is missing",
"domain": "global",
"reason": "required"
}
],
"status": "INVALID_ARGUMENT"
}
}
>
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:06:16.295Z:
JOB_MESSAGE_BASIC: Finished operation Read from
BigQuery/ReadFromBigQuery/Read+Read from
BigQuery/ReadFromBigQuery/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+Count
messages+Measure
time+Count/CombineGlobally(CountCombineFn)/KeyWithVoid+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Reify+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:06:16.375Z:
JOB_MESSAGE_DEBUG: Executing failure step failure41
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:06:16.409Z:
JOB_MESSAGE_ERROR: Workflow failed. Causes: S02:Read from
BigQuery/ReadFromBigQuery/Read+Read from
BigQuery/ReadFromBigQuery/_PassThroughThenCleanup/ParDo(PassThrough)/ParDo(PassThrough)+Count
messages+Measure
time+Count/CombineGlobally(CountCombineFn)/KeyWithVoid+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey+Count/CombineGlobally(CountCombineFn)/CombinePerKey/Combine/Partial+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Reify+Count/CombineGlobally(CountCombineFn)/CombinePerKey/GroupByKey/Write
failed., Internal Issue (b69c7ce2745aa77): 63963027:24514
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:06:16.475Z:
JOB_MESSAGE_DETAILED: Cleaning up.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:06:16.542Z:
JOB_MESSAGE_DEBUG: Starting **** pool teardown.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:06:16.584Z:
JOB_MESSAGE_BASIC: Stopping **** pool...
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:07:09.943Z:
JOB_MESSAGE_DETAILED: Autoscaling: Resized **** pool from 5 to 0.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:07:09.993Z:
JOB_MESSAGE_BASIC: Worker pool stopped.
INFO:apache_beam.runners.dataflow.dataflow_runner:2020-09-24T16:07:10.026Z:
JOB_MESSAGE_DEBUG: Tearing down pending resources...
INFO:apache_beam.runners.dataflow.dataflow_runner:Job
2020-09-24_09_00_44-9936382243129295363 is in state JOB_STATE_FAILED
Traceback (most recent call last):
File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
exec(code, run_globals)
File
"<https://ci-beam.apache.org/job/beam_PerformanceTests_BiqQueryIO_Read_Python/ws/src/sdks/python/apache_beam/io/gcp/bigquery_read_perf_test.py";,>
line 139, in <module>
BigQueryReadPerfTest().run()
File
"<https://ci-beam.apache.org/job/beam_PerformanceTests_BiqQueryIO_Read_Python/ws/src/sdks/python/apache_beam/testing/load_tests/load_test.py";,>
line 152, in run
self.result.wait_until_finish(duration=self.timeout_ms)
File
"<https://ci-beam.apache.org/job/beam_PerformanceTests_BiqQueryIO_Read_Python/ws/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py";,>
line 1650, in wait_until_finish
self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow
pipeline failed. State: FAILED, Error:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/dataflow_****/batch****.py",
line 649, in do_work
work_executor.execute()
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
226, in execute
self._split_task)
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
234, in _perform_source_split_considering_api_limits
desired_bundle_size)
File "/usr/local/lib/python3.7/site-packages/dataflow_****/executor.py", line
271, in _perform_source_split
for split in source.split(desired_bundle_size):
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py",
line 807, in split
schema, metadata_list = self._export_files(bq)
File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery.py",
line 879, in _export_files
include_header=False)
File "/usr/local/lib/python3.7/site-packages/apache_beam/utils/retry.py",
line 236, in wrapper
return fun(*args, **kwargs)
File
"/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/bigquery_tools.py",
line 787, in perform_extract_job
response = self.client.jobs.Insert(request)
File
"/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py",
line 346, in Insert
upload=upload, upload_config=upload_config)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 731, in _RunMethod
return self.ProcessHttpResponse(method_config, http_response, request)
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 737, in ProcessHttpResponse
self.__ProcessHttpResponse(method_config, http_response, request))
File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py",
line 604, in __ProcessHttpResponse
http_response, method_config=method_config, request=request)
apitools.base.py.exceptions.HttpBadRequestError: HttpError accessing
<https://bigquery.googleapis.com/bigquery/v2/projects/apache-beam-testing/jobs?alt=json>:
response: <{'vary': 'Origin, X-Origin, Referer', 'content-type':
'application/json; charset=UTF-8', 'date': 'Thu, 24 Sep 2020 16:06:13 GMT',
'server': 'ESF', 'cache-control': 'private', 'x-xss-protection': '0',
'x-frame-options': 'SAMEORIGIN', 'x-content-type-options': 'nosniff',
'transfer-encoding': 'chunked', 'status': '400', 'content-length': '267',
'-content-encoding': 'gzip'}>, content <{
"error": {
"code": 400,
"message": "Required parameter is missing",
"errors": [
{
"message": "Required parameter is missing",
"domain": "global",
"reason": "required"
}
],
"status": "INVALID_ARGUMENT"
}
}
>
> Task :sdks:python:apache_beam:testing:load_tests:run FAILED
FAILURE: Build failed with an exception.
* Where:
Build file
'<https://ci-beam.apache.org/job/beam_PerformanceTests_BiqQueryIO_Read_Python/ws/src/sdks/python/apache_beam/testing/load_tests/build.gradle'>
line: 58
* What went wrong:
Execution failed for task ':sdks:python:apache_beam:testing:load_tests:run'.
> error occurred
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug
option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with
Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See
https://docs.gradle.org/6.6.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 8m 13s
5 actionable tasks: 5 executed
Publishing build scan...
https://gradle.com/s/sy4btkaemc5oo
Build step 'Invoke Gradle script' changed build result to FAILURE
Build step 'Invoke Gradle script' marked build as failure
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
