quentin-sommer opened a new issue, #34270:
URL: https://github.com/apache/beam/issues/34270
### What happened?
Since ~2 weeks my streaming job started getting this error regularly. It
happened again today with latest beam (2.63.0) running on GCP Dataflow.
It makes the job retry the failed unit of work forever until it gives up and
halt all processing, forcing me to re-deploy
My WriteToBigQuery config
```py
WriteToBigQuery(table=self.table.table_id,
project=self.table.project,
dataset=self.table.dataset_id,
insert_retry_strategy=RetryStrategy.RETRY_ON_TRANSIENT_ERROR,
write_disposition=BigQueryDisposition.WRITE_APPEND,
create_disposition=BigQueryDisposition.CREATE_NEVER,
method=WriteToBigQuery.Method.STREAMING_INSERTS,
batch_size=1900,
triggering_frequency=10,
with_auto_sharding=True,
).with_output_types(WriteResult)
```
Stacktrace:
```
Error message from worker: generic::unknown: urllib3.exceptions.SSLError:
EOF occurred in violation of protocol (_ssl.c:2427)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\",
line 667, in send
resp = conn.urlopen(
^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py\", line
841, in urlopen
retries = retries.increment(
^^^^^^^^^^^^^^^^^^
File \"/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py\",
line 519, in increment
raise MaxRetryError(_pool, url, reason) from reason # type:
ignore[arg-type]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
urllib3.exceptions.MaxRetryError:
HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries
exceeded with url: /bigquery/v2/projects/REDACTED/insertAll?prettyPrint=false
(Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol
(_ssl.c:2427)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
\"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py\",
line 144, in retry_target
result = target()
^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/cloud/_http/__init__.py\",
line 482, in api_request
response = self._make_request(
^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/cloud/_http/__init__.py\",
line 341, in _make_request
return self._do_request(
^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/cloud/_http/__init__.py\",
line 379, in _do_request
return self.http.request(
^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/auth/transport/requests.py\",
line 537, in request
response = super(AuthorizedSession, self).request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\",
line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/usr/local/lib/python3.11/site-packages/requests/sessions.py\",
line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File \"/usr/local/lib/python3.11/site-packages/requests/adapters.py\",
line 698, in send
raise SSLError(e, request=request)
requests.exceptions.SSLError:
HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max retries
exceeded with url: /bigquery/v2/projects/REDACTED/insertAll?prettyPrint=false
(Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol
(_ssl.c:2427)')))
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File \"apache_beam/runners/common.py\", line 1501, in
apache_beam.runners.common.DoFnRunner.process
File \"apache_beam/runners/common.py\", line 917, in
apache_beam.runners.common.PerWindowInvoker.invoke_process
File \"apache_beam/runners/common.py\", line 1061, in
apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery.py\",
line 1640, in process
return self._flush_batch(destination)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery.py\",
line 1683, in _flush_batch
passed, errors = self.bigquery_wrapper.insert_rows(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py\",
line 1293, in insert_rows
result, errors = self._insert_all_rows(
^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/utils/retry.py\", line
311, in wrapper
raise exn.with_traceback(exn_traceback)
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/utils/retry.py\", line
298, in wrapper
return fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py\",
line 744, in _insert_all_rows
errors = self.gcp_bq_client.insert_rows_json(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/cloud/bigquery/client.py\",
line 3889, in insert_rows_json
response = self._call_api(
^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/cloud/bigquery/client.py\",
line 837, in _call_api
return call()
^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py\",
line 293, in retry_wrapped_func
return retry_target(
^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py\",
line 153, in retry_target
_retry_error_helper(
File
\"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_base.py\",
line 221, in _retry_error_helper
raise final_exc from source_exc
google.api_core.exceptions.RetryError: Timeout of 600.0s exceeded, last
exception: HTTPSConnectionPool(host='bigquery.googleapis.com', port=443): Max
retries exceeded with url:
/bigquery/v2/projects/REDACTED/insertAll?prettyPrint=false (Caused by
SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:2427)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py\",
line 313, in _execute
response = task()
^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py\",
line 388, in <lambda>
lambda: self.create_worker().do_instruction(request), request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py\",
line 658, in do_instruction
return getattr(self, request_type)(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/sdk_worker.py\",
line 696, in process_bundle
bundle_processor.process_bundle(instruction_id))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/runners/worker/bundle_processor.py\",
line 1271, in process_bundle
self.ops[element.transform_id].process_timer(
File \"apache_beam/runners/worker/operations.py\", line 974, in
apache_beam.runners.worker.operations.DoOperation.process_timer
File \"apache_beam/runners/common.py\", line 1553, in
apache_beam.runners.common.DoFnRunner.process_user_timer
File \"apache_beam/runners/common.py\", line 1591, in
apache_beam.runners.common.DoFnRunner._reraise_augmented
File \"apache_beam/runners/common.py\", line 1550, in
apache_beam.runners.common.DoFnRunner.process_user_timer
File \"apache_beam/runners/common.py\", line 645, in
apache_beam.runners.common.DoFnInvoker.invoke_user_timer
File \"apache_beam/runners/common.py\", line 1686, in
apache_beam.runners.common._OutputHandler.handle_process_outputs
File \"apache_beam/runners/common.py\", line 1799, in
apache_beam.runners.common._OutputHandler._write_value_to_tag
File \"apache_beam/runners/worker/operations.py\", line 263, in
apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File \"apache_beam/runners/worker/operations.py\", line 950, in
apache_beam.runners.worker.operations.DoOperation.process
File \"apache_beam/runners/worker/operations.py\", line 951, in
apache_beam.runners.worker.operations.DoOperation.process
File \"apache_beam/runners/common.py\", line 1503, in
apache_beam.runners.common.DoFnRunner.process
File \"apache_beam/runners/common.py\", line 1591, in
apache_beam.runners.common.DoFnRunner._reraise_augmented
File \"apache_beam/runners/common.py\", line 1501, in
apache_beam.runners.common.DoFnRunner.process
File \"apache_beam/runners/common.py\", line 689, in
apache_beam.runners.common.SimpleInvoker.invoke_process
File \"apache_beam/runners/common.py\", line 1686, in
apache_beam.runners.common._OutputHandler.handle_process_outputs
File \"apache_beam/runners/common.py\", line 1799, in
apache_beam.runners.common._OutputHandler._write_value_to_tag
File \"apache_beam/runners/worker/operations.py\", line 263, in
apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File \"apache_beam/runners/worker/operations.py\", line 950, in
apache_beam.runners.worker.operations.DoOperation.process
File \"apache_beam/runners/worker/operations.py\", line 951, in
apache_beam.runners.worker.operations.DoOperation.process
File \"apache_beam/runners/common.py\", line 1503, in
apache_beam.runners.common.DoFnRunner.process
File \"apache_beam/runners/common.py\", line 1591, in
apache_beam.runners.common.DoFnRunner._reraise_augmented
File \"apache_beam/runners/common.py\", line 1501, in
apache_beam.runners.common.DoFnRunner.process
File \"apache_beam/runners/common.py\", line 689, in
apache_beam.runners.common.SimpleInvoker.invoke_process
File \"apache_beam/runners/common.py\", line 1686, in
apache_beam.runners.common._OutputHandler.handle_process_outputs
File \"apache_beam/runners/common.py\", line 1799, in
apache_beam.runners.common._OutputHandler._write_value_to_tag
File \"apache_beam/runners/worker/operations.py\", line 263, in
apache_beam.runners.worker.operations.SingletonElementConsumerSet.receive
File \"apache_beam/runners/worker/operations.py\", line 950, in
apache_beam.runners.worker.operations.DoOperation.process
File \"apache_beam/runners/worker/operations.py\", line 951, in
apache_beam.runners.worker.operations.DoOperation.process
File \"apache_beam/runners/common.py\", line 1503, in
apache_beam.runners.common.DoFnRunner.process
File \"apache_beam/runners/common.py\", line 1612, in
apache_beam.runners.common.DoFnRunner._reraise_augmented
File \"apache_beam/runners/common.py\", line 1501, in
apache_beam.runners.common.DoFnRunner.process
File \"apache_beam/runners/common.py\", line 917, in
apache_beam.runners.common.PerWindowInvoker.invoke_process
File \"apache_beam/runners/common.py\", line 1061, in
apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery.py\",
line 1640, in process
return self._flush_batch(destination)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery.py\",
line 1683, in _flush_batch
passed, errors = self.bigquery_wrapper.insert_rows(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py\",
line 1293, in insert_rows
result, errors = self._insert_all_rows(
^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/utils/retry.py\", line
311, in wrapper
raise exn.with_traceback(exn_traceback)
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/utils/retry.py\", line
298, in wrapper
return fun(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/apache_beam/io/gcp/bigquery_tools.py\",
line 744, in _insert_all_rows
errors = self.gcp_bq_client.insert_rows_json(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/cloud/bigquery/client.py\",
line 3889, in insert_rows_json
response = self._call_api(
^^^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/cloud/bigquery/client.py\",
line 837, in _call_api
return call()
^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py\",
line 293, in retry_wrapped_func
return retry_target(
^^^^^^^^^^^^^
File
\"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_unary.py\",
line 153, in retry_target
_retry_error_helper(
File
\"/usr/local/lib/python3.11/site-packages/google/api_core/retry/retry_base.py\",
line 221, in _retry_error_helper
raise final_exc from source_exc
RuntimeError: google.api_core.exceptions.RetryError: Timeout of 600.0s
exceeded, last exception: HTTPSConnectionPool(host='bigquery.googleapis.com',
port=443): Max retries exceeded with url:
/bigquery/v2/projects/REDACTED/insertAll?prettyPrint=false (Caused by
SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol
(_ssl.c:2427)'))) [while running 'sessions to
bq/WriteToBQ/_StreamToBigQuery/StreamInsertRows/ParDo(BigQueryWriteFn)-ptransform-82']
passed through:
==>
dist_proc/dax/workflow/worker/fnapi_service_impl.cc:1331
```
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [x] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
