[
https://issues.apache.org/jira/browse/BEAM-7437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16853808#comment-16853808
]
Tanay Tummalapalli edited comment on BEAM-7437 at 6/1/19 8:08 PM:
------------------------------------------------------------------
I am getting consistent failures on the Integration Tests for BigQuery with the
Dataflow runner, for the classes `BigQueryFileLoadsIT` and
`BigQueryStreamingInsertTransformIntegrationTests` which seem to be related to
recent b64 changes. The traceback is:
{code:java}
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 649, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 150, in execute
test_shuffle_sink=self._test_shuffle_sink)
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 116, in create_operation
is_streaming=False)
File "apache_beam/runners/worker/operations.py", line 962, in
apache_beam.runners.worker.operations.create_operation
op = BatchGroupAlsoByWindowsOperation(
File "dataflow_worker/shuffle_operations.py", line 219, in
dataflow_worker.shuffle_operations.BatchGroupAlsoByWindowsOperation.__init__
self.windowing = deserialize_windowing_strategy(self.spec.window_fn)
File "dataflow_worker/shuffle_operations.py", line 207, in
dataflow_worker.shuffle_operations.deserialize_windowing_strategy
return pickler.loads(serialized_data)
File
"/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line
248, in loads
c = base64.b64decode(encoded)
File "/usr/lib/python2.7/base64.py", line 78, in b64decode
raise TypeError(msg)
TypeError: Incorrect padding
{code}
The failure is ONLY on the master branch. I tested the 2.13.0RC2 branch as
well. It passed.
I need to investigate this further.
was (Author: ttanay):
I am getting consistent failures on the Integration Tests for BigQuery with the
Dataflow runner, for the classes `BigQueryFileLoadsIT` and
`BigQueryStreamingInsertTransformIntegrationTests` which seem to be related to
recent b64 changes. The traceback is:
```
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py",
line 649, in do_work
work_executor.execute()
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 150, in execute
test_shuffle_sink=self._test_shuffle_sink)
File "/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py",
line 116, in create_operation
is_streaming=False)
File "apache_beam/runners/worker/operations.py", line 962, in
apache_beam.runners.worker.operations.create_operation
op = BatchGroupAlsoByWindowsOperation(
File "dataflow_worker/shuffle_operations.py", line 219, in
dataflow_worker.shuffle_operations.BatchGroupAlsoByWindowsOperation.__init__
self.windowing = deserialize_windowing_strategy(self.spec.window_fn)
File "dataflow_worker/shuffle_operations.py", line 207, in
dataflow_worker.shuffle_operations.deserialize_windowing_strategy
return pickler.loads(serialized_data)
File
"/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler.py", line
248, in loads
c = base64.b64decode(encoded)
File "/usr/lib/python2.7/base64.py", line 78, in b64decode
raise TypeError(msg)
TypeError: Incorrect padding
```
The failure is ONLY on the master branch. I tested the 2.13.0RC2 branch as
well. It passed.
I need to investigate this further.
> Integration Test for BQ streaming inserts for streaming pipelines
> -----------------------------------------------------------------
>
> Key: BEAM-7437
> URL: https://issues.apache.org/jira/browse/BEAM-7437
> Project: Beam
> Issue Type: Test
> Components: io-python-gcp
> Affects Versions: 2.12.0
> Reporter: Tanay Tummalapalli
> Assignee: Tanay Tummalapalli
> Priority: Minor
>
> Integration Test for BigQuery Sink using Streaming Inserts for streaming
> pipelines.
> Integration tests currently exist for batch pipelines, it can also be added
> for streaming pipelines using TestStream. This will be a precursor to the
> failing integration test to be added for [BEAM-6611|
> https://issues.apache.org/jira/browse/BEAM-6611].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
