[
https://issues.apache.org/jira/browse/BEAM-11830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17286218#comment-17286218
]
Alex Amato edited comment on BEAM-11830 at 2/18/21, 9:41 PM:
-------------------------------------------------------------
Test history for particular failure:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 0:
invalid continuation byte
[https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/testReport/junit/apache_beam.transforms.validate_runner_xlang_test/ValidateRunnerXlangTest/test_prefix/history/]
Not immediately clear if any of these commits would cause this...
[https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/changes]
The only python change was this rollback
[https://github.com/apache/beam/commit/9c60fd546458f48912285d38bce52a31d2eb4472
]
this change there is some calls to encode objects with utf-8. If its being
encoded incorrectly, or that's the thing being decoded later on, perhaps that
could cause this decode error? But its not obvious to me this is what is
happening here
[~robertwb] Do you that this rollback may have led to this?
(Other theory: Could also be related to changes in the data ingested by the
pipeline, if that was modified. No evidence found for this so far)
was (Author: [email protected]):
Test history for particular failure:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 0:
invalid continuation byte
[https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/testReport/junit/apache_beam.transforms.validate_runner_xlang_test/ValidateRunnerXlangTest/test_prefix/history/]
Not immediately clear if any of these commits would cause this...
[https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/changes]
The only python change was this rollback
https://github.com/apache/beam/commit/9c60fd546458f48912285d38bce52a31d2eb4472
Could also be related to changes in the data ingested by the pipeline, if that
was modified.
> ValidateRunnerXlangTest tsts failing in beam_PostCommit_XVR_Dataflow
> --------------------------------------------------------------------
>
> Key: BEAM-11830
> URL: https://issues.apache.org/jira/browse/BEAM-11830
> Project: Beam
> Issue Type: Test
> Components: test-failures
> Reporter: Alex Amato
> Priority: P2
>
> [https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/#showFailuresLink]
> [Test
> Result|https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/testReport/]
> (8 failures / +6)
> *
> [apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest.test_partition|https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/testReport/junit/apache_beam.transforms.validate_runner_xlang_test/ValidateRunnerXlangTest/test_partition/]
> *
> [apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest.test_combine_per_key|https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/testReport/junit/apache_beam.transforms.validate_runner_xlang_test/ValidateRunnerXlangTest/test_combine_per_key/]
> *
> [apache_beam.io.external.generate_sequence_test.XlangGenerateSequenceTest.test_generate_sequence|https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/testReport/junit/apache_beam.io.external.generate_sequence_test/XlangGenerateSequenceTest/test_generate_sequence/]
> *
> [apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest.test_prefix|https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/testReport/junit/apache_beam.transforms.validate_runner_xlang_test/ValidateRunnerXlangTest/test_prefix/]
> *
> [apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest.test_group_by_key|https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/testReport/junit/apache_beam.transforms.validate_runner_xlang_test/ValidateRunnerXlangTest/test_group_by_key/]
> *
> [apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest.test_cogroup_by_key|https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/testReport/junit/apache_beam.transforms.validate_runner_xlang_test/ValidateRunnerXlangTest/test_cogroup_by_key/]
>
> A few tests such as
> (apache_beam.transforms.validate_runner_xlang_test.ValidateRunnerXlangTest.test_combine_per_key)
> fail with this error:
> Dataflow pipeline failed. State: FAILED, Error: Traceback (most recent call
> last): File
> "/usr/local/lib/python3.6/site-packages/dataflow_worker/batchworker.py", line
> 649, in do_work work_executor.execute() File
> "/usr/local/lib/python3.6/site-packages/dataflow_worker/executor.py", line
> 179, in execute op.start() File "dataflow_worker/native_operations.py", line
> 38, in dataflow_worker.native_operations.NativeReadOperation.start File
> "dataflow_worker/native_operations.py", line 39, in
> dataflow_worker.native_operations.NativeReadOperation.start File
> "dataflow_worker/native_operations.py", line 44, in
> dataflow_worker.native_operations.NativeReadOperation.start File
> "dataflow_worker/native_operations.py", line 48, in
> dataflow_worker.native_operations.NativeReadOperation.start File
> "/usr/local/lib/python3.6/site-packages/dataflow_worker/inmemory.py", line
> 108, in __iter__ yield self._source.coder.decode(value) File
> "/usr/local/lib/python3.6/site-packages/apache_beam/coders/coders.py", line
> 456, in decode return self.get_impl().decode(encoded) File
> "apache_beam/coders/coder_impl.py", line 226, in
> apache_beam.coders.coder_impl.StreamCoderImpl.decode File
> "apache_beam/coders/coder_impl.py", line 228, in
> apache_beam.coders.coder_impl.StreamCoderImpl.decode File
> "apache_beam/coders/coder_impl.py", line 862, in
> apache_beam.coders.coder_impl.AbstractComponentCoderImpl.decode_from_stream
> File "apache_beam/coders/coder_impl.py", line 261, in
> apache_beam.coders.coder_impl.CallbackCoderImpl.decode_from_stream File
> "/usr/local/lib/python3.6/site-packages/apache_beam/coders/coders.py", line
> 413, in decode return value.decode('utf-8') UnicodeDecodeError: 'utf-8' codec
> can't decode byte 0xdf in position 0: invalid continuation byte
>
> First fails at #389
> https://ci-beam.apache.org/job/beam_PostCommit_XVR_Dataflow/389/changes
>
> =========
> apache_beam.io.external.generate_sequence_test.XlangGenerateSequenceTest.test_generate_sequence
> fails with a different error
> Dataflow pipeline failed. State: FAILED, Error: Traceback (most recent call
> last): File "apache_beam/runners/common.py", line 1239, in
> apache_beam.runners.common.DoFnRunner.process File
> "apache_beam/runners/common.py", line 768, in
> apache_beam.runners.common.PerWindowInvoker.invoke_process File
> "apache_beam/runners/common.py", line 893, in
> apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Dataflow/src/sdks/python/apache_beam/transforms/core.py",
> line 1590, in <lambda> wrapper = lambda x, *args, **kwargs: [fn(x, *args,
> **kwargs)] File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_XVR_Dataflow/src/sdks/python/apache_beam/testing/util.py",
> line 197, in _equal raise BeamAssertException(msg)
> apache_beam.testing.util.BeamAssertException: Failed assert: [1, 2, 3, 4, 5,
> 6, 7, 8, 9] == [127], unexpected elements [127], missing elements [1, 2, 3,
> 4, 5, 6, 7, 8, 9]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
