[ https://issues.apache.org/jira/browse/BEAM-6522?focusedWorklogId=387764&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-387764 ]
ASF GitHub Bot logged work on BEAM-6522: ---------------------------------------- Author: ASF GitHub Bot Created on: 15/Feb/20 00:10 Start Date: 15/Feb/20 00:10 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10876: [BEAM-6522] Remove the warnings associated with BEAM-6522. URL: https://github.com/apache/beam/pull/10876 Post-Commit Tests Status (on master branch) ------------------------------------------------------------------------------------------------ Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/) XLang | --- | --- | --- | [![Build Status](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | --- Pre-Commit Tests Status (on master branch) ------------------------------------------------------------------------------------------------ --- |Java | Python | Go | Website --- | --- | --- | --- | --- Non-portable | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[![Build Status](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/) Portable | --- | [![Build Status](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/badge/icon)](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | --- See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 387764) Time Spent: 10h 40m (was: 10.5h) > Dill fails to pickle avro.RecordSchema classes on Python 3. > ------------------------------------------------------------ > > Key: BEAM-6522 > URL: https://issues.apache.org/jira/browse/BEAM-6522 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core > Reporter: Robbe > Assignee: Valentyn Tymofieiev > Priority: Major > Fix For: 2.16.0 > > Time Spent: 10h 40m > Remaining Estimate: 0h > > The avroio module still has 4 failing tests. This is actually 2 times the > same 2 tests, both for Avro and Fastavro. > *apache_beam.io.avroio_test.TestAvro.test_sink_transform* > *apache_beam.io.avroio_test.TestFastAvro.test_sink_transform* > fail with: > {code:java} > Traceback (most recent call last): > File "/home/robbe/workspace/beam/sdks/python/apache_beam/io/avroio_test.py", > line 432, in test_sink_transform > | avroio.WriteToAvro(path, self.SCHEMA, use_fastavro=self.use_fastavro) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/pvalue.py", line > 112, in __or__ > return self.pipeline.apply(ptransform, self) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/pipeline.py", line > 515, in apply > pvalueish_result = self.runner.apply(transform, pvalueish, self._options) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/runners/runner.py", > line 193, in apply > return m(transform, input, options) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/runners/runner.py", > line 199, in apply_PTransform > return transform.expand(input) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/io/avroio.py", line > 528, in expand > return pcoll | beam.io.iobase.Write(self._sink) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/pvalue.py", line > 112, in __or__ > return self.pipeline.apply(ptransform, self) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/pipeline.py", line > 515, in apply > pvalueish_result = self.runner.apply(transform, pvalueish, self._options) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/runners/runner.py", > line 193, in apply > return m(transform, input, options) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/runners/runner.py", > line 199, in apply_PTransform > return transform.expand(input) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/io/iobase.py", line > 960, in expand > return pcoll | WriteImpl(self.sink) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/pvalue.py", line > 112, in __or__ > return self.pipeline.apply(ptransform, self) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/pipeline.py", line > 515, in apply > pvalueish_result = self.runner.apply(transform, pvalueish, self._options) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/runners/runner.py", > line 193, in apply > return m(transform, input, options) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/runners/runner.py", > line 199, in apply_PTransform > return transform.expand(input) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/io/iobase.py", line > 979, in expand > lambda _, sink: sink.initialize_write(), self.sink) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/transforms/core.py", > line 1103, in Map > pardo = FlatMap(wrapper, *args, **kwargs) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/transforms/core.py", > line 1054, in FlatMap > pardo = ParDo(CallableWrapperDoFn(fn), *args, **kwargs) > File "/home/robbe/workspace/beam/sdks/python/apache_beam/transforms/core.py", > line 864, in __init__ > super(ParDo, self).__init__(fn, *args, **kwargs) > File > "/home/robbe/workspace/beam/sdks/python/apache_beam/transforms/ptransform.py", > line 646, in __init__ > self.args = pickler.loads(pickler.dumps(self.args)) > File > "/home/robbe/workspace/beam/sdks/python/apache_beam/internal/pickler.py", > line 247, in loads > return dill.loads(s) > File > "/home/robbe/workspace/beam/sdks/python/.eggs/dill-0.2.9-py3.5.egg/dill/_dill.py", > line 317, in loads > return load(file, ignore) > File > "/home/robbe/workspace/beam/sdks/python/.eggs/dill-0.2.9-py3.5.egg/dill/_dill.py", > line 305, in load > obj = pik.load() > File > "/home/robbe/workspace/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/avro/schema.py", > line 173, in __setitem__ > % (key, value, self)) > Exception: Attempting to map key 'favorite_color' to value <avro.schema.Field > object at 0x7f8f72d0d0b8> in ImmutableDict {} > {code} > > *apache_beam.io.avroio_test.TestAvro.test_split_points* > *apache_beam.io.avroio_test.TestFastAvro.test_split_points* > fail with: > > {code:java} > Traceback (most recent call last): > File "/home/robbe/workspace/beam/sdks/python/apache_beam/io/avroio_test.py", > line 308, in test_split_points > self.assertEquals(split_points_report[-10:], [(2, 1)] * 10) > AssertionError: Lists differ: [(10, 1), (10, 1), (10, 1), (10, 1), (10, 1[42 > chars], 1)] != [(2, 1), (2, 1), (2, 1), (2, 1), (2, 1), (2[32 chars], 1)] > First differing element 0: > (10, 1) > (2, 1) > + [(2, 1), (2, 1), (2, 1), (2, 1), (2, 1), (2, 1), (2, 1), (2, 1), (2, 1), > (2, 1)] > - [(10, 1), > - (10, 1), > - (10, 1), > - (10, 1), > - (10, 1), > - (10, 1), > - (10, 1), > - (10, 1), > - (10, 1), > - (10, 1)] > {code} -- This message was sent by Atlassian Jira (v8.3.4#803005)