[jira] [Commented] (BEAM-5628) Several VcfIO tests fail in Python 3 with TypeError: cannot use a string pattern on a bytes-like object

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648695#comment-16648695
 ] 

Valentyn Tymofieiev commented on BEAM-5628:
---

cc [~arostami], one of VCF IO authors.

> Several VcfIO tests fail in Python 3 with  TypeError: cannot use a string 
> pattern on a bytes-like object
> 
>
> Key: BEAM-5628
> URL: https://issues.apache.org/jira/browse/BEAM-5628
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Simon
>Priority: Major
>
> ERROR: test_read_after_splitting (apache_beam.io.vcfio_test.VcfSourceTest)
> "
>  --
> Traceback (most recent call last):
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio_test.py"",
>  line 336, in test_read_after_splitting
> ] split_records.extend(source_test_utils.read_from_source(*source_info))
> ]   File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils.py"",
>  line 101, in read_from_source
>  for value in reader:
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio.py"",
>  line 264, in read_records
>  for line in record_iterator:
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio.py"",
>  line 330, in __next__
>  record = next(self._vcf_reader)
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/vcf/parser.py"",
>  line 543, in __next__
>  row = self._row_pattern.split(line.rstrip())
>  TypeError: cannot use a string pattern on a bytes-like object
> "



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5628) Several VcfIO tests fail in Python 3 with TypeError: cannot use a string pattern on a bytes-like object

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648694#comment-16648694
 ] 

Valentyn Tymofieiev edited comment on BEAM-5628 at 10/13/18 2:20 AM:
-

Looks like VCF IO is using another package (PyVCF). We need to understand 
whether the issue is because we don't use PyVCF correctly (VCF IO issue), or 
because PyVCF itself is not Python3-compatible as it claims to be.


was (Author: tvalentyn):
Looks like VCF IO is using another package (PyVCF). We need to understand 
whether the issue is because we don't use PyVCF correctly (VCF IO issue), or 
because PyVCF itself is not Python3-compatible.

> Several VcfIO tests fail in Python 3 with  TypeError: cannot use a string 
> pattern on a bytes-like object
> 
>
> Key: BEAM-5628
> URL: https://issues.apache.org/jira/browse/BEAM-5628
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Simon
>Priority: Major
>
> ERROR: test_read_after_splitting (apache_beam.io.vcfio_test.VcfSourceTest)
> "
>  --
> Traceback (most recent call last):
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio_test.py"",
>  line 336, in test_read_after_splitting
> ] split_records.extend(source_test_utils.read_from_source(*source_info))
> ]   File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils.py"",
>  line 101, in read_from_source
>  for value in reader:
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio.py"",
>  line 264, in read_records
>  for line in record_iterator:
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio.py"",
>  line 330, in __next__
>  record = next(self._vcf_reader)
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/vcf/parser.py"",
>  line 543, in __next__
>  row = self._row_pattern.split(line.rstrip())
>  TypeError: cannot use a string pattern on a bytes-like object
> "



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5628) Several VcfIO tests fail in Python 3 with TypeError: cannot use a string pattern on a bytes-like object

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648694#comment-16648694
 ] 

Valentyn Tymofieiev commented on BEAM-5628:
---

Looks like VCF IO is using another package (PyVCF). We need to understand 
whether the issue is because we don't use PyVCF correctly (VCF IO issue), or 
because PyVCF itself is not Python3-compatible.

> Several VcfIO tests fail in Python 3 with  TypeError: cannot use a string 
> pattern on a bytes-like object
> 
>
> Key: BEAM-5628
> URL: https://issues.apache.org/jira/browse/BEAM-5628
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Simon
>Priority: Major
>
> ERROR: test_read_after_splitting (apache_beam.io.vcfio_test.VcfSourceTest)
> "
>  --
> Traceback (most recent call last):
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio_test.py"",
>  line 336, in test_read_after_splitting
> ] split_records.extend(source_test_utils.read_from_source(*source_info))
> ]   File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils.py"",
>  line 101, in read_from_source
>  for value in reader:
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio.py"",
>  line 264, in read_records
>  for line in record_iterator:
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio.py"",
>  line 330, in __next__
>  record = next(self._vcf_reader)
>File 
> ""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/vcf/parser.py"",
>  line 543, in __next__
>  row = self._row_pattern.split(line.rstrip())
>  TypeError: cannot use a string pattern on a bytes-like object
> "



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5623) Several IO tests hang indefinitely during execution on Python 3.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648693#comment-16648693
 ] 

Valentyn Tymofieiev commented on BEAM-5623:
---

What exactly happens when the test is hanging?

> Several IO tests hang indefinitely during execution on Python 3.
> 
>
> Key: BEAM-5623
> URL: https://issues.apache.org/jira/browse/BEAM-5623
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> test_read_empty_single_file_no_eol_gzip 
> (apache_beam.io.textio_test.TextSourceTest) 
> Also several tests cases in tfrecordio_test, for example:
> test_process_auto (apache_beam.io.tfrecordio_test.TestReadAllFromTFRecord)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5744) Investigate negative numbers represented as 'long' in Python SDK + Direct runner

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648688#comment-16648688
 ] 

Valentyn Tymofieiev commented on BEAM-5744:
---

cc [~altay] [~charleschen] [~juta] [~robertwb]

> Investigate negative numbers represented as 'long' in Python SDK + Direct 
> runner
> 
>
> Key: BEAM-5744
> URL: https://issues.apache.org/jira/browse/BEAM-5744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.8.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5744) Investigate negative numbers represented as 'long' in Python SDK + Direct runner

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648681#comment-16648681
 ] 

Valentyn Tymofieiev commented on BEAM-5744:
---

https://github.com/apache/beam/pull/6685 illustrates the failure.

18:13:43 ==
18:13:43 ERROR: test_assert_that_passes_order_does_not_matter_with_negatives 
(apache_beam.testing.util_test.UtilTest)
18:13:43 --
18:13:43 Traceback (most recent call last):
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/testing/util_test.py",
 line 47, in test_assert_that_passes_order_does_not_matter_with_negatives
18:13:43 assert_that(p | Create([1, -2, 3]), equal_to([-2, 1, 3]))
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/pipeline.py",
 line 423, in __exit__
18:13:43 self.run().wait_until_finish()
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/testing/test_pipeline.py",
 line 104, in run
18:13:43 result = super(TestPipeline, self).run(test_runner_api)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/pipeline.py",
 line 403, in run
18:13:43 self.to_runner_api(), self.runner, self._options).run(False)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/pipeline.py",
 line 416, in run
18:13:43 return self.runner.run_pipeline(self)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/direct/direct_runner.py",
 line 138, in run_pipeline
18:13:43 return runner.run_pipeline(pipeline)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
 line 231, in run_pipeline
18:13:43 return self.run_via_runner_api(pipeline.to_runner_api())
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
 line 234, in run_via_runner_api
18:13:43 return self.run_stages(*self.create_stages(pipeline_proto))
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
 line 1017, in run_stages
18:13:43 pcoll_buffers, safe_coders)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
 line 1137, in run_stage
18:13:43 self._progress_frequency).process_bundle(data_input, data_output)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
 line 1393, in process_bundle
18:13:43 result_future = 
self._controller.control_handler.push(process_bundle)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
 line 1265, in push
18:13:43 response = self.worker.do_instruction(request)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/worker/sdk_worker.py",
 line 212, in do_instruction
18:13:43 request.instruction_id)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/worker/sdk_worker.py",
 line 234, in process_bundle
18:13:43 processor.process_bundle(instruction_id)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/worker/bundle_processor.py",
 line 420, in process_bundle
18:13:43 ].process_encoded(data.data)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/worker/bundle_processor.py",
 line 125, in process_encoded
18:13:43 self.output(decoded_value)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/worker/operations.py",
 line 169, in output
18:13:43 cython.cast(Receiver, 
self.receivers[output_index]).receive(windowed_value)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/worker/operations.py",
 line 89, in receive
18:13:43 cython.cast(Operation, consumer).process(windowed_value)
18:13:43   File 
"/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/apache_beam/runners/worker/operations.py",
 line 481, i

[jira] [Commented] (BEAM-5744) Investigate negative numbers represented as 'long' in Python SDK + Direct runner

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648659#comment-16648659
 ] 

Valentyn Tymofieiev commented on BEAM-5744:
---

It appears that sometimes negative numbers are sometimes represented as `long`, 
which I observed in 
https://github.com/apache/beam/pull/6602#issuecomment-429468709, opened this 
issue to investigate if this is a regression in 2.8.0, since such behavior 
would be incompatible with https://github.com/apache/beam/pull/6602 on Python 
2, and https://github.com/apache/beam/pull/6602 is included in 2.8.0 release.

> Investigate negative numbers represented as 'long' in Python SDK + Direct 
> runner
> 
>
> Key: BEAM-5744
> URL: https://issues.apache.org/jira/browse/BEAM-5744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.8.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5744) Investigate negative numbers represented as 'long' in Python SDK + Direct runner

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5744:
--
Summary: Investigate negative numbers represented as 'long' in Python SDK + 
Direct runner  (was: Investigate negative numbers investigated as 'long' in 
Python SDK + Direct runner)

> Investigate negative numbers represented as 'long' in Python SDK + Direct 
> runner
> 
>
> Key: BEAM-5744
> URL: https://issues.apache.org/jira/browse/BEAM-5744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.8.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5744) Investigate negative numbers investigated as 'long' in Python SDK + Direct runner

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5744:
-

 Summary: Investigate negative numbers investigated as 'long' in 
Python SDK + Direct runner
 Key: BEAM-5744
 URL: https://issues.apache.org/jira/browse/BEAM-5744
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Ahmet Altay
 Fix For: 2.8.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5744) Investigate negative numbers investigated as 'long' in Python SDK + Direct runner

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-5744:
-

Assignee: Valentyn Tymofieiev  (was: Ahmet Altay)

> Investigate negative numbers investigated as 'long' in Python SDK + Direct 
> runner
> -
>
> Key: BEAM-5744
> URL: https://issues.apache.org/jira/browse/BEAM-5744
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.8.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5615) Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword argument for this function

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5615:
--
Affects Version/s: (was: 2.8.0)

> Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword 
> argument for this function
> -
>
> Key: BEAM-5615
> URL: https://issues.apache.org/jira/browse/BEAM-5615
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> ERROR: test_top (apache_beam.transforms.combiners_test.CombineTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners_test.py",
>  line 89, in test_top
> names)  # Note parameter passed to comparator.
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 467, in apply
> label or transform.label)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 477, in apply
> return self.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform.py",
>  line 759, in expand
> return self._fn(pcoll, *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 185, in Of
> TopCombineFn(n, compare, key, reverse), *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1251, in expand
> default_value = combine_fn.apply([], *self.args, **self.kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 623, in apply
> *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 362, in extract_output
> self._sort_buffer(buffer, lt)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 295, in _sort_buffer
> key=self._key_fn)
> TypeError: 'cmp' is an invalid keyword argument for this function



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-5615) Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword argument for this function

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reopened BEAM-5615:
---

> Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword 
> argument for this function
> -
>
> Key: BEAM-5615
> URL: https://issues.apache.org/jira/browse/BEAM-5615
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Affects Versions: 2.8.0
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> ERROR: test_top (apache_beam.transforms.combiners_test.CombineTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners_test.py",
>  line 89, in test_top
> names)  # Note parameter passed to comparator.
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 467, in apply
> label or transform.label)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 477, in apply
> return self.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform.py",
>  line 759, in expand
> return self._fn(pcoll, *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 185, in Of
> TopCombineFn(n, compare, key, reverse), *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1251, in expand
> default_value = combine_fn.apply([], *self.args, **self.kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 623, in apply
> *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 362, in extract_output
> self._sort_buffer(buffer, lt)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 295, in _sort_buffer
> key=self._key_fn)
> TypeError: 'cmp' is an invalid keyword argument for this function



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5615) Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword argument for this function

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5615:
--
Affects Version/s: 2.8.0

> Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword 
> argument for this function
> -
>
> Key: BEAM-5615
> URL: https://issues.apache.org/jira/browse/BEAM-5615
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Affects Versions: 2.8.0
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 3.5h
>  Remaining Estimate: 0h
>
> ERROR: test_top (apache_beam.transforms.combiners_test.CombineTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners_test.py",
>  line 89, in test_top
> names)  # Note parameter passed to comparator.
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 467, in apply
> label or transform.label)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 477, in apply
> return self.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform.py",
>  line 759, in expand
> return self._fn(pcoll, *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 185, in Of
> TopCombineFn(n, compare, key, reverse), *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1251, in expand
> default_value = combine_fn.apply([], *self.args, **self.kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 623, in apply
> *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 362, in extract_output
> self._sort_buffer(buffer, lt)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 295, in _sort_buffer
> key=self._key_fn)
> TypeError: 'cmp' is an invalid keyword argument for this function



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5720) Default coder breaks with large ints on Python 3

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5720:
--
Priority: Minor  (was: Major)

> Default coder breaks with large ints on Python 3
> 
>
> Key: BEAM-5720
> URL: https://issues.apache.org/jira/browse/BEAM-5720
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Minor
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> The test for `int` includes greater than 64-bit values, which causes an 
> overflow error later in the code. We need to only use that coding scheme for 
> machine-sized ints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5663) Add tox suites for various Python 3 versions

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648346#comment-16648346
 ] 

Valentyn Tymofieiev commented on BEAM-5663:
---

Lowered the priority to 'Minor' for now - I suggest reserving 'Major' 
priorities for issues that we must finish for Beam to be viable on Python 3.   

> Add tox suites for various Python 3 versions
> 
>
> Key: BEAM-5663
> URL: https://issues.apache.org/jira/browse/BEAM-5663
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Priority: Minor
>
> Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test 
> failings across various Python 3 versions. It will be valuable to add tox 
> suites for Python 3.4, 3.5, 3.6 and 3.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5612) Add tox suites to exercise unit tests using Python3 interpreter with cython, and with gcp dependencies.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5612:
--
Priority: Minor  (was: Major)

> Add tox suites to exercise unit tests using Python3 interpreter with cython, 
> and with gcp dependencies.
> ---
>
> Key: BEAM-5612
> URL: https://issues.apache.org/jira/browse/BEAM-5612
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Minor
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5663) Add tox suites for various Python 3 versions

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5663:
--
Priority: Minor  (was: Major)

> Add tox suites for various Python 3 versions
> 
>
> Key: BEAM-5663
> URL: https://issues.apache.org/jira/browse/BEAM-5663
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Priority: Minor
>
> Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test 
> failings across various Python 3 versions. It will be valuable to add tox 
> suites for Python 3.4, 3.5, 3.6 and 3.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5091) Exercise Python 3 SDK harness container in ValidatesContainer Jenkins test suite.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5091:
--
Priority: Major  (was: Minor)

> Exercise Python 3 SDK harness container in ValidatesContainer Jenkins test 
> suite.
> -
>
> Key: BEAM-5091
> URL: https://issues.apache.org/jira/browse/BEAM-5091
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5731) Disable compare parameter in Top.Of() combiner when executing in Python 3.

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5731:
-

 Summary: Disable compare parameter in Top.Of() combiner when 
executing in Python 3.
 Key: BEAM-5731
 URL: https://issues.apache.org/jira/browse/BEAM-5731
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Valentyn Tymofieiev


Removing `compare` parameter in Top.Of() will be a more consistent API with 
`sorted()` function in Python 3. We should also evaluate whether similar 
changes would be  reasonable in other places in Beam SDK.

See also: https://github.com/apache/beam/pull/6570#issuecomment-429176130.

cc [~altay] [~juta] [~robertwb]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5623) Several IO tests hang indefinitely during execution on Python 3.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16648308#comment-16648308
 ] 

Valentyn Tymofieiev commented on BEAM-5623:
---

FWIW, I noticed that this error happens on _gzip tests.

> Several IO tests hang indefinitely during execution on Python 3.
> 
>
> Key: BEAM-5623
> URL: https://issues.apache.org/jira/browse/BEAM-5623
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> test_read_empty_single_file_no_eol_gzip 
> (apache_beam.io.textio_test.TextSourceTest) 
> Also several tests cases in tfrecordio_test, for example:
> test_process_auto (apache_beam.io.tfrecordio_test.TestReadAllFromTFRecord)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5720) Default coder breaks with large ints on Python 3

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-5720:
-

Assignee: Robert Bradshaw  (was: Valentyn Tymofieiev)

> Default coder breaks with large ints on Python 3
> 
>
> Key: BEAM-5720
> URL: https://issues.apache.org/jira/browse/BEAM-5720
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Robert Bradshaw
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> The test for `int` includes greater than 64-bit values, which causes an 
> overflow error later in the code. We need to only use that coding scheme for 
> machine-sized ints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reopened BEAM-5627:
---

[~rakeshkumar] Feel free to close if this error is completely gone.

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647145#comment-16647145
 ] 

Valentyn Tymofieiev edited comment on BEAM-5627 at 10/11/18 10:57 PM:
--

[~rakeshkumar] Did you check if there are any other tests in IO module that 
still fail with this error?


was (Author: tvalentyn):
[~rakeshkumar] Did you check if there any other tests in IO module that fail 
with this error?

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647145#comment-16647145
 ] 

Valentyn Tymofieiev commented on BEAM-5627:
---

[~rakeshkumar] Did you check if there any other tests in IO module that fail 
with this error?

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 2h
>  Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5663) Add tox suites for various Python 3 versions

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647072#comment-16647072
 ] 

Valentyn Tymofieiev commented on BEAM-5663:
---

Thanks [~yifanzou]. I think we should find a way to parallelize the test 
execution, before adding more test suites, since slow precommits will impact 
developer productivity.

I heard from [~udim] that Bazel has better isolation of test environments and 
natively supports parallelizing test execution, on test suite and test level. 

> Add tox suites for various Python 3 versions
> 
>
> Key: BEAM-5663
> URL: https://issues.apache.org/jira/browse/BEAM-5663
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Priority: Major
>
> Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test 
> failings across various Python 3 versions. It will be valuable to add tox 
> suites for Python 3.4, 3.5, 3.6 and 3.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5722) Flake in Python integration test - Operation cancel not allowed for job. Job is not yet ready for canceling.

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5722:
-

 Summary: Flake in Python integration test - Operation cancel not 
allowed for job. Job is not yet ready for canceling.
 Key: BEAM-5722
 URL: https://issues.apache.org/jira/browse/BEAM-5722
 Project: Beam
  Issue Type: Bug
  Components: test-failures
Reporter: Valentyn Tymofieiev
Assignee: Anton Kedin


12:42:33 ==
12:42:33 ERROR: test_big_query_new_types 
(apache_beam.io.gcp.big_query_query_to_table_it_test.BigQueryQueryToTableIT)
12:42:33 --
12:42:33 Traceback (most recent call last):
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_it_test.py",
 line 167, in test_big_query_new_types
12:42:33 big_query_query_to_table_pipeline.run_bq_pipeline(options)
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/io/gcp/big_query_query_to_table_pipeline.py",
 line 67, in run_bq_pipeline
12:42:33 result = p.run()
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/testing/test_pipeline.py",
 line 104, in run
12:42:33 result = super(TestPipeline, self).run(test_runner_api)
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 416, in run
12:42:33 return self.runner.run_pipeline(self)
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
 line 68, in run_pipeline
12:42:33 self.result.cancel()
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1181, in cancel
12:42:33 self.job_id(), 'JOB_STATE_CANCELLED'):
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py",
 line 184, in wrapper
12:42:33 return fun(*args, **kwargs)
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
 line 600, in modify_job_state
12:42:33 self._client.projects_locations_jobs.Update(request)
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",
 line 682, in Update
12:42:33 config, request, global_params=global_params)
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
 line 722, in _RunMethod
12:42:33 return self.ProcessHttpResponse(method_config, http_response, 
request)
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
 line 728, in ProcessHttpResponse
12:42:33 self.__ProcessHttpResponse(method_config, http_response, request))
12:42:33   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
 line 599, in __ProcessHttpResponse
12:42:33 http_response, method_config=method_config, request=request)
12:42:33 HttpBadRequestError: HttpError accessing 
:
 response: <{'status': '400', 'content-length': '316', 'x-xss-protection': '1; 
mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding': 
'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', 
'-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Thu, 11 Oct 
2018 19:36:17 GMT', 'x-frame-options': 'SAMEORIGIN', 'content-type': 
'application/json; charset=UTF-8'}>, content <{
12:42:33   "error": {
12:42:33 "code": 400,
12:42:33 "message": "(f4d3cf3db38ed95d): Workflow modification failed. 
Causes: (5bf98356c427cb5d): Operation cancel not allowed for job 
2018-10-11_12_34_08-11945601871177836156. Job is not yet ready for canceling. 
Please retry in a few minutes.",
12:42:33 "status": "FAILED_PRECONDITION"
12:42:33   }
12:42:33 }
12:42:33 >



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5720) Default coder breaks with large ints on Python 3

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646961#comment-16646961
 ] 

Valentyn Tymofieiev commented on BEAM-5720:
---

Thank you. Which test are you referring to?
Also, in the future, let's file Python 3 issues as subtasks of BEAM-1251 to 
simplify tracking.

> Default coder breaks with large ints on Python 3
> 
>
> Key: BEAM-5720
> URL: https://issues.apache.org/jira/browse/BEAM-5720
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> The test for `int` includes greater than 64-bit values, which causes an 
> overflow error later in the code. We need to only use that coding scheme for 
> machine-sized ints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5720) Default coder breaks with large ints on Python 3

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5720:
--
Issue Type: Sub-task  (was: Improvement)
Parent: BEAM-1251

> Default coder breaks with large ints on Python 3
> 
>
> Key: BEAM-5720
> URL: https://issues.apache.org/jira/browse/BEAM-5720
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robert Bradshaw
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> The test for `int` includes greater than 64-bit values, which causes an 
> overflow error later in the code. We need to only use that coding scheme for 
> machine-sized ints. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5663) Add tox suites for various Python 3 versions

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16646940#comment-16646940
 ] 

Valentyn Tymofieiev commented on BEAM-5663:
---

Hey [~alanmyrvold], [~markliu], [~yifanzou] 

Do you know  how do we configure the version of Python interpreter on Jenkins?

We eventually would like to have multiple versions of the interpreter, so I 
wonder what the best practices are to achieve that.

One option may be to simply install multiple versions of interpreter on Jenkins 
machines and rely on tox to discern, another - try to run test suites in a 
docker container with all necessary dependencies as desired. 

Do you have any advice here or perhaps you're working on something similar 
already?  

> Add tox suites for various Python 3 versions
> 
>
> Key: BEAM-5663
> URL: https://issues.apache.org/jira/browse/BEAM-5663
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Priority: Major
>
> Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test 
> failings across various Python 3 versions. It will be valuable to add tox 
> suites for Python 3.4, 3.5, 3.6 and 3.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5663) Add tox suites for various Python 3 versions

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645955#comment-16645955
 ] 

Valentyn Tymofieiev edited comment on BEAM-5663 at 10/11/18 7:09 PM:
-

Another interesting question is whether we can easily make these suites run in 
parallel so that we don't slow down precommits too much.



was (Author: tvalentyn):
Another interesting question whether we can easily make these suites run in 
parallel so that we don't slow down precommits too much.


> Add tox suites for various Python 3 versions
> 
>
> Key: BEAM-5663
> URL: https://issues.apache.org/jira/browse/BEAM-5663
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Priority: Major
>
> Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test 
> failings across various Python 3 versions. It will be valuable to add tox 
> suites for Python 3.4, 3.5, 3.6 and 3.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5663) Add tox suites for various Python 3 versions

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645955#comment-16645955
 ] 

Valentyn Tymofieiev commented on BEAM-5663:
---

Another interesting question whether we can easily make these suites run in 
parallel so that we don't slow down precommits too much.


> Add tox suites for various Python 3 versions
> 
>
> Key: BEAM-5663
> URL: https://issues.apache.org/jira/browse/BEAM-5663
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Priority: Major
>
> Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test 
> failings across various Python 3 versions. It will be valuable to add tox 
> suites for Python 3.4, 3.5, 3.6 and 3.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5243) beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python setup.py nosetests

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645754#comment-16645754
 ] 

Valentyn Tymofieiev edited comment on BEAM-5243 at 10/11/18 12:16 AM:
--

Happened today again: 
https://builds.apache.org/job/beam_PostCommit_Python_Verify/6235/


was (Author: tvalentyn):
Happened today again: 
https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/

> beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python 
> setup.py nosetests
> -
>
> Key: BEAM-5243
> URL: https://issues.apache.org/jira/browse/BEAM-5243
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, test-failures
>Reporter: Andrew Pilloud
>Assignee: Ahmet Altay
>Priority: Major
>
> It isn't clear to me what exactly failed, logs are full of stack traces.
>  [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/151/]
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/5844/]
>  
>  *01:00:38* ERROR: InvocationError for command 
> '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27-cython/bin/python
>  setup.py nosetests' (exited with code -11)*01:00:38* 
> ___ summary 
> *01:00:38* ERROR: py27-cython: commands 
> failed*01:00:38*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5243) beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python setup.py nosetests

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645754#comment-16645754
 ] 

Valentyn Tymofieiev commented on BEAM-5243:
---

Happened today again: 
https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/

> beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python 
> setup.py nosetests
> -
>
> Key: BEAM-5243
> URL: https://issues.apache.org/jira/browse/BEAM-5243
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, test-failures
>Reporter: Andrew Pilloud
>Assignee: Ahmet Altay
>Priority: Major
>
> It isn't clear to me what exactly failed, logs are full of stack traces.
>  [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/151/]
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/5844/]
>  
>  *01:00:38* ERROR: InvocationError for command 
> '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27-cython/bin/python
>  setup.py nosetests' (exited with code -11)*01:00:38* 
> ___ summary 
> *01:00:38* ERROR: py27-cython: commands 
> failed*01:00:38*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Reopened] (BEAM-3040) Python precommit timed out after 150 minutes

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reopened BEAM-3040:
---

> Python precommit timed out after 150 minutes
> 
>
> Key: BEAM-3040
> URL: https://issues.apache.org/jira/browse/BEAM-3040
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: Not applicable
>
>
> https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/143/consoleFull
> Within about 10 minutes it reaches this point:
> {code}
> ...
> 2017-10-10T03:33:33.591 [INFO] --- findbugs-maven-plugin:3.0.4:check 
> (default) @ beam-sdks-python ---
> 2017-10-10T03:33:33.702 [INFO] 
> 2017-10-10T03:33:33.702 [INFO] --- exec-maven-plugin:1.5.0:exec 
> (setuptools-test) @ beam-sdks-python ---
> {code}
> and the final output is like this:
> {code}
> ...
> 2017-10-10T03:33:33.702 [INFO] --- exec-maven-plugin:1.5.0:exec 
> (setuptools-test) @ beam-sdks-python ---
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/docs
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/setup.py
> lint create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/lint
> py27 create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/py27
> py27cython create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/py27cython
> py27cython installdeps: nose==1.3.7, grpcio-tools==1.3.5, cython==0.25.2
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> lint installdeps: nose==1.3.7, pycodestyle==2.3.1, pylint==1.7.1
> py27 installdeps: nose==1.3.7, grpcio-tools==1.3.5
> lint inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> py27 inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> py27cython inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> py27 runtests: PYTHONHASHSEED='2225684666'
> py27 runtests: commands[0] | python --version
> py27 runtests: commands[1] | - find apache_beam -type f -name *.pyc -delete
> py27 runtests: commands[2] | pip install -e .[test]
> lint runtests: PYTHONHASHSEED='2225684666'
> lint runtests: commands[0] | time pip install -e .[test]
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> py27 runtests: commands[3] | python 
> apache_beam/examples/complete/autocomplete_test.py
> lint runtests: commands[1] | time 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/run_pylint.sh
> py27 runtests: commands[4] | python setup.py test
> docs runtests: PYTHONHASHSEED='2225684666'
> docs runtests: commands[0] | time pip install -e .[test,gcp,docs]
> docs runtests: commands[1] | time 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/generate_pydoc.sh
> py27gcp create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/py27gcp
> py27gcp installdeps: nose==1.3.7
> py27cython runtests: PYTHONHASHSEED='2225684666'
> py27cython runtests: commands[0] | python --version
> py27cython runtests: commands[1] | - find apache_beam -type f -name *.pyc 
> -delete
> py27cython runtests: commands[2] | - find apache_beam -type f -name *.c 
> -delete
> py27cython runtests: commands[3] | - find apache_beam -type f -name *.so 
> -delete
> py27cython runtests: commands[4] | - find target/build -type f -name *.c 
> -delete
> py27cython runtests: commands[5] | - find target/build -type f -name *.so 
> -delete
> py27cython runtests: commands[6] | time pip install -e .[test]
> py27gcp inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> py27gcp runtests: PYTHONHASHSEED='2225684666'
> py27gcp runtests: commands[0] | pip install -e .[test,gcp]
> py27gcp runtests: commands[1] | python --version
> py27gcp runtests: commands[2] | - find apache_beam -type f -name *.pyc -delete
> py27gcp runtests: commands[3] | python 
> apache_beam/examples/complete/autocomplete_test.py
> py27gcp runtests: commands[4] | python setup.py test
> py27cython runtests: commands[7] | python 
> apache_beam/examples/complete/autocomplete_test.py
> py27cython runtests: commands[8] | pyth

[jira] [Commented] (BEAM-3040) Python precommit timed out after 150 minutes

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16645492#comment-16645492
 ] 

Valentyn Tymofieiev commented on BEAM-3040:
---

~30 tests are skipped in 
https://github.com/apache/beam/blob/dc4830d03525586e42d3cc893bccfe23959196f0/sdks/python/apache_beam/runners/portability/portable_runner_test.py
 
due to this issue, which is now marked fixed. Not sure if it's safe to unskip 
them.

> Python precommit timed out after 150 minutes
> 
>
> Key: BEAM-3040
> URL: https://issues.apache.org/jira/browse/BEAM-3040
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Ahmet Altay
>Priority: Major
> Fix For: Not applicable
>
>
> https://builds.apache.org/job/beam_PreCommit_Python_MavenInstall/143/consoleFull
> Within about 10 minutes it reaches this point:
> {code}
> ...
> 2017-10-10T03:33:33.591 [INFO] --- findbugs-maven-plugin:3.0.4:check 
> (default) @ beam-sdks-python ---
> 2017-10-10T03:33:33.702 [INFO] 
> 2017-10-10T03:33:33.702 [INFO] --- exec-maven-plugin:1.5.0:exec 
> (setuptools-test) @ beam-sdks-python ---
> {code}
> and the final output is like this:
> {code}
> ...
> 2017-10-10T03:33:33.702 [INFO] --- exec-maven-plugin:1.5.0:exec 
> (setuptools-test) @ beam-sdks-python ---
> docs create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/docs
> GLOB sdist-make: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/setup.py
> lint create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/lint
> py27 create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/py27
> py27cython create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/py27cython
> py27cython installdeps: nose==1.3.7, grpcio-tools==1.3.5, cython==0.25.2
> docs installdeps: nose==1.3.7, grpcio-tools==1.3.5, Sphinx==1.5.5, 
> sphinx_rtd_theme==0.2.4
> lint installdeps: nose==1.3.7, pycodestyle==2.3.1, pylint==1.7.1
> py27 installdeps: nose==1.3.7, grpcio-tools==1.3.5
> lint inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> py27 inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> py27cython inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> py27 runtests: PYTHONHASHSEED='2225684666'
> py27 runtests: commands[0] | python --version
> py27 runtests: commands[1] | - find apache_beam -type f -name *.pyc -delete
> py27 runtests: commands[2] | pip install -e .[test]
> lint runtests: PYTHONHASHSEED='2225684666'
> lint runtests: commands[0] | time pip install -e .[test]
> docs inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> py27 runtests: commands[3] | python 
> apache_beam/examples/complete/autocomplete_test.py
> lint runtests: commands[1] | time 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/run_pylint.sh
> py27 runtests: commands[4] | python setup.py test
> docs runtests: PYTHONHASHSEED='2225684666'
> docs runtests: commands[0] | time pip install -e .[test,gcp,docs]
> docs runtests: commands[1] | time 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/generate_pydoc.sh
> py27gcp create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/py27gcp
> py27gcp installdeps: nose==1.3.7
> py27cython runtests: PYTHONHASHSEED='2225684666'
> py27cython runtests: commands[0] | python --version
> py27cython runtests: commands[1] | - find apache_beam -type f -name *.pyc 
> -delete
> py27cython runtests: commands[2] | - find apache_beam -type f -name *.c 
> -delete
> py27cython runtests: commands[3] | - find apache_beam -type f -name *.so 
> -delete
> py27cython runtests: commands[4] | - find target/build -type f -name *.c 
> -delete
> py27cython runtests: commands[5] | - find target/build -type f -name *.so 
> -delete
> py27cython runtests: commands[6] | time pip install -e .[test]
> py27gcp inst: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall/sdks/python/target/.tox/dist/apache-beam-2.3.0.dev.zip
> py27gcp runtests: PYTHONHASHSEED='2225684666'
> py27gcp runtests: commands[0] | pip install -e .[test,gcp]
> py27gcp runtests: commands[1] | python --version
> py27gcp runtests: commands[2] | - find apache_beam -ty

[jira] [Commented] (BEAM-5624) Avro IO does not work with avro-python3 package out-of-the-box on Python 3, several tests fail with AttributeError (module 'avro.schema' has no attribute 'parse')

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16643746#comment-16643746
 ] 

Valentyn Tymofieiev commented on BEAM-5624:
---

This issue is due to a small API change (see: 
https://github.com/apache/beam/pull/6616). 
However there are some troubling reports about bad experience with 
avro-python3, see [1,2].

That said, we may want to migrate to fastavro sooner than later, FYI [~udim] 
 [~chamikara] [~altay].

[1] https://github.com/common-workflow-language/cwltool/issues/524
[2] https://issues.apache.org/jira/browse/AVRO-2046 

> Avro IO does not work with avro-python3 package out-of-the-box on Python 3, 
> several tests fail with AttributeError (module 'avro.schema' has no attribute 
> 'parse') 
> ---
>
> Key: BEAM-5624
> URL: https://issues.apache.org/jira/browse/BEAM-5624
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Simon
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> ==
> ERROR: Failure: AttributeError (module 'avro.schema' has no attribute 'parse')
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/failure.py",
>  line 39, in runTest
> raise self.exc_val.with_traceback(self.tb)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/loader.py",
>  line 418, in loadTestsFromName
> addr.filename, addr.module)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/importer.py",
>  line 47, in importFromPath
> return self.importFromDir(dir_path, fqname)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/importer.py",
>  line 94, in importFromDir
> mod = load_module(part_fqname, fh, filename, desc)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/imp.py",
>  line 234, in load_module
> return load_source(name, filename, file)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/imp.py",
>  line 172, in load_source
> module = _load(spec)
>   File "", line 693, in _load
>   File "", line 673, in _load_unlocked
>   File "", line 673, in exec_module
>   File "", line 222, in _call_with_frames_removed
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/avroio_test.py",
>  line 54, in 
> class TestAvro(unittest.TestCase):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/avroio_test.py",
>  line 89, in TestAvro
> SCHEMA = avro.schema.parse('''
> AttributeError: module 'avro.schema' has no attribute 'parse'
> Note that we use a different implementation of avro/avro-python3 package 
> depending on Python version. We are also evaluating potential replacement of 
> avro with fastavro.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5617) Several tests fail on Python 3 with failed assert: == []

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642684#comment-16642684
 ] 

Valentyn Tymofieiev commented on BEAM-5617:
---

It appears that lines 1272-1274 are not executed for 
apache_beam.transforms.sideinputs_test.SideInputsTest, so failure may be 
somewhere else (for that particular test at least). One possible way to track 
this down would be to go over functions/methods that are invoked during test 
execution, and see when do the arguments passed to the functions/methods start 
to diverge between Python 2 and Python 3. I didn't get to the bottom of the 
issue yet, but will continue looking tomorrow, if we don't rootcause it by then.

> Several tests fail on Python 3 with failed assert:  == []
> --
>
> Key: BEAM-5617
> URL: https://issues.apache.org/jira/browse/BEAM-5617
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
>
> ==
> ERROR: test_iterable_side_input 
> (apache_beam.transforms.sideinputs_test.SideInputsTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 414, in invoke_process
> windowed_value, self.process_method(windowed_value.value))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1068, in 
> wrapper = lambda x: [fn(x)]
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
>  line 119, in _equal
> 'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
> apache_beam.testing.util.BeamAssertException: Failed assert: [3, 4, 6, 8] == 
> []



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5617) Several tests fail on Python 3 with failed assert: == []

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642684#comment-16642684
 ] 

Valentyn Tymofieiev edited comment on BEAM-5617 at 10/9/18 1:42 AM:


It appears that lines 1272-1274 are not executed for 
apache_beam.transforms.sideinputs_test.SideInputsTest.test_iterable_side_input, 
so failure may be somewhere else (for that particular test at least). One 
possible way to track this down would be to go over functions/methods that are 
invoked during test execution, and see when do the arguments passed to the 
functions/methods start to diverge between Python 2 and Python 3. I didn't get 
to the bottom of the issue yet, but will continue looking tomorrow, if we don't 
rootcause it by then.


was (Author: tvalentyn):
It appears that lines 1272-1274 are not executed for 
apache_beam.transforms.sideinputs_test.SideInputsTest, so failure may be 
somewhere else (for that particular test at least). One possible way to track 
this down would be to go over functions/methods that are invoked during test 
execution, and see when do the arguments passed to the functions/methods start 
to diverge between Python 2 and Python 3. I didn't get to the bottom of the 
issue yet, but will continue looking tomorrow, if we don't rootcause it by then.

> Several tests fail on Python 3 with failed assert:  == []
> --
>
> Key: BEAM-5617
> URL: https://issues.apache.org/jira/browse/BEAM-5617
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
>
> ==
> ERROR: test_iterable_side_input 
> (apache_beam.transforms.sideinputs_test.SideInputsTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 414, in invoke_process
> windowed_value, self.process_method(windowed_value.value))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1068, in 
> wrapper = lambda x: [fn(x)]
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
>  line 119, in _equal
> 'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
> apache_beam.testing.util.BeamAssertException: Failed assert: [3, 4, 6, 8] == 
> []



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5623) Several IO tests hang indefinitely during execution on Python 3.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5623:
--
Summary: Several IO tests hang indefinitely during execution on Python 3.  
(was: Several tests IO tests hang indefinitely during execution on Python 3.)

> Several IO tests hang indefinitely during execution on Python 3.
> 
>
> Key: BEAM-5623
> URL: https://issues.apache.org/jira/browse/BEAM-5623
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> test_read_empty_single_file_no_eol_gzip 
> (apache_beam.io.textio_test.TextSourceTest) 
> Also several tests cases in tfrecordio_test, for example:
> test_process_auto (apache_beam.io.tfrecordio_test.TestReadAllFromTFRecord)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5663) Add tox suites for various Python 3 versions

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642614#comment-16642614
 ] 

Valentyn Tymofieiev commented on BEAM-5663:
---

+1. Thanks for filing this issue.

> Add tox suites for various Python 3 versions
> 
>
> Key: BEAM-5663
> URL: https://issues.apache.org/jira/browse/BEAM-5663
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Manu Zhang
>Priority: Major
>
> Currently, Python 3.5.2 is set up for Jenkins tests but we've seen test 
> failings across various Python 3 versions. It will be valuable to add tox 
> suites for Python 3.4, 3.5, 3.6 and 3.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5626) Several IO tests fail in Python 3 with RuntimeError('dictionary changed size during iteration',)}

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev resolved BEAM-5626.
---
   Resolution: Fixed
Fix Version/s: 2.8.0

> Several IO tests fail in Python 3 with RuntimeError('dictionary changed size 
> during iteration',)}
> -
>
> Key: BEAM-5626
> URL: https://issues.apache.org/jira/browse/BEAM-5626
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Ruoyun Huang
>Priority: Major
> Fix For: 2.8.0
>
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
>  ERROR: test_delete_dir 
> (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem_test.py",
>  line 506, in test_delete_dir
>  self.fs.delete([url_t1])
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem.py",
>  line 370, in delete
>  raise BeamIOError("Delete operation failed", exceptions)
>  apache_beam.io.filesystem.BeamIOError: Delete operation failed with 
> exceptions {'hdfs://test_dir/new_dir1': RuntimeError('dictionary changed size 
> during iteration',   )}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5626) Several IO tests fail in Python 3 with RuntimeError('dictionary changed size during iteration',)}

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642600#comment-16642600
 ] 

Valentyn Tymofieiev commented on BEAM-5626:
---

Thanks for fixing this, [~ruoyun] !

> Several IO tests fail in Python 3 with RuntimeError('dictionary changed size 
> during iteration',)}
> -
>
> Key: BEAM-5626
> URL: https://issues.apache.org/jira/browse/BEAM-5626
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Ruoyun Huang
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
>  ERROR: test_delete_dir 
> (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem_test.py",
>  line 506, in test_delete_dir
>  self.fs.delete([url_t1])
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem.py",
>  line 370, in delete
>  raise BeamIOError("Delete operation failed", exceptions)
>  apache_beam.io.filesystem.BeamIOError: Delete operation failed with 
> exceptions {'hdfs://test_dir/new_dir1': RuntimeError('dictionary changed size 
> during iteration',   )}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5626) Several IO tests fail in Python 3 with RuntimeError('dictionary changed size during iteration',)}

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642598#comment-16642598
 ] 

Valentyn Tymofieiev commented on BEAM-5626:
---

Yes, I don't see this failure in IO tests anymore.

> Several IO tests fail in Python 3 with RuntimeError('dictionary changed size 
> during iteration',)}
> -
>
> Key: BEAM-5626
> URL: https://issues.apache.org/jira/browse/BEAM-5626
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Ruoyun Huang
>Priority: Major
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
>  ERROR: test_delete_dir 
> (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem_test.py",
>  line 506, in test_delete_dir
>  self.fs.delete([url_t1])
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem.py",
>  line 370, in delete
>  raise BeamIOError("Delete operation failed", exceptions)
>  apache_beam.io.filesystem.BeamIOError: Delete operation failed with 
> exceptions {'hdfs://test_dir/new_dir1': RuntimeError('dictionary changed size 
> during iteration',   )}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5089) Add an SDK harness container with Python 3 interpreter for portable pipelines.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642407#comment-16642407
 ] 

Valentyn Tymofieiev commented on BEAM-5089:
---

Remaining comments:
- Beam SDK should use new container when run in Python 3 environments.
- We should exercise new container in a validates container test suite.

> Add an SDK harness container with Python 3 interpreter for portable pipelines.
> --
>
> Key: BEAM-5089
> URL: https://issues.apache.org/jira/browse/BEAM-5089
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5617) Several tests fail on Python 3 with failed assert: == []

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642025#comment-16642025
 ] 

Valentyn Tymofieiev commented on BEAM-5617:
---

Thanks for your investigation, I'll try to take a look at this today.

> Several tests fail on Python 3 with failed assert:  == []
> --
>
> Key: BEAM-5617
> URL: https://issues.apache.org/jira/browse/BEAM-5617
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
>
> ==
> ERROR: test_iterable_side_input 
> (apache_beam.transforms.sideinputs_test.SideInputsTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 414, in invoke_process
> windowed_value, self.process_method(windowed_value.value))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1068, in 
> wrapper = lambda x: [fn(x)]
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
>  line 119, in _equal
> 'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
> apache_beam.testing.util.BeamAssertException: Failed assert: [3, 4, 6, 8] == 
> []



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5618) Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' and 'EmptySideInput'

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642015#comment-16642015
 ] 

Valentyn Tymofieiev commented on BEAM-5618:
---

Please take a look at 
https://docs.google.com/document/d/1s1BJVCY65LB_SYK1SU1u7NbZiFANoq-nEYaEvzRbYlA 
and let me know if you were able to reproduce this. The command line will be 
similar to one in "Run an Individual Test" section of the guide.


> Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' 
> and 'EmptySideInput'
> --
>
> Key: BEAM-5618
> URL: https://issues.apache.org/jira/browse/BEAM-5618
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_do_with_side_input_as_arg 
> (apache_beam.transforms.ptransform_test.PTransformTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 529, in invoke_process
> windowed_value, additional_args, additional_kwargs, output_processor)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 598, in _invoke_per_window
> windowed_value, self.process_method(*args_for_process))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 135, in 
> lambda x, addon: [x + addon], pvalue.AsSingleton(side))
> TypeError: unsupported operand type(s) for +: 'int' and 'EmptySideInput'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5617) Several tests fail on Python 3 with failed assert: == []

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16641461#comment-16641461
 ] 

Valentyn Tymofieiev commented on BEAM-5617:
---

Hi [~Juta], on high level, CombineGlobally expresses a functionality to combine 
a collection of elements, to a single element, using a commutative and 
associative function that knows how to combine several elements. For example, 
using an addition function (+) we could combine a sequence of integers into a 
single number (sum). "typed" function seems to capture the  following intent: 
if the arguments have an annotation that describes their type , then attach the 
type to the result, otherwise return result as is. 

So we may have an issue related to type annotataions (typehints). 

Were you able to find a part of code which evaluates differently under Python 2 
and Python 3, given exactly the same input? We may need to find out where the 
divergence happens first, as we trace the execution of test code. 

What makes you think that the culprit is in:  
https://github.com/apache/beam/blob/8fdbc0973949123a8fa0a27aa0f9773b1c24c6db/sdks/python/apache_beam/transforms/core.py#L1272
 ?


> Several tests fail on Python 3 with failed assert:  == []
> --
>
> Key: BEAM-5617
> URL: https://issues.apache.org/jira/browse/BEAM-5617
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Juta Staes
>Priority: Major
>
> ==
> ERROR: test_iterable_side_input 
> (apache_beam.transforms.sideinputs_test.SideInputsTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 414, in invoke_process
> windowed_value, self.process_method(windowed_value.value))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1068, in 
> wrapper = lambda x: [fn(x)]
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
>  line 119, in _equal
> 'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
> apache_beam.testing.util.BeamAssertException: Failed assert: [3, 4, 6, 8] == 
> []



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5620) Some tests use assertItemsEqual method, not available in Python 3

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640200#comment-16640200
 ] 

Valentyn Tymofieiev commented on BEAM-5620:
---

So, to sum up, I would go with 
https://github.com/apache/beam/blob/c34c367f5da6f9bef8a46471195470923a201af9/sdks/python/apache_beam/coders/coders_test_common.py#L61

> Some tests use assertItemsEqual method, not available in Python 3
> -
>
> Key: BEAM-5620
> URL: https://issues.apache.org/jira/browse/BEAM-5620
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> See: 
> https://github.com/apache/beam/search?q=assertItemsEqual&unscoped_q=assertItemsEqual



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5620) Some tests use assertItemsEqual method, not available in Python 3

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639308#comment-16639308
 ] 

Valentyn Tymofieiev commented on BEAM-5620:
---

It's possible although in this case some of our test classes will inherit from 
unittest.TestCase, while others will inherit from something like 
apache_beam.testing.util.TestCase, which may cause confusion. Everytime someone 
reading the code would see the latter, they may be wondering why do we have two 
classes, and which one needs to be used. 

While if we have a patch in an individual class, the goal is self-explanatory, 
but I agree that this is not a perfect solution.

Another possibility is to use six.assertCountEqual, but according to the 
approach we have chosen in Beam [1] , we'd like to use native idioms of Python 
3 when possible. This is why we eliminated the usages of six, and removed this 
dependency. 

[1] 
https://docs.google.com/document/d/1xDG0MWVlDKDPu_IW9gtMvxi2S9I0GB0VDTkPhjXT0nE

> Some tests use assertItemsEqual method, not available in Python 3
> -
>
> Key: BEAM-5620
> URL: https://issues.apache.org/jira/browse/BEAM-5620
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> See: 
> https://github.com/apache/beam/search?q=assertItemsEqual&unscoped_q=assertItemsEqual



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5661) [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such file or directory

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16639009#comment-16639009
 ] 

Valentyn Tymofieiev commented on BEAM-5661:
---

I don't think so. It sounds like this is a flake of Docker itself. [~herohde] 
may have some advice here.

> [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such 
> file or directory
> ---
>
> Key: BEAM-5661
> URL: https://issues.apache.org/jira/browse/BEAM-5661
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins Job|https://builds.apache.org/job/beam_PostCommit_Py_ValCont/902/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/pmjbkhxaeryx4/console-log?task=:beam-sdks-python-container:docker#L2]
>  * [Test source 
> code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/sdks/python/container/build.gradle#L61]
> Initial investigation:
> {{failed to get digest 
> sha256:4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed: open 
> /var/lib/docker/image/aufs/imagedb/content/sha256/4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed:
>  no such file or directory}}
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5661) [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such file or directory

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-5661:
-

Assignee: Henning Rohde  (was: Valentyn Tymofieiev)

> [beam_PostCommit_Py_ValCont] [:beam-sdks-python-container:docker] no such 
> file or directory
> ---
>
> Key: BEAM-5661
> URL: https://issues.apache.org/jira/browse/BEAM-5661
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Henning Rohde
>Priority: Major
>  Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins Job|https://builds.apache.org/job/beam_PostCommit_Py_ValCont/902/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/pmjbkhxaeryx4/console-log?task=:beam-sdks-python-container:docker#L2]
>  * [Test source 
> code|https://github.com/apache/beam/blob/845f8d0abcc5a8d7f93457c27aff0feeb1a867d5/sdks/python/container/build.gradle#L61]
> Initial investigation:
> {{failed to get digest 
> sha256:4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed: open 
> /var/lib/docker/image/aufs/imagedb/content/sha256/4ee4ea2f0113e98b49d8e376ce847feb374ddf2b8ea775502459d8a1b8a3eaed:
>  no such file or directory}}
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5243) beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python setup.py nosetests

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638998#comment-16638998
 ] 

Valentyn Tymofieiev edited comment on BEAM-5243 at 10/4/18 10:55 PM:
-

[~swegner]

It seems that we are hit by this flake ~once per month, most recently in 
https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/197/

The symptoms are that a test suite started by `python setup.py nosetests` is 
passing but then returns a non-zero exit code "-11", which we interpret as 
failure:

Ran 1726 tests in 261.611s
OK (SKIP=324)
ERROR: InvocationError for command 
'/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27/bin/python
 setup.py nosetests' (exited with code -11)


Exit code -11 may in some cases be a segmentation fault as per Stackoverflow 
Search.

This may be an issue with "nose" package, or some side-effects of some of our 
tests.

I suggest following further investigations avenues:
- Try to reproduce this locally, in a virtual environment, that has nose 
package and Beam SDK installed via something like  `for i in `seq 100`; do 
python setup.py nosetests || echo "Last exit code was $?"; done`.
- See if the tests that we are running make a difference whether the error 
appears.
- See if versions of nose makes a difference. 
- Reach out to nose maintainers.




was (Author: tvalentyn):
[~swegner]

It seems that we are hit by this flake ~once per month, most recently in 
https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/197/

The symptoms are that a test suite started by `python setup.py nosetests` is 
passing but then returns a non-zero exit code "-11", which we interpret as 
failure:

Ran 1726 tests in 261.611s
OK (SKIP=324)
ERROR: InvocationError for command 
'/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27/bin/python
 setup.py nosetests' (exited with code -11)


Exit code -11 may in some cases be a segmentation fault as per Stackoverflow 
Search.

This may be an issue with "nose" package, or some side-effects of some of our 
tests.

I suggest following further investigations avenues:
- Try to reproduce this locally  via something like  `for i in `seq 100`; do 
python setup.py nosetests || echo "Last exit code was $?"; done`.
- See if the tests that we are running make a difference whether the error 
appears.
- See if versions of nose makes a difference. 
- Reach out to nose maintainers.



> beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python 
> setup.py nosetests
> -
>
> Key: BEAM-5243
> URL: https://issues.apache.org/jira/browse/BEAM-5243
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Andrew Pilloud
>Assignee: Ahmet Altay
>Priority: Major
>
> It isn't clear to me what exactly failed, logs are full of stack traces.
>  [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/151/]
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/5844/]
>  
>  *01:00:38* ERROR: InvocationError for command 
> '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27-cython/bin/python
>  setup.py nosetests' (exited with code -11)*01:00:38* 
> ___ summary 
> *01:00:38* ERROR: py27-cython: commands 
> failed*01:00:38*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5243) beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python setup.py nosetests

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638998#comment-16638998
 ] 

Valentyn Tymofieiev commented on BEAM-5243:
---

[~swegner]

It seems that we are hit by this flake ~once per month, most recently in 
https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/197/

The symptoms are that a test suite started by `python setup.py nosetests` is 
passing but then returns a non-zero exit code "-11", which we interpret as 
failure:

Ran 1726 tests in 261.611s
OK (SKIP=324)
ERROR: InvocationError for command 
'/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27/bin/python
 setup.py nosetests' (exited with code -11)


Exit code -11 may in some cases be a segmentation fault as per Stackoverflow 
Search.

This may be an issue with "nose" package, or some side-effects of some of our 
tests.

I suggest following further investigations avenues:
- Try to reproduce this locally  via something like  `for i in `seq 100`; do 
python setup.py nosetests || echo "Last exit code was $?"; done`.
- See if the tests that we are running make a difference whether the error 
appears.
- See if versions of nose makes a difference. 
- Reach out to nose maintainers.



> beam_Release_Gradle_NightlySnapshot InvocationError py27-cython/bin/python 
> setup.py nosetests
> -
>
> Key: BEAM-5243
> URL: https://issues.apache.org/jira/browse/BEAM-5243
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Andrew Pilloud
>Assignee: Ahmet Altay
>Priority: Major
>
> It isn't clear to me what exactly failed, logs are full of stack traces.
>  [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/151/]
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/5844/]
>  
>  *01:00:38* ERROR: InvocationError for command 
> '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27-cython/bin/python
>  setup.py nosetests' (exited with code -11)*01:00:38* 
> ___ summary 
> *01:00:38* ERROR: py27-cython: commands 
> failed*01:00:38*



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5652) [beam_Release_Gradle_NightlySnapshot] [:testPython2] python setup.py nosetests' (exited with code -11)

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16638980#comment-16638980
 ] 

Valentyn Tymofieiev commented on BEAM-5652:
---

This is a duplicate of https://issues.apache.org/jira/browse/BEAM-5243?, I'll 
add some comments there.

> [beam_Release_Gradle_NightlySnapshot] [:testPython2] python setup.py 
> nosetests' (exited with code -11)
> --
>
> Key: BEAM-5652
> URL: https://issues.apache.org/jira/browse/BEAM-5652
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Labels: currently-failing
> Fix For: Not applicable
>
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/197/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/ysct2bmi66p5c/console-log?task=:beam-sdks-python:testPython2#L4784]
>  * [Test source 
> code|https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/build.gradle#L130]
> Initial investigation:
> It seems that all tests passed, but the overall command failed. I'm not sure 
> why.
> Ran 1726 tests in 261.611s
> OK (SKIP=324)
> ERROR: InvocationError for command 
> '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27/bin/python
>  setup.py nosetests' (exited with code -11)
> ___ summary 
> 
> ERROR:   py27: commands failed
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5652) [beam_Release_Gradle_NightlySnapshot] [:testPython2] python setup.py nosetests' (exited with code -11)

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev resolved BEAM-5652.
---
   Resolution: Duplicate
Fix Version/s: Not applicable

> [beam_Release_Gradle_NightlySnapshot] [:testPython2] python setup.py 
> nosetests' (exited with code -11)
> --
>
> Key: BEAM-5652
> URL: https://issues.apache.org/jira/browse/BEAM-5652
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Scott Wegner
>Assignee: Valentyn Tymofieiev
>Priority: Major
>  Labels: currently-failing
> Fix For: Not applicable
>
>
> _Use this form to file an issue for test failure:_
>  * [Jenkins 
> Job|https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/197/]
>  * [Gradle Build 
> Scan|https://scans.gradle.com/s/ysct2bmi66p5c/console-log?task=:beam-sdks-python:testPython2#L4784]
>  * [Test source 
> code|https://github.com/apache/beam/blob/766a1dc5b6523a0b7a3ce0a9ab78901cf17c9013/sdks/python/build.gradle#L130]
> Initial investigation:
> It seems that all tests passed, but the overall command failed. I'm not sure 
> why.
> Ran 1726 tests in 261.611s
> OK (SKIP=324)
> ERROR: InvocationError for command 
> '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/python/target/.tox/py27/bin/python
>  setup.py nosetests' (exited with code -11)
> ___ summary 
> 
> ERROR:   py27: commands failed
> 
> _After you've filled out the above details, please [assign the issue to an 
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
>  Assignee should [treat test failures as 
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
>  helping to fix the issue or find a more appropriate owner. See [Apache Beam 
> Post-Commit 
> Policies|https://beam.apache.org/contribute/postcommits-policies]._



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5608) KeyError on Python 3 when checking environment var BEAM_EXPERIMENTAL_PY3

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev resolved BEAM-5608.
---
   Resolution: Fixed
Fix Version/s: 2.8.0

> KeyError on Python 3 when checking environment var BEAM_EXPERIMENTAL_PY3
> 
>
> Key: BEAM-5608
> URL: https://issues.apache.org/jira/browse/BEAM-5608
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Pablo Estrada
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.8.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3055) Retry downloading required test artifacts with a backoff when download fails.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16637371#comment-16637371
 ] 

Valentyn Tymofieiev commented on BEAM-3055:
---

As per my previous comment this happened after Gradle migration.

> Retry downloading required test artifacts with a backoff when download fails.
> -
>
> Key: BEAM-3055
> URL: https://issues.apache.org/jira/browse/BEAM-3055
> Project: Beam
>  Issue Type: Improvement
>  Components: test-failures, testing
>Reporter: Valentyn Tymofieiev
>Assignee: Jason Kuster
>Priority: Major
> Fix For: Not applicable
>
>
> When Maven fails to download a required artifact for a test, the test fails. 
> Is it possible to configure Maven to retry the download with a backoff up to 
> N number of attempts?
> Example test failure:
> https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/15004/console
> 2017-10-11T19:01:21.382 [INFO] 
> 
> 2017-10-11T19:01:21.382 [INFO] BUILD FAILURE
> 2017-10-11T19:01:21.382 [INFO] 
> 
> 2017-10-11T19:01:21.383 [INFO] Total time: 55:20 min
> 2017-10-11T19:01:21.383 [INFO] Finished at: 2017-10-11T19:01:21+00:00
> 2017-10-11T19:01:23.807 [INFO] Final Memory: 261M/2068M
> 2017-10-11T19:01:23.807 [INFO] 
> 
> 2017-10-11T19:01:23.836 [ERROR] Failed to execute goal on project 
> beam-sdks-java-io-hcatalog: Could not resolve dependencies for project 
> org.apache.beam:beam-sdks-java-io-hcatalog:jar:2.2.0-SNAPSHOT: The following 
> artifacts could not be resolved: org.apache.hive:hive-metastore:jar:2.1.0, 
> javolution:javolution:jar:5.5.1: Could not transfer artifact 
> org.apache.hive:hive-metastore:jar:2.1.0 from/to central 
> (https://repo.maven.apache.org/maven2): GET request of: 
> org/apache/hive/hive-metastore/2.1.0/hive-metastore-2.1.0.jar from central 
> failed: Connection reset -> [Help 1].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5627:
--
Summary: Several IO tests fail in Python 3  when accessing a temporary file 
with  TypeError: a bytes-like object is required, not 'str'  (was: Several IO 
tests fail in Python 3  with  TypeError: a bytes-like object is required, not 
'str')

> Several IO tests fail in Python 3  when accessing a temporary file with  
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_split_at_fraction_exhaustive 
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
>  --
>  Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 120, in test_split_at_fraction_exhaustive
>  source = self._create_source(data)
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 43, in _create_source
>  source = LineSource(self._create_file_with_data(data))
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
>  line 35, in _create_file_with_data
>  f.write(line + '\n')
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
>  line 622, in func_wrapper
>  return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
>  ERROR: test_file_sink_writing 
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 121, in 
> test_file_sink_writing
>   init_token, writer_results = self._common_init(sink)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
>   writer1 = sink.open_writer(init_token, '1')
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/options/value_provider.py", line 133, in _f
>   return fnc(self, *args, **kwargs)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink.py", line 385, in __init__
>   self.temp_handle = self.sink.open(temp_shard_path)
> File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/   
>apache_beam/io/filebasedsink_test.py", line 82, in open
>   file_handle.write('[start]')
>   TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5627) Several IO tests fail in Python 3 with TypeError: a bytes-like object is required, not 'str'

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5627:
--
Description: 
ERROR: test_split_at_fraction_exhaustive 
(apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
 --
 Traceback (most recent call last):
   File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
 line 120, in test_split_at_fraction_exhaustive
 source = self._create_source(data)
   File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
 line 43, in _create_source
 source = LineSource(self._create_file_with_data(data))
   File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
 line 35, in _create_file_with_data
 f.write(line + '\n')
   File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
 line 622, in func_wrapper
 return func(*args, **kwargs)
TypeError: a bytes-like object is required, not 'str'


Also similar:



==
 ERROR: test_file_sink_writing 
(apache_beam.io.filebasedsink_test.TestFileBasedSink)
--
Traceback (most recent call last):
   File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink_test.py", line 121, in test_file_sink_writing
  init_token, writer_results = self._common_init(sink)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink_test.py", line 103, in _common_init
  writer1 = sink.open_writer(init_token, '1')
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/options/value_provider.py", line 133, in _f
  return fnc(self, *args, **kwargs)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink.py", line 185, in open_writer
return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink.py", line 385, in __init__
  self.temp_handle = self.sink.open(temp_shard_path)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink_test.py", line 82, in open
  file_handle.write('[start]')
  TypeError: a bytes-like object is required, not 'str'

  was:
==
 ERROR: test_file_sink_writing 
(apache_beam.io.filebasedsink_test.TestFileBasedSink)
--
Traceback (most recent call last):
   File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink_test.py", line 121, in test_file_sink_writing
  init_token, writer_results = self._common_init(sink)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink_test.py", line 103, in _common_init
  writer1 = sink.open_writer(init_token, '1')
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/options/value_provider.py", line 133, in _f
  return fnc(self, *args, **kwargs)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink.py", line 185, in open_writer
return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink.py", line 385, in __init__
  self.temp_handle = self.sink.open(temp_shard_path)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink_test.py", line 82, in open
  file_handle.write('[start]')
  TypeError: a bytes-like object is required, not 'str'


> Several IO tests fail in Python 3  with  TypeError: a bytes-like object is 
> required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_split_at_fraction_exhaustive

[jira] [Created] (BEAM-5629) Several typehints tests fail on Python 3 with ValueError: no signature found for builtin

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5629:
-

 Summary: Several typehints tests fail on Python 3 with ValueError: 
no signature found for builtin 
 Key: BEAM-5629
 URL: https://issues.apache.org/jira/browse/BEAM-5629
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


"Traceback (most recent call last):
  File ""/usr/lib/python3.5/inspect.py"", line 1088, in getfullargspec
sigcls=Signature)
  File ""/usr/lib/python3.5/inspect.py"", line 2231, in _signature_from_callable
skip_bound_arg=skip_bound_arg)
  File ""/usr/lib/python3.5/inspect.py"", line 2061, in _signature_from_builtin
raise ValueError(""no signature found for builtin {!r}"".format(func))
ValueError: no signature found for builtin 




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5628) Several VcfIO tests fail in Python 3 with TypeError: cannot use a string pattern on a bytes-like object

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5628:
-

 Summary: Several VcfIO tests fail in Python 3 with  TypeError: 
cannot use a string pattern on a bytes-like object
 Key: BEAM-5628
 URL: https://issues.apache.org/jira/browse/BEAM-5628
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


ERROR: test_read_after_splitting (apache_beam.io.vcfio_test.VcfSourceTest)
"
 --
Traceback (most recent call last):
   File 
""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio_test.py"",
 line 336, in test_read_after_splitting
] split_records.extend(source_test_utils.read_from_source(*source_info))
]   File 
""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils.py"",
 line 101, in read_from_source
 for value in reader:
   File 
""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio.py"",
 line 264, in read_records
 for line in record_iterator:
   File 
""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/vcfio.py"",
 line 330, in __next__
 record = next(self._vcf_reader)
   File 
""/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/vcf/parser.py"",
 line 543, in __next__
 row = self._row_pattern.split(line.rstrip())
 TypeError: cannot use a string pattern on a bytes-like object
"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5627) Several IO tests fail in Python 3 with TypeError: a bytes-like object is required, not 'str'

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5627:
-

 Summary: Several IO tests fail in Python 3  with  TypeError: a 
bytes-like object is required, not 'str'
 Key: BEAM-5627
 URL: https://issues.apache.org/jira/browse/BEAM-5627
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


==
 ERROR: test_file_sink_writing 
(apache_beam.io.filebasedsink_test.TestFileBasedSink)
--
Traceback (most recent call last):
   File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink_test.py", line 121, in test_file_sink_writing
  init_token, writer_results = self._common_init(sink)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink_test.py", line 103, in _common_init
  writer1 = sink.open_writer(init_token, '1')
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/options/value_provider.py", line 133, in _f
  return fnc(self, *args, **kwargs)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink.py", line 185, in open_writer
return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink.py", line 385, in __init__
  self.temp_handle = self.sink.open(temp_shard_path)
File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/ 
 apache_beam/io/filebasedsink_test.py", line 82, in open
  file_handle.write('[start]')
  TypeError: a bytes-like object is required, not 'str'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5626) Several IO tests fail in Python 3 with RuntimeError('dictionary changed size during iteration',)}

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5626:
--
Summary: Several IO tests fail in Python 3 with RuntimeError('dictionary 
changed size during iteration',)}  (was: Several IO tests fail in Python  with 
RuntimeError('dictionary changed size during iteration',)})

> Several IO tests fail in Python 3 with RuntimeError('dictionary changed size 
> during iteration',)}
> -
>
> Key: BEAM-5626
> URL: https://issues.apache.org/jira/browse/BEAM-5626
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
>  ERROR: test_delete_dir 
> (apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem_test.py",
>  line 506, in test_delete_dir
>  self.fs.delete([url_t1])
>File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem.py",
>  line 370, in delete
>  raise BeamIOError("Delete operation failed", exceptions)
>  apache_beam.io.filesystem.BeamIOError: Delete operation failed with 
> exceptions {'hdfs://test_dir/new_dir1': RuntimeError('dictionary changed size 
> during iteration',   )}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5626) Several IO tests fail in Python with RuntimeError('dictionary changed size during iteration',)}

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5626:
-

 Summary: Several IO tests fail in Python  with 
RuntimeError('dictionary changed size during iteration',)}
 Key: BEAM-5626
 URL: https://issues.apache.org/jira/browse/BEAM-5626
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


 ERROR: test_delete_dir 
(apache_beam.io.hadoopfilesystem_test.HadoopFileSystemTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem_test.py",
 line 506, in test_delete_dir
 self.fs.delete([url_t1])
   File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/hadoopfilesystem.py",
 line 370, in delete
 raise BeamIOError("Delete operation failed", exceptions)
 apache_beam.io.filesystem.BeamIOError: Delete operation failed with exceptions 
{'hdfs://test_dir/new_dir1': RuntimeError('dictionary changed size during 
iteration',   )}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5624) Avro IO does not work with avro-python3 package out-of-the-box on Python 3, several tests fail with AttributeError (module 'avro.schema' has no attribute 'parse')

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5624:
-

 Summary: Avro IO does not work with avro-python3 package 
out-of-the-box on Python 3, several tests fail with AttributeError (module 
'avro.schema' has no attribute 'parse') 
 Key: BEAM-5624
 URL: https://issues.apache.org/jira/browse/BEAM-5624
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


==
ERROR: Failure: AttributeError (module 'avro.schema' has no attribute 'parse')
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/failure.py",
 line 39, in runTest
raise self.exc_val.with_traceback(self.tb)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/loader.py",
 line 418, in loadTestsFromName
addr.filename, addr.module)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/importer.py",
 line 47, in importFromPath
return self.importFromDir(dir_path, fqname)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/site-packages/nose/importer.py",
 line 94, in importFromDir
mod = load_module(part_fqname, fh, filename, desc)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/imp.py",
 line 234, in load_module
return load_source(name, filename, file)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/imp.py",
 line 172, in load_source
module = _load(spec)
  File "", line 693, in _load
  File "", line 673, in _load_unlocked
  File "", line 673, in exec_module
  File "", line 222, in _call_with_frames_removed
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/avroio_test.py",
 line 54, in 
class TestAvro(unittest.TestCase):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/avroio_test.py",
 line 89, in TestAvro
SCHEMA = avro.schema.parse('''
AttributeError: module 'avro.schema' has no attribute 'parse'

Note that we use a different implementation of avro/avro-python3 package 
depending on Python version. We are also evaluating potential replacement of 
avro with fastavro.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5623) Several tests IO tests hang indefinitely during execution on Python 3.

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5623:
-

 Summary: Several tests IO tests hang indefinitely during execution 
on Python 3.
 Key: BEAM-5623
 URL: https://issues.apache.org/jira/browse/BEAM-5623
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


test_read_empty_single_file_no_eol_gzip 
(apache_beam.io.textio_test.TextSourceTest) 

Also several tests cases in tfrecordio_test, for example:

test_process_auto (apache_beam.io.tfrecordio_test.TestReadAllFromTFRecord)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5618) Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' and 'EmptySideInput'

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636376#comment-16636376
 ] 

Valentyn Tymofieiev commented on BEAM-5618:
---

ERROR: test_as_singleton_without_unique_labels 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 677, in process
self.do_fn_invoker.invoke_process(windowed_value)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 414, in invoke_process
windowed_value, self.process_method(windowed_value.value))
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
 line 1068, in 
wrapper = lambda x: [fn(x)]
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/sideinputs_test.py",
 line 224, in match
equal_to([expected_singleton])([actual_singleton1])
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
 line 119, in _equal
'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
apache_beam.testing.util.BeamAssertException: Failed assert: [2] == 
[]



> Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' 
> and 'EmptySideInput'
> --
>
> Key: BEAM-5618
> URL: https://issues.apache.org/jira/browse/BEAM-5618
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_do_with_side_input_as_arg 
> (apache_beam.transforms.ptransform_test.PTransformTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 529, in invoke_process
> windowed_value, additional_args, additional_kwargs, output_processor)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 598, in _invoke_per_window
> windowed_value, self.process_method(*args_for_process))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
>  line 135, in 
> lambda x, addon: [x + addon], pvalue.AsSingleton(side))
> TypeError: unsupported operand type(s) for +: 'int' and 'EmptySideInput'



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5621) Several tests fail on Python 3 with TypeError: unorderable types: str() < int()

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636375#comment-16636375
 ] 

Valentyn Tymofieiev commented on BEAM-5621:
---

A few other examples:

==
ERROR: test_target_duration (apache_beam.transforms.util_test.BatchElementsTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/util_test.py",
 line 103, in test_target_duration
target_batch_overhead=None, target_batch_duration_secs=10, clock=clock)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/util.py",
 line 226, in __init__
if max(0, target_batch_overhead, target_batch_duration_secs) == 0:
TypeError: unorderable types: NoneType() > int()

==
ERROR: test_target_overhead (apache_beam.transforms.util_test.BatchElementsTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/util_test.py",
 line 117, in test_target_overhead
target_batch_overhead=.05, target_batch_duration_secs=None, clock=clock)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/util.py",
 line 226, in __init__
if max(0, target_batch_overhead, target_batch_duration_secs) == 0:
TypeError: unorderable types: NoneType() > float()

==
ERROR: test_reshuffle_window_fn_preserved 
(apache_beam.transforms.util_test.ReshuffleTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 677, in process
self.do_fn_invoker.invoke_process(windowed_value)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 414, in invoke_process
windowed_value, self.process_method(windowed_value.value))
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
 line 1068, in 
wrapper = lambda x: [fn(x)]
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
 line 115, in _equal
sorted_expected = sorted(expected)
TypeError: unorderable types: list() < InAnyOrder()


> Several tests fail on Python 3 with TypeError: unorderable types: str() < 
> int()
> ---
>
> Key: BEAM-5621
> URL: https://issues.apache.org/jira/browse/BEAM-5621
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ==
> ERROR: test_remove_duplicates 
> (apache_beam.transforms.ptransform_test.PTransformTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 414, in invoke_process
> windowed_value, self.process_method(windowed_value.value))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1068, in 
> wrapper = lambda x: [fn(x)]
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
>  line 115, in _equal
> sorted_expected = sorted(expected)
> TypeError: unorderable types: str() < int()



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5622) Several tests fail on Python 3 with: Runtime type violation detected

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5622:
-

 Summary: Several tests fail on Python 3 with: Runtime type 
violation detected
 Key: BEAM-5622
 URL: https://issues.apache.org/jira/browse/BEAM-5622
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


==
FAIL: test_combine_runtime_type_check_violation_using_decorators 
(apache_beam.transforms.ptransform_test.PTransformTypeCheckTestCase)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
 line 1543, in test_combine_runtime_type_check_violation_using_decorators
"Runtime type violation detected within "
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
 line 911, in assertStartswith
'"%s" does not start with "%s"' % (msg, prefix))
AssertionError: False is not true : "Runtime type violation detected within 
Mul/CombinePerKey: Type-hint for return type violated. Expected an instance of 
, instead found 
25252525252525252525252525252525252525252525252525, an instance of ." does not start with "Runtime type violation detected within 
Mul/CombinePerKey: Type-hint for return type violated. Expected an instance of 
, instead found"

==
FAIL: test_combine_runtime_type_check_violation_using_methods 
(apache_beam.transforms.ptransform_test.PTransformTypeCheckTestCase)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
 line 1597, in test_combine_runtime_type_check_violation_using_methods
"Runtime type violation detected within "
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
 line 911, in assertStartswith
'"%s" does not start with "%s"' % (msg, prefix))
AssertionError: False is not true : "Runtime type violation detected within 
ParDo(SortJoin/KeyWithVoid): Type-hint for argument: 'v' violated. Expected an 
instance of , instead found 0, an instance of . 
[while running 'SortJoin/KeyWithVoid']" does not start with "Runtime type 
violation detected within ParDo(SortJoin/KeyWithVoid): Type-hint for argument: 
'v' violated. Expected an instance of , instead found 0, an 
instance of ."





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5615) Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword argument for this function

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636370#comment-16636370
 ] 

Valentyn Tymofieiev edited comment on BEAM-5615 at 10/3/18 3:01 AM:


Possibly related: https://issues.apache.org/jira/browse/BEAM-5620


was (Author: tvalentyn):
Possibly related: https://issues.apache.org/jira/browse/BEAM-5615

> Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword 
> argument for this function
> -
>
> Key: BEAM-5615
> URL: https://issues.apache.org/jira/browse/BEAM-5615
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_top (apache_beam.transforms.combiners_test.CombineTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners_test.py",
>  line 89, in test_top
> names)  # Note parameter passed to comparator.
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 467, in apply
> label or transform.label)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 477, in apply
> return self.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform.py",
>  line 759, in expand
> return self._fn(pcoll, *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 185, in Of
> TopCombineFn(n, compare, key, reverse), *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1251, in expand
> default_value = combine_fn.apply([], *self.args, **self.kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 623, in apply
> *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 362, in extract_output
> self._sort_buffer(buffer, lt)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 295, in _sort_buffer
> key=self._key_fn)
> TypeError: 'cmp' is an invalid keyword argument for this function



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5621) Several tests fail on Python 3 with TypeError: unorderable types: str() < int()

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636371#comment-16636371
 ] 

Valentyn Tymofieiev commented on BEAM-5621:
---

Possibly related: https://issues.apache.org/jira/browse/BEAM-5615

> Several tests fail on Python 3 with TypeError: unorderable types: str() < 
> int()
> ---
>
> Key: BEAM-5621
> URL: https://issues.apache.org/jira/browse/BEAM-5621
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ==
> ERROR: test_remove_duplicates 
> (apache_beam.transforms.ptransform_test.PTransformTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 414, in invoke_process
> windowed_value, self.process_method(windowed_value.value))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1068, in 
> wrapper = lambda x: [fn(x)]
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
>  line 115, in _equal
> sorted_expected = sorted(expected)
> TypeError: unorderable types: str() < int()



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5615) Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword argument for this function

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636370#comment-16636370
 ] 

Valentyn Tymofieiev commented on BEAM-5615:
---

Possibly related: https://issues.apache.org/jira/browse/BEAM-5615

> Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword 
> argument for this function
> -
>
> Key: BEAM-5615
> URL: https://issues.apache.org/jira/browse/BEAM-5615
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_top (apache_beam.transforms.combiners_test.CombineTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners_test.py",
>  line 89, in test_top
> names)  # Note parameter passed to comparator.
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 467, in apply
> label or transform.label)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 477, in apply
> return self.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform.py",
>  line 759, in expand
> return self._fn(pcoll, *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 185, in Of
> TopCombineFn(n, compare, key, reverse), *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1251, in expand
> default_value = combine_fn.apply([], *self.args, **self.kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 623, in apply
> *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 362, in extract_output
> self._sort_buffer(buffer, lt)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 295, in _sort_buffer
> key=self._key_fn)
> TypeError: 'cmp' is an invalid keyword argument for this function



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5615) Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword argument for this function

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636370#comment-16636370
 ] 

Valentyn Tymofieiev edited comment on BEAM-5615 at 10/3/18 3:01 AM:


Possibly related: https://issues.apache.org/jira/browse/BEAM-5621


was (Author: tvalentyn):
Possibly related: https://issues.apache.org/jira/browse/BEAM-5620

> Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword 
> argument for this function
> -
>
> Key: BEAM-5615
> URL: https://issues.apache.org/jira/browse/BEAM-5615
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-harness
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_top (apache_beam.transforms.combiners_test.CombineTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners_test.py",
>  line 89, in test_top
> names)  # Note parameter passed to comparator.
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 467, in apply
> label or transform.label)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 477, in apply
> return self.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform.py",
>  line 759, in expand
> return self._fn(pcoll, *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 185, in Of
> TopCombineFn(n, compare, key, reverse), *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
>  line 111, in __or__
> return self.pipeline.apply(ptransform, self)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
>  line 513, in apply
> pvalueish_result = self.runner.apply(transform, pvalueish)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 193, in apply
> return m(transform, input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
>  line 199, in apply_PTransform
> return transform.expand(input)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1251, in expand
> default_value = combine_fn.apply([], *self.args, **self.kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 623, in apply
> *args, **kwargs)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 362, in extract_output
> self._sort_buffer(buffer, lt)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
>  line 295, in _sort_buffer
> key=self._key_fn)
> TypeError: 'cmp' is an invalid keyword argument for this function



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5621) Several tests fail on Python 3 with TypeError: unorderable types: str() < int()

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5621:
-

 Summary: Several tests fail on Python 3 with TypeError: 
unorderable types: str() < int()
 Key: BEAM-5621
 URL: https://issues.apache.org/jira/browse/BEAM-5621
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


==
ERROR: test_remove_duplicates 
(apache_beam.transforms.ptransform_test.PTransformTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 677, in process
self.do_fn_invoker.invoke_process(windowed_value)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 414, in invoke_process
windowed_value, self.process_method(windowed_value.value))
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
 line 1068, in 
wrapper = lambda x: [fn(x)]
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
 line 115, in _equal
sorted_expected = sorted(expected)
TypeError: unorderable types: str() < int()




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5620) Some tests use assertItemsEqual method, not available in Python 3

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-5620:
-

Assignee: (was: Ahmet Altay)

> Some tests use assertItemsEqual method, not available in Python 3
> -
>
> Key: BEAM-5620
> URL: https://issues.apache.org/jira/browse/BEAM-5620
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> See: 
> https://github.com/apache/beam/search?q=assertItemsEqual&unscoped_q=assertItemsEqual



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5612) Add tox suites to exercise unit tests using Python3 interpreter with cython, and with gcp dependencies.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-5612:
-

Assignee: (was: Ahmet Altay)

> Add tox suites to exercise unit tests using Python3 interpreter with cython, 
> and with gcp dependencies.
> ---
>
> Key: BEAM-5612
> URL: https://issues.apache.org/jira/browse/BEAM-5612
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5620) Some tests use assertItemsEqual method, not available in Python 3

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5620:
-

 Summary: Some tests use assertItemsEqual method, not available in 
Python 3
 Key: BEAM-5620
 URL: https://issues.apache.org/jira/browse/BEAM-5620
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Ahmet Altay


See: 
https://github.com/apache/beam/search?q=assertItemsEqual&unscoped_q=assertItemsEqual



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5618) Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' and 'EmptySideInput'

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5618:
-

 Summary: Several tests fail on Python 3 with: unsupported operand 
type(s) for +: 'int' and 'EmptySideInput'
 Key: BEAM-5618
 URL: https://issues.apache.org/jira/browse/BEAM-5618
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


ERROR: test_do_with_side_input_as_arg 
(apache_beam.transforms.ptransform_test.PTransformTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 677, in process
self.do_fn_invoker.invoke_process(windowed_value)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 529, in invoke_process
windowed_value, additional_args, additional_kwargs, output_processor)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 598, in _invoke_per_window
windowed_value, self.process_method(*args_for_process))
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py",
 line 135, in 
lambda x, addon: [x + addon], pvalue.AsSingleton(side))
TypeError: unsupported operand type(s) for +: 'int' and 'EmptySideInput'





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-5617) Several SideInput tests fail on Python 3 with failed assert: == []

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-5617:
--
Summary: Several SideInput tests fail on Python 3 with failed assert:  == []  (was: Several SideInput tests fail on Python 3 with 
failed assert:  == [])

> Several SideInput tests fail on Python 3 with failed assert:  list> == []
> 
>
> Key: BEAM-5617
> URL: https://issues.apache.org/jira/browse/BEAM-5617
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ==
> ERROR: test_iterable_side_input 
> (apache_beam.transforms.sideinputs_test.SideInputsTest)
> --
> Traceback (most recent call last):
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 677, in process
> self.do_fn_invoker.invoke_process(windowed_value)
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
>  line 414, in invoke_process
> windowed_value, self.process_method(windowed_value.value))
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
>  line 1068, in 
> wrapper = lambda x: [fn(x)]
>   File 
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
>  line 119, in _equal
> 'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
> apache_beam.testing.util.BeamAssertException: Failed assert: [3, 4, 6, 8] == 
> []



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5617) Several SideInput tests fail on Python 3 with failed assert: == []

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5617:
-

 Summary: Several SideInput tests fail on Python 3 with failed 
assert:  == []
 Key: BEAM-5617
 URL: https://issues.apache.org/jira/browse/BEAM-5617
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


==
ERROR: test_iterable_side_input 
(apache_beam.transforms.sideinputs_test.SideInputsTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 677, in process
self.do_fn_invoker.invoke_process(windowed_value)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 414, in invoke_process
windowed_value, self.process_method(windowed_value.value))
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
 line 1068, in 
wrapper = lambda x: [fn(x)]
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
 line 119, in _equal
'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
apache_beam.testing.util.BeamAssertException: Failed assert: [3, 4, 6, 8] == []




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5616) Several tests fail on Python 3 with Failed assert: [] == [nan]

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5616:
-

 Summary: Several tests fail on Python 3 with  Failed assert: 
[] == [nan]
 Key: BEAM-5616
 URL: https://issues.apache.org/jira/browse/BEAM-5616
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev


==
ERROR: test_global_fanout (apache_beam.transforms.combiners_test.CombineTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 677, in process
self.do_fn_invoker.invoke_process(windowed_value)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py",
 line 414, in invoke_process
windowed_value, self.process_method(windowed_value.value))
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
 line 1068, in 
wrapper = lambda x: [fn(x)]
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/testing/util.py",
 line 119, in _equal
'Failed assert: %r == %r' % (sorted_expected, sorted_actual))
apache_beam.testing.util.BeamAssertException: Failed assert: [49.5] == [nan]





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5615) Several tests fail on Python 3 with TypeError: 'cmp' is an invalid keyword argument for this function

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5615:
-

 Summary: Several tests fail on Python 3 with TypeError: 'cmp' is 
an invalid keyword argument for this function
 Key: BEAM-5615
 URL: https://issues.apache.org/jira/browse/BEAM-5615
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-harness
Reporter: Valentyn Tymofieiev


ERROR: test_top (apache_beam.transforms.combiners_test.CombineTest)
--
Traceback (most recent call last):
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners_test.py",
 line 89, in test_top
names)  # Note parameter passed to comparator.
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
 line 111, in __or__
return self.pipeline.apply(ptransform, self)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
 line 467, in apply
label or transform.label)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
 line 477, in apply
return self.apply(transform, pvalueish)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
 line 513, in apply
pvalueish_result = self.runner.apply(transform, pvalueish)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
 line 193, in apply
return m(transform, input)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
 line 199, in apply_PTransform
return transform.expand(input)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform.py",
 line 759, in expand
return self._fn(pcoll, *args, **kwargs)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
 line 185, in Of
TopCombineFn(n, compare, key, reverse), *args, **kwargs)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pvalue.py",
 line 111, in __or__
return self.pipeline.apply(ptransform, self)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/pipeline.py",
 line 513, in apply
pvalueish_result = self.runner.apply(transform, pvalueish)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
 line 193, in apply
return m(transform, input)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/runner.py",
 line 199, in apply_PTransform
return transform.expand(input)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
 line 1251, in expand
default_value = combine_fn.apply([], *self.args, **self.kwargs)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/core.py",
 line 623, in apply
*args, **kwargs)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
 line 362, in extract_output
self._sort_buffer(buffer, lt)
  File 
"/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/combiners.py",
 line 295, in _sort_buffer
key=self._key_fn)
TypeError: 'cmp' is an invalid keyword argument for this function





--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5315) Finish Python 3 porting for io module

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16636297#comment-16636297
 ] 

Valentyn Tymofieiev commented on BEAM-5315:
---

Note that Datastore dependency currently used in Beam is not 
python3-compatible, so we'll have to skip Datastore tests.

https://issues.apache.org/jira/browse/BEAM-4543

> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Matthias Feys
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5612) Add tox suites to exercise unit tests using Python3 interpreter with cython, and with gcp dependencies.

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5612:
-

 Summary: Add tox suites to exercise unit tests using Python3 
interpreter with cython, and with gcp dependencies.
 Key: BEAM-5612
 URL: https://issues.apache.org/jira/browse/BEAM-5612
 Project: Beam
  Issue Type: Sub-task
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Ahmet Altay






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-5255) Fix over-aggressive division futurization.

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev resolved BEAM-5255.
---
   Resolution: Fixed
Fix Version/s: 2.7.0

> Fix over-aggressive division futurization.
> --
>
> Key: BEAM-5255
> URL: https://issues.apache.org/jira/browse/BEAM-5255
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.6.0
>Reporter: Robert Bradshaw
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> When converting from Python 2 to Python 3, `a / b` becomes `a // b` only for 
> ints, but it is incorrect to do this substitution for floating point 
> division. 
>  
> I noticed this change in the microbenchmarks, but we should do an audit to 
> make sure we haven't broken things elsewhere. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5255) Fix over-aggressive division futurization.

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16629399#comment-16629399
 ] 

Valentyn Tymofieiev commented on BEAM-5255:
---

I did an audit of all  initial futurization  PRs from that series, looking for  
incorrect introduction of integer division (using //) for incorrect implicit 
introduction of float division (via from __future__ import division).

I did not find any regressions besides a regression in microbenchmarks, which 
is now fixed. 

There was one suspicious place, see the lines below, where the underlying types 
of  arguments of division are  not obvious, and we are forcing float division 
where previously an integer division may have been used, for example if  
SourceBundle weights are integers. 

https://github.com/apache/beam/blob/6f6feaaeebfc82302ba83c52d087b06a12a5b119/sdks/python/apache_beam/io/concat_source.py#L142
https://github.com/apache/beam/blob/6f6feaaeebfc82302ba83c52d087b06a12a5b119/sdks/python/apache_beam/io/concat_source.py#L212
https://github.com/apache/beam/blob/6f6feaaeebfc82302ba83c52d087b06a12a5b119/sdks/python/apache_beam/io/concat_source.py#L254

As far as I can tell, we are still doing the right thing, since the goal of 
that computation is ultimately to compute fraction of consumed positions in the 
source, which should be a real number between 0.0 and 1.0.


> Fix over-aggressive division futurization.
> --
>
> Key: BEAM-5255
> URL: https://issues.apache.org/jira/browse/BEAM-5255
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 2.6.0
>Reporter: Robert Bradshaw
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: 2.7.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> When converting from Python 2 to Python 3, `a / b` becomes `a // b` only for 
> ints, but it is incorrect to do this substitution for floating point 
> division. 
>  
> I noticed this change in the microbenchmarks, but we should do an audit to 
> make sure we haven't broken things elsewhere. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5502) Object stager tests are not hermetic

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5502:
-

 Summary: Object stager tests are not hermetic
 Key: BEAM-5502
 URL: https://issues.apache.org/jira/browse/BEAM-5502
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Valentyn Tymofieiev
Assignee: Ahmet Altay


As per discussion in https://github.com/apache/beam/pull/6451, 
test_with_setup_file fails on Python 3, however it does not fail on its own or 
when only running the runners tests. It only fails when running the full test 
suite, so it seems like there is a conflict with another test.

cc [~RobbeSneyders].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5502) Object stager tests are not hermetic

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-5502:
-

Assignee: Valentyn Tymofieiev  (was: Ahmet Altay)

> Object stager tests are not hermetic
> 
>
> Key: BEAM-5502
> URL: https://issues.apache.org/jira/browse/BEAM-5502
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> As per discussion in https://github.com/apache/beam/pull/6451, 
> test_with_setup_file fails on Python 3, however it does not fail on its own 
> or when only running the runners tests. It only fails when running the full 
> test suite, so it seems like there is a conflict with another test.
> cc [~RobbeSneyders].



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-5492) Python Dataflow integration tests should export the pipeline console output to Jenkins Test Result section

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-5492:
-

 Summary: Python Dataflow integration tests should export the 
pipeline console output to Jenkins Test Result section
 Key: BEAM-5492
 URL: https://issues.apache.org/jira/browse/BEAM-5492
 Project: Beam
  Issue Type: New Feature
  Components: testing
Reporter: Valentyn Tymofieiev
Assignee: Jason Kuster


I'm looking at a flake in a postcommit suite that happened last week in Python 
Dataflow integration test (https://issues.apache.org/jira/browse/BEAM-5415), 
and the existing logs are not sufficient to debug the problem. The test suite 
failed, however: 
1. Test Result section in Jenkins postcommit only includes unit tests, and does 
not include any integration tests.
2. The only available information about the failure is buried in the Full 
Console Log and is not very informative (copy-pasted below). 
3. There is a line in the console log pointing to: "XML: 
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/nosetests.xml",
 however as a non-committer I don't have access to Jenkins machines.

This makes rootcausing integration tests failures quite difficult, since we 
cannot even reliably identify an ID of a failed Dataflow job from the test 
output.

It would be helpful to capture and persist in Jenkins the invocation of 
integration test, and all the console output that SDK produces during test 
execution.

Sample available logs from console output.

13:41:54 test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... 
ERROR
13:41:54 
13:41:54 ==
13:41:54 ERROR: test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
13:41:54 --
13:41:54 Traceback (most recent call last):
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
 line 812, in run
13:41:54 test(orig)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 45, in __call__
13:41:54 return self.run(*arg, **kwarg)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 133, in run
13:41:54 self.runTest(result)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 151, in runTest
13:41:54 test(result)
13:41:54   File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
13:41:54 return self.run(*args, **kwds)
13:41:54   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
13:41:54 testMethod()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py",
 line 161, in test_leader_board_it
13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts))
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py",
 line 345, in run
13:41:54 'total_score': 'INTEGER',
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 414, in __exit__
13:41:54 self.run().wait_until_finish()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 394, in run
13:41:54 self.to_runner_api(), self.runner, self._options).run(False)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 407, in run
13:41:54 return self.runner.run_pipeline(self)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
 line 68, in run_pipeline
13:41:54 self.result.cancel()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1167, in cancel
13:41:54 self._update_job()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1078, in _update_job
13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id())
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/

[jira] [Commented] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618264#comment-16618264
 ] 

Valentyn Tymofieiev commented on BEAM-5415:
---

I don't think https://github.com/apache/beam/pull/6403 is a culprit, but we do 
have a flaky test that we need to investigate.

[~huygaa] [~chamikara] [~charleschen]: Do you know how to reliably identify a 
relevant Dataflow job ID for a failed integration test from a stream of Jenkins 
test logs? In this case, for: test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT).
 

> [beam_PostCommit_Python_Verify] test is failing
> ---
>
> Key: BEAM-5415
> URL: https://issues.apache.org/jira/browse/BEAM-5415
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Failing job url: 
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]
> Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618246#comment-16618246
 ] 

Valentyn Tymofieiev commented on BEAM-5415:
---

Actually, after looking again at Jenkins log, I am no longer sure if 13:13:17 
WARNING:root:Timing out on waiting for job 
2018-09-17_13_12_39-8645011813028270201 after 30 seconds is relevant, there are 
a few more of those:


13:02:07 WARNING:root:Timing out on waiting for job 
2018-09-17_12_58_58-3598041514621079800 after 182 seconds
13:03:08 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_57-1488988588266282762 after 603 seconds
13:05:03 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_51-1885862207900782284 after 724 seconds


> [beam_PostCommit_Python_Verify] test is failing
> ---
>
> Key: BEAM-5415
> URL: https://issues.apache.org/jira/browse/BEAM-5415
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Failing job url: 
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]
> Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618246#comment-16618246
 ] 

Valentyn Tymofieiev edited comment on BEAM-5415 at 9/17/18 10:46 PM:
-

Actually, after looking again at Jenkins log, I am no longer sure if "13:13:17 
WARNING:root:Timing out on waiting for job 
2018-09-17_13_12_39-8645011813028270201 after 30 seconds" is relevant, there 
are a few more of those:


13:02:07 WARNING:root:Timing out on waiting for job 
2018-09-17_12_58_58-3598041514621079800 after 182 seconds
13:03:08 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_57-1488988588266282762 after 603 seconds
13:05:03 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_51-1885862207900782284 after 724 seconds



was (Author: tvalentyn):
Actually, after looking again at Jenkins log, I am no longer sure if 13:13:17 
WARNING:root:Timing out on waiting for job 
2018-09-17_13_12_39-8645011813028270201 after 30 seconds is relevant, there are 
a few more of those:


13:02:07 WARNING:root:Timing out on waiting for job 
2018-09-17_12_58_58-3598041514621079800 after 182 seconds
13:03:08 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_57-1488988588266282762 after 603 seconds
13:05:03 WARNING:root:Timing out on waiting for job 
2018-09-17_12_52_51-1885862207900782284 after 724 seconds


> [beam_PostCommit_Python_Verify] test is failing
> ---
>
> Key: BEAM-5415
> URL: https://issues.apache.org/jira/browse/BEAM-5415
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Failing job url: 
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]
> Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618224#comment-16618224
 ] 

Valentyn Tymofieiev commented on BEAM-5415:
---

>From Dataflow worker logs: 
I  finish  
I  Stopping BeamFnMapTaskExecutor, grabbing final metric updates 
I  Stopping BeamFnMapTaskExecutor, grabbing final metric updates 
I  Setting node annotation to enable volume controller attach/detach 
I  Pipeline stage consuming pubsub took 2m55s and default ack deadline is 10s. 
Consider increasing ack deadline for subscription 
projects/apache-beam-testing/subscriptions/psit_subscription_inputa3ed93d9-f6b0-420b-b5ee-cb887e9b51de
 
I  Setting node annotation to enable volume controller attach/detach 
I  Dataflow cancel 844138762903-comp...@developer.gserviceaccount.com 
{"@type":"type.googleapis.com/google.cloud.audit.AuditLog","authenticationInfo":{"principalEmail":"844138762903-comp...@developer.gserviceaccount.com"},"requestMetadata":{"callerIp":"35.225.248.94","requestAttributes":{"time":"2018-09-17T20:18:05.162196575Z","auth":{}},"destinationAttributes":{}},"s…
 Dataflow cancel  844138762903-comp...@developer.gserviceaccount.com 
E  Lost lease for work with id 6186087141253459847 with expiration time: 
1537215545000, now: 1537215486508 
E  Lost lease for work with id 6186087141253459847 with expiration time: 
1537215545000, now: 1537215486508 
I  E0917 20:18:06.508733   193 work_service_client.cc:692] Lost lease for work 
with id 6186087141253459847 with expiration time: 1537215545000, now: 
1537215486508 
W  Unable to update setup work item 4601443631075438240 error: 
INVALID_ARGUMENT: Http(400) Bad Request 


> [beam_PostCommit_Python_Verify] test is failing
> ---
>
> Key: BEAM-5415
> URL: https://issues.apache.org/jira/browse/BEAM-5415
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Failing job url: 
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]
> Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618218#comment-16618218
 ] 

Valentyn Tymofieiev commented on BEAM-5415:
---

*13:13:17* WARNING:root:Timing out on waiting for job 
2018-09-17_13_12_39-8645011813028270201 after 30 seconds

> [beam_PostCommit_Python_Verify] test is failing
> ---
>
> Key: BEAM-5415
> URL: https://issues.apache.org/jira/browse/BEAM-5415
> Project: Beam
>  Issue Type: Bug
>  Components: test-failures
>Reporter: Batkhuyag Batsaikhan
>Assignee: Valentyn Tymofieiev
>Priority: Major
>
> Failing job url: 
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6003/]
> Most likely culprit PR: https://github.com/apache/beam/pull/6403



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618217#comment-16618217
 ] 

Valentyn Tymofieiev edited comment on BEAM-5415 at 9/17/18 10:14 PM:
-

13:41:54 test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... 
ERROR
13:41:54 
13:41:54 ==
13:41:54 ERROR: test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)
13:41:54 --
13:41:54 Traceback (most recent call last):
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
 line 812, in run
13:41:54 test(orig)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 45, in __call__
13:41:54 return self.run(*arg, **kwarg)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 133, in run
13:41:54 self.runTest(result)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 151, in runTest
13:41:54 test(result)
13:41:54   File "/usr/lib/python2.7/unittest/case.py", line 393, in __call__
13:41:54 return self.run(*args, **kwds)
13:41:54   File "/usr/lib/python2.7/unittest/case.py", line 329, in run
13:41:54 testMethod()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py",
 line 161, in test_leader_board_it
13:41:54 self.test_pipeline.get_full_options_as_args(**extra_opts))
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py",
 line 345, in run
13:41:54 'total_score': 'INTEGER',
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 414, in __exit__
13:41:54 self.run().wait_until_finish()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 394, in run
13:41:54 self.to_runner_api(), self.runner, self._options).run(False)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 407, in run
13:41:54 return self.runner.run_pipeline(self)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
 line 68, in run_pipeline
13:41:54 self.result.cancel()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1167, in cancel
13:41:54 self._update_job()
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1078, in _update_job
13:41:54 self._job = self._runner.dataflow_client.get_job(self.job_id())
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py",
 line 184, in wrapper
13:41:54 return fun(*args, **kwargs)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
 line 629, in get_job
13:41:54 response = self._client.projects_locations_jobs.Get(request)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",
 line 659, in Get
13:41:54 config, request, global_params=global_params)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
 line 720, in _RunMethod
13:41:54 http, http_request, **opts)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
 line 346, in MakeRequest
13:41:54 check_response_func=check_response_func)
13:41:54   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/sit

[jira] [Commented] (BEAM-5415) [beam_PostCommit_Python_Verify] test is failing

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16618217#comment-16618217
 ] 

Valentyn Tymofieiev commented on BEAM-5415:
---

*13:41:54* test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT) ... 
ERROR*13:41:54* *13:41:54* 
==*13:41:54*
 ERROR: test_leader_board_it 
(apache_beam.examples.complete.game.leader_board_it_test.LeaderBoardIT)*13:41:54*
 
--*13:41:54*
 Traceback (most recent call last):*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/plugins/multiprocess.py",
 line 812, in run*13:41:54* test(orig)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 45, in __call__*13:41:54* return self.run(*arg, **kwarg)*13:41:54*   
File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 133, in run*13:41:54* self.runTest(result)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/nose/case.py",
 line 151, in runTest*13:41:54* test(result)*13:41:54*   File 
"/usr/lib/python2.7/unittest/case.py", line 393, in __call__*13:41:54* 
return self.run(*args, **kwds)*13:41:54*   File 
"/usr/lib/python2.7/unittest/case.py", line 329, in run*13:41:54* 
testMethod()*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board_it_test.py",
 line 161, in test_leader_board_it*13:41:54* 
self.test_pipeline.get_full_options_as_args(**extra_opts))*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/examples/complete/game/leader_board.py",
 line 345, in run*13:41:54* 'total_score': 'INTEGER',*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 414, in __exit__*13:41:54* self.run().wait_until_finish()*13:41:54*   
File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 394, in run*13:41:54* self.to_runner_api(), self.runner, 
self._options).run(False)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/pipeline.py",
 line 407, in run*13:41:54* return self.runner.run_pipeline(self)*13:41:54* 
  File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/test_dataflow_runner.py",
 line 68, in run_pipeline*13:41:54* self.result.cancel()*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1167, in cancel*13:41:54* self._update_job()*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/dataflow_runner.py",
 line 1078, in _update_job*13:41:54* self._job = 
self._runner.dataflow_client.get_job(self.job_id())*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/utils/retry.py",
 line 184, in wrapper*13:41:54* return fun(*args, **kwargs)*13:41:54*   
File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/apiclient.py",
 line 629, in get_job*13:41:54* response = 
self._client.projects_locations_jobs.Get(request)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py",
 line 659, in Get*13:41:54* config, request, 
global_params=global_params)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/base_api.py",
 line 720, in _RunMethod*13:41:54* http, http_request, **opts)*13:41:54*   
File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/apitools/base/py/http_wrapper.py",
 line 346, in MakeRequest*13:41:54* 
check_response_func=check_response_func)*13:41:54*   File 
"/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Verify/src/sdks/python/build/gradleenv/local/li

[jira] [Commented] (BEAM-5323) Finish Python 3 porting for utils module

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16615074#comment-16615074
 ] 

Valentyn Tymofieiev commented on BEAM-5323:
---

This is now completed with https://github.com/apache/beam/pull/6359?

> Finish Python 3 porting for utils module
> 
>
> Key: BEAM-5323
> URL: https://issues.apache.org/jira/browse/BEAM-5323
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Robbe
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5323) Finish Python 3 porting for utils module

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-5323:
-

Assignee: (was: Ahmet Altay)

> Finish Python 3 porting for utils module
> 
>
> Key: BEAM-5323
> URL: https://issues.apache.org/jira/browse/BEAM-5323
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5320) Finish Python 3 porting for testing module

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-5320:
-

Assignee: (was: Ahmet Altay)

> Finish Python 3 porting for testing module
> --
>
> Key: BEAM-5320
> URL: https://issues.apache.org/jira/browse/BEAM-5320
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-5315) Finish Python 3 porting for io module

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-5315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev reassigned BEAM-5315:
-

Assignee: (was: Ahmet Altay)

> Finish Python 3 porting for io module
> -
>
> Key: BEAM-5315
> URL: https://issues.apache.org/jira/browse/BEAM-5315
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)