[jira] [Resolved] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'
[
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh Kumar resolved BEAM-5627.
Resolution: Fixed
Fix Version/s: Not applicable
> Several IO tests fail in Python 3 when accessing a temporary file with
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 120, in test_split_at_fraction_exhaustive
> source = self._create_source(data)
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 43, in _create_source
> source = LineSource(self._create_file_with_data(data))
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 35, in _create_file_with_data
> f.write(line + '\n')
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
> line 622, in func_wrapper
> return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
> ERROR: test_file_sink_writing
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 121, in
> test_file_sink_writing
> init_token, writer_results = self._common_init(sink)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
> writer1 = sink.open_writer(init_token, '1')
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/options/value_provider.py", line 133, in _f
> return fnc(self, *args, **kwargs)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 385, in __init__
> self.temp_handle = self.sink.open(temp_shard_path)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 82, in open
> file_handle.write('[start]')
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'
[
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16647144#comment-16647144
]
Rakesh Kumar commented on BEAM-5627:
[~ruoyun] thanks for fixing this.
> Several IO tests fail in Python 3 when accessing a temporary file with
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> ERROR: test_split_at_fraction_exhaustive
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 120, in test_split_at_fraction_exhaustive
> source = self._create_source(data)
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 43, in _create_source
> source = LineSource(self._create_file_with_data(data))
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 35, in _create_file_with_data
> f.write(line + '\n')
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
> line 622, in func_wrapper
> return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
> ERROR: test_file_sink_writing
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 121, in
> test_file_sink_writing
> init_token, writer_results = self._common_init(sink)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
> writer1 = sink.open_writer(init_token, '1')
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/options/value_provider.py", line 133, in _f
> return fnc(self, *args, **kwargs)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 385, in __init__
> self.temp_handle = self.sink.open(temp_shard_path)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 82, in open
> file_handle.write('[start]')
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'
[
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh Kumar reassigned BEAM-5627:
--
Assignee: Rakesh Kumar
> Several IO tests fail in Python 3 when accessing a temporary file with
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Rakesh Kumar
>Priority: Major
>
> ERROR: test_split_at_fraction_exhaustive
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 120, in test_split_at_fraction_exhaustive
> source = self._create_source(data)
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 43, in _create_source
> source = LineSource(self._create_file_with_data(data))
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 35, in _create_file_with_data
> f.write(line + '\n')
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
> line 622, in func_wrapper
> return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
> ERROR: test_file_sink_writing
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 121, in
> test_file_sink_writing
> init_token, writer_results = self._common_init(sink)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
> writer1 = sink.open_writer(init_token, '1')
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/options/value_provider.py", line 133, in _f
> return fnc(self, *args, **kwargs)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 385, in __init__
> self.temp_handle = self.sink.open(temp_shard_path)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 82, in open
> file_handle.write('[start]')
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'
[
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640777#comment-16640777
]
Rakesh Kumar commented on BEAM-5627:
[~tvalentyn] can you also assign this Jira ticket to me. Currently, I am not
able to assign this to me.
> Several IO tests fail in Python 3 when accessing a temporary file with
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_split_at_fraction_exhaustive
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 120, in test_split_at_fraction_exhaustive
> source = self._create_source(data)
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 43, in _create_source
> source = LineSource(self._create_file_with_data(data))
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 35, in _create_file_with_data
> f.write(line + '\n')
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
> line 622, in func_wrapper
> return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
> ERROR: test_file_sink_writing
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 121, in
> test_file_sink_writing
> init_token, writer_results = self._common_init(sink)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
> writer1 = sink.open_writer(init_token, '1')
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/options/value_provider.py", line 133, in _f
> return fnc(self, *args, **kwargs)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 385, in __init__
> self.temp_handle = self.sink.open(temp_shard_path)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 82, in open
> file_handle.write('[start]')
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'
[
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640578#comment-16640578
]
Rakesh Kumar edited comment on BEAM-5627 at 10/6/18 6:11 AM:
-
[~tvalentyn] can you also add the command and tips to reproduce it?
was (Author: rakeshkumar):
[~tvalentyn] can you also add the command to reproduce it?
> Several IO tests fail in Python 3 when accessing a temporary file with
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_split_at_fraction_exhaustive
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 120, in test_split_at_fraction_exhaustive
> source = self._create_source(data)
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 43, in _create_source
> source = LineSource(self._create_file_with_data(data))
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 35, in _create_file_with_data
> f.write(line + '\n')
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
> line 622, in func_wrapper
> return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
> ERROR: test_file_sink_writing
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 121, in
> test_file_sink_writing
> init_token, writer_results = self._common_init(sink)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
> writer1 = sink.open_writer(init_token, '1')
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/options/value_provider.py", line 133, in _f
> return fnc(self, *args, **kwargs)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 385, in __init__
> self.temp_handle = self.sink.open(temp_shard_path)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 82, in open
> file_handle.write('[start]')
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5627) Several IO tests fail in Python 3 when accessing a temporary file with TypeError: a bytes-like object is required, not 'str'
[
https://issues.apache.org/jira/browse/BEAM-5627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640578#comment-16640578
]
Rakesh Kumar commented on BEAM-5627:
[~tvalentyn] can you also add the command to reproduce it?
> Several IO tests fail in Python 3 when accessing a temporary file with
> TypeError: a bytes-like object is required, not 'str'
> --
>
> Key: BEAM-5627
> URL: https://issues.apache.org/jira/browse/BEAM-5627
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: Major
>
> ERROR: test_split_at_fraction_exhaustive
> (apache_beam.io.source_test_utils_test.SourceTestUtilsTest)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 120, in test_split_at_fraction_exhaustive
> source = self._create_source(data)
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 43, in _create_source
> source = LineSource(self._create_file_with_data(data))
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/io/source_test_utils_test.py",
> line 35, in _create_file_with_data
> f.write(line + '\n')
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/target/.tox/py3/lib/python3.5/tempfile.py",
> line 622, in func_wrapper
> return func(*args, **kwargs)
> TypeError: a bytes-like object is required, not 'str'
> Also similar:
> ==
> ERROR: test_file_sink_writing
> (apache_beam.io.filebasedsink_test.TestFileBasedSink)
> --
> Traceback (most recent call last):
>File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 121, in
> test_file_sink_writing
> init_token, writer_results = self._common_init(sink)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 103, in _common_init
> writer1 = sink.open_writer(init_token, '1')
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/options/value_provider.py", line 133, in _f
> return fnc(self, *args, **kwargs)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 185, in open_writer
> return FileBasedSinkWriter(self, os.path.join(init_result, uid) + suffix)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink.py", line 385, in __init__
> self.temp_handle = self.sink.open(temp_shard_path)
> File
> "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/
>apache_beam/io/filebasedsink_test.py", line 82, in open
> file_handle.write('[start]')
> TypeError: a bytes-like object is required, not 'str'
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5618) Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' and 'EmptySideInput'
[ https://issues.apache.org/jira/browse/BEAM-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640573#comment-16640573 ] Rakesh Kumar edited comment on BEAM-5618 at 10/6/18 6:00 AM: - [~tvalentyn]: can you share the test command to reproduce this error? was (Author: rakeshkumar): [~tvalentyn]: can you share the command to reproduce this error? > Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' > and 'EmptySideInput' > -- > > Key: BEAM-5618 > URL: https://issues.apache.org/jira/browse/BEAM-5618 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Major > > ERROR: test_do_with_side_input_as_arg > (apache_beam.transforms.ptransform_test.PTransformTest) > -- > Traceback (most recent call last): > File > "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py", > line 677, in process > self.do_fn_invoker.invoke_process(windowed_value) > File > "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py", > line 529, in invoke_process > windowed_value, additional_args, additional_kwargs, output_processor) > File > "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py", > line 598, in _invoke_per_window > windowed_value, self.process_method(*args_for_process)) > File > "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py", > line 135, in > lambda x, addon: [x + addon], pvalue.AsSingleton(side)) > TypeError: unsupported operand type(s) for +: 'int' and 'EmptySideInput' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5618) Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' and 'EmptySideInput'
[ https://issues.apache.org/jira/browse/BEAM-5618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16640573#comment-16640573 ] Rakesh Kumar commented on BEAM-5618: [~tvalentyn]: can you share the command to reproduce this error? > Several tests fail on Python 3 with: unsupported operand type(s) for +: 'int' > and 'EmptySideInput' > -- > > Key: BEAM-5618 > URL: https://issues.apache.org/jira/browse/BEAM-5618 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Valentyn Tymofieiev >Priority: Major > > ERROR: test_do_with_side_input_as_arg > (apache_beam.transforms.ptransform_test.PTransformTest) > -- > Traceback (most recent call last): > File > "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py", > line 677, in process > self.do_fn_invoker.invoke_process(windowed_value) > File > "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py", > line 529, in invoke_process > windowed_value, additional_args, additional_kwargs, output_processor) > File > "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/runners/common.py", > line 598, in _invoke_per_window > windowed_value, self.process_method(*args_for_process)) > File > "/usr/local/google/home/valentyn/projects/beam/clean_head/beam/sdks/python/apache_beam/transforms/ptransform_test.py", > line 135, in > lambda x, addon: [x + addon], pvalue.AsSingleton(side)) > TypeError: unsupported operand type(s) for +: 'int' and 'EmptySideInput' -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5497) Provide support for Gevent
Rakesh Kumar created BEAM-5497:
--
Summary: Provide support for Gevent
Key: BEAM-5497
URL: https://issues.apache.org/jira/browse/BEAM-5497
Project: Beam
Issue Type: Improvement
Components: runner-flink
Reporter: Rakesh Kumar
Assignee: Aljoscha Krettek
[Gevent|http://www.gevent.org/] is basically used to make parallel network
calls. We are using gevent in one of the transformation methods to call
internal services. Before using the gevent we also patch it as mentioned
[here|https://github.com/grpc/grpc/issues/4629#issuecomment-376962677].
The transformation method is making multiple network call in parallel. Here is
the code snippet:
{code}
/__init__.py
import gevent.monkey
gevent.monkey.patch_all()
/transform.py
from gevent import Greenlet
from gevent import joinall
def filter_out_invalid_users(events):
key, user_id_data_pairs = events
user_ids = [user_id for user_id, data in user_id_data_pairs]
jobs = []
id_chunks = utils.chunk_list_evenly(user_ids, BATCH_SIZE)
for id_chunk in id_chunks:
jobs.append(Greenlet.spawn(_call_users_service, # _call_user_service_
method is making the network call.
list(id_chunk)))
"""
Here we increase the timeout based on the number of greenlets we are
running, to account for yielding
among greenlets
"""
join_timeout = GREENLET_TIMEOUT + len(jobs) * GREENLET_TIMEOUT * 0.1
joinall(jobs, timeout=join_timeout)
successful_jobs = [job for job in jobs if job.successful()]
valid_user_ids = []
for job in successful_jobs:
network_response = job.get()
valid_user_ids.append(network_response.user_id)
yield valid_user_ids
def _call_users_service(user_ids):
# make network call and return response
..
..
return network_response
{code}
This allows pipelines to start and partially run. However the tasks produce a
stream of gevent exceptions and do not make any progress.:
{code}
Exception greenlet.error: error('cannot switch to a different thread',) in
'grpc._cython.cygrpc.run_loop' ignored
Traceback (most recent call last):
File "src/gevent/event.py", line 240, in gevent._event.Event.wait
File "src/gevent/event.py", line 140, in gevent._event._AbstractLinkable._wait
File "src/gevent/event.py", line 117, in
gevent._event._AbstractLinkable._wait_core
File "src/gevent/event.py", line 119, in
gevent._event._AbstractLinkable._wait_core
File "src/gevent/_greenlet_primitives.py", line 59, in
gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 59, in
gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/_greenlet_primitives.py", line 63, in
gevent.__greenlet_primitives.SwitchOutGreenletWithLoop.switch
File "src/gevent/__greenlet_primitives.pxd", line 35, in
gevent.__greenlet_primitives._greenlet_switch
greenlet.error: cannot switch to a different thread
{code}
The alternative approach is to use multiprocess module as shown
[here|https://github.com/apache/beam/blob/59c85b44d156bb7b4462d80fcb5591f860235708/sdks/python/apache_beam/internal/util.py#L117]
&
[here|https://github.com/apache/beam/blob/7bd73a51b670755bbb19e1291003722d5d16bdc5/sdks/python/apache_beam/io/filebasedsink.py#L313].
Gevent is lightweight and good for parallelizing IO/Network bound jobs and it
can efficiently use resources and it can scale well in case heavy load.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
