[
https://issues.apache.org/jira/browse/BEAM-12914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hourany updated BEAM-12914:
------------------------------------
Description:
The order and/or type of opcodes used to build lambda functions have changed in
Python 3.9 (see example below). This causes the any test that relies on
`trivial_inference.infer_return_type` to fail because the function returns a
default of `Any` in cases where the type should normally be inferable. Tests
cases that fail include:
*
[testBuildListUnpack|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/typehints/trivial_inference_test.py#L39]
*
[testBuildTupleUnpack|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/typehints/trivial_inference_test.py#L45]
*
[testBuildTupleUnpackWithCall|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/typehints/trivial_inference_test.py#L305]
*
[test_pardo_type_inference|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/transforms/ptransform_test.py#L2540]
Test failure messages are all variations of:
{noformat}
./sdks/python/apache_beam/transforms/ptransform_test.py::PTransformTypeCheckTestCase::test_pardo_type_inference
Failed: [undefined]AssertionError: <class 'int'> != Any
./sdks/python/apache_beam/typehints/trivial_inference_test.py::TrivialInferenceTest::testBuildListUnpack
Failed: [undefined]AssertionError: List[int] != Any
{noformat}
etc
h3. Example
{code:python}
# Python3.7
In [1]: from dis import dis
In [2]: from apache_beam.typehints import typehints
In [3]: beam_type_hints = [typehints.List[int]]
In [4]: dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
1 0 LOAD_FAST 0 (_list)
2 LOAD_FAST 0 (_list)
4 LOAD_FAST 0 (_list)
6 BUILD_LIST_UNPACK 3
8 LOAD_GLOBAL 0 (beam_type_hints)
10 CALL_FUNCTION 1
12 RETURN_VALUE
{code}
----
Whereas in Python 3.9
{code:python}
# Python 3.9
In [1]: from dis import dis
In [2]: from apache_beam.typehints import typehints
In [3]: beam_type_hints = [typehints.List[int]]
In [4]: dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
<>:1: SyntaxWarning: 'list' object is not callable; perhaps you missed a comma?
<ipython-input-4-6499114c3d61>:1: SyntaxWarning: 'list' object is not callable;
perhaps you missed a comma?
dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
1 0 BUILD_LIST 0
2 LOAD_FAST 0 (_list)
4 LIST_EXTEND 1
6 LOAD_FAST 0 (_list)
8 LIST_EXTEND 1
10 LOAD_FAST 0 (_list)
12 LIST_EXTEND 1
14 LOAD_GLOBAL 0 (beam_type_hints)
16 CALL_FUNCTION 1
18 RETURN_VALUE
{code}
The `SyntaxWarning` is most likely raising because enhancment in
[bpo-15248|https://bugs.python.org/issue15248] misses the fact that the lambda
expression is a function, not a list we're calling, and isn't relevant to the
issue.
was:
The [Build List
Unpack,|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/typehints/trivial_inference_test.py#L39]
[Build Tuple
Unapck|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/typehints/trivial_inference_test.py#L45],
and [Build Tuple Unpack With
Call|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/typehints/trivial_inference_test.py#L305]
tests in [Trivial Inference
Test|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/typehints/trivial_inference_test.py#L33]
fail in Python 3.9 because the opcodes lambda functions use in Python 3.9 have
changed.
{code:python}
# Python3.7
In [1]: from dis import dis
In [2]: from apache_beam.typehints import typehints
In [3]: beam_type_hints = [typehints.List[int]]
In [4]: dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
1 0 LOAD_FAST 0 (_list)
2 LOAD_FAST 0 (_list)
4 LOAD_FAST 0 (_list)
6 BUILD_LIST_UNPACK 3
8 LOAD_GLOBAL 0 (beam_type_hints)
10 CALL_FUNCTION 1
12 RETURN_VALUE
{code}
----
Whereas in Python 3.9
{code:python}
# Python 3.9
In [1]: from dis import dis
In [2]: from apache_beam.typehints import typehints
In [3]: beam_type_hints = [typehints.List[int]]
In [4]: dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
<>:1: SyntaxWarning: 'list' object is not callable; perhaps you missed a comma?
<ipython-input-4-6499114c3d61>:1: SyntaxWarning: 'list' object is not callable;
perhaps you missed a comma?
dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
1 0 BUILD_LIST 0
2 LOAD_FAST 0 (_list)
4 LIST_EXTEND 1
6 LOAD_FAST 0 (_list)
8 LIST_EXTEND 1
10 LOAD_FAST 0 (_list)
12 LIST_EXTEND 1
14 LOAD_GLOBAL 0 (beam_type_hints)
16 CALL_FUNCTION 1
18 RETURN_VALUE
{code}
The `SyntaxWarning` is most likely raising because enhancment in
[bpo-15248|https://bugs.python.org/issue15248] misses the fact that the lambda
expression is a function, not a list we're calling, and isn't relevant to the
issue.
This causes type inference to return `Any` instead of the proper type as shown
by the test failures
{code}
./sdks/python/apache_beam/typehints/trivial_inference_test.py::TrivialInferenceTest::testBuildListUnpack
Failed: [undefined]AssertionError: List[int] != Any
{code}
> Trivial Type Inference Fails On Lambda Functions in Python 3.9
> --------------------------------------------------------------
>
> Key: BEAM-12914
> URL: https://issues.apache.org/jira/browse/BEAM-12914
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Affects Versions: 2.32.0
> Environment: Python 3.9.7
> Linux Mint 20.2, Kernel: 5.4.0-84-generic
> Reporter: Jonathan Hourany
> Priority: P2
>
> The order and/or type of opcodes used to build lambda functions have changed
> in Python 3.9 (see example below). This causes the any test that relies on
> `trivial_inference.infer_return_type` to fail because the function returns a
> default of `Any` in cases where the type should normally be inferable. Tests
> cases that fail include:
> *
> [testBuildListUnpack|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/typehints/trivial_inference_test.py#L39]
>
> *
> [testBuildTupleUnpack|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/typehints/trivial_inference_test.py#L45]
> *
> [testBuildTupleUnpackWithCall|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/typehints/trivial_inference_test.py#L305]
> *
> [test_pardo_type_inference|https://github.com/apache/beam/blob/8072cc0bcfd4eee08a95902e13b9bf1dc2338693/sdks/python/apache_beam/transforms/ptransform_test.py#L2540]
> Test failure messages are all variations of:
> {noformat}
> ./sdks/python/apache_beam/transforms/ptransform_test.py::PTransformTypeCheckTestCase::test_pardo_type_inference
> Failed: [undefined]AssertionError: <class 'int'> != Any
> ./sdks/python/apache_beam/typehints/trivial_inference_test.py::TrivialInferenceTest::testBuildListUnpack
> Failed: [undefined]AssertionError: List[int] != Any
> {noformat}
> etc
> h3. Example
> {code:python}
> # Python3.7
> In [1]: from dis import dis
> In [2]: from apache_beam.typehints import typehints
> In [3]: beam_type_hints = [typehints.List[int]]
> In [4]: dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
> 1 0 LOAD_FAST 0 (_list)
> 2 LOAD_FAST 0 (_list)
> 4 LOAD_FAST 0 (_list)
> 6 BUILD_LIST_UNPACK 3
> 8 LOAD_GLOBAL 0 (beam_type_hints)
> 10 CALL_FUNCTION 1
> 12 RETURN_VALUE
> {code}
> ----
> Whereas in Python 3.9
> {code:python}
> # Python 3.9
> In [1]: from dis import dis
> In [2]: from apache_beam.typehints import typehints
> In [3]: beam_type_hints = [typehints.List[int]]
> In [4]: dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
> <>:1: SyntaxWarning: 'list' object is not callable; perhaps you missed a
> comma?
> <ipython-input-4-6499114c3d61>:1: SyntaxWarning: 'list' object is not
> callable; perhaps you missed a comma?
> dis(lambda _list: [*_list, *_list, *_list](beam_type_hints))
> 1 0 BUILD_LIST 0
> 2 LOAD_FAST 0 (_list)
> 4 LIST_EXTEND 1
> 6 LOAD_FAST 0 (_list)
> 8 LIST_EXTEND 1
> 10 LOAD_FAST 0 (_list)
> 12 LIST_EXTEND 1
> 14 LOAD_GLOBAL 0 (beam_type_hints)
> 16 CALL_FUNCTION 1
> 18 RETURN_VALUE
> {code}
> The `SyntaxWarning` is most likely raising because enhancment in
> [bpo-15248|https://bugs.python.org/issue15248] misses the fact that the
> lambda expression is a function, not a list we're calling, and isn't relevant
> to the issue.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
