Re: Python 3.11 support in Apache Beam

[Anand Inguva via dev]

Thanks Ahmet. I will check them out soon.

On Thu, Apr 13, 2023 at 6:24 PM Ahmet Altay  wrote:

> I forgot to add the link, [1] was meant to be :
> https://docs.python.org/3/whatsnew/3.11.html#faster-cpython
>
> On Thu, Apr 13, 2023 at 10:17 AM Anand Inguva 
> wrote:
>
>> Yes Ahmet. That would be great.
>>
>> There are some load tests defined in the
>> https://github.com/apache/beam/blob/master/.test-infra which could be
>> useful for performance testing of Beam between 3.10 and 3.11. Do you
>> suggest any other tests?
>>
>
> I have not looked at the full list. I do not think we will see much in IO
> bound pipelines, or pipelines that do most of their work with a C extension
> library already. Maybe some of the load tests like pardo load tests?
>
> If feasible, we could convert benchmarks to run on 3.11 and see which ones
> will see a larger improvement.
>
> Also apparently there is a potential regression of using up to 20% more
> memory (
> https://docs.python.org/3/whatsnew/3.11.html#will-cpython-3-11-use-more-memory).
> I wonder if that will negatively impact us. If feasible, it would be useful
> to understand that as well.
>
>
>>
>> On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay  wrote:
>>
>>> Thank you, this is great!
>>>
>>> Python 3.11 announcement had a claim about performance [1]:
>>>
>>> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
>>> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
>>> Linux. Depending on your workload, the overall speedup could be 10-60%."
>>>
>>> Have we measured this in Beam? Are we seeing any benefits? If not, why?
>>> If yes, this would be a cool blog post as well.
>>>
>>> Ahmet
>>>
>>>
>>> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev 
>>> wrote:
>>>
 Python 3.11 support has been merged at
 https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0
 release.

 Please let me know if you have any questions.

 Thanks,
 Anand

 On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Thanks a lot Anand. I'll take a look at the PRs.
>
> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva 
> wrote:
>
>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>> that updates the build dependencies of Apache Beam.
>>
>> Several GCP dependencies needed to be updated as well. I covered them
>> in the PR: https://github.com/apache/beam/pull/24599
>>
>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva 
>> wrote:
>>
>>> Yes, we may need to update all of them
>>> .
>>> I can add more information once I dig into the issue(most likely next
>>> week). I will comment on my findings on the issue:
>>> https://github.com/apache/beam/issues/24569 and will periodically
>>> update this thread.
>>>
>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva 
 wrote:

> Yes, it is related to protobuf only. But I think the update of
> these dependencies are required for Python 3.11 since the newer 
> versions
> have support for Python 3.11 wheels.
>
 Assuming you refer to protobuf. Yes, there are no wheels for 3.10
 for protobuf==3.x.x and that can cause friction.
 https://pypi.org/project/protobuf/3.20.3/#files

 I would probably narrow the problem further to demonstrate which
 stubs are not being generated, and if reason not obvious we can also 
 ask
 for feedback from protobuf maintainers. Also - do we by chance need to
 update some other deps from
 https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
 for this to work?

 Also: tracking issue for protobuf4 support in Beam:
 https://github.com/apache/beam/issues/24569.

 If we use older versions of these packages, then we have to depend
> on installing those packages on Python 3.11 from source distributions 
> which
> is not desired.
>
> I am working parallely on that issue in a different PR
> https://github.com/apache/beam/pull/24599 but I think this issue
> should be a blocker for Python 3.11 update.
>
> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Hi Anand,
>>
>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hi all,
>>>
>>> We are planning to work on adding support for Python 3.11[1] to
>>> Apache Beam Python SDK.
>>>
>>> As part of this effort, we are going to update 

Re: Python 3.11 support in Apache Beam

[Ahmet Altay via dev]

I forgot to add the link, [1] was meant to be :
https://docs.python.org/3/whatsnew/3.11.html#faster-cpython

On Thu, Apr 13, 2023 at 10:17 AM Anand Inguva 
wrote:

> Yes Ahmet. That would be great.
>
> There are some load tests defined in the
> https://github.com/apache/beam/blob/master/.test-infra which could be
> useful for performance testing of Beam between 3.10 and 3.11. Do you
> suggest any other tests?
>

I have not looked at the full list. I do not think we will see much in IO
bound pipelines, or pipelines that do most of their work with a C extension
library already. Maybe some of the load tests like pardo load tests?

If feasible, we could convert benchmarks to run on 3.11 and see which ones
will see a larger improvement.

Also apparently there is a potential regression of using up to 20% more
memory (
https://docs.python.org/3/whatsnew/3.11.html#will-cpython-3-11-use-more-memory).
I wonder if that will negatively impact us. If feasible, it would be useful
to understand that as well.


>
> On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay  wrote:
>
>> Thank you, this is great!
>>
>> Python 3.11 announcement had a claim about performance [1]:
>>
>> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
>> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
>> Linux. Depending on your workload, the overall speedup could be 10-60%."
>>
>> Have we measured this in Beam? Are we seeing any benefits? If not, why?
>> If yes, this would be a cool blog post as well.
>>
>> Ahmet
>>
>>
>> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev 
>> wrote:
>>
>>> Python 3.11 support has been merged at
>>> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0
>>> release.
>>>
>>> Please let me know if you have any questions.
>>>
>>> Thanks,
>>> Anand
>>>
>>> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev 
>>> wrote:
>>>
 Thanks a lot Anand. I'll take a look at the PRs.

 On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva 
 wrote:

> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
> that updates the build dependencies of Apache Beam.
>
> Several GCP dependencies needed to be updated as well. I covered them
> in the PR: https://github.com/apache/beam/pull/24599
>
> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva 
> wrote:
>
>> Yes, we may need to update all of them
>> .
>> I can add more information once I dig into the issue(most likely next
>> week). I will comment on my findings on the issue:
>> https://github.com/apache/beam/issues/24569 and will periodically
>> update this thread.
>>
>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva 
>>> wrote:
>>>
 Yes, it is related to protobuf only. But I think the update of
 these dependencies are required for Python 3.11 since the newer 
 versions
 have support for Python 3.11 wheels.

>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10
>>> for protobuf==3.x.x and that can cause friction.
>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>
>>> I would probably narrow the problem further to demonstrate which
>>> stubs are not being generated, and if reason not obvious we can also ask
>>> for feedback from protobuf maintainers. Also - do we by chance need to
>>> update some other deps from
>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>> for this to work?
>>>
>>> Also: tracking issue for protobuf4 support in Beam:
>>> https://github.com/apache/beam/issues/24569.
>>>
>>> If we use older versions of these packages, then we have to depend
 on installing those packages on Python 3.11 from source distributions 
 which
 is not desired.

 I am working parallely on that issue in a different PR
 https://github.com/apache/beam/pull/24599 but I think this issue
 should be a blocker for Python 3.11 update.

 On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
 valen...@google.com> wrote:

> Hi Anand,
>
> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
> dev@beam.apache.org> wrote:
>
>> Hi all,
>>
>> We are planning to work on adding support for Python 3.11[1] to
>> Apache Beam Python SDK.
>>
>> As part of this effort, we are going to update the python build
>> dependencies defined at [2].
>>
>> Right now, there is an error with the newer version of
>> protobuf(4.21.11). It is not generating _urn files.
>>
>> It can be reproduced by
>>
>

Re: Python 3.11 support in Apache Beam

[Anand Inguva via dev]

Yes Ahmet. That would be great.

There are some load tests defined in the
https://github.com/apache/beam/blob/master/.test-infra which could be
useful for performance testing of Beam between 3.10 and 3.11. Do you
suggest any other tests?

On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay  wrote:

> Thank you, this is great!
>
> Python 3.11 announcement had a claim about performance [1]:
>
> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
> Linux. Depending on your workload, the overall speedup could be 10-60%."
>
> Have we measured this in Beam? Are we seeing any benefits? If not, why? If
> yes, this would be a cool blog post as well.
>
> Ahmet
>
>
> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev 
> wrote:
>
>> Python 3.11 support has been merged at
>> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release.
>>
>> Please let me know if you have any questions.
>>
>> Thanks,
>> Anand
>>
>> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> Thanks a lot Anand. I'll take a look at the PRs.
>>>
>>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva 
>>> wrote:
>>>
 I was able to spin up a PR: https://github.com/apache/beam/pull/24599
 that updates the build dependencies of Apache Beam.

 Several GCP dependencies needed to be updated as well. I covered them
 in the PR: https://github.com/apache/beam/pull/24599

 On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva 
 wrote:

> Yes, we may need to update all of them
> .
> I can add more information once I dig into the issue(most likely next
> week). I will comment on my findings on the issue:
> https://github.com/apache/beam/issues/24569 and will periodically
> update this thread.
>
> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva 
>> wrote:
>>
>>> Yes, it is related to protobuf only. But I think the update of these
>>> dependencies are required for Python 3.11 since the newer versions have
>>> support for Python 3.11 wheels.
>>>
>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>> protobuf==3.x.x and that can cause friction.
>> https://pypi.org/project/protobuf/3.20.3/#files
>>
>> I would probably narrow the problem further to demonstrate which
>> stubs are not being generated, and if reason not obvious we can also ask
>> for feedback from protobuf maintainers. Also - do we by chance need to
>> update some other deps from
>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>> for this to work?
>>
>> Also: tracking issue for protobuf4 support in Beam:
>> https://github.com/apache/beam/issues/24569.
>>
>> If we use older versions of these packages, then we have to depend on
>>> installing those packages on Python 3.11 from source distributions 
>>> which is
>>> not desired.
>>>
>>> I am working parallely on that issue in a different PR
>>> https://github.com/apache/beam/pull/24599 but I think this issue
>>> should be a blocker for Python 3.11 update.
>>>
>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>>> valen...@google.com> wrote:
>>>
 Hi Anand,

 On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
 dev@beam.apache.org> wrote:

> Hi all,
>
> We are planning to work on adding support for Python 3.11[1] to
> Apache Beam Python SDK.
>
> As part of this effort, we are going to update the python build
> dependencies defined at [2].
>
> Right now, there is an error with the newer version of
> protobuf(4.21.11). It is not generating _urn files.
>
> It can be reproduced by
>

> 1. python setup.py sdist
> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
> 3. switch to python interpreter and run import apache_beam as beam
>
 I think the error you are describing is related to protobuf 4, so
 the repro should focus on the portion where generation of stubs is
 happening. Presumably some stubs are not generated on protobuf 4 + 
 Python
 3.11?


>
> will lead to *ImportError: cannot import name
> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'.  
> *Running
> `python gen_protos.py` to forcefully generate files didn't help 
> either.
>
> If you have encountered this error and found a resolution, please
> let me know(that would be super helpful).
>
> I am going to work on this soon. 

Re: Python 3.11 support in Apache Beam

[Ahmet Altay via dev]

Thank you, this is great!

Python 3.11 announcement had a claim about performance [1]:

"CPython 3.11 is an average of 25% faster than CPython 3.10 as measured
with the pyperformance benchmark suite, when compiled with GCC on Ubuntu
Linux. Depending on your workload, the overall speedup could be 10-60%."

Have we measured this in Beam? Are we seeing any benefits? If not, why? If
yes, this would be a cool blog post as well.

Ahmet


On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev 
wrote:

> Python 3.11 support has been merged at
> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release.
>
> Please let me know if you have any questions.
>
> Thanks,
> Anand
>
> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev 
> wrote:
>
>> Thanks a lot Anand. I'll take a look at the PRs.
>>
>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva 
>> wrote:
>>
>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>>> that updates the build dependencies of Apache Beam.
>>>
>>> Several GCP dependencies needed to be updated as well. I covered them in
>>> the PR: https://github.com/apache/beam/pull/24599
>>>
>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva 
>>> wrote:
>>>
 Yes, we may need to update all of them
 .
 I can add more information once I dig into the issue(most likely next
 week). I will comment on my findings on the issue:
 https://github.com/apache/beam/issues/24569 and will periodically
 update this thread.

 On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev 
 wrote:

> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva 
> wrote:
>
>> Yes, it is related to protobuf only. But I think the update of these
>> dependencies are required for Python 3.11 since the newer versions have
>> support for Python 3.11 wheels.
>>
> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
> protobuf==3.x.x and that can cause friction.
> https://pypi.org/project/protobuf/3.20.3/#files
>
> I would probably narrow the problem further to demonstrate which stubs
> are not being generated, and if reason not obvious we can also ask for
> feedback from protobuf maintainers. Also - do we by chance need to
> update some other deps from
> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
> for this to work?
>
> Also: tracking issue for protobuf4 support in Beam:
> https://github.com/apache/beam/issues/24569.
>
> If we use older versions of these packages, then we have to depend on
>> installing those packages on Python 3.11 from source distributions which 
>> is
>> not desired.
>>
>> I am working parallely on that issue in a different PR
>> https://github.com/apache/beam/pull/24599 but I think this issue
>> should be a blocker for Python 3.11 update.
>>
>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
>> valen...@google.com> wrote:
>>
>>> Hi Anand,
>>>
>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>>> dev@beam.apache.org> wrote:
>>>
 Hi all,

 We are planning to work on adding support for Python 3.11[1] to
 Apache Beam Python SDK.

 As part of this effort, we are going to update the python build
 dependencies defined at [2].

 Right now, there is an error with the newer version of
 protobuf(4.21.11). It is not generating _urn files.

 It can be reproduced by

>>>
 1. python setup.py sdist
 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
 3. switch to python interpreter and run import apache_beam as beam

>>> I think the error you are describing is related to protobuf 4, so
>>> the repro should focus on the portion where generation of stubs is
>>> happening. Presumably some stubs are not generated on protobuf 4 + 
>>> Python
>>> 3.11?
>>>
>>>

 will lead to *ImportError: cannot import name
 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'.  
 *Running
 `python gen_protos.py` to forcefully generate files didn't help either.

 If you have encountered this error and found a resolution, please
 let me know(that would be super helpful).

 I am going to work on this soon. Please let me know if you want to
 collaborate.

 Thanks,
 Anand Inguva

 *[1] *https://github.com/apache/beam/pull/24721
 [2]
 https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt

>>>

Re: Python 3.11 support in Apache Beam

[Anand Inguva via dev]

Python 3.11 support has been merged at
https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 release.

Please let me know if you have any questions.

Thanks,
Anand

On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev 
wrote:

> Thanks a lot Anand. I'll take a look at the PRs.
>
> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva 
> wrote:
>
>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
>> that updates the build dependencies of Apache Beam.
>>
>> Several GCP dependencies needed to be updated as well. I covered them in
>> the PR: https://github.com/apache/beam/pull/24599
>>
>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva 
>> wrote:
>>
>>> Yes, we may need to update all of them
>>> .
>>> I can add more information once I dig into the issue(most likely next
>>> week). I will comment on my findings on the issue:
>>> https://github.com/apache/beam/issues/24569 and will periodically
>>> update this thread.
>>>
>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev 
>>> wrote:
>>>
 On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva 
 wrote:

> Yes, it is related to protobuf only. But I think the update of these
> dependencies are required for Python 3.11 since the newer versions have
> support for Python 3.11 wheels.
>
 Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
 protobuf==3.x.x and that can cause friction.
 https://pypi.org/project/protobuf/3.20.3/#files

 I would probably narrow the problem further to demonstrate which stubs
 are not being generated, and if reason not obvious we can also ask for
 feedback from protobuf maintainers. Also - do we by chance need to
 update some other deps from
 https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
 for this to work?

 Also: tracking issue for protobuf4 support in Beam:
 https://github.com/apache/beam/issues/24569.

 If we use older versions of these packages, then we have to depend on
> installing those packages on Python 3.11 from source distributions which 
> is
> not desired.
>
> I am working parallely on that issue in a different PR
> https://github.com/apache/beam/pull/24599 but I think this issue
> should be a blocker for Python 3.11 update.
>
> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev <
> valen...@google.com> wrote:
>
>> Hi Anand,
>>
>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Hi all,
>>>
>>> We are planning to work on adding support for Python 3.11[1] to
>>> Apache Beam Python SDK.
>>>
>>> As part of this effort, we are going to update the python build
>>> dependencies defined at [2].
>>>
>>> Right now, there is an error with the newer version of
>>> protobuf(4.21.11). It is not generating _urn files.
>>>
>>> It can be reproduced by
>>>
>>
>>> 1. python setup.py sdist
>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>> 3. switch to python interpreter and run import apache_beam as beam
>>>
>> I think the error you are describing is related to protobuf 4, so the
>> repro should focus on the portion where generation of stubs is happening.
>> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>>
>>
>>>
>>> will lead to *ImportError: cannot import name
>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'.  *Running
>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>
>>> If you have encountered this error and found a resolution, please
>>> let me know(that would be super helpful).
>>>
>>> I am going to work on this soon. Please let me know if you want to
>>> collaborate.
>>>
>>> Thanks,
>>> Anand Inguva
>>>
>>> *[1] *https://github.com/apache/beam/pull/24721
>>> [2]
>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>
>>

Re: Python 3.11 support in Apache Beam

[Valentyn Tymofieiev via dev]

Thanks a lot Anand. I'll take a look at the PRs.

On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva  wrote:

> I was able to spin up a PR: https://github.com/apache/beam/pull/24599
> that updates the build dependencies of Apache Beam.
>
> Several GCP dependencies needed to be updated as well. I covered them in
> the PR: https://github.com/apache/beam/pull/24599
>
> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva 
> wrote:
>
>> Yes, we may need to update all of them
>> .
>> I can add more information once I dig into the issue(most likely next
>> week). I will comment on my findings on the issue:
>> https://github.com/apache/beam/issues/24569 and will periodically update
>> this thread.
>>
>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva 
>>> wrote:
>>>
 Yes, it is related to protobuf only. But I think the update of these
 dependencies are required for Python 3.11 since the newer versions have
 support for Python 3.11 wheels.

>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>>> protobuf==3.x.x and that can cause friction.
>>> https://pypi.org/project/protobuf/3.20.3/#files
>>>
>>> I would probably narrow the problem further to demonstrate which stubs
>>> are not being generated, and if reason not obvious we can also ask for
>>> feedback from protobuf maintainers. Also - do we by chance need to
>>> update some other deps from
>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>>> for this to work?
>>>
>>> Also: tracking issue for protobuf4 support in Beam:
>>> https://github.com/apache/beam/issues/24569.
>>>
>>> If we use older versions of these packages, then we have to depend on
 installing those packages on Python 3.11 from source distributions which is
 not desired.

 I am working parallely on that issue in a different PR
 https://github.com/apache/beam/pull/24599 but I think this issue
 should be a blocker for Python 3.11 update.

 On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev 
 wrote:

> Hi Anand,
>
> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
> dev@beam.apache.org> wrote:
>
>> Hi all,
>>
>> We are planning to work on adding support for Python 3.11[1] to
>> Apache Beam Python SDK.
>>
>> As part of this effort, we are going to update the python build
>> dependencies defined at [2].
>>
>> Right now, there is an error with the newer version of
>> protobuf(4.21.11). It is not generating _urn files.
>>
>> It can be reproduced by
>>
>
>> 1. python setup.py sdist
>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>> 3. switch to python interpreter and run import apache_beam as beam
>>
> I think the error you are describing is related to protobuf 4, so the
> repro should focus on the portion where generation of stubs is happening.
> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>
>
>>
>> will lead to *ImportError: cannot import name
>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'.  *Running
>> `python gen_protos.py` to forcefully generate files didn't help either.
>>
>> If you have encountered this error and found a resolution, please let
>> me know(that would be super helpful).
>>
>> I am going to work on this soon. Please let me know if you want to
>> collaborate.
>>
>> Thanks,
>> Anand Inguva
>>
>> *[1] *https://github.com/apache/beam/pull/24721
>> [2]
>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>
>

Re: Python 3.11 support in Apache Beam

[Anand Inguva via dev]

I was able to spin up a PR: https://github.com/apache/beam/pull/24599 that
updates the build dependencies of Apache Beam.

Several GCP dependencies needed to be updated as well. I covered them in
the PR: https://github.com/apache/beam/pull/24599

On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva  wrote:

> Yes, we may need to update all of them
> .
> I can add more information once I dig into the issue(most likely next
> week). I will comment on my findings on the issue:
> https://github.com/apache/beam/issues/24569 and will periodically update
> this thread.
>
> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev 
> wrote:
>
>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva 
>> wrote:
>>
>>> Yes, it is related to protobuf only. But I think the update of these
>>> dependencies are required for Python 3.11 since the newer versions have
>>> support for Python 3.11 wheels.
>>>
>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
>> protobuf==3.x.x and that can cause friction.
>> https://pypi.org/project/protobuf/3.20.3/#files
>>
>> I would probably narrow the problem further to demonstrate which stubs
>> are not being generated, and if reason not obvious we can also ask for
>> feedback from protobuf maintainers. Also - do we by chance need to
>> update some other deps from
>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
>> for this to work?
>>
>> Also: tracking issue for protobuf4 support in Beam:
>> https://github.com/apache/beam/issues/24569.
>>
>> If we use older versions of these packages, then we have to depend on
>>> installing those packages on Python 3.11 from source distributions which is
>>> not desired.
>>>
>>> I am working parallely on that issue in a different PR
>>> https://github.com/apache/beam/pull/24599 but I think this issue should
>>> be a blocker for Python 3.11 update.
>>>
>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev 
>>> wrote:
>>>
 Hi Anand,

 On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev <
 dev@beam.apache.org> wrote:

> Hi all,
>
> We are planning to work on adding support for Python 3.11[1] to Apache
> Beam Python SDK.
>
> As part of this effort, we are going to update the python build
> dependencies defined at [2].
>
> Right now, there is an error with the newer version of
> protobuf(4.21.11). It is not generating _urn files.
>
> It can be reproduced by
>

> 1. python setup.py sdist
> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
> 3. switch to python interpreter and run import apache_beam as beam
>
 I think the error you are describing is related to protobuf 4, so the
 repro should focus on the portion where generation of stubs is happening.
 Presumably some stubs are not generated on protobuf 4 + Python 3.11?


>
> will lead to *ImportError: cannot import name
> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'.  *Running
> `python gen_protos.py` to forcefully generate files didn't help either.
>
> If you have encountered this error and found a resolution, please let
> me know(that would be super helpful).
>
> I am going to work on this soon. Please let me know if you want to
> collaborate.
>
> Thanks,
> Anand Inguva
>
> *[1] *https://github.com/apache/beam/pull/24721
> [2]
> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>

Re: Python 3.11 support in Apache Beam

[Anand Inguva via dev]

Yes, we may need to update all of them
.
I can add more information once I dig into the issue(most likely next
week). I will comment on my findings on the issue:
https://github.com/apache/beam/issues/24569 and will periodically update
this thread.

On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev 
wrote:

> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva 
> wrote:
>
>> Yes, it is related to protobuf only. But I think the update of these
>> dependencies are required for Python 3.11 since the newer versions have
>> support for Python 3.11 wheels.
>>
> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
> protobuf==3.x.x and that can cause friction.
> https://pypi.org/project/protobuf/3.20.3/#files
>
> I would probably narrow the problem further to demonstrate which stubs are
> not being generated, and if reason not obvious we can also ask for feedback
> from protobuf maintainers. Also - do we by chance need to update some other
> deps from
> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
> for this to work?
>
> Also: tracking issue for protobuf4 support in Beam:
> https://github.com/apache/beam/issues/24569.
>
> If we use older versions of these packages, then we have to depend on
>> installing those packages on Python 3.11 from source distributions which is
>> not desired.
>>
>> I am working parallely on that issue in a different PR
>> https://github.com/apache/beam/pull/24599 but I think this issue should
>> be a blocker for Python 3.11 update.
>>
>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev 
>> wrote:
>>
>>> Hi Anand,
>>>
>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev 
>>> wrote:
>>>
 Hi all,

 We are planning to work on adding support for Python 3.11[1] to Apache
 Beam Python SDK.

 As part of this effort, we are going to update the python build
 dependencies defined at [2].

 Right now, there is an error with the newer version of
 protobuf(4.21.11). It is not generating _urn files.

 It can be reproduced by

>>>
 1. python setup.py sdist
 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
 3. switch to python interpreter and run import apache_beam as beam

>>> I think the error you are describing is related to protobuf 4, so the
>>> repro should focus on the portion where generation of stubs is happening.
>>> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>>>
>>>

 will lead to *ImportError: cannot import name
 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'.  *Running
 `python gen_protos.py` to forcefully generate files didn't help either.

 If you have encountered this error and found a resolution, please let
 me know(that would be super helpful).

 I am going to work on this soon. Please let me know if you want to
 collaborate.

 Thanks,
 Anand Inguva

 *[1] *https://github.com/apache/beam/pull/24721
 [2]
 https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt

>>>

Re: Python 3.11 support in Apache Beam

[Valentyn Tymofieiev via dev]

On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva  wrote:

> Yes, it is related to protobuf only. But I think the update of these
> dependencies are required for Python 3.11 since the newer versions have
> support for Python 3.11 wheels.
>
Assuming you refer to protobuf. Yes, there are no wheels for 3.10 for
protobuf==3.x.x and that can cause friction.
https://pypi.org/project/protobuf/3.20.3/#files

I would probably narrow the problem further to demonstrate which stubs are
not being generated, and if reason not obvious we can also ask for feedback
from protobuf maintainers. Also - do we by chance need to update some other
deps from
https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33
for this to work?

Also: tracking issue for protobuf4 support in Beam:
https://github.com/apache/beam/issues/24569.

If we use older versions of these packages, then we have to depend on
> installing those packages on Python 3.11 from source distributions which is
> not desired.
>
> I am working parallely on that issue in a different PR
> https://github.com/apache/beam/pull/24599 but I think this issue should
> be a blocker for Python 3.11 update.
>
> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev 
> wrote:
>
>> Hi Anand,
>>
>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev 
>> wrote:
>>
>>> Hi all,
>>>
>>> We are planning to work on adding support for Python 3.11[1] to Apache
>>> Beam Python SDK.
>>>
>>> As part of this effort, we are going to update the python build
>>> dependencies defined at [2].
>>>
>>> Right now, there is an error with the newer version of
>>> protobuf(4.21.11). It is not generating _urn files.
>>>
>>> It can be reproduced by
>>>
>>
>>> 1. python setup.py sdist
>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>>> 3. switch to python interpreter and run import apache_beam as beam
>>>
>> I think the error you are describing is related to protobuf 4, so the
>> repro should focus on the portion where generation of stubs is happening.
>> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>>
>>
>>>
>>> will lead to *ImportError: cannot import name
>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'.  *Running
>>> `python gen_protos.py` to forcefully generate files didn't help either.
>>>
>>> If you have encountered this error and found a resolution, please let me
>>> know(that would be super helpful).
>>>
>>> I am going to work on this soon. Please let me know if you want to
>>> collaborate.
>>>
>>> Thanks,
>>> Anand Inguva
>>>
>>> *[1] *https://github.com/apache/beam/pull/24721
>>> [2]
>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>>
>>

Re: Python 3.11 support in Apache Beam

[Anand Inguva via dev]

Yes, it is related to protobuf only. But I think the update of these
dependencies are required for Python 3.11 since the newer versions have
support for Python 3.11 wheels. If we use older versions of these packages,
then we have to depend on installing those packages on Python 3.11 from
source distributions which is not desired.

I am working parallely on that issue in a different PR
https://github.com/apache/beam/pull/24599 but I think this issue should be
a blocker for Python 3.11 update.

On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev 
wrote:

> Hi Anand,
>
> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev 
> wrote:
>
>> Hi all,
>>
>> We are planning to work on adding support for Python 3.11[1] to Apache
>> Beam Python SDK.
>>
>> As part of this effort, we are going to update the python build
>> dependencies defined at [2].
>>
>> Right now, there is an error with the newer version of protobuf(4.21.11).
>> It is not generating _urn files.
>>
>> It can be reproduced by
>>
>
>> 1. python setup.py sdist
>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
>> 3. switch to python interpreter and run import apache_beam as beam
>>
> I think the error you are describing is related to protobuf 4, so the
> repro should focus on the portion where generation of stubs is happening.
> Presumably some stubs are not generated on protobuf 4 + Python 3.11?
>
>
>>
>> will lead to *ImportError: cannot import name 'beam_runner_api_pb2_urns'
>> from 'apache_beam.portability.api'.  *Running `python gen_protos.py` to
>> forcefully generate files didn't help either.
>>
>> If you have encountered this error and found a resolution, please let me
>> know(that would be super helpful).
>>
>> I am going to work on this soon. Please let me know if you want to
>> collaborate.
>>
>> Thanks,
>> Anand Inguva
>>
>> *[1] *https://github.com/apache/beam/pull/24721
>> [2]
>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>>
>

Re: Python 3.11 support in Apache Beam

[Valentyn Tymofieiev via dev]

Hi Anand,

On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev 
wrote:

> Hi all,
>
> We are planning to work on adding support for Python 3.11[1] to Apache
> Beam Python SDK.
>
> As part of this effort, we are going to update the python build
> dependencies defined at [2].
>
> Right now, there is an error with the newer version of protobuf(4.21.11).
> It is not generating _urn files.
>
> It can be reproduced by
>

> 1. python setup.py sdist
> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
> 3. switch to python interpreter and run import apache_beam as beam
>
I think the error you are describing is related to protobuf 4, so the repro
should focus on the portion where generation of stubs is happening.
Presumably some stubs are not generated on protobuf 4 + Python 3.11?


>
> will lead to *ImportError: cannot import name 'beam_runner_api_pb2_urns'
> from 'apache_beam.portability.api'.  *Running `python gen_protos.py` to
> forcefully generate files didn't help either.
>
> If you have encountered this error and found a resolution, please let me
> know(that would be super helpful).
>
> I am going to work on this soon. Please let me know if you want to
> collaborate.
>
> Thanks,
> Anand Inguva
>
> *[1] *https://github.com/apache/beam/pull/24721
> [2]
> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt
>

Python 3.11 support in Apache Beam

[Anand Inguva via dev]

Hi all,

We are planning to work on adding support for Python 3.11[1] to Apache Beam
Python SDK.

As part of this effort, we are going to update the python build
dependencies defined at [2].

Right now, there is an error with the newer version of protobuf(4.21.11).
It is not generating _urn files.

It can be reproduced by

1. python setup.py sdist
2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz
3. switch to python interpreter and run import apache_beam as beam

will lead to *ImportError: cannot import name 'beam_runner_api_pb2_urns'
from 'apache_beam.portability.api'.  *Running `python gen_protos.py` to
forcefully generate files didn't help either.

If you have encountered this error and found a resolution, please let me
know(that would be super helpful).

I am going to work on this soon. Please let me know if you want to
collaborate.

Thanks,
Anand Inguva

*[1] *https://github.com/apache/beam/pull/24721
[2]
https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt