Thanks Ahmet. I will check them out soon. On Thu, Apr 13, 2023 at 6:24 PM Ahmet Altay <al...@google.com> wrote:
> I forgot to add the link, [1] was meant to be : > https://docs.python.org/3/whatsnew/3.11.html#faster-cpython > > On Thu, Apr 13, 2023 at 10:17 AM Anand Inguva <ananding...@google.com> > wrote: > >> Yes Ahmet. That would be great. >> >> There are some load tests defined in the >> https://github.com/apache/beam/blob/master/.test-infra which could be >> useful for performance testing of Beam between 3.10 and 3.11. Do you >> suggest any other tests? >> > > I have not looked at the full list. I do not think we will see much in IO > bound pipelines, or pipelines that do most of their work with a C extension > library already. Maybe some of the load tests like pardo load tests? > > If feasible, we could convert benchmarks to run on 3.11 and see which ones > will see a larger improvement. > > Also apparently there is a potential regression of using up to 20% more > memory ( > https://docs.python.org/3/whatsnew/3.11.html#will-cpython-3-11-use-more-memory). > I wonder if that will negatively impact us. If feasible, it would be useful > to understand that as well. > > >> >> On Wed, Apr 12, 2023 at 8:04 PM Ahmet Altay <al...@google.com> wrote: >> >>> Thank you, this is great! >>> >>> Python 3.11 announcement had a claim about performance [1]: >>> >>> "CPython 3.11 is an average of 25% faster than CPython 3.10 as measured >>> with the pyperformance benchmark suite, when compiled with GCC on Ubuntu >>> Linux. Depending on your workload, the overall speedup could be 10-60%." >>> >>> Have we measured this in Beam? Are we seeing any benefits? If not, why? >>> If yes, this would be a cool blog post as well. >>> >>> Ahmet >>> >>> >>> On Wed, Apr 5, 2023 at 1:12 PM Anand Inguva via dev <dev@beam.apache.org> >>> wrote: >>> >>>> Python 3.11 support has been merged at >>>> https://github.com/apache/beam/pull/26121 targeting Beam 2.47.0 >>>> release. >>>> >>>> Please let me know if you have any questions. >>>> >>>> Thanks, >>>> Anand >>>> >>>> On Tue, Feb 21, 2023 at 6:04 PM Valentyn Tymofieiev < >>>> valen...@google.com> wrote: >>>> >>>>> Thanks a lot Anand. I'll take a look at the PRs. >>>>> >>>>> On Tue, Feb 21, 2023 at 1:56 PM Anand Inguva <ananding...@google.com> >>>>> wrote: >>>>> >>>>>> I was able to spin up a PR: https://github.com/apache/beam/pull/24599 >>>>>> that updates the build dependencies of Apache Beam. >>>>>> >>>>>> Several GCP dependencies needed to be updated as well. I covered them >>>>>> in the PR: https://github.com/apache/beam/pull/24599 >>>>>> >>>>>> On Thu, Feb 9, 2023 at 3:29 PM Anand Inguva <ananding...@google.com> >>>>>> wrote: >>>>>> >>>>>>> Yes, we may need to update all of them >>>>>>> <https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33>. >>>>>>> I can add more information once I dig into the issue(most likely next >>>>>>> week). I will comment on my findings on the issue: >>>>>>> https://github.com/apache/beam/issues/24569 and will periodically >>>>>>> update this thread. >>>>>>> >>>>>>> On Tue, Feb 7, 2023 at 5:47 PM Valentyn Tymofieiev < >>>>>>> valen...@google.com> wrote: >>>>>>> >>>>>>>> On Tue, Feb 7, 2023 at 2:35 PM Anand Inguva <ananding...@google.com> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Yes, it is related to protobuf only. But I think the update of >>>>>>>>> these dependencies are required for Python 3.11 since the newer >>>>>>>>> versions >>>>>>>>> have support for Python 3.11 wheels. >>>>>>>>> >>>>>>>> Assuming you refer to protobuf. Yes, there are no wheels for 3.10 >>>>>>>> for protobuf==3.x.x and that can cause friction. >>>>>>>> https://pypi.org/project/protobuf/3.20.3/#files >>>>>>>> >>>>>>>> I would probably narrow the problem further to demonstrate which >>>>>>>> stubs are not being generated, and if reason not obvious we can also >>>>>>>> ask >>>>>>>> for feedback from protobuf maintainers. Also - do we by chance need to >>>>>>>> update some other deps from >>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt#L28-L33 >>>>>>>> for this to work? >>>>>>>> >>>>>>>> Also: tracking issue for protobuf4 support in Beam: >>>>>>>> https://github.com/apache/beam/issues/24569. >>>>>>>> >>>>>>>> If we use older versions of these packages, then we have to depend >>>>>>>>> on installing those packages on Python 3.11 from source distributions >>>>>>>>> which >>>>>>>>> is not desired. >>>>>>>>> >>>>>>>>> I am working parallely on that issue in a different PR >>>>>>>>> https://github.com/apache/beam/pull/24599 but I think this issue >>>>>>>>> should be a blocker for Python 3.11 update. >>>>>>>>> >>>>>>>>> On Tue, Feb 7, 2023 at 5:25 PM Valentyn Tymofieiev < >>>>>>>>> valen...@google.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Anand, >>>>>>>>>> >>>>>>>>>> On Tue, Feb 7, 2023 at 1:35 PM Anand Inguva via dev < >>>>>>>>>> dev@beam.apache.org> wrote: >>>>>>>>>> >>>>>>>>>>> Hi all, >>>>>>>>>>> >>>>>>>>>>> We are planning to work on adding support for Python 3.11[1] to >>>>>>>>>>> Apache Beam Python SDK. >>>>>>>>>>> >>>>>>>>>>> As part of this effort, we are going to update the python build >>>>>>>>>>> dependencies defined at [2]. >>>>>>>>>>> >>>>>>>>>>> Right now, there is an error with the newer version of >>>>>>>>>>> protobuf(4.21.11). It is not generating _urn files. >>>>>>>>>>> >>>>>>>>>>> It can be reproduced by >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> 1. python setup.py sdist >>>>>>>>>>> 2. pip install dist/apache-beam-x.xx.x.dev0.tar.gz >>>>>>>>>>> 3. switch to python interpreter and run import apache_beam as >>>>>>>>>>> beam >>>>>>>>>>> >>>>>>>>>> I think the error you are describing is related to protobuf 4, so >>>>>>>>>> the repro should focus on the portion where generation of stubs is >>>>>>>>>> happening. Presumably some stubs are not generated on protobuf 4 + >>>>>>>>>> Python >>>>>>>>>> 3.11? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> will lead to *ImportError: cannot import name >>>>>>>>>>> 'beam_runner_api_pb2_urns' from 'apache_beam.portability.api'. >>>>>>>>>>> *Running >>>>>>>>>>> `python gen_protos.py` to forcefully generate files didn't help >>>>>>>>>>> either. >>>>>>>>>>> >>>>>>>>>>> If you have encountered this error and found a resolution, >>>>>>>>>>> please let me know(that would be super helpful). >>>>>>>>>>> >>>>>>>>>>> I am going to work on this soon. Please let me know if you want >>>>>>>>>>> to collaborate. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Anand Inguva >>>>>>>>>>> >>>>>>>>>>> *[1] *https://github.com/apache/beam/pull/24721 >>>>>>>>>>> [2] >>>>>>>>>>> https://github.com/apache/beam/blob/master/sdks/python/build-requirements.txt >>>>>>>>>>> >>>>>>>>>>
