Hi Xingbo, > So I guess you want to run ci tests on the m1 > environment, but the current version of miniconda cannot meet this > requirement, so there is a pre-step that must drop python 3.7?
In short yes + local wheel build fix on the M1. In a bit more detailed please let me elaborate. You're right about the arm64 build creation, Flink is doing that for quite some time already and wheels are running fine. There are gaps however. >From user perspective: * we're building wheels for ARM64 but we're not executing any python tests on ARM64 but only on X64. I think it's a must to test platform packages on the related platform since not all python packages are ARM compatible or must be built/used in a different way. >From dev perspective: * Local MacOs M1 wheel or python source distro builds are simply failing with error messages. Just to give an example GRPC pip install is failing and require the following changes: https://github.com/apache/flink/pull/23359/files#diff-c4c4916cd6ab7f87a4b154ef9603f9ba6d3b74a00096be9adacb7df363b5efcfR28-R32 I can't really imagine how Azure is able to build MacOS ARM64 wheels... * Python tests are not running on any kind of ARM CPUs which makes any bug hunting/development hard or nearly impossible In order to give ARM support with local development + test execution possibility new miniconda is needed. New miniconda runs 3.8+. There is already a PR which is under development: https://github.com/apache/flink/pull/23359 This fills all the mentioned gaps: local test/development/test execution on ARM CPUs + the previously mentioned 3+ years not actively supported python eliminated. Hope this helps. G On Wed, Sep 6, 2023 at 9:00 AM Xingbo Huang <hxbks...@gmail.com> wrote: > Hi Gyala and Gabor, > > Pyflink has provided arm64 wheel packages for Apple silicon since 1.16[1]. > The use of Miniconda is only related to ci testing and packaging on linux > platform, and building mac platform wheels are dependent on > cibuildwheel[2]. So I guess you want to run ci tests on the m1 > environment, but the current version of miniconda cannot meet this > requirement, so there is a pre-step that must drop python 3.7? > > [1] https://pypi.org/project/apache-flink/1.16.2/#files > [2] > > https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-python-wheels.yml#L30 > > Best, > Xingbo > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2023年9月6日周三 14:36写道： > > > Hi Xingbo, > > > > *Constraint:* > > I personally not found any miniconda version which provides arm64 support > > together with python 3.7. > > [image: image.png] > > > > At the moment I think new platform support means 3.7 drop. > > > > I fully to agree with Gyula, if we start now maybe we can release it in > > half a year however *3.7 active support already ended in 27 Jun 2020*. > > At the moment any python development/test execution on MacOS M1 is just > > not working as-is just like any kind of python test execution on any ARM > > CPU. > > > > Gains: > > * We can release a working version in half a year hopefully and not > > shifting support to 1+ year > > * MacOS M1 local development would work finally which is essential for > > user engagement > > * It would be possible to execute python tests on ARM64 machines > > * We can shake up the python development story because it's not the most > > loved area > > > > BR, > > G > > > > > > On Wed, Sep 6, 2023 at 8:06 AM Gyula Fóra <gyula.f...@gmail.com> wrote: > > > >> Hi Xingbo! > >> > >> I think we have to analyze what we gain by dropping 3.7 and upgrading > to a > >> miniconda version with a multiarch support. > >> > >> If this is what we need to get Apple silicon support then I think it's > >> worth doing it already in 1.19. Keep in mind that 1.18 is not even > >> released > >> yet so if we delay this to 1.20 that is basically 1 year from now. > >> Making this change can increase the adoption instantly if we enable new > >> platforms. > >> > >> Cheers, > >> Gyula > >> > >> On Wed, Sep 6, 2023 at 4:46 AM Xingbo Huang <hxbks...@gmail.com> wrote: > >> > >> > Hi Gabor, > >> > > >> > Thanks for bringing this up. In my opinion, it is a bit aggressive to > >> > directly drop Python 3.7 in 1.19. Python 3.7 is still used a lot[1], > >> and as > >> > far as I know, many Pyflink users are still using python 3.7 as their > >> > default interpreter. I prefer to deprecate Python 3.7 in 1.19 just > like > >> we > >> > deprecated Python 3.6 in 1.16[2] and dropped Python 3.6 in 1.17[3]. > >> > > >> > For the support of Python 3.11, I am very supportive of the > >> implementation > >> > in 1.19 (many users have this appeal, and I originally wanted to > >> support it > >> > in 1.18). > >> > > >> > Regarding the miniconda upgrade, I tend to upgrade miniconda to the > >> latest > >> > version that can support python 3.7 to 3.11 at the same time. > >> > > >> > [1] https://w3techs.com/technologies/history_details/pl-python/3 > >> > [2] https://issues.apache.org/jira/browse/FLINK-28195 > >> > [3] https://issues.apache.org/jira/browse/FLINK-27929 > >> > > >> > Best, > >> > Xingbo > >> > > >> > Jing Ge <j...@ververica.com.invalid> 于2023年9月5日周二 04:10写道： > >> > > >> > > +1 > >> > > > >> > > @Dian should we add support of python 3.11 > >> > > > >> > > Best regards, > >> > > Jing > >> > > > >> > > On Mon, Sep 4, 2023 at 3:39 PM Gabor Somogyi < > >> gabor.g.somo...@gmail.com> > >> > > wrote: > >> > > > >> > > > Thanks for all the responses! > >> > > > > >> > > > Based on the suggestions I've created the following jiras and > >> started > >> > to > >> > > > work on them: > >> > > > * https://issues.apache.org/jira/browse/FLINK-33029 > >> > > > * https://issues.apache.org/jira/browse/FLINK-33030 > >> > > > > >> > > > The reason why I've split them is to separate the concerns and > >> reduce > >> > the > >> > > > amount of code in a PR to help reviewers. > >> > > > > >> > > > BR, > >> > > > G > >> > > > > >> > > > > >> > > > On Mon, Sep 4, 2023 at 12:57 PM Sergey Nuyanzin < > >> snuyan...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > +1, > >> > > > > Thanks for looking into this. > >> > > > > > >> > > > > On Mon, Sep 4, 2023 at 8:38 AM Gyula Fóra <gyula.f...@gmail.com > > > >> > > wrote: > >> > > > > > >> > > > > > +1 > >> > > > > > Thanks for looking into this. > >> > > > > > > >> > > > > > Gyula > >> > > > > > > >> > > > > > On Mon, Sep 4, 2023 at 8:26 AM Matthias Pohl < > >> > matthias.p...@aiven.io > >> > > > > > .invalid> > >> > > > > > wrote: > >> > > > > > > >> > > > > > > Thanks Gabor for looking into it. It sounds reasonable to me > >> as > >> > > well. > >> > > > > > > > >> > > > > > > +1 > >> > > > > > > > >> > > > > > > On Sun, Sep 3, 2023 at 5:44 PM Márton Balassi < > >> > > > > balassi.mar...@gmail.com> > >> > > > > > > wrote: > >> > > > > > > > >> > > > > > > > Hi Gabor, > >> > > > > > > > > >> > > > > > > > Thanks for bringing this up. Similarly to when we dropped > >> > Python > >> > > > 3.6 > >> > > > > > due > >> > > > > > > to > >> > > > > > > > its end of life (and added 3.10) in Flink 1.17 [1,2], it > >> makes > >> > > > sense > >> > > > > to > >> > > > > > > > proceed to remove 3.7 and add 3.11 instead. > >> > > > > > > > > >> > > > > > > > +1. > >> > > > > > > > > >> > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-27929 > >> > > > > > > > [2] https://github.com/apache/flink/pull/21699 > >> > > > > > > > > >> > > > > > > > Best, > >> > > > > > > > Marton > >> > > > > > > > > >> > > > > > > > On Fri, Sep 1, 2023 at 10:39 AM Gabor Somogyi < > >> > > > > > gabor.g.somo...@gmail.com > >> > > > > > > > > >> > > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > > Hi All, > >> > > > > > > > > > >> > > > > > > > > I've analyzed through part of the pyflink code and found > >> some > >> > > > > > > improvement > >> > > > > > > > > possibilities. > >> > > > > > > > > I would like to hear voices on the idea. > >> > > > > > > > > > >> > > > > > > > > Intention: > >> > > > > > > > > * upgrade several python related versions to eliminate > >> > > > end-of-life > >> > > > > > > issues > >> > > > > > > > > and keep up with bugfixes > >> > > > > > > > > * start to add python arm64 support > >> > > > > > > > > > >> > > > > > > > > Actual situation: > >> > > > > > > > > * Flink supports the following python versions: 3.7, > 3.8, > >> > 3.9, > >> > > > 3.10 > >> > > > > > > > > * We use miniconda 4.7.10 (python package management > >> system > >> > and > >> > > > > > > > environment > >> > > > > > > > > management system) which supports the following python > >> > > versions: > >> > > > > 3.7, > >> > > > > > > > 3.8, > >> > > > > > > > > 3.9, 3.10 > >> > > > > > > > > * Our python framework is not supporting anything but > >> x86_64 > >> > > > > > > > > > >> > > > > > > > > Issues: > >> > > > > > > > > * Python 3.7.17 is the latest security patch of the 3.7 > >> line. > >> > > > This > >> > > > > > > > version > >> > > > > > > > > is end-of-life and is no longer supported: > >> > > > > > > > > https://www.python.org/downloads/release/python-3717/ > >> > > > > > > > > * Miniconda 4.7.10 is released on 2019-07-29 which is 4 > >> years > >> > > old > >> > > > > > > already > >> > > > > > > > > and not supporting too many architectures (x86_64 and > >> > ppc64le) > >> > > > > > > > > * The latest miniconda which has real multi-arch feature > >> set > >> > > > > supports > >> > > > > > > the > >> > > > > > > > > following python versions: 3.8, 3.9, 3.10, 3.11 and no > 3.7 > >> > > > support > >> > > > > > > > > > >> > > > > > > > > Suggestion to solve the issues: > >> > > > > > > > > * In 1.19 drop python 3.7 support and upgrade miniconda > to > >> > the > >> > > > > latest > >> > > > > > > > > version which opens the door to other platform + python > >> 3.11 > >> > > > > support > >> > > > > > > > > > >> > > > > > > > > Please note python 3.11 support is not > initiated/discussed > >> > > here. > >> > > > > > > > > > >> > > > > > > > > BR, > >> > > > > > > > > G > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > -- > >> > > > > Best regards, > >> > > > > Sergey > >> > > > > > >> > > > > >> > > > >> > > >> > > >
