Hi Xingbo, Thanks for your support!
I agree that multi-platform in general is a good idea. We start with ARM but this opens the door for all others. G On Wed, Sep 6, 2023 at 10:46 AM Xingbo Huang <hxbks...@gmail.com> wrote: > Hi Gabor, > > Thanks a lot for the detailed explanation. Building a test environment with > multi-platforms is indeed what we have been lacking, > not only for M1 users, but also for Windows users. At 1.16, I remember that > Azure did not provide the m1 environment, > and the github action just planned m1 in the annual plan at that > time[1][2]. > > +1 for drop 3.7 support in 1.19 > > [1] https://github.com/actions/runner-images/issues/2187 > [2] https://github.com/github/roadmap/issues/528 > > Best, > Xingbo > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2023年9月6日周三 16:07写道： > > > Hi Xingbo, > > > > > So I guess you want to run ci tests on the m1 > > > environment, but the current version of miniconda cannot meet this > > > requirement, so there is a pre-step that must drop python 3.7? > > > > In short yes + local wheel build fix on the M1. In a bit more detailed > > please let me elaborate. > > > > You're right about the arm64 build creation, Flink is doing that for > quite > > some time already and wheels are running fine. > > There are gaps however. > > > > From user perspective: > > * we're building wheels for ARM64 but we're not executing any python > tests > > on ARM64 but only on X64. > > I think it's a must to test platform packages on the related platform > since > > not all python packages are ARM compatible or must be built/used in a > > different way. > > > > From dev perspective: > > * Local MacOs M1 wheel or python source distro builds are simply failing > > with error messages. > > Just to give an example GRPC pip install is failing and require the > > following changes: > > > > > https://github.com/apache/flink/pull/23359/files#diff-c4c4916cd6ab7f87a4b154ef9603f9ba6d3b74a00096be9adacb7df363b5efcfR28-R32 > > I can't really imagine how Azure is able to build MacOS ARM64 wheels... > > * Python tests are not running on any kind of ARM CPUs which makes any > bug > > hunting/development hard or nearly impossible > > > > In order to give ARM support with local development + test execution > > possibility new miniconda is needed. New miniconda runs 3.8+. > > > > There is already a PR which is under development: > > https://github.com/apache/flink/pull/23359 > > This fills all the mentioned gaps: local test/development/test execution > on > > ARM CPUs + the previously mentioned 3+ years not actively supported > python > > eliminated. > > > > Hope this helps. > > > > G > > > > > > On Wed, Sep 6, 2023 at 9:00 AM Xingbo Huang <hxbks...@gmail.com> wrote: > > > > > Hi Gyala and Gabor, > > > > > > Pyflink has provided arm64 wheel packages for Apple silicon since > > 1.16[1]. > > > The use of Miniconda is only related to ci testing and packaging on > linux > > > platform, and building mac platform wheels are dependent on > > > cibuildwheel[2]. So I guess you want to run ci tests on the m1 > > > environment, but the current version of miniconda cannot meet this > > > requirement, so there is a pre-step that must drop python 3.7? > > > > > > [1] https://pypi.org/project/apache-flink/1.16.2/#files > > > [2] > > > > > > > > > https://github.com/apache/flink/blob/master/tools/azure-pipelines/build-python-wheels.yml#L30 > > > > > > Best, > > > Xingbo > > > > > > Gabor Somogyi <gabor.g.somo...@gmail.com> 于2023年9月6日周三 14:36写道： > > > > > > > Hi Xingbo, > > > > > > > > *Constraint:* > > > > I personally not found any miniconda version which provides arm64 > > support > > > > together with python 3.7. > > > > [image: image.png] > > > > > > > > At the moment I think new platform support means 3.7 drop. > > > > > > > > I fully to agree with Gyula, if we start now maybe we can release it > in > > > > half a year however *3.7 active support already ended in 27 Jun > 2020*. > > > > At the moment any python development/test execution on MacOS M1 is > just > > > > not working as-is just like any kind of python test execution on any > > ARM > > > > CPU. > > > > > > > > Gains: > > > > * We can release a working version in half a year hopefully and not > > > > shifting support to 1+ year > > > > * MacOS M1 local development would work finally which is essential > for > > > > user engagement > > > > * It would be possible to execute python tests on ARM64 machines > > > > * We can shake up the python development story because it's not the > > most > > > > loved area > > > > > > > > BR, > > > > G > > > > > > > > > > > > On Wed, Sep 6, 2023 at 8:06 AM Gyula Fóra <gyula.f...@gmail.com> > > wrote: > > > > > > > >> Hi Xingbo! > > > >> > > > >> I think we have to analyze what we gain by dropping 3.7 and > upgrading > > > to a > > > >> miniconda version with a multiarch support. > > > >> > > > >> If this is what we need to get Apple silicon support then I think > it's > > > >> worth doing it already in 1.19. Keep in mind that 1.18 is not even > > > >> released > > > >> yet so if we delay this to 1.20 that is basically 1 year from now. > > > >> Making this change can increase the adoption instantly if we enable > > new > > > >> platforms. > > > >> > > > >> Cheers, > > > >> Gyula > > > >> > > > >> On Wed, Sep 6, 2023 at 4:46 AM Xingbo Huang <hxbks...@gmail.com> > > wrote: > > > >> > > > >> > Hi Gabor, > > > >> > > > > >> > Thanks for bringing this up. In my opinion, it is a bit aggressive > > to > > > >> > directly drop Python 3.7 in 1.19. Python 3.7 is still used a > lot[1], > > > >> and as > > > >> > far as I know, many Pyflink users are still using python 3.7 as > > their > > > >> > default interpreter. I prefer to deprecate Python 3.7 in 1.19 just > > > like > > > >> we > > > >> > deprecated Python 3.6 in 1.16[2] and dropped Python 3.6 in > 1.17[3]. > > > >> > > > > >> > For the support of Python 3.11, I am very supportive of the > > > >> implementation > > > >> > in 1.19 (many users have this appeal, and I originally wanted to > > > >> support it > > > >> > in 1.18). > > > >> > > > > >> > Regarding the miniconda upgrade, I tend to upgrade miniconda to > the > > > >> latest > > > >> > version that can support python 3.7 to 3.11 at the same time. > > > >> > > > > >> > [1] https://w3techs.com/technologies/history_details/pl-python/3 > > > >> > [2] https://issues.apache.org/jira/browse/FLINK-28195 > > > >> > [3] https://issues.apache.org/jira/browse/FLINK-27929 > > > >> > > > > >> > Best, > > > >> > Xingbo > > > >> > > > > >> > Jing Ge <j...@ververica.com.invalid> 于2023年9月5日周二 04:10写道： > > > >> > > > > >> > > +1 > > > >> > > > > > >> > > @Dian should we add support of python 3.11 > > > >> > > > > > >> > > Best regards, > > > >> > > Jing > > > >> > > > > > >> > > On Mon, Sep 4, 2023 at 3:39 PM Gabor Somogyi < > > > >> gabor.g.somo...@gmail.com> > > > >> > > wrote: > > > >> > > > > > >> > > > Thanks for all the responses! > > > >> > > > > > > >> > > > Based on the suggestions I've created the following jiras and > > > >> started > > > >> > to > > > >> > > > work on them: > > > >> > > > * https://issues.apache.org/jira/browse/FLINK-33029 > > > >> > > > * https://issues.apache.org/jira/browse/FLINK-33030 > > > >> > > > > > > >> > > > The reason why I've split them is to separate the concerns and > > > >> reduce > > > >> > the > > > >> > > > amount of code in a PR to help reviewers. > > > >> > > > > > > >> > > > BR, > > > >> > > > G > > > >> > > > > > > >> > > > > > > >> > > > On Mon, Sep 4, 2023 at 12:57 PM Sergey Nuyanzin < > > > >> snuyan...@gmail.com> > > > >> > > > wrote: > > > >> > > > > > > >> > > > > +1, > > > >> > > > > Thanks for looking into this. > > > >> > > > > > > > >> > > > > On Mon, Sep 4, 2023 at 8:38 AM Gyula Fóra < > > gyula.f...@gmail.com > > > > > > > >> > > wrote: > > > >> > > > > > > > >> > > > > > +1 > > > >> > > > > > Thanks for looking into this. > > > >> > > > > > > > > >> > > > > > Gyula > > > >> > > > > > > > > >> > > > > > On Mon, Sep 4, 2023 at 8:26 AM Matthias Pohl < > > > >> > matthias.p...@aiven.io > > > >> > > > > > .invalid> > > > >> > > > > > wrote: > > > >> > > > > > > > > >> > > > > > > Thanks Gabor for looking into it. It sounds reasonable > to > > me > > > >> as > > > >> > > well. > > > >> > > > > > > > > > >> > > > > > > +1 > > > >> > > > > > > > > > >> > > > > > > On Sun, Sep 3, 2023 at 5:44 PM Márton Balassi < > > > >> > > > > balassi.mar...@gmail.com> > > > >> > > > > > > wrote: > > > >> > > > > > > > > > >> > > > > > > > Hi Gabor, > > > >> > > > > > > > > > > >> > > > > > > > Thanks for bringing this up. Similarly to when we > > dropped > > > >> > Python > > > >> > > > 3.6 > > > >> > > > > > due > > > >> > > > > > > to > > > >> > > > > > > > its end of life (and added 3.10) in Flink 1.17 [1,2], > it > > > >> makes > > > >> > > > sense > > > >> > > > > to > > > >> > > > > > > > proceed to remove 3.7 and add 3.11 instead. > > > >> > > > > > > > > > > >> > > > > > > > +1. > > > >> > > > > > > > > > > >> > > > > > > > [1] https://issues.apache.org/jira/browse/FLINK-27929 > > > >> > > > > > > > [2] https://github.com/apache/flink/pull/21699 > > > >> > > > > > > > > > > >> > > > > > > > Best, > > > >> > > > > > > > Marton > > > >> > > > > > > > > > > >> > > > > > > > On Fri, Sep 1, 2023 at 10:39 AM Gabor Somogyi < > > > >> > > > > > gabor.g.somo...@gmail.com > > > >> > > > > > > > > > > >> > > > > > > > wrote: > > > >> > > > > > > > > > > >> > > > > > > > > Hi All, > > > >> > > > > > > > > > > > >> > > > > > > > > I've analyzed through part of the pyflink code and > > found > > > >> some > > > >> > > > > > > improvement > > > >> > > > > > > > > possibilities. > > > >> > > > > > > > > I would like to hear voices on the idea. > > > >> > > > > > > > > > > > >> > > > > > > > > Intention: > > > >> > > > > > > > > * upgrade several python related versions to > eliminate > > > >> > > > end-of-life > > > >> > > > > > > issues > > > >> > > > > > > > > and keep up with bugfixes > > > >> > > > > > > > > * start to add python arm64 support > > > >> > > > > > > > > > > > >> > > > > > > > > Actual situation: > > > >> > > > > > > > > * Flink supports the following python versions: 3.7, > > > 3.8, > > > >> > 3.9, > > > >> > > > 3.10 > > > >> > > > > > > > > * We use miniconda 4.7.10 (python package management > > > >> system > > > >> > and > > > >> > > > > > > > environment > > > >> > > > > > > > > management system) which supports the following > python > > > >> > > versions: > > > >> > > > > 3.7, > > > >> > > > > > > > 3.8, > > > >> > > > > > > > > 3.9, 3.10 > > > >> > > > > > > > > * Our python framework is not supporting anything > but > > > >> x86_64 > > > >> > > > > > > > > > > > >> > > > > > > > > Issues: > > > >> > > > > > > > > * Python 3.7.17 is the latest security patch of the > > 3.7 > > > >> line. > > > >> > > > This > > > >> > > > > > > > version > > > >> > > > > > > > > is end-of-life and is no longer supported: > > > >> > > > > > > > > > https://www.python.org/downloads/release/python-3717/ > > > >> > > > > > > > > * Miniconda 4.7.10 is released on 2019-07-29 which > is > > 4 > > > >> years > > > >> > > old > > > >> > > > > > > already > > > >> > > > > > > > > and not supporting too many architectures (x86_64 > and > > > >> > ppc64le) > > > >> > > > > > > > > * The latest miniconda which has real multi-arch > > feature > > > >> set > > > >> > > > > supports > > > >> > > > > > > the > > > >> > > > > > > > > following python versions: 3.8, 3.9, 3.10, 3.11 and > no > > > 3.7 > > > >> > > > support > > > >> > > > > > > > > > > > >> > > > > > > > > Suggestion to solve the issues: > > > >> > > > > > > > > * In 1.19 drop python 3.7 support and upgrade > > miniconda > > > to > > > >> > the > > > >> > > > > latest > > > >> > > > > > > > > version which opens the door to other platform + > > python > > > >> 3.11 > > > >> > > > > support > > > >> > > > > > > > > > > > >> > > > > > > > > Please note python 3.11 support is not > > > initiated/discussed > > > >> > > here. > > > >> > > > > > > > > > > > >> > > > > > > > > BR, > > > >> > > > > > > > > G > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > -- > > > >> > > > > Best regards, > > > >> > > > > Sergey > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > > > > > > > >
