Hi All! One thing we could do already now is to add a simple example on how to execute Python jobs like java jobs (with the right main class, args etc).
It would be similar to https://github.com/apache/flink-kubernetes-operator/tree/main/examples/flink-sql-runner-example but slightly simpler as we don't need a maven module most likely. Unfortunately I cannot do it myself as @Geng Biao <biaoge...@gmail.com> pointed out that Flink Python on M1 macbook is unsupported so cannot really test this locally. Cheers, Gyula On Wed, Jul 6, 2022 at 4:56 AM Dian Fu <dian0511...@gmail.com> wrote: > Thanks for the confirmation Matyas! > > On Tue, Jul 5, 2022 at 3:00 PM Őrhidi Mátyás <matyas.orh...@gmail.com> > wrote: > > > Yes, this is the plan Dian. Appreciate your assistance! > > > > Best, > > Matyas > > > > On Tue, Jul 5, 2022 at 8:55 AM Dian Fu <dian0511...@gmail.com> wrote: > > > >> Hi Matyas, > >> > >> According to the release schedule defined in [1], it seems that the > >> feature freeze of v1.2 may occur at the beginning of September, is this > >> correct? If this is the case, I think it should be reasonable to make > it in > >> v1.2 for Python support. > >> > >> Regards, > >> Dian > >> > >> [1] > >> > https://cwiki.apache.org/confluence/display/FLINK/Release+Schedule+and+Planning > >> > >> On Tue, Jul 5, 2022 at 2:10 PM Őrhidi Mátyás <matyas.orh...@gmail.com> > >> wrote: > >> > >>> Both sql and py support is requested frequently. I guess we should aim > >>> to support both in v1.2. > >>> > >>> Matyas > >>> > >>> On Tue, Jul 5, 2022 at 6:26 AM Gyula Fóra <gyula.f...@gmail.com> > wrote: > >>> > >>>> Thank you for the info and help Dian :) > >>>> > >>>> Gyula > >>>> > >>>> On Tue, 5 Jul 2022 at 05:14, Yang Wang <danrtsey...@gmail.com> wrote: > >>>> > >>>> > Thanks Dian for the confirmation and nice help. > >>>> > > >>>> > Best, > >>>> > Yang > >>>> > > >>>> > Dian Fu <dian0511...@gmail.com> 于2022年7月5日周二 09:27写道： > >>>> > > >>>> > > @Yang, Yes, you are right. Python jobs could be seen as special > JAR > >>>> jobs > >>>> > > whose main class is always > >>>> `org.apache.flink.client.python.PythonDriver`. > >>>> > > What we could do in Flink K8s operator is to make it more > >>>> convenient and > >>>> > > handle properly for the different kinds of dependencies[1]. > >>>> > > > >>>> > > @Gyula, I can help on this. I will find some time to investigate > >>>> this in > >>>> > > the following days and will let you know when there is any > progress. > >>>> > > > >>>> > > Regards, > >>>> > > Dian > >>>> > > > >>>> > > [1] > >>>> > > > >>>> > > >>>> > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/dependency_management/ > >>>> > > > >>>> > > On Mon, Jul 4, 2022 at 11:52 AM Yang Wang <danrtsey...@gmail.com> > >>>> wrote: > >>>> > > > >>>> > >> AFAIK, the python job could be considered as a special case of > jar > >>>> job. > >>>> > >> The user jar is flink-python-*.jar and is located in the opt > >>>> directory. > >>>> > >> The python script is just the argument of this user jar. So I > >>>> believe > >>>> > the > >>>> > >> users already could submit python jobs via Flink Kubernetes > >>>> operator. > >>>> > >> However, they need some manual operations, including specify the > >>>> user > >>>> > >> jar, download python script via init container, etc. > >>>> > >> > >>>> > >> What we could do in the Flink kubernetes operator is to make the > >>>> > >> submission more convenient by introducing a new field(e.g. > >>>> pyScript). > >>>> > >> > >>>> > >> cc @Dian Fu <dian0511...@gmail.com> @biaoge...@gmail.com > >>>> > >> <biaoge...@gmail.com> WDYT? > >>>> > >> > >>>> > >> Best, > >>>> > >> Yang > >>>> > >> > >>>> > >> Gyula Fóra <gyf...@apache.org> 于2022年7月4日周一 00:12写道： > >>>> > >> > >>>> > >>> Hi Devs! > >>>> > >>> > >>>> > >>> Would anyone with a good understanding of the Python execution > >>>> layer be > >>>> > >>> interested in working on adding Python job support for the Flink > >>>> > >>> Kubernetes > >>>> > >>> Operator? > >>>> > >>> > >>>> > >>> This is a feature request that comes up often ( > >>>> > >>> https://issues.apache.org/jira/browse/FLINK-28364) and it would > >>>> be a > >>>> > >>> great > >>>> > >>> way to fill some missing feature gaps on the operator :) > >>>> > >>> > >>>> > >>> I am of course happy to help or work together with someone on > >>>> this but > >>>> > I > >>>> > >>> have zero experience with the Python API at this stage and don't > >>>> want > >>>> > to > >>>> > >>> miss some obvious requirements. > >>>> > >>> > >>>> > >>> Cheers, > >>>> > >>> Gyula > >>>> > >>> > >>>> > >> > >>>> > > >>>> > >>> >
