Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

On Thu, Apr 25, 2019 at 6:04 AM jincheng sun wrote: > > Hi Robert, > > In addition to the questions described by Dian, I also want to know what > difficult problems Py4j's solution will encounter in add UDF support, which > you mentioned as follows: > >> Using something like Py4j is an easy way

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Robert, In addition to the questions described by Dian, I also want to know what difficult problems Py4j's solution will encounter in add UDF support, which you mentioned as follows: Using something like Py4j is an easy way to get up an running, especially > for a very faithful API, but the

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Thanks everyone for the discussion here. Regarding to the Java/Scala UDF and the built-in UDF to execute in the current Flink way (directly in JVM, not via RPC), I share the same thoughts with Max and Robert and I think it will not be a big problem. From the design doc, I guess the main

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Thanks for the meeting summary, Stephan. Sound like you covered a lot of ground. Some more comments below, adding onto what Max has said. On Wed, Apr 24, 2019 at 3:20 PM Maximilian Michels wrote: > > Hi Stephan, > > This is excited! Thanks for sharing. The inter-process communication > code

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Stephan, This is excited! Thanks for sharing. The inter-process communication code looks like the most natural choice as a common ground. To go further, there are indeed some challenges to solve. => Biggest question is whether the language-independent DAG is expressive enough to capture

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Stephan， Thanks for your summary, from the points of my view, we are on the same page about the conclusion of the discussion! I completely agree that we can divide the support of the Python Table API into short-term and long-term goals, and the design of short-term goals should be smoothly

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi all! Below are my notes on the discussion last week on how to collaborate between Beam and Flink. The discussion was between Tyler, Kenn, Luke, Ahmed, Xiaowei, Shaoxuan, Jincheng, and me. This represents my understanding of the discussion, please augment this where I missed something or where

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi everyone, Thank you for all of your feedback and comments in google doc! I have updated the google doc and add the UDFs part. For a short summary: - Python TableAPI - Flink introduces a set of Python Table API Interfaces which align with Flink Java Table API. It uses Py4j framework to

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Stephan, Thanks for your suggestion and summarize. :) ==> The FLIP should probably reflect the full goal rather than the > first implementation step only, this would make sure everyone understands > what the final goal of the effort is. I totally agree that we can implement the

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

One more thought: The FLIP is very much centered on the CLI and it looks like it has mainly batch jobs and session clusters in mind. In very many cases, especially in streaming cases, the CLI (or shell) is not the entry point for a program. See for example the use of Flink jobs on Kubernetes

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi all! I think that all the opinions and ideas are not actually in conflict, so let me summarize what I understand is the proposal: *(1) Long-term goal: Full Python Table API with UDFs* To break the implementation effort up into stages, the first step would be the API without UDFs.

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

One more thing It's better to mention that Flink table API is a superset of Flink SQL, such as: - AddColumns/DropColums/RenameColumns, the detail can be found in Google doc - Interactive

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Thomas, Thanks for your quick reply. I will share my thoughts further. :) 1. Without the support of the User-defined function, the Python Table API can do a lot of things: (Of course, we need to support UDF on the Python Table API.） Do you have use cases where the Python table API can be

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hu Shuyi, I would recommend to start with the following: https://docs.google.com/presentation/d/1AkU-QXSflau-RSeolB4TSLy0_mg0xwb398Czw7aqVGw/edit#slide=id.p https://www.youtube.com/watch?v=VsGQ2LFeTHY=PL4dEBWmGSIU_9JTGnkGVg6-BwaV0FMxyJ=12=0s For follow-up, there is plenty of knowledge available

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Jincheng, > > Yes, we can add use case examples in both google doc and FLIP， I had > already add the simple usage in the google doc, here I want to know which > kind of examples you want? :) > Do you have use cases where the Python table API can be applied without UDF support? (And where the

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Thomas, I agreed that UDF support is important. As some of us might not be familiar with the effort on Apache Beam, It will be great if you can share some design documents on Beam's portability layer and multi-language support, and the current status. Thanks a lot. Regards Shuyi On Wed, Apr

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Dawid, Thanks for your feedback! Yes, you are right, there is no problem with two-way communication in our proposal. The two solutions proposed already support for two-way communication between Python and Java. Similar to the interface of from_collection/from_elements, our proposal can solve

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Thomas, Glad to see your feedback! Yes, we can add use case examples in both google doc and FLIP， I had already add the simple usage in the google doc, here I want to know which kind of examples you want? :) The very short answer to UDF support is Yes. As you said, we need UDF support on the

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Shaoxuan, Yes, I've seen your message and I am not saying it already contradicts. I agree as long as we just define DAG/pipeline/logical plan it is a reasonable thing to do. No doubts about that. I have a feeling though it mentions at some points things that might be in the area of

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

David, This proposal does not contradict with what we have discussed. Please check my reply in https://lists.apache.org/thread.html/f6f8116b4b38b0b2d70ed45b990d6bb1bcb33611fde6fdf32ec0e840@%3Cdev.flink.apache.org%3E on 2019/02/21. "Beam Python API and Flink Python TableAPI describe the

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi all, Thank you very much Jincheng for the very thorough proposal. I was following the discussion very briefly, but I have an impression that the consensus in the previous discussion[1] was that we do not want to have an independent, flink specific multi language support but we want to

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Shuyi, Glad to see your feedback and port more requirements about multi-language! I think the Flink community is very much looking forward to more language support, of course, Golang should be in the future support list. Since the topic of supporting Python on Flink has been researched and

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi Jeff, thank you for your encouragement and valuable comments on the google doc. I saw that you participated in many pySpark discussions a long time ago, I am very grateful and look forward to your follow-up comments on pyFlink! Thanks, Jincheng Jeff Zhang 于2019年4月2日周二 下午10:53写道： > Thanks

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Thanks for putting this proposal together. It would be nice, if you could share a few use case examples (maybe add them as section to the FLIP?). The reason I ask: The table API is immensely useful, but it isn't clear to me what value other language bindings provide without UDF support. With

Re: [DISCUSS] FLIP-38 Support python language in flink TableAPI

Thanks jincheng for driving this. Overall I agree with the approach, just left a few comments for details. jincheng sun 于2019年4月2日周二 下午4:03写道： > Hi All, > As Xianda brought up in the previous email, There are a large number of > data analysis users who want flink to support Python. At the

[DISCUSS] FLIP-38 Support python language in flink TableAPI

Hi All, As Xianda brought up in the previous email, There are a large number of data analysis users who want flink to support Python. At the Flink API level, we have DataStreamAPI/DataSetAPI/TableAPI, the Table API will become the first-class citizen. Table API is declarative and can be