Hi all, I have updated the FLIP and removed content relate to UDAF and also changed the title of the FLIP to "Flink Python User-Defined Stateless Function for Table". Does it make sense to you?
Regards, Dian > 在 2019年9月6日，下午6:09，Dian Fu <dian0511...@gmail.com> 写道： > > Hi all, > > Thanks a lot for the discussion here. It makes sense to limit the scope of > this FLIP to only ScalarFunction. I'll update the FLIP and remove the content > relating to UDAF. > > Thanks, > Dian > >> 在 2019年9月6日，下午6:02，jincheng sun <sunjincheng...@gmail.com> 写道： >> >> Hi， >> >> Sure, for ensure the 1.10 relesae of flink, let's split the FLIPs, and >> FLIP-58 only do the stateless part. >> >> Cheers, >> Jincheng >> >> Aljoscha Krettek <aljos...@apache.org> 于2019年9月6日周五 下午5:53写道： >> >>> Hi, >>> >>> Regarding stateful functions and MapView/DataView/ListView: I think it’s >>> best to keep that for a later FLIP and focus on a more basic version. >>> Supporting stateful functions, especially with MapView can potentially be >>> very slow so we have to see what we can do there. >>> >>> For the method names, I don’t know. If FLIP-64 passes they have to be >>> changed. So we could use the final names right away, but I’m also fine with >>> using the old method names for now. >>> >>> Best, >>> Aljoscha >>> >>>> On 5. Sep 2019, at 12:40, jincheng sun <sunjincheng...@gmail.com> wrote: >>>> >>>> Hi Aljoscha, >>>> >>>> Thanks for your comments! >>>> >>>> Regarding to the FLIP scope, it seems that we have agreed on the design >>> of >>>> the stateless function support. >>>> What do you think about starting the development of the stateless >>> function >>>> support firstly and continue the discussion of stateful function support? >>>> Or you think we should split the current FLIP into two FLIPs and discuss >>>> the stateful function support in another thread? >>>> >>>> Currently, the Python DataView/MapView/ListView interfaces design follow >>>> the Java/Scala naming conversions. >>>> Of couse, We can continue to discuss whether there are better solutions, >>>> i.e. using annotations. >>>> >>>> Regarding to the magic logic to support DataView/MapView/ListView, it >>> will >>>> be done by the framework and is transparent for users. >>>> Per my understanding, the magic logic is unavoidable no matter what the >>>> interfaces will be. >>>> >>>> Regarding to the catalog support of python function:1) If it's stored in >>>> memory as temporary object, just as you said, users can call >>>> TableEnvironment.register_function(will change to >>>> register_temporary_function in FLIP-64) >>>> 2) If it's persisted in external storage, users can call >>>> Catalog.create_function. There will be no API change per my >>> understanding. >>>> >>>> What do you think? >>>> Best,Jincheng >>>> >>>> Aljoscha Krettek <aljos...@apache.org> 于2019年9月5日周四 下午5:32写道： >>>> >>>>> Hi, >>>>> >>>>> Another thing to consider is the Scope of the FLIP. Currently, we try to >>>>> support (stateful) AggregateFunctions. I have some concerns about >>> whether >>>>> or not DataView/MapView/ListView is a good interface because it requires >>>>> quite some magic from the runners to make it work, such as messing with >>> the >>>>> TypeInformation and injecting objects at runtime. If the FLIP aims for >>> the >>>>> minimum of ScalarFunctions and the whole execution harness, that should >>> be >>>>> easier to agree on. >>>>> >>>>> Another point is the naming of the new methods. I think Timo hinted at >>> the >>>>> fact that we have to consider catalog support for functions. There is >>>>> ongoing work about differentiating between temporary objects and objects >>>>> that are stored in a catalog (FLIP-64 [1]). With this in mind, the >>> method >>>>> for registering functions should be called register_temporary_function() >>>>> and so on. Unless we want to already think about mixing Python and Java >>>>> functions in the catalog, which is outside the scope of this FLIP, I >>> think. >>>>> >>>>> Best, >>>>> Aljoscha >>>>> >>>>> [1] >>>>> >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-64%3A+Support+for+Temporary+Objects+in+Table+module >>>>> >>>>> >>>>>> On 5. Sep 2019, at 05:01, jincheng sun <sunjincheng...@gmail.com> >>> wrote: >>>>>> >>>>>> Hi Aljoscha, >>>>>> >>>>>> That's a good points, so far, most of the code will live in >>> flink-python >>>>>> module, and the rules and relNodes will be put into the both blink and >>>>>> flink planner modules, some of the common interface of required by >>>>> planners >>>>>> will be placed in flink-table-common. I think you are right, we should >>>>> try >>>>>> to ensure the changes of this feature is minimal. For more detail we >>>>> would >>>>>> follow this principle when review the PRs. >>>>>> >>>>>> Great thanks for your questions and remind! >>>>>> >>>>>> Best, >>>>>> Jincheng >>>>>> >>>>>> >>>>>> Aljoscha Krettek <aljos...@apache.org> 于2019年9月4日周三 下午8:58写道： >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> Things looks interesting so far! >>>>>>> >>>>>>> I had one question: Where will most of the support code for this live? >>>>>>> Will this add the required code to flink-table-common or the different >>>>>>> runners? Can we implement this in such a way that only a minimal >>> amount >>>>> of >>>>>>> support code is required in the parts of the Table API (and Table API >>>>>>> runners) that are not python specific? >>>>>>> >>>>>>> Best, >>>>>>> Aljoscha >>>>>>> >>>>>>>> On 4. Sep 2019, at 14:14, Timo Walther <twal...@apache.org> wrote: >>>>>>>> >>>>>>>> Hi Jincheng, >>>>>>>> >>>>>>>> 2. Serializability of functions: "#2 is very convenient for users" >>>>> means >>>>>>> only until they have the first backwards-compatibility issue, after >>> that >>>>>>> they will find it not so convinient anymore and will ask why the >>>>> framework >>>>>>> allowed storing such objects in a persistent storage. I don't want to >>> be >>>>>>> picky about it, but wanted to raise awareness that sometimes it is ok >>> to >>>>>>> limit use cases to guide users for devloping backwards-compatible >>>>> programs. >>>>>>>> >>>>>>>> Thanks for the explanation fo the remaining items. It sounds >>> reasonable >>>>>>> to me. Regarding the example with `getKind()`, I actually meant >>>>>>> `org.apache.flink.table.functions.ScalarFunction#getKind` we don't >>> allow >>>>>>> users to override this property. And I think we should do something >>>>> similar >>>>>>> for the getLanguage property. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Timo >>>>>>>> >>>>>>>> On 03.09.19 15:01, jincheng sun wrote: >>>>>>>>> Hi Timo, >>>>>>>>> >>>>>>>>> Thanks for the quick reply ! :) >>>>>>>>> I have added more example for #3 and #5 to the FLIP. That are great >>>>>>>>> suggestions ! >>>>>>>>> >>>>>>>>> Regarding 2: >>>>>>>>> >>>>>>>>> There are two kind Serialization for CloudPickle(Which is different >>>>> from >>>>>>>>> Java): >>>>>>>>> 1) For class and function which can be imported, CloudPickle only >>>>>>>>> serialize the full path of the class and function (just like java >>>>> class >>>>>>>>> name). >>>>>>>>> 2) For the class and function which can not be imported, CloudPickle >>>>>>> will >>>>>>>>> serialize the full content of the class and function. >>>>>>>>> For #2, It means that we can not just store the full path of the >>> class >>>>>>> and >>>>>>>>> function. >>>>>>>>> >>>>>>>>> The above serialization is recursive. >>>>>>>>> >>>>>>>>> However, there is indeed an problem of backwards compatibility when >>>>> the >>>>>>>>> module path of the parent class changed. But I think this is an rare >>>>>>> case >>>>>>>>> and acceptable. i.e., For Flink framework we never change the user >>>>>>>>> interface module path if we want to keep backwards compatibility. >>> For >>>>>>> user >>>>>>>>> code, if they change the interface of UDF's parent, they should >>>>>>> re-register >>>>>>>>> their functions. >>>>>>>>> >>>>>>>>> If we do not want support #2, we can store the full path of class >>> and >>>>>>>>> function, in that case we have no backwards compatibility problem. >>>>> But I >>>>>>>>> think the #2 is very convenient for users. >>>>>>>>> >>>>>>>>> What do you think? >>>>>>>>> >>>>>>>>> Regarding 4: >>>>>>>>> As I mentioned earlier, there may be built-in Python functions and I >>>>>>> think >>>>>>>>> language is a "function" concept. Function and Language are >>> orthogonal >>>>>>>>> concepts. >>>>>>>>> We may have R, GO and other language functions in the future, not >>> only >>>>>>>>> user-defined, but also built-in functions. >>>>>>>>> >>>>>>>>> You are right that users will not set this method and for Python >>>>>>> functions, >>>>>>>>> it will be set in the code-generated Java function by the framework. >>>>>>> So, I >>>>>>>>> think we should declare the getLanguage() in FunctionDefinition for >>>>> now. >>>>>>>>> (I'm not pretty sure what do you mean by saying that getKind() is >>>>> final >>>>>>> in >>>>>>>>> UserDefinedFunction?) >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Jincheng >>>>>>>>> >>>>>>>>> Timo Walther <twal...@apache.org> 于2019年9月3日周二 下午6:01写道： >>>>>>>>> >>>>>>>>>> Hi Jincheng, >>>>>>>>>> >>>>>>>>>> thanks for your response. >>>>>>>>>> >>>>>>>>>> 2. Serializability of functions: Using some arbitrary serialization >>>>>>>>>> format for shipping a function to worker sounds fine to me. But >>> once >>>>> we >>>>>>>>>> store functions a the catalog we need to think about backwards >>>>>>>>>> compatibility and evolution of interfaces etc. I'm not sure if >>>>>>>>>> CloudPickle is the right long-term storage format for this. If we >>>>> don't >>>>>>>>>> think about this in advance, we are basically violating our code >>>>>>> quality >>>>>>>>>> guide [1] of never use Java Serialization but in the Python-way. We >>>>> are >>>>>>>>>> using the RPC serialization for persistence. >>>>>>>>>> >>>>>>>>>> 3. TableEnvironment: Can you add some example to the FLIP? Because >>>>> API >>>>>>>>>> code like the following is not covered there: >>>>>>>>>> >>>>>>>>>> self.t_env.register_function("add_one", udf(lambda i: i + 1, >>>>>>>>>> DataTypes.BIGINT(), >>>>>>>>>> DataTypes.BIGINT())) >>>>>>>>>> self.t_env.register_function("subtract_one", udf(SubtractOne(), >>>>>>>>>> DataTypes.BIGINT(), >>>>>>>>>> DataTypes.BIGINT())) >>>>>>>>>> self.t_env.register_function("add", add) >>>>>>>>>> >>>>>>>>>> 4. FunctionDefinition: Your response still doesn't answer my >>> question >>>>>>>>>> entirely. Why do we need FunctionDefinition.getLanguage() if this >>> is >>>>> a >>>>>>>>>> "user-defined function" concept and not a "function" concept. In >>> any >>>>>>>>>> case, all users should not be able to set this method. So it must >>> be >>>>>>>>>> final in UserDefinedFunction similar to getKind(). >>>>>>>>>> >>>>>>>>>> 5. Function characteristics: If UserDefinedFunction is defined in >>>>>>>>>> Python, why is it not used in your example in FLIP-58. You could >>> you >>>>>>>>>> extend the example to show how to specify these attributes in the >>>>> FLIP? >>>>>>>>>> >>>>>>>>>> Regards, >>>>>>>>>> Timo >>>>>>>>>> >>>>>>>>>> [1] >>>>>>> >>> https://flink.apache.org/contributing/code-style-and-quality-java.html >>>>>>>>>> >>>>>>>>>> On 02.09.19 15:35, jincheng sun wrote: >>>>>>>>>>> Hi Timo, >>>>>>>>>>> >>>>>>>>>>> Great thanks for your feedback. I would like to share my thoughts >>>>> with >>>>>>>>>> you >>>>>>>>>>> inline. :) >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Jincheng >>>>>>>>>>> >>>>>>>>>>> Timo Walther <twal...@apache.org> 于2019年9月2日周一 下午5:04写道： >>>>>>>>>>> >>>>>>>>>>>> Hi all, >>>>>>>>>>>> >>>>>>>>>>>> the FLIP looks awesome. However, I would like to discuss the >>>>> changes >>>>>>> to >>>>>>>>>>>> the user-facing parts again. Some feedback: >>>>>>>>>>>> >>>>>>>>>>>> 1. DataViews: With the current non-annotation design for >>> DataViews, >>>>>>> we >>>>>>>>>>>> cannot perform eager state declaration, right? At which point >>>>> during >>>>>>>>>>>> execution do we know which state is required by the function? We >>>>>>> need to >>>>>>>>>>>> instantiate the function first, right? >>>>>>>>>>>> >>>>>>>>>>>>> We will analysis the Python AggregateFunction and extract the >>>>>>> DataViews >>>>>>>>>>> used in the Python AggregateFunction. This can be done >>>>>>>>>>> by instantiate a Python AggregateFunction, creating an accumulator >>>>> by >>>>>>>>>>> calling method create_accumulator and then analysis the created >>>>>>>>>>> accumulator. This is actually similar to the way that Java >>>>>>>>>>> AggregateFunction processing codegen logic. The extracted >>> DataViews >>>>>>> can >>>>>>>>>>> then be used to construct the StateDescriptors in the operator, >>>>> i.e., >>>>>>> we >>>>>>>>>>> should have hold the state spec and the state descriptor id in >>> Java >>>>>>>>>>> operator and Python worker can access the state by specifying the >>>>>>>>>>> corresponding state descriptor id. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 2. Serializability of functions: How do we ensure serializability >>>>> of >>>>>>>>>>>> functions for catalog persistence? In the Scala/Java API, we >>> would >>>>>>> like >>>>>>>>>>>> to register classes instead of instances soon. This is the only >>> way >>>>>>> to >>>>>>>>>>>> store a function properly in a catalog or we need some >>>>>>>>>>>> serialization/deserialization logic in the function interfaces to >>>>>>>>>>>> convert an instance to string properties. >>>>>>>>>>>> >>>>>>>>>>>>> The Python function will be serialized with CloudPickle anyway >>> in >>>>>>> the >>>>>>>>>>> Python API as we need to transfer it to the Python worker which >>> can >>>>>>> then >>>>>>>>>>> deserialize it for execution. The serialized Python function can >>> be >>>>>>>>>> stored >>>>>>>>>>> into catalog. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 3. TableEnvironment: What is the signature of >>>>>>> `register_function(self, >>>>>>>>>>>> name, function)`? Does it accept both a class and function? Like >>>>>>> `class >>>>>>>>>>>> Sum` and `def split()`? Could you add some examples for >>> registering >>>>>>> both >>>>>>>>>>>> kinds of functions? >>>>>>>>>>>> >>>>>>>>>>>>> It has been already supported which you mentioned. You can find >>> an >>>>>>>>>>> example in the POC code: >>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> >>> https://github.com/dianfu/flink/commit/93f41ba173482226af7513fdec5acba72b274489#diff-34f619b31a7e38604e22a42a441fbe2fR26 >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 4. FunctionDefinition: Function definition is not a user-defined >>>>>>>>>>>> function definition. It is the highest interface for both >>>>>>> user-defined >>>>>>>>>>>> and built-in functions. I'm not sure if getLanguage() should be >>>>> part >>>>>>> of >>>>>>>>>>>> this interface or one-level down which would be >>>>>>> `UserDefinedFunction`. >>>>>>>>>>>> Built-in functions will never be implemented in a different >>>>>>> language. In >>>>>>>>>>>> any case, I would vote for removing the UNKNOWN language, because >>>>> it >>>>>>>>>>>> does not solve anything. Why should a user declare a function >>> that >>>>>>> the >>>>>>>>>>>> runtime can not handle? I also find the term `JAVA` confusing for >>>>>>> Scala >>>>>>>>>>>> users. How about `FunctionLanguage.JVM` instead? >>>>>>>>>>>> >>>>>>>>>>>>> Actually we may have built-in Python functions in the future. >>>>>>> Regarding >>>>>>>>>>> to the following expression: py_udf1(a, b) + py_udf2(c), if there >>> is >>>>>>>>>>> built-in Python >>>>>>>>>>> funciton for '+' operator, then we don't need to mix using Java >>> and >>>>>>>>>> Python >>>>>>>>>>> UDFs. In this way, we can improve the execution performance. >>>>>>>>>>> Regarding to removing FunctionLanguage.UNKNOWN and renaming >>>>>>>>>>> FunctionLanguage.Java to FunctionLanguage.JVM, it makes more sense >>>>> to >>>>>>> me. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> 5. Function characteristics: In the current design, function >>>>> classes >>>>>>> do >>>>>>>>>>>> not extend from any upper class. How can users declare >>>>>>> characteristics >>>>>>>>>>>> that are present in `FunctionDefinition` like determinism, >>>>>>> requirements, >>>>>>>>>>>> or soon also monotonism. >>>>>>>>>>>> >>>>>>>>>>>>> Actually we have defined 'UserDefinedFunction' which is the base >>>>>>> class >>>>>>>>>>> for all user-defined functions. >>>>>>>>>>> We can define the deterministic, requirements, etc in this class. >>>>>>>>>>> Currently, we have already supported to define the deterministic. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> Thanks, >>>>>>>>>>>> Timo >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On 02.09.19 03:38, Shaoxuan Wang wrote: >>>>>>>>>>>>> Hi Jincheng, Fudian, and Aljoscha, >>>>>>>>>>>>> I am assuming the proposed python UDX can also be applied to >>> Flink >>>>>>> SQL. >>>>>>>>>>>>> Is this correct? If yes, I would suggest to title the FLIP as >>>>> "Flink >>>>>>>>>>>> Python >>>>>>>>>>>>> User-Defined Function" or "Flink Python User-Defined Function >>> for >>>>>>>>>> Table". >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> Shaoxuan >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, Aug 28, 2019 at 12:22 PM jincheng sun < >>>>>>>>>> sunjincheng...@gmail.com> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks for the feedback Bowen! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Great thanks for create the FLIP and bring up the VOTE Dian! >>>>>>>>>>>>>> >>>>>>>>>>>>>> Best, Jincheng >>>>>>>>>>>>>> >>>>>>>>>>>>>> Dian Fu <dian0511...@gmail.com> 于2019年8月28日周三 上午11:32写道： >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I have started a voting thread [1]. Thanks a lot for your help >>>>>>> during >>>>>>>>>>>>>>> creating the FLIP @Jincheng. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Bowen, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Very appreciated for your comments. I have replied you in the >>>>>>> design >>>>>>>>>>>> doc. >>>>>>>>>>>>>>> As it seems that the comments doesn't affect the overall >>> design, >>>>>>> I'll >>>>>>>>>>>> not >>>>>>>>>>>>>>> cancel the vote for now and we can continue the discussion in >>>>> the >>>>>>>>>>>> design >>>>>>>>>>>>>>> doc. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> >>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-58-Flink-Python-User-Defined-Function-for-Table-API-td32295.html >>>>>>>>>>>>>>> < >>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> >>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-FLIP-58-Flink-Python-User-Defined-Function-for-Table-API-td32295.html >>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>> Dian >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> 在 2019年8月28日，上午11:05，Bowen Li <bowenl...@gmail.com> 写道： >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Jincheng and Dian, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Sorry for being late to the party. I took a glance at the >>>>>>> proposal, >>>>>>>>>>>>>> LGTM >>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>> general, and I left only a couple comments. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>> Bowen >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Aug 26, 2019 at 8:05 PM Dian Fu < >>> dian0511...@gmail.com >>>>>> >>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> Hi Jincheng, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks! It works. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Dian >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> 在 2019年8月27日，上午10:55，jincheng sun < >>> sunjincheng...@gmail.com> >>>>>>> 写道： >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Dian, can you check if you have edit access? :) >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Dian Fu <dian0511...@gmail.com> 于2019年8月26日周一 上午10:52写道： >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi Jincheng, >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Appreciated for the kind tips and offering of help. >>>>> Definitely >>>>>>>>>> need >>>>>>>>>>>>>>> it! >>>>>>>>>>>>>>>>>>> Could you grant me write permission for confluence? My Id: >>>>>>> Dian >>>>>>>>>> Fu >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> Dian >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> 在 2019年8月26日，上午9:53，jincheng sun < >>> sunjincheng...@gmail.com >>>>>> >>>>>>> 写道： >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks for your feedback Hequn & Dian. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Dian, I am glad to see that you want help to create the >>>>> FLIP! >>>>>>>>>>>>>>>>>>>> Everyone will have first time, and I am very willing to >>>>> help >>>>>>> you >>>>>>>>>>>>>>>>> complete >>>>>>>>>>>>>>>>>>>> your first FLIP creation. Here some tips: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> - First I'll give your account write permission for >>>>>>> confluence. >>>>>>>>>>>>>>>>>>>> - Before create the FLIP, please have look at the FLIP >>>>>>> Template >>>>>>>>>>>>>> [1], >>>>>>>>>>>>>>>>>>> (It's >>>>>>>>>>>>>>>>>>>> better to know more about FLIP by reading [2]) >>>>>>>>>>>>>>>>>>>> - Create Flink Python UDFs related JIRAs after completing >>>>> the >>>>>>>>>> VOTE >>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>> FLIP.(I think you also can bring up the VOTE thread, if >>> you >>>>>>>>>> want! >>>>>>>>>>>> ) >>>>>>>>>>>>>>>>>>>> Any problems you encounter during this period，feel free >>> to >>>>>>> tell >>>>>>>>>> me >>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>> can solve them together. :) >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>> Jincheng >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>> >>> https://cwiki.apache.org/confluence/display/FLINK/FLIP+Template >>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> >>> https://cwiki.apache.org/confluence/display/FLINK/Flink+Improvement+Proposals >>>>>>>>>>>>>>>>>>>> Hequn Cheng <chenghe...@gmail.com> 于2019年8月23日周五 >>>>> 上午11:54写道： >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> +1 for starting the vote. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks Jincheng a lot for the discussion. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best, Hequn >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> On Fri, Aug 23, 2019 at 10:06 AM Dian Fu < >>>>>>>>>> dian0511...@gmail.com> >>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>> Hi Jincheng, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> +1 to start the FLIP create and VOTE on this feature. >>> I'm >>>>>>>>>>>> willing >>>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>> help >>>>>>>>>>>>>>>>>>>>>> on the FLIP create if you don't mind. As I haven't >>>>> created >>>>>>> a >>>>>>>>>>>> FLIP >>>>>>>>>>>>>>>>>>> before, >>>>>>>>>>>>>>>>>>>>>> it will be great if you could help on this. :) >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>>> Dian >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> 在 2019年8月22日，下午11:41，jincheng sun < >>>>>>> sunjincheng...@gmail.com> >>>>>>>>>>>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thanks a lot for your feedback. If there are no more >>>>>>>>>>>> suggestions >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>> comments, I think it's better to initiate a vote to >>>>>>> create a >>>>>>>>>>>>>> FLIP >>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>> Apache Flink Python UDFs. >>>>>>>>>>>>>>>>>>>>>>> What do you think? >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Best, Jincheng >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> jincheng sun <sunjincheng...@gmail.com> 于2019年8月15日周四 >>>>>>>>>>>>>> 上午12:54写道： >>>>>>>>>>>>>>>>>>>>>>>> Hi Thomas, >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Thanks for your confirmation and the very important >>>>>>> reminder >>>>>>>>>>>>>>> about >>>>>>>>>>>>>>>>>>>>>> bundle >>>>>>>>>>>>>>>>>>>>>>>> processing. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I have had add the description about how to perform >>>>>>> bundle >>>>>>>>>>>>>>>>> processing >>>>>>>>>>>>>>>>>>>>>> from >>>>>>>>>>>>>>>>>>>>>>>> the perspective of checkpoint and watermark. Feel >>> free >>>>> to >>>>>>>>>>>> leave >>>>>>>>>>>>>>>>>>>>>> comments if >>>>>>>>>>>>>>>>>>>>>>>> there are anything not describe clearly. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>> Jincheng >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Dian Fu <dian0511...@gmail.com> 于2019年8月14日周三 >>>>> 上午10:08写道： >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Hi Thomas, >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Thanks a lot the suggestions. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> Regarding to bundle processing, there is a section >>>>>>>>>>>>>>> "Checkpoint"[1] >>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>> design doc which talks about how to handle the >>>>>>> checkpoint. >>>>>>>>>>>>>>>>>>>>>>>>> However, I think you are right that we should talk >>>>> more >>>>>>>>>> about >>>>>>>>>>>>>>> it, >>>>>>>>>>>>>>>>>>>>> such >>>>>>>>>>>>>>>>>>>>>> as >>>>>>>>>>>>>>>>>>>>>>>>> what's bundle processing, how it affects the >>>>> checkpoint >>>>>>> and >>>>>>>>>>>>>>>>>>>>> watermark, >>>>>>>>>>>>>>>>>>>>>> how >>>>>>>>>>>>>>>>>>>>>>>>> to handle the checkpoint and watermark, etc. >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> >>> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit#heading=h.urladt565yo3 >>>>>>>>>>>>>>>>>>>>>>>>> < >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> >>> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit#heading=h.urladt565yo3 >>>>>>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>>>>>> Dian >>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> 在 2019年8月14日，上午1:01，Thomas Weise <t...@apache.org> >>>>> 写道： >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Hi Jincheng, >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> Thanks for putting this together. The proposal is >>>>> very >>>>>>>>>>>>>>> detailed, >>>>>>>>>>>>>>>>>>>>>>>>> thorough >>>>>>>>>>>>>>>>>>>>>>>>>> and for me as a Beam Flink runner contributor easy >>> to >>>>>>>>>>>>>>> understand >>>>>>>>>>>>>>>>> :) >>>>>>>>>>>>>>>>>>>>>>>>>> One thing that you should probably detail more is >>> the >>>>>>>>>> bundle >>>>>>>>>>>>>>>>>>>>>>>>> processing. It >>>>>>>>>>>>>>>>>>>>>>>>>> is critically important for performance that >>> multiple >>>>>>>>>>>>>> elements >>>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>>>>>>>>>>>> processed in a bundle. The default bundle size in >>> the >>>>>>>>>> Flink >>>>>>>>>>>>>>>>> runner >>>>>>>>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>>>>> 1s or >>>>>>>>>>>>>>>>>>>>>>>>>> 1000 elements, whichever comes first. And for >>>>>>> streaming, >>>>>>>>>> you >>>>>>>>>>>>>>> can >>>>>>>>>>>>>>>>>>>>> find >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>> logic necessary to align the bundle processing with >>>>>>>>>>>>>> watermarks >>>>>>>>>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>>>>>>> checkpointing here: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> >>> https://github.com/apache/beam/blob/release-2.14.0/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/ExecutableStageDoFnOperator.java >>>>>>>>>>>>>>>>>>>>>>>>>> Thomas >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>> On Tue, Aug 13, 2019 at 7:05 AM jincheng sun < >>>>>>>>>>>>>>>>>>>>>> sunjincheng...@gmail.com> >>>>>>>>>>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Hi all, >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> The Python Table API(without Python UDF support) >>> has >>>>>>>>>>>> already >>>>>>>>>>>>>>>>> been >>>>>>>>>>>>>>>>>>>>>>>>> supported >>>>>>>>>>>>>>>>>>>>>>>>>>> and will be available in the coming release 1.9. >>>>>>>>>>>>>>>>>>>>>>>>>>> As Python UDF is very important for Python users, >>>>> we'd >>>>>>>>>> like >>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>>>>>> start >>>>>>>>>>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>>>>>>>>>>>> discussion about the Python UDF support in the >>>>> Python >>>>>>>>>> Table >>>>>>>>>>>>>>> API. >>>>>>>>>>>>>>>>>>>>>>>>>>> Aljoscha Krettek, Dian Fu and I have discussed >>>>> offline >>>>>>>>>> and >>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>>>>> drafted a >>>>>>>>>>>>>>>>>>>>>>>>>>> design doc[1]. It includes the following items: >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> - The user-defined function interfaces. >>>>>>>>>>>>>>>>>>>>>>>>>>> - The user-defined function execution >>> architecture. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> As mentioned by many guys in the previous >>> discussion >>>>>>>>>>>>>>> thread[2], >>>>>>>>>>>>>>>>> a >>>>>>>>>>>>>>>>>>>>>>>>>>> portability framework was introduced in Apache >>> Beam >>>>> in >>>>>>>>>>>>>> latest >>>>>>>>>>>>>>>>>>>>>>>>> releases. It >>>>>>>>>>>>>>>>>>>>>>>>>>> provides well-defined, language-neutral data >>>>>>> structures >>>>>>>>>> and >>>>>>>>>>>>>>>>>>>>> protocols >>>>>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>> language-neutral user-defined function execution. >>>>> This >>>>>>>>>>>>>> design >>>>>>>>>>>>>>> is >>>>>>>>>>>>>>>>>>>>>> based >>>>>>>>>>>>>>>>>>>>>>>>> on >>>>>>>>>>>>>>>>>>>>>>>>>>> Beam's portability framework. We will introduce >>> how >>>>> to >>>>>>>>>> make >>>>>>>>>>>>>>> use >>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>>>>>>>>>> Beam's >>>>>>>>>>>>>>>>>>>>>>>>>>> portability framework for user-defined function >>>>>>>>>> execution: >>>>>>>>>>>>>>> data >>>>>>>>>>>>>>>>>>>>>>>>>>> transmission, state access, checkpoint, metrics, >>>>>>> logging, >>>>>>>>>>>>>> etc. >>>>>>>>>>>>>>>>>>>>>>>>>>> Considering that the design relies on Beam's >>>>>>> portability >>>>>>>>>>>>>>>>> framework >>>>>>>>>>>>>>>>>>>>>> for >>>>>>>>>>>>>>>>>>>>>>>>>>> Python user-defined function execution and not all >>>>> the >>>>>>>>>>>>>>>>>>> contributors >>>>>>>>>>>>>>>>>>>>>> in >>>>>>>>>>>>>>>>>>>>>>>>>>> Flink community are familiar with Beam's >>> portability >>>>>>>>>>>>>>> framework, >>>>>>>>>>>>>>>>> we >>>>>>>>>>>>>>>>>>>>>> have >>>>>>>>>>>>>>>>>>>>>>>>>>> done a prototype[3] for proof of concept and also >>>>>>> ease of >>>>>>>>>>>>>>>>>>>>>>>>> understanding of >>>>>>>>>>>>>>>>>>>>>>>>>>> the design. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Welcome any feedback. >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>>>>> Jincheng >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> >>> https://docs.google.com/document/d/1WpTyCXAQh8Jr2yWfz7MWCD2-lou05QaQFb810ZvTefY/edit?usp=sharing >>>>>>>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>> >>>>> >>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-FLIP-38-Support-python-language-in-flink-TableAPI-td28061.html >>>>>>>>>>>>>>>>>>>>>>>>>>> [3] >>> https://github.com/dianfu/flink/commits/udf_poc >>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>> >>>>> >>> >>> >
