hi tiger: Thanks for starting a valuable discussion, If udf is only a management module(CURD), that's easy, The key is in multiple deployment modes (on yarn|k8s|standalone...) How these udfs work together with the user's job? This is a difficult problem. Do you have any relevant ideas and designs for this?
Best, Huajie Wang tiger <[email protected]> 于2022年10月9日周日 17:18写道： > Hello everyone > > > As previously discussed in the group, an issue has been created over here > and suggestions are welcome. > > > Regarding the development of specific features, as I don't have permission > to create a branch, could @Huajie help to create a new branch based on the > 1.2.3-release branch? For example udf-management to facilitate development. > > > > > > > > > > > > > > > > > > At 2022-10-08 17:18:35, "功夫熊猫" <[email protected]> wrote: > >Hi all, > > > >BackGround:I've been in contact with StreamPark for a while, and I've had > a pretty good experience in terms of ease of use and stability. At present, > StreamPark itself supports UDF functions, but it seems that there is no > unified management menu for UDF, so I would like to add a new menu for UDF > management, which is used for the management of UDF. > > > >Main implementation ideas： > >Currently, we mainly create UDF through restful api, then select UDF when > creating the job, and associate UDF ids (mainly to get UDF JAR storage path > later), and finally achieve dynamic loading through yarn.provided.lib.dirs > parameter. > >Note: This feature is currently only implemented based on SQL jobs in > Yarn Application mode; the JAR is saved on top of HDFS. > > > > > >Main APIs: > >Add UDF > > > >Query UDF (list) > >Edit UDF > >Delete UDF > > > > > >Follow up plan: > >Basic functional development at the API level is implemented first, > followed by front-end UI-related development. > > > > > >Best wishes > >tiger >
