An alternative solution would be to offer the flink-ml libraries as optional dependencies on the download page. Similar to how we offer the different SQL formats and Hadoop releases [1].
[1] https://flink.apache.org/downloads.html Cheers, Till On Mon, Feb 3, 2020 at 10:19 AM Hequn Cheng <he...@apache.org> wrote: > Thank you all for your feedback and suggestions! > > Best, Hequn > > On Mon, Feb 3, 2020 at 5:07 PM Becket Qin <becket....@gmail.com> wrote: > > > Thanks for bringing up the discussion, Hequn. > > > > +1 on adding `flink-ml-api` and `flink-ml-lib` into opt. This would make > > it much easier for the users to try out some simple ml tasks. > > > > Thanks, > > > > Jiangjie (Becket) Qin > > > > On Mon, Feb 3, 2020 at 4:34 PM jincheng sun <sunjincheng...@gmail.com> > > wrote: > > > >> Thank you for pushing forward @Hequn Cheng <he...@apache.org> ! > >> > >> Hi @Becket Qin <becket....@gmail.com> , Do you have any concerns on > >> this ? > >> > >> Best, > >> Jincheng > >> > >> Hequn Cheng <he...@apache.org> 于2020年2月3日周一 下午2:09写道： > >> > >>> Hi everyone, > >>> > >>> Thanks for the feedback. As there are no objections, I've opened a JIRA > >>> issue(FLINK-15847[1]) to address this issue. > >>> The implementation details can be discussed in the issue or in the > >>> following PR. > >>> > >>> Best, > >>> Hequn > >>> > >>> [1] https://issues.apache.org/jira/browse/FLINK-15847 > >>> > >>> On Wed, Jan 8, 2020 at 9:15 PM Hequn Cheng <chenghe...@gmail.com> > wrote: > >>> > >>> > Hi Jincheng, > >>> > > >>> > Thanks a lot for your feedback! > >>> > Yes, I agree with you. There are cases that multi jars need to be > >>> > uploaded. I will prepare another discussion later. Maybe with a > simple > >>> > design doc. > >>> > > >>> > Best, Hequn > >>> > > >>> > On Wed, Jan 8, 2020 at 3:06 PM jincheng sun < > sunjincheng...@gmail.com> > >>> > wrote: > >>> > > >>> >> Thanks for bring up this discussion Hequn! > >>> >> > >>> >> +1 for include `flink-ml-api` and `flink-ml-lib` in opt. > >>> >> > >>> >> BTW: I think would be great if bring up a discussion for upload > >>> multiple > >>> >> Jars at the same time. as PyFlink JOB also can have the benefit if > we > >>> do > >>> >> that improvement. > >>> >> > >>> >> Best, > >>> >> Jincheng > >>> >> > >>> >> > >>> >> Hequn Cheng <chenghe...@gmail.com> 于2020年1月8日周三 上午11:50写道： > >>> >> > >>> >> > Hi everyone, > >>> >> > > >>> >> > FLIP-39[1] rebuilds Flink ML pipeline on top of TableAPI which > moves > >>> >> Flink > >>> >> > ML a step further. Base on it, users can develop their ML jobs and > >>> more > >>> >> and > >>> >> > more machine learning platforms are providing ML services. > >>> >> > > >>> >> > However, the problem now is the jars of flink-ml-api and > >>> flink-ml-lib > >>> >> are > >>> >> > only exist on maven repo. Whenever users want to submit ML jobs, > >>> they > >>> >> can > >>> >> > only depend on the ml modules and package a fat jar. This would be > >>> >> > inconvenient especially for the machine learning platforms on > which > >>> >> nearly > >>> >> > all jobs depend on Flink ML modules and have to package a fat jar. > >>> >> > > >>> >> > Given this, it would be better to include jars of flink-ml-api and > >>> >> > flink-ml-lib in the `opt` folder, so that users can directly use > the > >>> >> jars > >>> >> > with the binary release. For example, users can move the jars into > >>> the > >>> >> > `lib` folder or use -j to upload the jars. (Currently, -j only > >>> support > >>> >> > upload one jar. Supporting multi jars for -j can be discussed in > >>> another > >>> >> > discussion.) > >>> >> > > >>> >> > Putting the jars in the `opt` folder instead of the `lib` folder > is > >>> >> because > >>> >> > currently, the ml jars are still optional for the Flink project by > >>> >> default. > >>> >> > > >>> >> > What do you think? Welcome any feedback! > >>> >> > > >>> >> > Best, > >>> >> > > >>> >> > Hequn > >>> >> > > >>> >> > [1] > >>> >> > > >>> >> > > >>> >> > >>> > https://cwiki.apache.org/confluence/display/FLINK/FLIP-39+Flink+ML+pipeline+and+ML+libs > >>> >> > > >>> >> > >>> > > >>> > >> >
