[
https://issues.apache.org/jira/browse/FLINK-14319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993113#comment-16993113
]
Leo Zhang commented on FLINK-14319:
-----------------------------------
Please join the discussion on dev mailing list: [DISCUSS] Register user jar
files in \{Stream}ExecutionEnvironment
> Register user jar files in {Stream}ExecutionEnvironment
> --------------------------------------------------------
>
> Key: FLINK-14319
> URL: https://issues.apache.org/jira/browse/FLINK-14319
> Project: Flink
> Issue Type: New Feature
> Components: API / DataSet, API / DataStream
> Reporter: Leo Zhang
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I see that there are some use cases in which people want to implement
> applications based on loading external jars for now, not just SQL but also
> streaming ones. And the related API proposals have been issued in the
> sub-task FLINK-14055 under task FLINK-10232 Add a SQL DDL.
> To support this sub-task FLINK-14055 , we need the other new task for
> \{Stream}ExecutionEnvironment::registerUserJarFile() interface which will be
> addressed in this issue.
> Here is the plan.
> *Design*
> * Add interface
> {code:java}
> void registerUserJarFile(String jarFile)
> {code}
> into _StreamExecutionEnvironment_ ( in module flink-streaming-java). The
> affected classes are _StreamGraph_, _StreamGraphGenerator_,
> _StreamingJobGraphGenerator_ to support getting and setting a list of user
> jars. And all they are in module flink-streaming-java.
> * Add interface
> {code:java}
> void registerUserJarFile(String jarFile)
> {code}
> into _ExecutionEnvironment_ (in module flink-java). The affected classes is
> _Plan_, in module flink-core, to support getting and setting a list of user
> jars.
> * Add interface
> {code:java}
> void addUserJars(List<Path> userJars, JobGraph jobGraph)
> {code}
> into _JobGraphGenerator_ and add the user jars within the method _JobGraph_
> _compileJobGraph(OptimizedPlan program, JobID jobId)_ so that user jars can
> be shipped with user's program and submitted to cluster. _JobGraphGenerator_
> is in module flink-optimizer.
> * Add interface
> {code:java}
> void registerUserJarFile(String jarFile)
> {code}
> into \{Stream}ExecutionEnvironment (in module flink-scala and
> flink-streaming-scala) and just use the wrapped _javaEnv_ to achieve
> registration.
> *Testing*
> * One test case for adding local user jars both in the streaming and batch
> jobs. We need to process test classes into a jar before testing. For this
> purpose, we can add a goal in process-test-classes for this testing case in
> the pom file. The affected module is flink-tests.
> * Another test case for adding use jars in HDFS. The same idea with the
> previous one. The affected module is flink-fs-tests.
> * Note that python API is not included in this issue just as registering
> cached files. But we still need to modify some python test cases in order to
> avoid building error as lacking some methods declared in java. The affected
> files are
> _flink-python/pyflink/dataset/tests/test_execution_environment_completeness.py_
> and
> _flink-python/pyflink/datastream/tests/test_stream_execution_environment_completeness.py_.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
