[
https://issues.apache.org/jira/browse/FLINK-14319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993546#comment-16993546
]
Jeff Zhang commented on FLINK-14319:
------------------------------------
Is it possible to register package as well ? e.g. User may want to user any
kind flink connector at runtime which has transitive dependencies. It would be
nice to allow user just specify the package and flink download it including its
transitive dependencie, then register it to env.
> Register user jar files in {Stream}ExecutionEnvironment
> --------------------------------------------------------
>
> Key: FLINK-14319
> URL: https://issues.apache.org/jira/browse/FLINK-14319
> Project: Flink
> Issue Type: New Feature
> Components: API / DataSet, API / DataStream
> Reporter: Leo Zhang
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I see that there are some use cases in which people want to implement
> applications based on loading external jars for now, not just SQL but also
> streaming ones. And the related API proposals have been issued in the
> sub-task FLINK-14055 under task FLINK-10232 Add a SQL DDL.
> To support this sub-task FLINK-14055 , we need the other new task for
> \{Stream}ExecutionEnvironment::registerUserJarFile() interface which will be
> addressed in this issue.
> Here is the plan.
> *Design*
> * Add interface
> {code:java}
> void registerUserJarFile(String jarFile)
> {code}
> into _StreamExecutionEnvironment_ ( in module flink-streaming-java). The
> affected classes are _StreamGraph_, _StreamGraphGenerator_,
> _StreamingJobGraphGenerator_ to support getting and setting a list of user
> jars. And all they are in module flink-streaming-java.
> * Add interface
> {code:java}
> void registerUserJarFile(String jarFile)
> {code}
> into _ExecutionEnvironment_ (in module flink-java). The affected classes is
> _Plan_, in module flink-core, to support getting and setting a list of user
> jars.
> * Add interface
> {code:java}
> void addUserJars(List<Path> userJars, JobGraph jobGraph)
> {code}
> into _JobGraphGenerator_ and add the user jars within the method _JobGraph_
> _compileJobGraph(OptimizedPlan program, JobID jobId)_ so that user jars can
> be shipped with user's program and submitted to cluster. _JobGraphGenerator_
> is in module flink-optimizer.
> * Add interface
> {code:java}
> void registerUserJarFile(String jarFile)
> {code}
> into \{Stream}ExecutionEnvironment (in module flink-scala and
> flink-streaming-scala) and just use the wrapped _javaEnv_ to achieve
> registration.
> *Testing*
> * One test case for adding local user jars both in the streaming and batch
> jobs. We need to process test classes into a jar before testing. For this
> purpose, we can add a goal in process-test-classes for this testing case in
> the pom file. The affected module is flink-tests.
> * Another test case for adding use jars in HDFS. The same idea with the
> previous one. The affected module is flink-fs-tests.
> * Note that python API is not included in this issue just as registering
> cached files. But we still need to modify some python test cases in order to
> avoid building error as lacking some methods declared in java. The affected
> files are
> _flink-python/pyflink/dataset/tests/test_execution_environment_completeness.py_
> and
> _flink-python/pyflink/datastream/tests/test_stream_execution_environment_completeness.py_.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
