[
https://issues.apache.org/jira/browse/FLINK-14319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leo Zhang updated FLINK-14319:
------------------------------
Fix Version/s: (was: 1.10.0)
> Register user jar files in {Stream}ExecutionEnvironment
> --------------------------------------------------------
>
> Key: FLINK-14319
> URL: https://issues.apache.org/jira/browse/FLINK-14319
> Project: Flink
> Issue Type: New Feature
> Components: API / DataSet, API / DataStream
> Reporter: Leo Zhang
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I see that there are some use cases in which people want to implement their
> own SQL application based on loading external jars for now. And the related
> API proposals have been issued in the task FLINK-10232 Add a SQL DDL . And
> the related sub-task FLINK-14055 is unresolved and its status is still open.
> I feel like it's better to split this task FLINK-14055 into two goals, one
> for DDL and the other new task for
> \{Stream}ExecutionEnvironment::registerUserJarFile() interface which will be
> addressed in this issue.
> Here is the plan.
> *Design*
> * Add _void_ _registerUserJarFile(String jarFile)_ into
> _StreamExecutionEnvironment_ ( in module flink-streaming-java). The affected
> classes are _StreamGraph_, _StreamGraphGenerator_,
> _StreamingJobGraphGenerator_ to support getting and setting a list of user
> jars. And all they are in module flink-streaming-java.
> * Add _void_ _registerUserJarFile(String jarFile)_ into
> _ExecutionEnvironment_ (in module flink-java). The affected classes is
> _Plan_, in module flink-core, to support getting and setting a list of user
> jars.
> * Add _void addUserJars(List<Path> userJars, JobGraph jobGraph)_ into
> _JobGraphGenerator_ and add the user jars within the method _JobGraph_
> _compileHobGraph(OptimizedPlan program, JobID jobId)_ so that user jars can
> be shipped with user's program and submitted to cluster. _JobGraphGenerator_
> is in module flink-optimizer.
> * Add _void_ _registerUserJarFile(String jarFile)_ into
> \{Stream}ExecutionEnvironment (in module flink-scala and
> flink-streaming-scala) and just use the wrapped _javaEnv_ to achieve
> registration.
> *Testing*
> * One test case for adding local user jars both in the streaming and batch
> jobs. We need to process test classes into a jar before testing. For this
> purpose, we can add a goal in process-test-classes for this testing case in
> the pom file. The affected module is flink-tests.
> * Another test case for adding use jars in HDFS. The same idea with the
> previous one. The affected module is flink-fs-tests.
> * Note that python API is not included in this issue just as registering
> cached files. But we still need to modify some python test cases in order to
> avoid building error as lacking some methods declared in java. The affected
> files are
> _flink-python/pyflink/dataset/tests/test_execution_environment_completeness.py_
> and
> _flink-python/pyflink/datastream/tests/test_stream_execution_environment_completeness.py_.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
