[
https://issues.apache.org/jira/browse/FLINK-14319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leo Zhang updated FLINK-14319:
------------------------------
Description:
I see that there are some use cases in which people want to implement
applications based on loading external jars for now, not just SQL but also
streaming ones. And the related API proposals have been issued in the sub-task
FLINK-14055 under task FLINK-10232 Add a SQL DDL.
To support this sub-task FLINK-14055 , we need the other new task for
\{Stream}ExecutionEnvironment::registerUserJarFile() interface which will be
addressed in this issue.
Here is the plan.
*Design*
* Add interface
{code:java}
void registerUserJarFile(String jarFile)
{code}
into _StreamExecutionEnvironment_ ( in module flink-streaming-java). The
affected classes are _StreamGraph_, _StreamGraphGenerator_,
_StreamingJobGraphGenerator_ to support getting and setting a list of user
jars. And all they are in module flink-streaming-java.
* Add _void_ _registerUserJarFile(String jarFile)_ into _ExecutionEnvironment_
(in module flink-java). The affected classes is _Plan_, in module flink-core,
to support getting and setting a list of user jars.
{code:java}
void registerUserJarFile(String jarFile){code}
* Add interface
{code:java}
void addUserJars(List<Path> userJars, JobGraph jobGraph)
{code}
into _JobGraphGenerator_ and add the user jars within the method _JobGraph_
_compileJobGraph(OptimizedPlan program, JobID jobId)_ so that user jars can be
shipped with user's program and submitted to cluster. _JobGraphGenerator_ is in
module flink-optimizer.
* Add interface
{code:java}
void registerUserJarFile(String jarFile)
{code}
into \{Stream}ExecutionEnvironment (in module flink-scala and
flink-streaming-scala) and just use the wrapped _javaEnv_ to achieve
registration.
*Testing*
* One test case for adding local user jars both in the streaming and batch
jobs. We need to process test classes into a jar before testing. For this
purpose, we can add a goal in process-test-classes for this testing case in the
pom file. The affected module is flink-tests.
* Another test case for adding use jars in HDFS. The same idea with the
previous one. The affected module is flink-fs-tests.
* Note that python API is not included in this issue just as registering
cached files. But we still need to modify some python test cases in order to
avoid building error as lacking some methods declared in java. The affected
files are
_flink-python/pyflink/dataset/tests/test_execution_environment_completeness.py_
and
_flink-python/pyflink/datastream/tests/test_stream_execution_environment_completeness.py_.
was:
I see that there are some use cases in which people want to implement their
own SQL application based on loading external jars for now. And the related API
proposals have been issued in the task FLINK-10232 Add a SQL DDL . And the
related sub-task FLINK-14055 is unresolved and its status is still open.
I feel like it's better to split this task FLINK-14055 into two goals, one for
DDL and the other new task for
\{Stream}ExecutionEnvironment::registerUserJarFile() interface which will be
addressed in this issue.
Here is the plan.
*Design*
* Add _void_ _registerUserJarFile(String jarFile)_ into
_StreamExecutionEnvironment_ ( in module flink-streaming-java). The affected
classes are _StreamGraph_, _StreamGraphGenerator_, _StreamingJobGraphGenerator_
to support getting and setting a list of user jars. And all they are in module
flink-streaming-java.
* Add _void_ _registerUserJarFile(String jarFile)_ into _ExecutionEnvironment_
(in module flink-java). The affected classes is _Plan_, in module flink-core,
to support getting and setting a list of user jars.
* Add _void addUserJars(List<Path> userJars, JobGraph jobGraph)_ into
_JobGraphGenerator_ and add the user jars within the method _JobGraph_
_compileHobGraph(OptimizedPlan program, JobID jobId)_ so that user jars can be
shipped with user's program and submitted to cluster. _JobGraphGenerator_ is in
module flink-optimizer.
* Add _void_ _registerUserJarFile(String jarFile)_ into
\{Stream}ExecutionEnvironment (in module flink-scala and flink-streaming-scala)
and just use the wrapped _javaEnv_ to achieve registration.
*Testing*
* One test case for adding local user jars both in the streaming and batch
jobs. We need to process test classes into a jar before testing. For this
purpose, we can add a goal in process-test-classes for this testing case in the
pom file. The affected module is flink-tests.
* Another test case for adding use jars in HDFS. The same idea with the
previous one. The affected module is flink-fs-tests.
* Note that python API is not included in this issue just as registering
cached files. But we still need to modify some python test cases in order to
avoid building error as lacking some methods declared in java. The affected
files are
_flink-python/pyflink/dataset/tests/test_execution_environment_completeness.py_
and
_flink-python/pyflink/datastream/tests/test_stream_execution_environment_completeness.py_.
> Register user jar files in {Stream}ExecutionEnvironment
> --------------------------------------------------------
>
> Key: FLINK-14319
> URL: https://issues.apache.org/jira/browse/FLINK-14319
> Project: Flink
> Issue Type: New Feature
> Components: API / DataSet, API / DataStream
> Reporter: Leo Zhang
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> I see that there are some use cases in which people want to implement
> applications based on loading external jars for now, not just SQL but also
> streaming ones. And the related API proposals have been issued in the
> sub-task FLINK-14055 under task FLINK-10232 Add a SQL DDL.
> To support this sub-task FLINK-14055 , we need the other new task for
> \{Stream}ExecutionEnvironment::registerUserJarFile() interface which will be
> addressed in this issue.
> Here is the plan.
> *Design*
> * Add interface
> {code:java}
> void registerUserJarFile(String jarFile)
> {code}
> into _StreamExecutionEnvironment_ ( in module flink-streaming-java). The
> affected classes are _StreamGraph_, _StreamGraphGenerator_,
> _StreamingJobGraphGenerator_ to support getting and setting a list of user
> jars. And all they are in module flink-streaming-java.
>
> * Add _void_ _registerUserJarFile(String jarFile)_ into
> _ExecutionEnvironment_ (in module flink-java). The affected classes is
> _Plan_, in module flink-core, to support getting and setting a list of user
> jars.
>
> {code:java}
> void registerUserJarFile(String jarFile){code}
>
> * Add interface
> {code:java}
> void addUserJars(List<Path> userJars, JobGraph jobGraph)
> {code}
> into _JobGraphGenerator_ and add the user jars within the method _JobGraph_
> _compileJobGraph(OptimizedPlan program, JobID jobId)_ so that user jars can
> be shipped with user's program and submitted to cluster. _JobGraphGenerator_
> is in module flink-optimizer.
> * Add interface
> {code:java}
> void registerUserJarFile(String jarFile)
> {code}
> into \{Stream}ExecutionEnvironment (in module flink-scala and
> flink-streaming-scala) and just use the wrapped _javaEnv_ to achieve
> registration.
> *Testing*
> * One test case for adding local user jars both in the streaming and batch
> jobs. We need to process test classes into a jar before testing. For this
> purpose, we can add a goal in process-test-classes for this testing case in
> the pom file. The affected module is flink-tests.
> * Another test case for adding use jars in HDFS. The same idea with the
> previous one. The affected module is flink-fs-tests.
> * Note that python API is not included in this issue just as registering
> cached files. But we still need to modify some python test cases in order to
> avoid building error as lacking some methods declared in java. The affected
> files are
> _flink-python/pyflink/dataset/tests/test_execution_environment_completeness.py_
> and
> _flink-python/pyflink/datastream/tests/test_stream_execution_environment_completeness.py_.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
