FangYongs opened a new pull request, #22938:
URL: https://github.com/apache/flink/pull/22938
## What is the purpose of the change
When flink perform `create temporary function` or `create system function`
with a jar file, it will register the jar file to `ResourceManager` and add it
to the `ClassLoader`. This cause the `SHOW JARS` statement can list the jar
files for these functions, and even when a job does not use these function, the
jar file will be aded to the job's classpath.
This PR aims to address this issue with three steps:
1. Create temporary or system function
a> Download the jar file from remote as needed, which is the same as
before for these functions.
b> Add jar files for temporary and system functions to a new class
`FunctionResourceManager` in `ResourceManager` when they are created.
c> Create a new classloader with the added jar files and validate the
function class
2. Job uses temporary or system function
a> Register the jar files for the functions to `ResourceManager`, which
will be added to the classpath of the job
b> Add the jar files to the classloader for the job
3. Drop temporary or system function
a> There is reference counter in `FunctionResourceManager` for each jar
file
b> Register or unregister function to the `FunctionResourceManager` will
update the reference counter
c> The jar file in `FunctionResourceManager` will be removed when the
reference counter comes to 0
In this way, the jar files for temporary or system function will not be
listed in `SHOW JARS` and will not be added to the classpath and classloader
when a job does not use it.
## Brief change log
- Added `FunctionResourceManager` to register and unregister jar files for
temporary and system functions
- Don't register jar files to `resourceInfos` and `userClassLoader` when
temporary and system functions are created, register jar files to
`FunctionResourceManager`
- Register jar files to `resourceInfos` and `userClassLoader` when the
functions are used in a job
## Verifying this change
This change added tests and can be verified as follows:
- Added `SHOW JARS;` in `function.q` after creating a temporary functions
and the result is empty
- Updated `FunctionITCase.testCreateTemporarySystemFunctionByUsingJar` to
show jars after creating system function
- Added `ResourceManagerTest.testRegisterFunctionResource` to verify the
jar file will no be added to classloader when it is registered in
`FunctionResourceManager`
- Added `ResourceManagerTest.testRegisterFunctionResource` to check the
jar file can be registered and unregistered in `FunctionResourceManager` with
reference counter
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / no) no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / no) no
- The serializers: (yes / no / don't know) no
- The runtime per-record code paths (performance sensitive): (yes / no /
don't know) no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
no
- The S3 file system connector: (yes / no / don't know) no
## Documentation
- Does this pull request introduce a new feature? (yes / no) no
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
