LadyForest opened a new pull request #14895:
URL: https://github.com/apache/flink/pull/14895
## What is the purpose of the change
*This pull request aims to support changing the resolution order of modules
more flexibly. This will introduce two methods `void useModules(String...
moduleNames)` and `List<ModuleEntry> listFullModules()` in both `ModuleManager`
and `TableEnvironment`.*
## Brief change log
- *The `ModuleEntry` is a POJO representing the current loaded module name
and whether the module is currently in use*.
- *The `usedModuleCnt` in `ModuleManager` is the current number of used
modules. It servers as a pivot to determine the position of loaded/used
module(s) to be inserted for `#loadModule` and `#useModules`. The key-value
pairs on the left represent the loaded and used modules, separated by the
pivot, with the declared order. The rest pairs on the right represent the
loaded but not used modules, and the order is not guaranteed*.
- *The data structure used for `modules` in `ModuleManager` is replaced
with `org.apache.commons.collections.map.ListOrderedMap`. It's backed by an
`ArrayList` to store the insertion order and provides API to support inserting
key-value mapping at a specific index. It's relatively more convenient to use
than `LinkedHashMap`. However, there is one pitfall that Flink currently
depends on `common-collections:3.2.2` and generics are not supported in this
version.*
- *Behavior description for `loadModule(name, module)` method*
1. Compare `modules.size()` with `usedModuleCnt`, if `modules.size() ==
usedModuleCnt`, then just put the element at the end.
2. If `modules.size() > usedModuleCnt`, which means there are modules
loaded but not used, by design, we should append the newly loaded module right
after the used modules and right before the unused modules. Since we don’t need
to keep order between unused modules, we can swap the current entry at `index
== usedModuleCnt` to the end and put the new module at `index ==
usedModuleCnt`. Finally, increase `usedModuleCnt`.
- *Behavior description for `unloadModule(name)` method*
1. Ensure the module name exists
2. Find the index of this name, say `index`, if `index <
usedModuleCnt`, which means unloading a used module, then after removing this
entry, decrease `usedModuleCnt`.
- *Behavior description for `useModules(name1, name2, ...)` method*
1. Perform a sanity check on names to ensure there are no duplicate names
and unloaded module names.
2. Loop names to do a swap on the map, and assign `names.length` to
`usedModuleCount` after the loop.
- *Behavior description for `listModules()`, `listFunctions()` and
`getFunctionDefinition(name)` methods*
1. These methods will only return the results of used modules.
- *Behavior description for `listFullModules()` method*
1. Return all loaded modules (including unused modules) with the status.
The result is ordered for used modules and there is no guarantee on unused
module's orders.
## Verifying this change
This change added tests and can be verified as follows:
- *Added `ModuleManagerTest` to verify all public methods in
`ModuleManager` behave correctly*.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (**yes** / no)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no**
/ don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** /
don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (**yes** / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs
/ **not documented**)
- Docs will be updated after
[FLINK-21300](https://issues.apache.org/jira/browse/FLINK-21300) finish.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
