twalthr opened a new pull request, #20633:
URL: https://github.com/apache/flink/pull/20633
## What is the purpose of the change
This PR improves the configuration story in various aspects. The main target
is to remove inconsistencies but also a simple architecture under the hood.
## Brief change log
StreamExecutionEnvironment.getConfiguration() was not only a ReadableConfig
but also a deep copy. This caused various problems downstream (Python API) and
makes accessing configuration expensive. Root configuration of `TableConfig`
should not be a snapshot but a reference that always reflects the current
status of StreamExecutionEnvironment. Otherwise, when merging e.g.
`PipelineOptions.JARS`, it is possible that entries get lost when an outdated
root configuration is used during merging.
StreamExecutionEnvironment.configure() should not contain merging logic. It
should be a set operation as all other options as well.
Table API should apply JAR file paths to TableConfig only. There is logic to
apply this to lower layers already.
The same logic should also apply for StreamTableEnvironment.
Also, `PipelineOptions.JARS` is not set by default if no jars have been
added.
## Verifying this change
This change added tests and can be verified as follows:
`DataStreamJavaITCase#testResourcePropagation`
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: yes
- The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
## Documentation
- Does this pull request introduce a new feature? no
- If yes, how is the feature documented? JavaDocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
