[
https://issues.apache.org/jira/browse/FLINK-21804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan updated FLINK-21804:
--------------------------------------
Description:
[Proposed
design|https://docs.google.com/document/d/10c6hZsOVxzUjeCLPSDpKGyZOYHi73yd92lCqRs1CyUE/edit#heading=h.5b9hthjg53vl]
!Changelog Backend _ writer loading.png|width=600!
* Black arrows - existing references/creations
* {color:red}Red{color} arrows - required references/creations
* {color:#00875A}Green{color} arrows - proposed references/creations to enable
required ones
was:
As of FLINK-21355, changelog writer factory is created inside the backend while
creating keyed backend.
Instead, it should be obtained from the environment
(env.getTaskStateManager.get...). Creation should be done closer to
TaskExecutor/Task creation, so that PluginManager is accessible and grouping
across subtasks is possible.
The previous approach (removed from FLINK-21355 PR) was to use
StateBackendLoader, pass the created factory to backend, and then to keyed
backend. The issue is that it is also used in JM and other environments when no
state is actually logged. It's also difficult to use PluginManager there.
Motivation to use PluginManager:
1. allow different changelog implementations
2. (possibly) allow using changelog in different places
However, this would require moving backend loading closer to TaskManagerRunner
where other pluginized components are initialized.
Probably, this should be done together with pluginizing state backends.
> Create and wire changelog writer with backend
> ---------------------------------------------
>
> Key: FLINK-21804
> URL: https://issues.apache.org/jira/browse/FLINK-21804
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / State Backends
> Reporter: Roman Khachatryan
> Priority: Major
> Fix For: 1.14.0
>
> Attachments: Changelog Backend _ writer loading.png
>
>
> [Proposed
> design|https://docs.google.com/document/d/10c6hZsOVxzUjeCLPSDpKGyZOYHi73yd92lCqRs1CyUE/edit#heading=h.5b9hthjg53vl]
> !Changelog Backend _ writer loading.png|width=600!
> * Black arrows - existing references/creations
> * {color:red}Red{color} arrows - required references/creations
> * {color:#00875A}Green{color} arrows - proposed references/creations to
> enable required ones
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
