[
https://issues.apache.org/jira/browse/FLINK-26759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510352#comment-17510352
]
fanrui commented on FLINK-26759:
--------------------------------
Hi [~martijnvisser] [~knaufk] , thanks for your clarification.
Currently, we meet 3 issues when upgrading:
# The devs on the business side do not want to change the old code (it takes
time).
# After using the new source, the state is not compatible. For business
scenarios with high accuracy requirements, they cannot accept state loss. If
users deal with the state compatibility of all jobs, it will cost a lot of
labor.
# For old jobs, the business side is worried about changing the code to
introduce other instability.
For the state compatibility when upgrading the new Source api from
SourceFunction, could you share some best practice? Thanks a lot.
> Legacy source support waiting for recordWriter to be available
> --------------------------------------------------------------
>
> Key: FLINK-26759
> URL: https://issues.apache.org/jira/browse/FLINK-26759
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common, Runtime / Checkpointing
> Affects Versions: 1.13.0, 1.14.0, 1.15.0
> Reporter: fanrui
> Priority: Major
>
> In order for Unaligned Checkpoint not to be blocked, StreamTask#processInput
> will check recordWriter.isAvailable(). If not available, the data will not be
> processed until recordWriter is available.
> The new Source api is compatible with the above logic, but Legacy Source is
> not compatible with the above logic. When using Unaligned Checkpoint, if the
> backpressure of Legacy Source is high, the Checkpoint duration of Legacy
> Source will be very long.
>
> Since legacy sources are often used in production, can we add logic to wait
> for recordWriter to be available for legacy source?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
