[
https://issues.apache.org/jira/browse/FLINK-26759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510187#comment-17510187
]
fanrui commented on FLINK-26759:
--------------------------------
Hi [~martijnvisser] , thanks for your comment.
I understand your thinking. Unfortunately, many companies still use Legacy
Source for most flink jobs. There are several reasons:
# From the business side, job stability is the most important thing. If it is
just to upgrade the Source API, they have no motivation to change the old code.
Changes may introduce some stability issues.
# The state compatibility of the old and new sources cannot be well resolved,
it is quite complicated for users to upgrade the source.
# There may only be new jobs or new SQL jobs that use the new Source API with
a higher probability.
I consulted with some of the big internet companies in China and they said that
less than 20% of their jobs in production use the new Source APIs. We have
thousands of flink-1.13 jobs in production and only 10% of them use the new
Source APIs. From a business perspective, they prefer to optimize the engine
without changing the business code.
I can also understand if Legacy Source isn't going to be maintained in the end.
Thanks a lot.
> Legacy source support waiting for recordWriter to be available
> --------------------------------------------------------------
>
> Key: FLINK-26759
> URL: https://issues.apache.org/jira/browse/FLINK-26759
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common, Runtime / Checkpointing
> Affects Versions: 1.13.0, 1.14.0, 1.15.0
> Reporter: fanrui
> Priority: Major
>
> In order for Unaligned Checkpoint not to be blocked, StreamTask#processInput
> will check recordWriter.isAvailable(). If not available, the data will not be
> processed until recordWriter is available.
> The new Source api is compatible with the above logic, but Legacy Source is
> not compatible with the above logic. When using Unaligned Checkpoint, if the
> backpressure of Legacy Source is high, the Checkpoint duration of Legacy
> Source will be very long.
>
> Since legacy sources are often used in production, can we add logic to wait
> for recordWriter to be available for legacy source?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
