Thank you for your reply
But in my local test environment (flink1.15 and flink1.16), when the chain of
writer and commiter is disabled, the back pressure can be reduced.
The specific phenomenon is as follows:
1. After ck-4 is completed, the commit execution is very slow
2. At this time, the [Sink
Hi wudi
I'm afraid it cannot reduce back pressure. If Writer and Commiter are not
chained together, and the Commiter task runs slowly, it can block the
upstream Writer tasks by back pressure too.
On the other hand, you can try to increase the parallelism of sink node to
speedup the Commiter opera
thanks for your replies.
I think that if Writer and Commiter are not chained together, data consistency
can be guaranteed, right?
Because when the Commiter does not block the Writer, at the next Checkpoint, if
the Commit is not completed, the SinkWriter.precommit will not be triggered
In additio
As far as I know that chain between committer and writer is also
required for correctness.
On 16/02/2023 14:53, weijie guo wrote:
Hi wu,
I don't think it is a good choice to directly change the strategy of
chain. Operator chain usually has better performance and resource
utilization. If we d
Hi wu,
I don't think it is a good choice to directly change the strategy of chain.
Operator chain usually has better performance and resource utilization. If
we directly change the chain policy between them, users can no longer chain
them together, which is not a good starting point.
Best regards
Thank you for your reply.
Currently in the custom Sink Connector, the Flink task will combine Writer and
Committer into one thread, and the thread name is similar: [Sink: Writer ->
Sink: Committer (1/1)#0].
In this way, when the Committer.commit() method is very slow, it will block the
SinkWrit
Hi
Do you mean how to disable `chain` in your custom sink connector? Can you
give an example of what you want?
On Wed, Feb 15, 2023 at 10:42 PM di wu <676366...@qq.com> wrote:
> Hello
>
> The current Sink operator will be split into two operations, Writer and
> Commiter. By default, they will b
Hello
The current Sink operator will be split into two operations, Writer and
Commiter. By default, they will be chained together and executed on the same
thread.
So sometimes when the commiter is very slow, it will block the data writer,
causing back pressure.
At present, FlinkSQL can be so
