[
https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16901911#comment-16901911
]
Purushotham Pushpavanthar commented on NIFI-6501:
-------------------------------------------------
[~leiwangouc], [~bende]
To guarantee that there is no data loss, the BinLogClient has to be
disconnected when the buffer is full even after Max Queue Offer Time has
expired. When the processor is triggered next, it'll have to check first
whether is the Queue.size() >= Max Queue Size, if the queue is full, don't do
anything. If not check if BinLogClient is disconnected, seek to binlog_filename
and binlog_position and establish connection, else continue processing events
from queue.
What do you guys think?
> Add config to limit buffer queue size in CaptureChangeMySQL
> -----------------------------------------------------------
>
> Key: NIFI-6501
> URL: https://issues.apache.org/jira/browse/NIFI-6501
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 1.9.2
> Reporter: Purushotham Pushpavanthar
> Assignee: Purushotham Pushpavanthar
> Priority: Critical
> Labels: easyfix
> Attachments: image-2019-08-02-11-29-10-829.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> CaptureChangeMySQL processor registers a listener with Blocking Queue as
> buffer with the BinLogClient. When the thread polling from the Queue is
> slower compared to writer, the queue grows uncontrollably and brings down the
> node.
> Since the flow writing to listeners in
> [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]]
> is blocking, we should initialize the queue with *initial size* and *queue
> offer timeout* specified by the user based in cluster configuration.
> [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
