[
https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898885#comment-16898885
]
Bryan Bende commented on NIFI-6501:
-----------------------------------
If the BinaryLogClient is also blocked waiting for the call to onEvent to
complete, then you can just set the offer timeout really high, and unless the
the downstream processors are truly stopped, then you should eventually get
space to complete the offer and not really lose date.
The main two reasons the blocking queue would be full are 1) the next processor
can't keep up and back-pressure happened on the queue between the processors,
which caused CaptureChangeMySql to temporarily not execute, which caused the
internal queue to fill up, or 2) the next processor is stopped which caused the
same thing. The difference is that in #1 there will be space at some point, it
will be going back and forth between back-pressure and no back-pressure, where
as #2 there will never be space until the downstream processor is started.
> Add config to limit buffer queue size in CaptureChangeMySQL
> -----------------------------------------------------------
>
> Key: NIFI-6501
> URL: https://issues.apache.org/jira/browse/NIFI-6501
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 1.9.2
> Reporter: Purushotham Pushpavanthar
> Assignee: Purushotham Pushpavanthar
> Priority: Critical
> Labels: easyfix
> Attachments: image-2019-08-02-11-29-10-829.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> CaptureChangeMySQL processor registers a listener with Blocking Queue as
> buffer with the BinLogClient. When the thread polling from the Queue is
> slower compared to writer, the queue grows uncontrollably and brings down the
> node.
> Since the flow writing to listeners in
> [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]]
> is blocking, we should initialize the queue with *initial size* and *queue
> offer timeout* specified by the user based in cluster configuration.
> [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
