[
https://issues.apache.org/jira/browse/NIFI-6501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16898521#comment-16898521
]
Lei Wang commented on NIFI-6501:
--------------------------------
Hi Purushotham,
I have rebuild the nifi-cdc-mysql-processors using your new code.
Seems it works, no OOM error again.
After the down stream is blocked， the processor will throw an error every
one minites because the queue is full:
!image-2019-08-02-11-29-10-829.png!
java.lang.RuntimeException: Unable to add event to the queue
at
org.apache.nifi.cdc.mysql.event.BinlogEventListener.onEvent(BinlogEventListener.java:59)
at
com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1055)
at
com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:913)
at
com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:559)
at
com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:793)
at java.lang.Thread.run(Thread.java:748)
I have two quesion:
* The frequency of the exception is the same with the property Max Queue
Offer Time. Is this right?
* Will the exception cause data loss?
> Add config to limit buffer queue size in CaptureChangeMySQL
> -----------------------------------------------------------
>
> Key: NIFI-6501
> URL: https://issues.apache.org/jira/browse/NIFI-6501
> Project: Apache NiFi
> Issue Type: Bug
> Components: Configuration
> Affects Versions: 1.9.2
> Reporter: Purushotham Pushpavanthar
> Assignee: Purushotham Pushpavanthar
> Priority: Critical
> Labels: easyfix
> Attachments: image-2019-08-02-11-29-10-829.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> CaptureChangeMySQL processor registers a listener with Blocking Queue as
> buffer with the BinLogClient. When the thread polling from the Queue is
> slower compared to writer, the queue grows uncontrollably and brings down the
> node.
> Since the flow writing to listeners in
> [mysql-binlog-connector-java|[https://github.com/shyiko/mysql-binlog-connector-java]]
> is blocking, we should initialize the queue with *initial size* and *queue
> offer timeout* specified by the user based in cluster configuration.
> [http://apache-nifi-developer-list.39713.n7.nabble.com/NiFi-Cluster-crashes-while-running-CaptureChangeMySQL-for-CDC-td20895.html]
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
