Nikolay Izhikov created IGNITE-17700:
----------------------------------------
Summary: Online CDC
Key: IGNITE-17700
URL: https://issues.apache.org/jira/browse/IGNITE-17700
Project: Ignite
Issue Type: Improvement
Reporter: Nikolay Izhikov
Right now, timeout between CDC event happens and consumer notified about it is
relatively big.
It sums from the following:
* {{DataStorageConfiguration#walForceArchiveTimeout}}.
* segment archivation time.
* {{CdcConfiguration#checkFrequence}}
* time to consume previous events.
Advantages of this way described in
[IEP-59|https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+Change+Data+Capture]
So if user wants to minimize this timeouts we can implement the following:
* Collects cdc event inside fixed-size buffer in node.
* Consume events from buffer by CDCConsumer inside node process.
* Store WALPointer of last event added to buffer.
* On buffer overflow stop push events to it.
** Fallback to WAL segments iteration
** When iteration catch up WAL tail switch back to the buffer.
* On node failover recover consumer with the cdc state as in ignite-cdc.
With this approach most of the time CDCConsumer will consume events in near
realtime fashion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
