Hi, >On the segment archiving, utility iterates it using existing WALIterator >Wait and respond to some specific events or data changes. It seems like this solution will have an unpredictable delay for synchronization for handling events.
Why can't we just implement a Debezium connector for Ignite, for example? https://debezium.io/documentation/reference/1.3/index.html. It is a pretty popular product that uses Kafka underneath. Evgenii ср, 14 окт. 2020 г. в 05:00, Pavel Kovalenko <[email protected]>: > This tool is also can be used to store snapshots in an external warehouse. > > > ср, 14 окт. 2020 г. в 14:57, Pavel Kovalenko <[email protected]>: > > > Hi Nikolay, > > > > The idea is good. But what do you think to integrate these ideas into > > WAL-G project? > > https://github.com/wal-g/wal-g > > It's a well-known tool that is already used to stream WAL for PostgreSQL, > > MySQL, and MongoDB. > > The advantages are integration with S3, GCP, Azure out of the box, > > encryption, and compression. > > > > > > ср, 14 окт. 2020 г. в 14:21, Nikolay Izhikov <[email protected]>: > > > >> Hello, Igniters. > >> > >> I want to start a discussion of the new feature [1] > >> > >> CDC - capture data change. The feature allows the consumer to receive > >> online notifications about data record changes. > >> > >> It can be used in the following scenarios: > >> * Export data into some warehouse, full-text search, or > >> distributed log system. > >> * Online statistics and analytics. > >> * Wait and respond to some specific events or data changes. > >> > >> Propose to implement new IgniteCDC application as follows: > >> * Run on the server node host. > >> * Watches for the appearance of the WAL archive segments. > >> * Iterates it using existing WALIterator and notifies consumer > of > >> each record from the segment. > >> > >> IgniteCDC features: > >> * Independence from the server node process (JVM) - issues and > >> failures of the consumer will not lead to server node instability. > >> * Notification guarantees and failover - i.e. CDC track and save > >> the pointer to the last consumed record. Continue notification from this > >> pointer in case of restart. > >> * Resilience for the consumer - it's not an issue when a > consumer > >> temporarily consumes slower than data appear. > >> > >> WDYT? > >> > >> [1] > >> > https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+CDC+-+Capture+Data+Change > > > > >
