This tool is also can be used to store snapshots in an external warehouse.
ср, 14 окт. 2020 г. в 14:57, Pavel Kovalenko <[email protected]>: > Hi Nikolay, > > The idea is good. But what do you think to integrate these ideas into > WAL-G project? > https://github.com/wal-g/wal-g > It's a well-known tool that is already used to stream WAL for PostgreSQL, > MySQL, and MongoDB. > The advantages are integration with S3, GCP, Azure out of the box, > encryption, and compression. > > > ср, 14 окт. 2020 г. в 14:21, Nikolay Izhikov <[email protected]>: > >> Hello, Igniters. >> >> I want to start a discussion of the new feature [1] >> >> CDC - capture data change. The feature allows the consumer to receive >> online notifications about data record changes. >> >> It can be used in the following scenarios: >> * Export data into some warehouse, full-text search, or >> distributed log system. >> * Online statistics and analytics. >> * Wait and respond to some specific events or data changes. >> >> Propose to implement new IgniteCDC application as follows: >> * Run on the server node host. >> * Watches for the appearance of the WAL archive segments. >> * Iterates it using existing WALIterator and notifies consumer of >> each record from the segment. >> >> IgniteCDC features: >> * Independence from the server node process (JVM) - issues and >> failures of the consumer will not lead to server node instability. >> * Notification guarantees and failover - i.e. CDC track and save >> the pointer to the last consumed record. Continue notification from this >> pointer in case of restart. >> * Resilience for the consumer - it's not an issue when a consumer >> temporarily consumes slower than data appear. >> >> WDYT? >> >> [1] >> https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+CDC+-+Capture+Data+Change > >
