Hell. Pavel. Thanks for the feedback.
> But what do you think to integrate these ideas into WAL-G project? Looked into WAL-G description and look like it’s use-case are more restricted then CDC itself. Self-definition - "WAL-G is an archival restoration tool for Postgres(beta for MySQL, MariaDB, and MongoDB)" Restoration ability is one of the CDC application. I mentioned other use-cases in the first letter. Anyway, integration with the well-known tools is good. I try to contact WAL-G community. > 14 окт. 2020 г., в 14:59, Pavel Kovalenko <[email protected]> написал(а): > > This tool is also can be used to store snapshots in an external warehouse. > > > ср, 14 окт. 2020 г. в 14:57, Pavel Kovalenko <[email protected]>: > >> Hi Nikolay, >> >> The idea is good. But what do you think to integrate these ideas into >> WAL-G project? >> https://github.com/wal-g/wal-g >> It's a well-known tool that is already used to stream WAL for PostgreSQL, >> MySQL, and MongoDB. >> The advantages are integration with S3, GCP, Azure out of the box, >> encryption, and compression. >> >> >> ср, 14 окт. 2020 г. в 14:21, Nikolay Izhikov <[email protected]>: >> >>> Hello, Igniters. >>> >>> I want to start a discussion of the new feature [1] >>> >>> CDC - capture data change. The feature allows the consumer to receive >>> online notifications about data record changes. >>> >>> It can be used in the following scenarios: >>> * Export data into some warehouse, full-text search, or >>> distributed log system. >>> * Online statistics and analytics. >>> * Wait and respond to some specific events or data changes. >>> >>> Propose to implement new IgniteCDC application as follows: >>> * Run on the server node host. >>> * Watches for the appearance of the WAL archive segments. >>> * Iterates it using existing WALIterator and notifies consumer of >>> each record from the segment. >>> >>> IgniteCDC features: >>> * Independence from the server node process (JVM) - issues and >>> failures of the consumer will not lead to server node instability. >>> * Notification guarantees and failover - i.e. CDC track and save >>> the pointer to the last consumed record. Continue notification from this >>> pointer in case of restart. >>> * Resilience for the consumer - it's not an issue when a consumer >>> temporarily consumes slower than data appear. >>> >>> WDYT? >>> >>> [1] >>> https://cwiki.apache.org/confluence/display/IGNITE/IEP-59+CDC+-+Capture+Data+Change >> >>
