Hi Chathuranga, As discussed offline let's look at Debizum[1] as well especially embedded Debizium[2], which removes the dependency on Kafka[3]
Also, shall we have a User Story review for the above functionality? [1] http://debezium.io/ [2] http://debezium.io/docs/embedded/ [3] Mail Thread : CDC (Change Data Capture) for Product-SP Best Regards, *Niveathika Rajendran,* *Software Engineer.* *Mobile : +94 077 903 7536* On Tue, Jul 24, 2018 at 12:25 PM Chathuranga Siriwardhana < [email protected]> wrote: > > > > > > > > *Hi all,I am currently working on the 'Change Data Capture for WSO2SP'. As > the first part of the project, I have to capture the change events(INSERT, > UPDATE, DELETE mainly) of a few databases using java. Mainly I have focused > on MySQL, MS-SQL server, ORACLE and PostgreSQL databases.For now, I have > used the following approaches to try to capture the change data of relevant > databases.DatabaseMethod UsedUsed connectors / > APIs.OutcomeStatusCommentsReferencesMySQLBinlog file based java event > listening. (Asynchronous)Mysql-binlog-connector-java(Apache 2.0 > License)Change events(INSERT, UPDATE, DELETE) can be captured along with > the relevant row data.Complete[1]OracleDBDatabase Change Notification > feature based java event listeningOracle jdbc driverChange events(INSERT, > UPDATE, DELETE) can be captured without row data.Partially > Complete[2]PostgreSQLListen Notify functionality of the Postgres with > table-wise configuration and polling in the program.Postgres jdbc > driverChange events(INSERT, UPDATE, DELETE) without row data.IncompleteThis > method was discarded because a trigger should be written into each table in > order to get change events from that table.[3], [4]PostgreSQLReplication > Data based java pollingPhysical and Logical replication APIChange > events(INSERT, UPDATE, DELETE) can be captured. Row data is not returning > for DELETE event.Partially Complete[5]MS-SQL ServerTransaction Log based > java pollingMs-sql jdbc connectorOn progress; Transaction log is > read.On-going.[6], [7][1] > https://github.com/shyiko/mysql-binlog-connector-java > <https://github.com/shyiko/mysql-binlog-connector-java>[2] > https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820 > <https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820>[3] > https://www.postgresql.org/docs/current/static/sql-notify.html > <https://www.postgresql.org/docs/current/static/sql-notify.html>[4] > https://www.postgresql.org/docs/current/static/sql-listen.html > <https://www.postgresql.org/docs/current/static/sql-listen.html>[5] > https://jdbc.postgresql.org/documentation/head/replication.html > <https://jdbc.postgresql.org/documentation/head/replication.html>[6]https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-2017 > <https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-2017>[7] > https://dzone.com/articles/how-use-sql-server-transaction > <https://dzone.com/articles/how-use-sql-server-transaction>*Please find > the shared Google Sheet containing above table. You may comment about the > existing approaches / mention any new approaches. > > Best regards, > Chathuranga Siriwardhana, > Software Engineering Intern. > Mobile: +94713604485 >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
