Hi all, I tried the CDC for MySQL with Debezium[1] and implemented as a simple java program using embedded debezium[2]. Currently I am trying the Embedded Debezium for Oracle. The respected java code can be found here.
https://github.com/chathuranga95/Embedded-Debezium-CDC/blob/master/src/main/java/io/debezium/App.java Please note that the runMysqlCDC() is returning the MySQL database change event details with row data and currently runOracleCDC() is not producing expected output. Find the details of configuration from [2]. I also got started with the user story for this functionality. The source will be as follows for MySQL bin-log CDC. @source(type=’cdc’ , url=’jdbc:mysql://root:root@localhost:3306/dbStudents’, @map(type='json', fail.on.missing.attribute='false' ,@attributes(operation=’$.operation, beforeID=’$.before.id’, beforeName=’$. before.name’, afterID=’$.after.id’, afterName=’$.after.name’))) define stream cdcEvent(operation String, beforeID String, beforeName String, afterID String, afterName String); [1] http://debezium.io/ [2] http://debezium.io/docs/embedded/ Best regards, Chathuranga Siriwardhana, Software Engineering Intern. Mobile: +94713604485 On Tue, Jul 24, 2018 at 6:38 PM Niveathika Rajendran <[email protected]> wrote: > Hi Chathuranga, > > As discussed offline let's look at Debizum[1] as well especially > embedded Debizium[2], which removes the dependency on Kafka[3] > > Also, shall we have a User Story review for the above functionality? > > [1] http://debezium.io/ > [2] http://debezium.io/docs/embedded/ > [3] Mail Thread : CDC (Change Data Capture) for Product-SP > > Best Regards, > *Niveathika Rajendran,* > *Software Engineer.* > *Mobile : +94 077 903 7536* > > > > > > On Tue, Jul 24, 2018 at 12:25 PM Chathuranga Siriwardhana < > [email protected]> wrote: > >> >> >> >> >> >> >> >> *Hi all,I am currently working on the 'Change Data Capture for WSO2SP'. >> As the first part of the project, I have to capture the change >> events(INSERT, UPDATE, DELETE mainly) of a few databases using java. Mainly >> I have focused on MySQL, MS-SQL server, ORACLE and PostgreSQL databases.For >> now, I have used the following approaches to try to capture the change data >> of relevant databases.DatabaseMethod UsedUsed connectors / >> APIs.OutcomeStatusCommentsReferencesMySQLBinlog file based java event >> listening. (Asynchronous)Mysql-binlog-connector-java(Apache 2.0 >> License)Change events(INSERT, UPDATE, DELETE) can be captured along with >> the relevant row data.Complete[1]OracleDBDatabase Change Notification >> feature based java event listeningOracle jdbc driverChange events(INSERT, >> UPDATE, DELETE) can be captured without row data.Partially >> Complete[2]PostgreSQLListen Notify functionality of the Postgres with >> table-wise configuration and polling in the program.Postgres jdbc >> driverChange events(INSERT, UPDATE, DELETE) without row data.IncompleteThis >> method was discarded because a trigger should be written into each table in >> order to get change events from that table.[3], [4]PostgreSQLReplication >> Data based java pollingPhysical and Logical replication APIChange >> events(INSERT, UPDATE, DELETE) can be captured. Row data is not returning >> for DELETE event.Partially Complete[5]MS-SQL ServerTransaction Log based >> java pollingMs-sql jdbc connectorOn progress; Transaction log is >> read.On-going.[6], [7][1] >> https://github.com/shyiko/mysql-binlog-connector-java >> <https://github.com/shyiko/mysql-binlog-connector-java>[2] >> https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820 >> <https://docs.oracle.com/cd/E11882_01/java.112/e16548/dbchgnf.htm#JJDBC28820>[3] >> https://www.postgresql.org/docs/current/static/sql-notify.html >> <https://www.postgresql.org/docs/current/static/sql-notify.html>[4] >> https://www.postgresql.org/docs/current/static/sql-listen.html >> <https://www.postgresql.org/docs/current/static/sql-listen.html>[5] >> https://jdbc.postgresql.org/documentation/head/replication.html >> <https://jdbc.postgresql.org/documentation/head/replication.html>[6]https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-2017 >> <https://docs.microsoft.com/en-us/sql/relational-databases/logs/the-transaction-log-sql-server?view=sql-server-2017>[7] >> https://dzone.com/articles/how-use-sql-server-transaction >> <https://dzone.com/articles/how-use-sql-server-transaction>*Please find >> the shared Google Sheet containing above table. You may comment about the >> existing approaches / mention any new approaches. >> >> Best regards, >> Chathuranga Siriwardhana, >> Software Engineering Intern. >> Mobile: +94713604485 >> >
_______________________________________________ Architecture mailing list [email protected] https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
