Matt, In the case of CaptureChangeMySQL, does the third party library query the regular tables which have been modified, or a special table which captures CDC events from regular tables? To me that would be the distinction that would make it more effective than just using ExecuteSQL against regular tables, since only a single table would be scanned, that is not a subject to other operations.
-----Original Message----- From: Matt Burgess <[email protected]> Sent: Friday, January 18, 2019 1:10 PM To: [email protected] Subject: Re: CaptureChangeMySQL and Triggers Vitaly, There has been a PR [1] around for a while to add SQL Server CDC support, I'm not sure what shape it's in now. For Oracle you might be able to use QueryDatabaseTable to access LogMiner tables (if LogMiner is installed). Speaking of installation, unless there's a product installed at the RDBMS side to push data somewhere (GoldenGate, e.g.), then the solution is necessarily polling. For example, the third-party library used by CaptureChangeMySQL does polling/querying under the hood, it just uses MySQL's native API rather than SQL over JDBC. Regards, Matt [1] https://github.com/apache/nifi/pull/2231 On Fri, Jan 18, 2019 at 12:07 PM Vitaly Krivoy <[email protected]> wrote: > > This is a follow-on question to Apache/HortonWorks, rather than an answer to > the question posted by Marcelo. Outside of CaptureChangeMySQL, are there > plans underway to add similar processors for other databases? I realize that > a database would have to produce a capture data change log for this feature > to be implemented inexpensively. One of the objections that I that I > constantly have to face in my organization to NiFi adaptation is that > ExecuteSQL would require polling thus affecting performance of the source > DBMS system. I realize that this objection is silly and if > modification/creation timestamp column in the table is indexed, selecting the > rows that have been added/modified after last run date in ExecuteSQL would > barely affect the server. But as a consulting architect, I have to deal with > non-technical clients who make their decisions based on buzz words and they > have heard of change data capture logs marketed advantages. Thanks. > > > > From: Marcelo Terres <[email protected]> > Sent: Thursday, January 17, 2019 5:51 AM > To: [email protected] > Subject: CaptureChangeMySQL and Triggers > > > > Hello. > > > > Is someone here using CaptureChangeMySQL processor to get data from a table > which data is generated and managed by triggers/stored procedures? > > > > I'm having some weird issues as data not being processed in case of specific > inserts and also some weird data being generated in case of simple operations > (3 objects in one update operation, for example). > > > > Thanks in advance, > > > > Regards, > > > > Marcelo H. Terres > > <[email protected]> > https://www.mundoopensource.com.br > https://twitter.com/mhterres > https://linkedin.com/in/marceloterres > > > > STATEMENT OF CONFIDENTIALITY The information contained in this email message > and any attachments may be confidential and legally privileged and is > intended for the use of the addressee(s) only. If you are not an intended > recipient, please: (1) notify me immediately by replying to this message; (2) > do not use, disseminate, distribute or reproduce any part of the message or > any attachment; and (3) destroy all copies of this message and any > attachments. STATEMENT OF CONFIDENTIALITY The information contained in this email message and any attachments may be confidential and legally privileged and is intended for the use of the addressee(s) only. If you are not an intended recipient, please: (1) notify me immediately by replying to this message; (2) do not use, disseminate, distribute or reproduce any part of the message or any attachment; and (3) destroy all copies of this message and any attachments.
