[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1026044085 > Thanks for picking up and working on this feature @davyam! I noted several areas for improvement in terms of code formatting. Hi @exceptionfactory! The improvements were made. Thanks for the feedback and suggestions! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1024972496 Hey @exceptionfactory! Thanks for the feedback! I'll take a look. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1024754843 Hi @joewitt! The last build failed because we weren't expecting different time zones in the unit tests (being more specific, xCommitTime in the flowfile content). Now, this problem was solved. When possible, please, start the build again. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1024517839 Hi @gerdansantos! As you suggested, the replication feedback now occurs after the flowfiles transfer to success relation completes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1024199981 @joewitt @pvillard31 @mattyb149 Hey guys, please, fire off again the workflow. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020539193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1021118807 > > Fired off the workflow run for new contributor (thanks!) and closed the old referenced PR as requested. Hopoefully this one makes quick progress into a merge/release. > > Thanks > > @joewitt > > The unit tests are failing because they are trying to validate the driver location. I will fix this, leaving this validation only in the integration tests. Hi @joewitt! The driver location validation was removed from Unit Tests. Please, fire off the workflow again. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020724700 > Fired off the workflow run for new contributor (thanks!) and closed the old referenced PR as requested. Hopoefully this one makes quick progress into a merge/release. > > Thanks @joewitt The unit tests are failing because they are trying to validate the driver location. I will fix this, leaving this validation only in the integration tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710:
URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020720083
> I have tested this code.
>
> ✅ Build without exceptions ✅ The mock tests seem enough ✅ The CDC start
without problems including PostgreSQL slots ✅ The Syncing Turn On process runs
fine ❌ The recovery process, dont conect and recovery from abrutal shutdown of
PostgreSQL of NiFi. I beleive the processor stat came be definied before
PostgreSQL lsn Commit, so the PostgreSQL cannot resync with the last past LSN.
✅If the stop process, the processor restart from last step without problems. ❌
The connection with a slot previously created don't run.
>
> Congrats for the evolution of this processor code.
Hi man! Thanks for the feedback!
About the recovery test, please give me more information about your
simulation. The PostgreSQL Replication Slot doesn't keep changes after the
feedback (setAppliedLSN and setFlushedLSN functions), which is performed as
soon as the change is received by the processor. Then, if you try to set the
Start LSN propriety with an LSN already received, that will not work and the
processor will start from the pending changes available in the slot. But you
can test the opposite, I mean, you can request an LSN after many unreceived
transactions, so the processor will skip them. Take a look at the integration
test testCDCStartLSN.
Talking about the test with an existing slot, this is working. I tested it
many times, including integration tests. I believe that, previously, you don't
create the slot with **pgoutput** output plugin. For example:
`SELECT pg_create_logical_replication_slot('slot_city', 'pgoutput');`
Probably this is the reason. Anyway, now I see that this information needs
to be included in the processor description. I will make this improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020553625 Hey @rrjanbiah @driesva @mathiasbosman, We believe that the problems and improvements asked in the last PR #4065 were made. Feel free to check and test. We are counting on your feedback too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logica
davyam commented on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020539193 @pvillard31 and @mattyb149 this is a new version of CaptureChangePostgreSQL processor. The code was refactored and many improvements were made. I really appreciate your feedback! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
