[GitHub] [nifi] davyam edited a comment on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via
davyam edited a comment on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1024972496 Hey @exceptionfactory! Thanks for the feedback! I'll take a look at the suggestions. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam edited a comment on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via
davyam edited a comment on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1024517839 Hi @gerdansantos! As you suggested, the replication feedback now occurs after the flowfiles transfer to success relation completes. Thanks man! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam edited a comment on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via
davyam edited a comment on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020539193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam edited a comment on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via
davyam edited a comment on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020720083 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam edited a comment on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via
davyam edited a comment on pull request #5710:
URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020720083
> I have tested this code.
>
> ✅ Build without exceptions ✅ The mock tests seem enough ✅ The CDC start
without problems including PostgreSQL slots ✅ The Syncing Turn On process runs
fine ❌ The recovery process, dont conect and recovery from abrutal shutdown of
PostgreSQL of NiFi. I beleive the processor stat came be definied before
PostgreSQL lsn Commit, so the PostgreSQL cannot resync with the last past LSN.
✅If the stop process, the processor restart from last step without problems. ❌
The connection with a slot previously created don't run.
>
> Congrats for the evolution of this processor code.
Hi man! Thanks for the feedback!
About the recovery test, please give me more information about your
simulation. The PostgreSQL Replication Slot doesn't keep changes after the
feedback (setAppliedLSN and setFlushedLSN functions), which is performed as
soon as the change is received by the processor. Then, if you try to set the
Start LSN property with an LSN already received, that will not work and the
processor will start from the pending changes available in the slot. But you
can test the opposite, I mean, you can request an LSN after many unreceived
transactions, so the processor will skip them and start from the informed LSN
(since the state is clean). Take a look at the integration test testCDCStartLSN.
Talking about the test with an existing slot, this is working. I tested it
many times, including integration tests. I believe that, previously, you don't
create the slot with **pgoutput** output plugin. For example:
`SELECT pg_create_logical_replication_slot('slot_city', 'pgoutput');`
Probably this is the reason. Anyway, now I see that this information needs
to be included in the processor description. I will make this improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] davyam edited a comment on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via
davyam edited a comment on pull request #5710:
URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020720083
> I have tested this code.
>
> ✅ Build without exceptions ✅ The mock tests seem enough ✅ The CDC start
without problems including PostgreSQL slots ✅ The Syncing Turn On process runs
fine ❌ The recovery process, dont conect and recovery from abrutal shutdown of
PostgreSQL of NiFi. I beleive the processor stat came be definied before
PostgreSQL lsn Commit, so the PostgreSQL cannot resync with the last past LSN.
✅If the stop process, the processor restart from last step without problems. ❌
The connection with a slot previously created don't run.
>
> Congrats for the evolution of this processor code.
Hi man! Thanks for the feedback!
About the recovery test, please give me more information about your
simulation. The PostgreSQL Replication Slot doesn't keep changes after the
feedback (setAppliedLSN and setFlushedLSN functions), which is performed as
soon as the change is received by the processor. Then, if you try to set the
Start LSN propriety with an LSN already received, that will not work and the
processor will start from the pending changes available in the slot. But you
can test the opposite, I mean, you can request an LSN after many unreceived
transactions, so the processor will skip them and start from the informed LSN
(since the state is clean). Take a look at the integration test testCDCStartLSN.
Talking about the test with an existing slot, this is working. I tested it
many times, including integration tests. I believe that, previously, you don't
create the slot with **pgoutput** output plugin. For example:
`SELECT pg_create_logical_replication_slot('slot_city', 'pgoutput');`
Probably this is the reason. Anyway, now I see that this information needs
to be included in the processor description. I will make this improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] davyam edited a comment on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via
davyam edited a comment on pull request #5710:
URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020720083
> I have tested this code.
>
> ✅ Build without exceptions ✅ The mock tests seem enough ✅ The CDC start
without problems including PostgreSQL slots ✅ The Syncing Turn On process runs
fine ❌ The recovery process, dont conect and recovery from abrutal shutdown of
PostgreSQL of NiFi. I beleive the processor stat came be definied before
PostgreSQL lsn Commit, so the PostgreSQL cannot resync with the last past LSN.
✅If the stop process, the processor restart from last step without problems. ❌
The connection with a slot previously created don't run.
>
> Congrats for the evolution of this processor code.
Hi man! Thanks for the feedback!
About the recovery test, please give me more information about your
simulation. The PostgreSQL Replication Slot doesn't keep changes after the
feedback (setAppliedLSN and setFlushedLSN functions), which is performed as
soon as the change is received by the processor. Then, if you try to set the
Start LSN propriety with an LSN already received, that will not work and the
processor will start from the pending changes available in the slot. But you
can test the opposite, I mean, you can request an LSN after many unreceived
transactions, so the processor will skip them and start from the informed LSN.
Take a look at the integration test testCDCStartLSN.
Talking about the test with an existing slot, this is working. I tested it
many times, including integration tests. I believe that, previously, you don't
create the slot with **pgoutput** output plugin. For example:
`SELECT pg_create_logical_replication_slot('slot_city', 'pgoutput');`
Probably this is the reason. Anyway, now I see that this information needs
to be included in the processor description. I will make this improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] davyam edited a comment on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via
davyam edited a comment on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020553625 Hey @rrjanbiah @driesva @mathiasbosman, We believe that the improvements asked in the last PR #4065 were made. Feel free to check and test. We are counting on your feedback too. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam edited a comment on pull request #5710: NIFI-4239 - Adding (REFACTORED) CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via
davyam edited a comment on pull request #5710: URL: https://github.com/apache/nifi/pull/5710#issuecomment-1020539193 Hi @pvillard31 and @mattyb149! This is a new version of CaptureChangePostgreSQL processor. The code was refactored and many improvements were made. I really appreciate your feedback! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
