[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-1020534864 Guys, as the code was refactored and the processor behavior was changed, we decided to start a new PR #5710. Then, we avoid misunderstandings with old messages. There are several improvements in this new PR #5710 and we hope that all the contributors keep doing tests and giving us feedback from there. ASAP this PR can be closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-1020534864 Guys, as the code was refactored and the processor behavior was changed, we decided to start a new PR #5710. Then, we avoid misunderstandings with old messages. There are several improvements in this new PR #5710 and we hope that all the contributors keep doing tests and giving us feedback from there. ASAP this PR can be closed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-1008315183 I'm working on some improvements and mock tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-908329436 Thanks @pvillard31 for the update. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-908329436 Thanks @pvillard31 for the update. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-907731534 We did some changes and we still working for improve the tests, as requested by @mattyb149 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-906432924 Hey guys, we will make the necessary changes next weekend. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-827629376 Hi @driesva I got, sorry for the delay... @gerdansantos and I are very busy with our works. In the next month, we gonna make an effort to update this PR... in the meanwhile, I will discuss with @gerdansantos how we can work together. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-822699700 @mathiasbosman @driesva Hey guys! Why are you creating a new PR copying the code instead of work in collaboration with me and @gerdansantos to improve this? As you say in the new PR, you are not Postgres experts, but we are. In the same way, I and @gerdansantos are not Java experts. So, we can work together to make a better processor. This is the community's spirit. Let us know if you need anything. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065:
URL: https://github.com/apache/nifi/pull/4065#issuecomment-671375800
> > > > With the replacement from JSON Simple to Jackson, the output
(content) changed too.
> > > > Example:
> > > > ```
> > > > {
> > > > "tupleData" : {
> > > > "codigo" : 4,
> > > > "nome" : "CDC",
> > > > "data_fund" : "1929-10-19"
> > > > },
> > > > "relationName" : "public.cidade",
> > > > "type" : "insert"
> > > > }
> > > > ```
> > >
> > >
> > > @davyam could you please possibly explain how to use Jackson JSON
format for a CDC event? Or maybe it is documented somewhere.
> > > Thanks!
> >
> >
> > > > With the replacement from JSON Simple to Jackson, the output
(content) changed too.
> > > > Example:
> > > > ```
> > > > {
> > > > "tupleData" : {
> > > > "codigo" : 4,
> > > > "nome" : "CDC",
> > > > "data_fund" : "1929-10-19"
> > > > },
> > > > "relationName" : "public.cidade",
> > > > "type" : "insert"
> > > > }
> > > > ```
> > >
> > >
> > > @davyam could you please possibly explain how to use Jackson JSON
format for a CDC event? Or maybe it is documented somewhere.
> > > Thanks!
> >
> >
> > @energy2522 if I understood your question, you are asking for more
detail about the CDC event that is returned in flowfile content in JSON type.
> > So, for each row in an event (INSERT, UPDATE, DELETE) we have a JSON in
this format:
> > ```
> > {
> > "tupleData" : { _JSON object with all of the fields and their values_
},
> > "relationName" : _Table's name_,
> > "type" : _Event type (INSERT, UPDATE, DELTE, BEGIN*, COMMIT*)_
> > }
> > ```
> >
> >
> > *BEGIN and COMMIT events are included if the "Include Begin/Commit
Events" property are checked.
> > More info here: [https://github.com/davyam/pgEasyReplication](url)
>
> @davyam I tried to use this processor in my local task. For now the
processor returns flowFiles with content in such JSON format
{"insert":{"cidade":{"codigo":4,"nome":"New York","data_fund":"2020-02-24"}}}
>
> But, as you wrote after replacement from JSON Simple to Jackson it should
be in another format. So, my question is should I somehow configure this
processor to get output in Jackson format?
>
> Thanks!
You are using the old version.
To build the last version of this NiFi processor follow these steps:
```
### Download repo
git clone https://github.com/gerdansantos/nifi.git
cd nifi
### Export MAVEN options
export MAVEN_OPTS="-Xms1024m -Xmx3076m -XX:MaxPermSize=256m"
### Export JAVA_HOME
# On MacOS
export JAVA_HOME=$(/usr/libexec/java_home)
# On Linux
# export JAVA_HOME=/usr/jdk64/jdk1.8.0_112
### Build bundle
cd nifi-nar-bundles/nifi-cdc/nifi-cdc-postgresql-bundle
# You can omit the -Pinclude-grpc if you're not on a platform that it
supports. Include -DskipTests to compile tests, but skip running them.
mvn -T C2.0 clean install -Pinclude-grpc -DskipTests
### Move NAR to NiFi libs dir
cp
nifi-cdc-postgresql-nar/target/nifi-cdc-postgresql-nar-1.12.0-SNAPSHOT.nar
/Users/davyam/Documents/PROGRAMS/nifi-1.12.0-SNAPSHOT/lib/
### Check processor in NiFi Canvas
cd /Users/davyam/Documents/PROGRAMS/nifi-1.12.0-SNAPSHOT
sh bin/nifi.sh run
# Wait a few minutes to access http page
http://localhost:8080/nifi/
# CNTRL+C to shuts down NiFi
```
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-671358403 > > https://github.com/davyam/pgEasyReplication > > @davyam unfortunately, I can't see this page:( This is one of my public repositories, look at my github. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-671357997 > https://github.com/davyam/pgEasyReplication This is one of my public repositories, look at my github. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065:
URL: https://github.com/apache/nifi/pull/4065#issuecomment-671348801
> > With the replacement from JSON Simple to Jackson, the output (content)
changed too.
> > Example:
> > ```
> > {
> > "tupleData" : {
> > "codigo" : 4,
> > "nome" : "CDC",
> > "data_fund" : "1929-10-19"
> > },
> > "relationName" : "public.cidade",
> > "type" : "insert"
> > }
> > ```
>
> @davyam could you please possibly explain how to use Jackson JSON format
for a CDC event? Or maybe it is documented somewhere.
>
> Thanks!
> > With the replacement from JSON Simple to Jackson, the output (content)
changed too.
> > Example:
> > ```
> > {
> > "tupleData" : {
> > "codigo" : 4,
> > "nome" : "CDC",
> > "data_fund" : "1929-10-19"
> > },
> > "relationName" : "public.cidade",
> > "type" : "insert"
> > }
> > ```
>
> @davyam could you please possibly explain how to use Jackson JSON format
for a CDC event? Or maybe it is documented somewhere.
>
> Thanks!
@energy2522 if I understood your question, you are asking for more detail
about the CDC event that is returned in flowfile content in JSON type.
So, for each row in an event (INSERT, UPDATE, DELETE) we have a JSON in this
format:
{
"tupleData" : { _JSON object with all the fields and their values_ },
"relationName" : _Table's name_,
"type" : _Event type (INSERT, UPDATE, DELTE, BEGIN*, COMMIT*)_
}
*BEGIN and COMMIT events are included if the "Include Begin/Commit Events"
property are checked.
More info here: [https://github.com/davyam/pgEasyReplication](url)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-671332330 > @davyam Do you know that this CDC solution for PostgreSQL supports DDL events like in CaptureChangeMySQL, there is a boolean property `Include DDL Events`? But in the `nar` file that @gerdansantos shared with us this field is absent. Hi @energy2522, sorry for the delay. So, the short answer is no, it does not support. This CDC solution uses the native PostgreSQL logical replication API, that only replicates data changes. If you need the initial state of subscribed tables (including CREATE TABLE and INSERT for existing rows) you can use the SNAPSHOT property. The CaptureChangePostgreSQL is based on pgEasyReplication, then you can find more info here [https://github.com/davyam/pgEasyReplication](url) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam commented on pull request #4065: NIFI-4239 - Adding CaptureChangePostgreSQL processor to capture data changes (INSERT/UPDATE/DELETE) in PostgreSQL tables via Logical Replication
davyam commented on pull request #4065: URL: https://github.com/apache/nifi/pull/4065#issuecomment-653900456 > Hello team! > Do you have any plans to release this PR in the nearest time? > > Thanks! Hi @yegorbond, Saying as PR developers, we wish that this PR be included in the next NiFi version. This PR can help a lot of people. ASAP we should make more adjusts based on the lastest @mattyb149 comments. But it that can take a while because I and @gerdansantos have a lot of work to do in our professional jobs. Thanks for the support. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
