[GitHub] [nifi] davyam edited a comment on pull request #3887: NIFI-6032 - CDC for Oracle using xstream
davyam edited a comment on pull request #3887: URL: https://github.com/apache/nifi/pull/3887#issuecomment-711200489 Hi @rkarthik29 I got it! It is working! https://user-images.githubusercontent.com/32827630/96371553-9c1e3c00-1138-11eb-9ae5-de97fb5595c1.png;> https://user-images.githubusercontent.com/32827630/96371561-a0e2f000-1138-11eb-809a-169baae40fa2.png;> I think that the problem was the StandardOracleClassLoaderService config. I changed the value of Database Driver Location(s) to **/usr/lib/oracle/12.2/client64/lib**, instead of the jar path. https://user-images.githubusercontent.com/32827630/96371704-704f8600-1139-11eb-937c-bdb286404b47.png;> Now I will do more tests. Thanks! Davy Machado This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam edited a comment on pull request #3887: NIFI-6032 - CDC for Oracle using xstream
davyam edited a comment on pull request #3887: URL: https://github.com/apache/nifi/pull/3887#issuecomment-710739387 Hi @rkarthik29 I am getting the following error: `org.apache.nifi.processor.exception.ProcessException: cannot attach to outbound server: nfxout` Steps that I did: **BUILD NIFI** ``` git clone https://github.com/rkarthik29/nifi-1.git -b NIFI-6032 cd nifi-1 export MAVEN_OPTS="-Xms1024m -Xmx3076m -XX:MaxPermSize=256m" mvn -T C2.0 clean install -Pinclude-grpc -Pinclude-oraclecdc # Error: [ERROR] Failed to execute goal on project nifi-splunk-processors: Could not resolve dependencies for project org.apache.nifi:nifi-splunk-processors...Could not find artifact com.splunk:splunk:jar:1.6.5.0 in splunk... # Solution: https://feitam.es/nifi-1-11-4-error-maven-could-not-resolve-dependencies-update-interval mvn -T C2.0 clean install -Pinclude-grpc -Pinclude-oraclecdc -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -Dmaven.wagon.http.ssl.ignore.validity.dates=true cd nifi-assembly/target/ tar -xvzf nifi-1.11.0-SNAPSHOT-bin.tar.gz sh nifi-1.11.0-SNAPSHOT/bin/nifi.sh run ``` **INSTALL ORACLE DRIVER** ``` # Download driver: https://www.oracle.com/br/database/technologies/instant-client/linux-x86-64-downloads.html yum install oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm ``` **CONFIG LIBS** ``` vim ~./bash_profile ... export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib:$LD_LIBRARY_PATH source ~./bash_profile vim nifi-1.11.0-SNAPSHOT/conf/bootstrap.conf ... java.arg.17=-Djava.library.path=/usr/lib/oracle/12.2/client64/lib ... sh nifi-1.11.0-SNAPSHOT/bin/nifi.sh run ``` **CREATE ORACLE XSTREAM OUTBOUND SERVER** ``` # Create Recovery Area dir: mkdir -p /opt/oracle/oradata/recovery_area chown -R oracle: /opt/oracle # Enable GoldenGate replication and archive log mode: su - oracle sqlplus /nolog CONNECT sys AS SYSDBA alter system set db_recovery_file_dest_size = 5G; alter system set db_recovery_file_dest = '/opt/oracle/oradata/recovery_area' scope=spfile; alter system set enable_goldengate_replication=true; shutdown immediate startup mount alter database archivelog; alter database open; # Should show "Database log mode: Archive Mode": archive log list exit; # Create Tablespace dir: mkdir -p /opt/oracle/oradata/tbs # Create an XStream admin user: sqlplus / as sysdba CREATE TABLESPACE xstream_adm_tbs DATAFILE '/opt/oracle/oradata/tbs/xstream_adm_tbs.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; CREATE USER xstrmadmin IDENTIFIED BY xsa DEFAULT TABLESPACE xstream_adm_tbs QUOTA UNLIMITED ON xstream_adm_tbs; GRANT CREATE SESSION TO xstrmadmin; BEGIN DBMS_XSTREAM_AUTH.GRANT_ADMIN_PRIVILEGE( grantee => 'xstrmadmin', privilege_type => 'CAPTURE', grant_select_privileges => TRUE ); END; / exit; # Create XStream user: sqlplus / as sysdba CREATE TABLESPACE xstream_tbs DATAFILE '/opt/oracle/oradata/tbs/xstream_tbs.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; CREATE USER xstrm IDENTIFIED BY xs DEFAULT TABLESPACE xstream_tbs QUOTA UNLIMITED ON xstream_tbs; GRANT CREATE SESSION TO xstrm; GRANT SELECT ON V_$DATABASE to xstrm; GRANT FLASHBACK ANY TABLE TO xstrm; exit; # Create NiFi user: sqlplus / as sysdba CREATE TABLESPACE nifi_tbs DATAFILE '/opt/oracle/oradata/tbs/nifi_tbs.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; CREATE USER nifi IDENTIFIED BY nifi DEFAULT TABLESPACE nifi_tbs QUOTA UNLIMITED ON nifi_tbs; CREATE TABLE nifi.tb_teste (id NUMBER PRIMARY KEY, name VARCHAR2(150)); ALTER TABLE nifi.tb_teste ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; exit; # Create an XStream Outbound server: sqlplus xstrmadmin/xsa DECLARE tables DBMS_UTILITY.UNCL_ARRAY; schemas DBMS_UTILITY.UNCL_ARRAY; BEGIN tables(1) := NULL; schemas(1) := 'nifi'; DBMS_XSTREAM_ADM.CREATE_OUTBOUND( server_name => 'nfxout', table_names => tables, schema_names=> schemas); END; / exit; # Alter the XStream Outbound server to allow the xstrm user to connect to it: sqlplus / as sysdba BEGIN DBMS_XSTREAM_ADM.ALTER_OUTBOUND( server_name => 'nfxout', connect_user => 'xstrm'); END; / exit; # Insert data: INSERT INTO nifi.tb_teste VALUES (1, 'Davy'); ``` NiFi Processor: https://user-images.githubusercontent.com/32827630/96327216-0dad9b80-100e-11eb-991d-00b687f9b3f5.png;>
[GitHub] [nifi] davyam edited a comment on pull request #3887: NIFI-6032 - CDC for Oracle using xstream
davyam edited a comment on pull request #3887: URL: https://github.com/apache/nifi/pull/3887#issuecomment-710739387 Hi @rkarthik29 I am getting the following error: `org.apache.nifi.processor.exception.ProcessException: cannot attach to outbound server: nfxout` Steps that I did: **BUILD NIFI** ``` git clone https://github.com/rkarthik29/nifi-1.git -b NIFI-6032 cd nifi-1 export MAVEN_OPTS="-Xms1024m -Xmx3076m -XX:MaxPermSize=256m" mvn -T C2.0 clean install -Pinclude-grpc -Pinclude-oraclecdc # Error: [ERROR] Failed to execute goal on project nifi-splunk-processors: Could not resolve dependencies for project org.apache.nifi:nifi-splunk-processors...Could not find artifact com.splunk:splunk:jar:1.6.5.0 in splunk... # Solution: mvn -T C2.0 clean install -Pinclude-grpc -Pinclude-oraclecdc -Dmaven.wagon.http.ssl.insecure=true -Dmaven.wagon.http.ssl.allowall=true -Dmaven.wagon.http.ssl.ignore.validity.dates=true cd nifi-assembly/target/ tar -xvzf nifi-1.11.0-SNAPSHOT-bin.tar.gz sh nifi-1.11.0-SNAPSHOT/bin/nifi.sh run ``` **INSTALL ORACLE DRIVER** ``` # Download driver: https://www.oracle.com/br/database/technologies/instant-client/linux-x86-64-downloads.html yum install oracle-instantclient12.2-basic-12.2.0.1.0-1.x86_64.rpm ``` **CONFIG LIBS** ``` vim ~./bash_profile ... export LD_LIBRARY_PATH=/usr/lib/oracle/12.2/client64/lib:$LD_LIBRARY_PATH source ~./bash_profile vim nifi-1.11.0-SNAPSHOT/conf/bootstrap.conf ... java.arg.17=-Djava.library.path=/usr/lib/oracle/12.2/client64/lib ... sh nifi-1.11.0-SNAPSHOT/bin/nifi.sh run ``` **CREATE ORACLE XSTREAM OUTBOUND SERVER** ``` # Create Recovery Area dir: mkdir -p /opt/oracle/oradata/recovery_area chown -R oracle: /opt/oracle # Enable GoldenGate replication and archive log mode: su - oracle sqlplus /nolog CONNECT sys AS SYSDBA alter system set db_recovery_file_dest_size = 5G; alter system set db_recovery_file_dest = '/opt/oracle/oradata/recovery_area' scope=spfile; alter system set enable_goldengate_replication=true; shutdown immediate startup mount alter database archivelog; alter database open; # Should show "Database log mode: Archive Mode": archive log list exit; # Create Tablespace dir: mkdir -p /opt/oracle/oradata/tbs # Create an XStream admin user: sqlplus / as sysdba CREATE TABLESPACE xstream_adm_tbs DATAFILE '/opt/oracle/oradata/tbs/xstream_adm_tbs.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; CREATE USER xstrmadmin IDENTIFIED BY xsa DEFAULT TABLESPACE xstream_adm_tbs QUOTA UNLIMITED ON xstream_adm_tbs; GRANT CREATE SESSION TO xstrmadmin; BEGIN DBMS_XSTREAM_AUTH.GRANT_ADMIN_PRIVILEGE( grantee => 'xstrmadmin', privilege_type => 'CAPTURE', grant_select_privileges => TRUE ); END; / exit; # Create XStream user: sqlplus / as sysdba CREATE TABLESPACE xstream_tbs DATAFILE '/opt/oracle/oradata/tbs/xstream_tbs.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; CREATE USER xstrm IDENTIFIED BY xs DEFAULT TABLESPACE xstream_tbs QUOTA UNLIMITED ON xstream_tbs; GRANT CREATE SESSION TO xstrm; GRANT SELECT ON V_$DATABASE to xstrm; GRANT FLASHBACK ANY TABLE TO xstrm; exit; # Create NiFi user: sqlplus / as sysdba CREATE TABLESPACE nifi_tbs DATAFILE '/opt/oracle/oradata/tbs/nifi_tbs.dbf' SIZE 25M REUSE AUTOEXTEND ON MAXSIZE UNLIMITED; CREATE USER nifi IDENTIFIED BY nifi DEFAULT TABLESPACE nifi_tbs QUOTA UNLIMITED ON nifi_tbs; CREATE TABLE nifi.tb_teste (id NUMBER PRIMARY KEY, name VARCHAR2(150)); ALTER TABLE nifi.tb_teste ADD SUPPLEMENTAL LOG DATA (ALL) COLUMNS; exit; # Create an XStream Outbound server: sqlplus xstrmadmin/xsa DECLARE tables DBMS_UTILITY.UNCL_ARRAY; schemas DBMS_UTILITY.UNCL_ARRAY; BEGIN tables(1) := NULL; schemas(1) := 'nifi'; DBMS_XSTREAM_ADM.CREATE_OUTBOUND( server_name => 'nfxout', table_names => tables, schema_names=> schemas); END; / exit; # Alter the XStream Outbound server to allow the xstrm user to connect to it: sqlplus / as sysdba BEGIN DBMS_XSTREAM_ADM.ALTER_OUTBOUND( server_name => 'nfxout', connect_user => 'xstrm'); END; / exit; # Insert data: INSERT INTO nifi.tb_teste VALUES (1, 'Davy'); ``` NiFi Processor: https://user-images.githubusercontent.com/32827630/96327216-0dad9b80-100e-11eb-991d-00b687f9b3f5.png;>
[GitHub] [nifi] davyam edited a comment on pull request #3887: NIFI-6032 - CDC for Oracle using xstream
davyam edited a comment on pull request #3887: URL: https://github.com/apache/nifi/pull/3887#issuecomment-708867523 @rkarthik29 Thanks for creating this PR! ASAP I will test this contribution and report the results. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [nifi] davyam edited a comment on pull request #3887: NIFI-6032 - CDC for Oracle using xstream
davyam edited a comment on pull request #3887: URL: https://github.com/apache/nifi/pull/3887#issuecomment-708871481 @rkarthik29 I built this PR, but the OracleChangeCapture did not appear on the processor list. Commands executed: ``` git clone https://github.com/rkarthik29/nifi-1.git cd nifi-1 git checkout NIFI-6032 export MAVEN_OPTS="-Xms1024m -Xmx3076m -XX:MaxPermSize=256m" mvn -T C2.0 clean install -Pinclude-grpc cd nifi-assembly/target/ tar -xvzf nifi-1.11.0-SNAPSHOT-bin.tar.gz sh nifi-1.11.0-SNAPSHOT/bin/nifi.sh run ``` So I had to copy the nars from **nifi-1/nifi-nar-bundles/nifi-oraclecdc-bundle** to the **nifi-1.11.0-SNAPSHOT/lib** directory manually. Please, check the parent pom.xml. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
