Re: [discuss] nifi 1.16.1
I have a few PRs for standardizing and updating our dependencies like Jackson and Avro. Might be good to get those included in 1.16.1 On Mon, Apr 4, 2022 at 3:02 PM Joe Witt wrote: > > Team, > > Sorry for the delays in wrapping up the 1.16 release from earlier last > week. We had great vote turnout and it was definitely a large release in > terms of features/etc.. > > Watching evolution on the 1.17 line since it is clear we can benefit from a > 1.16.1 release so I've started preparing that and will closely monitor > candidates for inclusion. > > The branch is pushed and is called 'support/nifi-1.16'. I've already > pulled in everything on 1.17 that looks like it can work in a pure 3rd > digit release. Some of this makes sense due to bugs found and or natural > concerns of scanners for things like springshell/etc.. In any case it is > always good to tighten up the release lines if time permits. > > I'll probably start the release process tomorrow or Wednesday. Will watch > for anything that comes in. > > Thanks
[discuss] nifi 1.16.1
Team, Sorry for the delays in wrapping up the 1.16 release from earlier last week. We had great vote turnout and it was definitely a large release in terms of features/etc.. Watching evolution on the 1.17 line since it is clear we can benefit from a 1.16.1 release so I've started preparing that and will closely monitor candidates for inclusion. The branch is pushed and is called 'support/nifi-1.16'. I've already pulled in everything on 1.17 that looks like it can work in a pure 3rd digit release. Some of this makes sense due to bugs found and or natural concerns of scanners for things like springshell/etc.. In any case it is always good to tighten up the release lines if time permits. I'll probably start the release process tomorrow or Wednesday. Will watch for anything that comes in. Thanks
[ANNOUNCE] Apache NiFi 1.16.0 release
Hello The Apache NiFi team would like to announce the release of Apache NiFi 1.16.0. This is a substantial feature, improvement, bug fix and security focused release with more than 400 JIRAs completed. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation, and system mediation logic. More details on Apache NiFi can be found here: https://nifi.apache.org/ The release artifacts can be downloaded from here: https://nifi.apache.org/download.html Maven artifacts have been made available and mirrored as per normal ASF artifact processes. Issues closed/resolved for this list can be found here: https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12350741 Release note highlights can be found here: https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.16.0 Thank you The Apache NiFi team
Re: Missing maven dependencies when building nifi
Thanks Joe! On Tue, 5 Apr 2022 at 00:53, Joe Witt wrote: > Phil > > I think you need to run 'git clean -fxd' from the nifi source root. > > Then try the build again. And allow the entire thing to run. > > Too many builds in different levels are leaving things around. > > Thanks > > On Mon, Apr 4, 2022 at 7:46 AM Phil H wrote: > > > So, about that error that flashed past... > > > > [INFO] > > > > [INFO] BUILD FAILURE > > [INFO] > > > > [INFO] Total time: 22:21 min (Wall Clock) > > [INFO] Finished at: 2022-04-05T00:37:46+10:00 > > [INFO] > > > > [ERROR] Failed to execute goal > org.apache.rat:apache-rat-plugin:0.13:check > > (default) on project nifi-poi-processors: Too many files with unapproved > > license: 1 See RAT report in: > > > > > D:\nifi.github\nifi\nifi-nar-bundles\nifi-poi-bundle\nifi-poi-processors\target\rat.txt > > -> [Help 1] > > > > The rat file contains the following: > > > > * > > Summary > > --- > > Generated at: 2022-04-05T00:33:56+10:00 > > > > Notes: 0 > > Binaries: 4 > > Archives: 0 > > Standards: 7 > > > > Apache Licensed: 6 > > Generated Documents: 0 > > > > JavaDocs are generated, thus a license header is optional. > > Generated files do not require license headers. > > > > 1 Unknown Licenses > > > > * > > > > Files with unapproved licenses: > > > > > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/Unsupported.xls > > > > * > > > > * > > Files with Apache License headers will be marked AL > > Binary files (which do not require any license headers) will be marked > B > > Compressed archives will be marked A > > Notices, licenses etc. will be marked N > > AL > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/pom.xml > > AL > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java > > AL > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/resources/docs/org.apache.nifi.processors.poi.ConvertExcelToCSVProcessor/additionalDetails.html > > AL > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor > > AL > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java > > B > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/CollegeScorecard.xlsx > > B > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/dataformatting.xlsx > > AL > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/logback-test.xml > > B > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/TwoSheets.xlsx > > !? > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/Unsupported.xls > > B > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/with-blank-cells.xlsx > > > > * > > > > Printing headers for text files without a valid license header... > > > > = > > == File: > > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/Unsupported.xls > > = > > -- some binary stuff -- > > > > > > On Tue, Apr 5, 2022 at 12:30 AM Phil H wrote: > > > > > Thanks Joe, it seems to be about half way through now - much further > than > > > previously. I thought I saw some red text flash past, but nothing > > > stopped. Does one of those switches you suggested pile all the bad > news > > up > > > to deliver at the end, or can maven have "red errors" that aren't > > terminal? > > > > > > TIA, > > > Phil > > > > > > On Mon, Apr 4, 2022 at 11:48 PM Joe Witt wrote: > > > > > >> Phil > > >> > > >> Here is a command I commonly run to ensure I have nothing other than > > >> what the source tree intends. > > >> > > >> mvn -T C1 clean package verify -Pcontrib-check,include-grpc > > >> > > >> This will do everything - build it all, run contrib check, and use all > > the > > >> cores. > > >> > > >> Thanks > > >> > > >> On Mon, Apr 4, 2022 at 6:38 AM Mike Thomsen > > >> wrote: > > >>
Re: Missing maven dependencies when building nifi
Phil I think you need to run 'git clean -fxd' from the nifi source root. Then try the build again. And allow the entire thing to run. Too many builds in different levels are leaving things around. Thanks On Mon, Apr 4, 2022 at 7:46 AM Phil H wrote: > So, about that error that flashed past... > > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 22:21 min (Wall Clock) > [INFO] Finished at: 2022-04-05T00:37:46+10:00 > [INFO] > > [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.13:check > (default) on project nifi-poi-processors: Too many files with unapproved > license: 1 See RAT report in: > > D:\nifi.github\nifi\nifi-nar-bundles\nifi-poi-bundle\nifi-poi-processors\target\rat.txt > -> [Help 1] > > The rat file contains the following: > > * > Summary > --- > Generated at: 2022-04-05T00:33:56+10:00 > > Notes: 0 > Binaries: 4 > Archives: 0 > Standards: 7 > > Apache Licensed: 6 > Generated Documents: 0 > > JavaDocs are generated, thus a license header is optional. > Generated files do not require license headers. > > 1 Unknown Licenses > > * > > Files with unapproved licenses: > > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/Unsupported.xls > > * > > * > Files with Apache License headers will be marked AL > Binary files (which do not require any license headers) will be marked B > Compressed archives will be marked A > Notices, licenses etc. will be marked N > AL > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/pom.xml > AL > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java > AL > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/resources/docs/org.apache.nifi.processors.poi.ConvertExcelToCSVProcessor/additionalDetails.html > AL > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor > AL > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java > B > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/CollegeScorecard.xlsx > B > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/dataformatting.xlsx > AL > > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/logback-test.xml > B > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/TwoSheets.xlsx > !? > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/Unsupported.xls > B > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/with-blank-cells.xlsx > > * > > Printing headers for text files without a valid license header... > > = > == File: > > D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/Unsupported.xls > = > -- some binary stuff -- > > > On Tue, Apr 5, 2022 at 12:30 AM Phil H wrote: > > > Thanks Joe, it seems to be about half way through now - much further than > > previously. I thought I saw some red text flash past, but nothing > > stopped. Does one of those switches you suggested pile all the bad news > up > > to deliver at the end, or can maven have "red errors" that aren't > terminal? > > > > TIA, > > Phil > > > > On Mon, Apr 4, 2022 at 11:48 PM Joe Witt wrote: > > > >> Phil > >> > >> Here is a command I commonly run to ensure I have nothing other than > >> what the source tree intends. > >> > >> mvn -T C1 clean package verify -Pcontrib-check,include-grpc > >> > >> This will do everything - build it all, run contrib check, and use all > the > >> cores. > >> > >> Thanks > >> > >> On Mon, Apr 4, 2022 at 6:38 AM Mike Thomsen > >> wrote: > >> > >> > Did you run "mvn install" from the root of the project before > >> > attempting to run a full test of the code base? > >> > > >> > On Mon, Apr 4, 2022 at 9:36 AM Phil H wrote: > >> > > > >> > > Hi there, > >> > > > >> > > So I got past this issue by running "mvn test" from the root of the > >> nifi > >> > > tree. After much compiling and
Re: Missing maven dependencies when building nifi
So, about that error that flashed past... [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 22:21 min (Wall Clock) [INFO] Finished at: 2022-04-05T00:37:46+10:00 [INFO] [ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.13:check (default) on project nifi-poi-processors: Too many files with unapproved license: 1 See RAT report in: D:\nifi.github\nifi\nifi-nar-bundles\nifi-poi-bundle\nifi-poi-processors\target\rat.txt -> [Help 1] The rat file contains the following: * Summary --- Generated at: 2022-04-05T00:33:56+10:00 Notes: 0 Binaries: 4 Archives: 0 Standards: 7 Apache Licensed: 6 Generated Documents: 0 JavaDocs are generated, thus a license header is optional. Generated files do not require license headers. 1 Unknown Licenses * Files with unapproved licenses: D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/Unsupported.xls * * Files with Apache License headers will be marked AL Binary files (which do not require any license headers) will be marked B Compressed archives will be marked A Notices, licenses etc. will be marked N AL D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/pom.xml AL D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessor.java AL D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/resources/docs/org.apache.nifi.processors.poi.ConvertExcelToCSVProcessor/additionalDetails.html AL D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/main/resources/META-INF/services/org.apache.nifi.processor.Processor AL D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/java/org/apache/nifi/processors/poi/ConvertExcelToCSVProcessorTest.java B D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/CollegeScorecard.xlsx B D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/dataformatting.xlsx AL D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/logback-test.xml B D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/TwoSheets.xlsx !? D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/Unsupported.xls B D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/with-blank-cells.xlsx * Printing headers for text files without a valid license header... = == File: D:/nifi.github/nifi/nifi-nar-bundles/nifi-poi-bundle/nifi-poi-processors/src/test/resources/Unsupported.xls = -- some binary stuff -- On Tue, Apr 5, 2022 at 12:30 AM Phil H wrote: > Thanks Joe, it seems to be about half way through now - much further than > previously. I thought I saw some red text flash past, but nothing > stopped. Does one of those switches you suggested pile all the bad news up > to deliver at the end, or can maven have "red errors" that aren't terminal? > > TIA, > Phil > > On Mon, Apr 4, 2022 at 11:48 PM Joe Witt wrote: > >> Phil >> >> Here is a command I commonly run to ensure I have nothing other than >> what the source tree intends. >> >> mvn -T C1 clean package verify -Pcontrib-check,include-grpc >> >> This will do everything - build it all, run contrib check, and use all the >> cores. >> >> Thanks >> >> On Mon, Apr 4, 2022 at 6:38 AM Mike Thomsen >> wrote: >> >> > Did you run "mvn install" from the root of the project before >> > attempting to run a full test of the code base? >> > >> > On Mon, Apr 4, 2022 at 9:36 AM Phil H wrote: >> > > >> > > Hi there, >> > > >> > > So I got past this issue by running "mvn test" from the root of the >> nifi >> > > tree. After much compiling and testing, I am now getting the attached >> > > error. I think I am following the contributor guide in having my >> local >> > > branch track origin/main (currently "4a90334728: NIFI-9846 Implement >> > > pagination listing for Azure List processors"). >> > > >> > > Is this some environmental issue on my machine somehow? Or does the >> main >> > > branch contain compilation errors, and I should be tracking something >> > else >> > > for my own branch? As I said in my previous email, I am new to >> github, >> > so >> > >
Re: SplitContent doesn’t support regex?
Phil, Yeah, that’s fine. We want to include the Jira number in the commit message, but you can include multiple by writing a message like: NIFI-3470, NIFI-1517: Addressed Thing #1 and Thing #2 Thanks -Mark > On Apr 4, 2022, at 10:27 AM, Phil H wrote: > > Whilst I try and get NiFi to build, let's circle back to JIRA. I found an > open issue that matches my requirement (NIFI-1517), however to implement my > solution, I'd also fix NIFI-3470 on the way (reading a configurable amount > of data to run the regex over, rather than byte-by-byte). > > So, what's the proper way to go about this from a JIRA perspective? I > assume my branch would be nifi-1517 as that's the feature I'm building, but > it would also "solve" 3470? > > TIA, > Phil > > > > > On Thu, Mar 17, 2022 at 9:12 AM Joe Witt wrote: > >> Phil >> >> I'd say if you have a good implementation in mind you should go for it. >> Sounds interesting. >> >> Thanks >> >> On Wed, Mar 16, 2022 at 3:59 PM Phil H wrote: >> >>> Hi, >>> >>> This seems like an odd omission - aside from performance (presumably?) is >>> there a reason why there isn’t a regex option for the byte sequence? I >> need >>> one but thought I’d ask before I built my own. >>> >>> Thanks >>> Phil >>> >>
Re: Missing maven dependencies when building nifi
Thanks Joe, it seems to be about half way through now - much further than previously. I thought I saw some red text flash past, but nothing stopped. Does one of those switches you suggested pile all the bad news up to deliver at the end, or can maven have "red errors" that aren't terminal? TIA, Phil On Mon, Apr 4, 2022 at 11:48 PM Joe Witt wrote: > Phil > > Here is a command I commonly run to ensure I have nothing other than > what the source tree intends. > > mvn -T C1 clean package verify -Pcontrib-check,include-grpc > > This will do everything - build it all, run contrib check, and use all the > cores. > > Thanks > > On Mon, Apr 4, 2022 at 6:38 AM Mike Thomsen > wrote: > > > Did you run "mvn install" from the root of the project before > > attempting to run a full test of the code base? > > > > On Mon, Apr 4, 2022 at 9:36 AM Phil H wrote: > > > > > > Hi there, > > > > > > So I got past this issue by running "mvn test" from the root of the > nifi > > > tree. After much compiling and testing, I am now getting the attached > > > error. I think I am following the contributor guide in having my local > > > branch track origin/main (currently "4a90334728: NIFI-9846 Implement > > > pagination listing for Azure List processors"). > > > > > > Is this some environmental issue on my machine somehow? Or does the > main > > > branch contain compilation errors, and I should be tracking something > > else > > > for my own branch? As I said in my previous email, I am new to github, > > so > > > I'm not sure of the conventions there. > > > > > > Thanks in advance > > > Phil > > > > > > (PS: this occur whethers I use JDK1.8 or JDK11, the two options I have > > > installed) > > > > > > > > > > > > [ERROR] Failed to execute goal > > > org.apache.maven.plugins:maven-compiler-plugin:3.9.0:compile > > > (default-compile) on project nifi-h2-database-migrator: Compilation > > > failure: Compilation failure: > > > [ERROR] > > > > > > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[19,35] > > > error: package org.apache.nifi.org.h2.jdbcx does not exist > > > [ERROR] > > > > > > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,8] > > > error: cannot find symbol > > > [ERROR] symbol: class JdbcDataSource > > > [ERROR] location: class H2DatabaseMigrator > > > [ERROR] > > > > > > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,49] > > > error: cannot find symbol > > > [ERROR] symbol: class JdbcDataSource > > > [ERROR] location: class H2DatabaseMigrator > > > > > > On Sat, Apr 2, 2022 at 10:20 PM Phil H wrote: > > > > > > > Hi there, > > > > > > > > Trying to build the nifi git repo for the first time and I'm getting > > the > > > > message below > > > > > > > > I'm used to working in offline development environments, so I'm a bit > > lost > > > > with this online dependency stuff. I note that > > > > https://repository.apache.org/snapshots/org/apache/nifi/ doesn't > > contain > > > > any similar subdirectories to that in the URL. > > > > > > > > Help! > > > > > > > > Regards, > > > > Phil > > > > > > > > [WARNING] The POM for org.apache.nifi:nifi-utils:jar:1.16.0-SNAPSHOT > is > > > > missing, no dependency information available > > > > Downloading from apache.snapshots: > > > > > > > https://repository.apache.org/snapshots/org/apache/nifi/nifi-utils/1.16.0-SNAPSHOT/nifi-utils-1.16.0-SNAPSHOT.jar > > > > [INFO] > > > > > > > > > > [INFO] Reactor Summary for nifi-standard-bundle 1.16.0-SNAPSHOT: > > > > [INFO] > > > > [INFO] nifi-standard-bundle ... SUCCESS > > [01:07 > > > > min] > > > > [INFO] nifi-standard-utils FAILURE [ > > > > 3.218 s] > > > > [INFO] nifi-standard-processors ... SKIPPED > > > > [INFO] nifi-standard-reporting-tasks .. SKIPPED > > > > [INFO] nifi-standard-content-viewer ... SKIPPED > > > > [INFO] nifi-jolt-transform-json-ui SKIPPED > > > > [INFO] nifi-standard-nar .. SKIPPED > > > > [INFO] > > > > > > > > > > [INFO] BUILD FAILURE > > > > [INFO] > > > > > > > > > > [INFO] Total time: 01:43 min > > > > [INFO] Finished at: 2022-04-02T21:58:47+10:00 > > > > [INFO] > > > > > > > > > > [ERROR] Failed to execute goal on project nifi-standard-utils: Could > > not > > > > resolve dependencies for project > > > > org.apache.nifi:nifi-standard-utils:jar:1.16.0-SNAPSHOT: Could not >
Re: SplitContent doesn’t support regex?
Whilst I try and get NiFi to build, let's circle back to JIRA. I found an open issue that matches my requirement (NIFI-1517), however to implement my solution, I'd also fix NIFI-3470 on the way (reading a configurable amount of data to run the regex over, rather than byte-by-byte). So, what's the proper way to go about this from a JIRA perspective? I assume my branch would be nifi-1517 as that's the feature I'm building, but it would also "solve" 3470? TIA, Phil On Thu, Mar 17, 2022 at 9:12 AM Joe Witt wrote: > Phil > > I'd say if you have a good implementation in mind you should go for it. > Sounds interesting. > > Thanks > > On Wed, Mar 16, 2022 at 3:59 PM Phil H wrote: > > > Hi, > > > > This seems like an odd omission - aside from performance (presumably?) is > > there a reason why there isn’t a regex option for the byte sequence? I > need > > one but thought I’d ask before I built my own. > > > > Thanks > > Phil > > >
Re: Missing maven dependencies when building nifi
Thanks Mike - that certainly moved the pieces along the board. Your patience for my ignorance is greatly appreciated ;-) Sidenote: I always thought mvn "install" was to do with packaging up binaries, and occurred after the "test" phase had completed? So, got a new new problem: [INFO] Running org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServerTest [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.551 s <<< FAILURE! - in org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServerTest [ERROR] org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServerTest.testRequestPeerListShouldUseTLS Time elapsed: 0.298 s <<< ERROR! java.io.IOException: Could not begin listening for incoming connections in order to load balance data across the cluster. Please verify the values of the 'nifi.cluster.load.balance.port' and 'nifi.cluster.load.balance.host' properties as well as the 'nifi.security.*' properties at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServerTest.testRequestPeerListShouldUseTLS(ConnectionLoadBalanceServerTest.groovy:77) Caused by: java.net.BindException: Address already in use: JVM_Bind at org.apache.nifi.controller.queue.clustered.server.ConnectionLoadBalanceServerTest.testRequestPeerListShouldUseTLS(ConnectionLoadBalanceServerTest.groovy:77) So I had a look at the test code, checked that nothing I was running elsewhere was binding on the port declared in the test (54321) and restarted the testing with "mvn install -rf :nifi-framework-core". It successfully ran the previous test (non-deterministic behaviour?? please no), but this time it hung up a few tests later: [INFO] Running org.apache.nifi.controller.state.server.TestZooKeeperStateServer [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.305 s <<< FAILURE! - in org.apache.nifi.controller.state.server.TestZooKeeperStateServer [ERROR] org.apache.nifi.controller.state.server.TestZooKeeperStateServer.testServerCreatePath Time elapsed: 0.272 s <<< ERROR! org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /test at org.apache.nifi.controller.state.server.TestZooKeeperStateServer.testServerCreatePath(TestZooKeeperStateServer.java:125) Re-running resulted in the same failure a second time (points for consistency!), but I'm not sure what to do about this one. Phil On Mon, Apr 4, 2022 at 11:38 PM Mike Thomsen wrote: > Did you run "mvn install" from the root of the project before > attempting to run a full test of the code base? > > On Mon, Apr 4, 2022 at 9:36 AM Phil H wrote: > > > > Hi there, > > > > So I got past this issue by running "mvn test" from the root of the nifi > > tree. After much compiling and testing, I am now getting the attached > > error. I think I am following the contributor guide in having my local > > branch track origin/main (currently "4a90334728: NIFI-9846 Implement > > pagination listing for Azure List processors"). > > > > Is this some environmental issue on my machine somehow? Or does the main > > branch contain compilation errors, and I should be tracking something > else > > for my own branch? As I said in my previous email, I am new to github, > so > > I'm not sure of the conventions there. > > > > Thanks in advance > > Phil > > > > (PS: this occur whethers I use JDK1.8 or JDK11, the two options I have > > installed) > > > > > > > > [ERROR] Failed to execute goal > > org.apache.maven.plugins:maven-compiler-plugin:3.9.0:compile > > (default-compile) on project nifi-h2-database-migrator: Compilation > > failure: Compilation failure: > > [ERROR] > > > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[19,35] > > error: package org.apache.nifi.org.h2.jdbcx does not exist > > [ERROR] > > > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,8] > > error: cannot find symbol > > [ERROR] symbol: class JdbcDataSource > > [ERROR] location: class H2DatabaseMigrator > > [ERROR] > > > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,49] > > error: cannot find symbol > > [ERROR] symbol: class JdbcDataSource > > [ERROR] location: class H2DatabaseMigrator > > > > On Sat, Apr 2, 2022 at 10:20 PM Phil H wrote: > > > > > Hi there, > > > > > > Trying to build the nifi git repo for the first time and I'm getting > the > > > message below > > > > > > I'm used to working in offline development environments, so I'm a bit > lost > > > with this online dependency stuff. I note that > > > https://repository.apache.org/snapshots/org/apache/nifi/ doesn't > contain > > > any similar subdirectories to that in the URL. > > > > > > Help! > > > > > > Regards, > > > Phil > > > > > >
Re: Missing maven dependencies when building nifi
Phil Here is a command I commonly run to ensure I have nothing other than what the source tree intends. mvn -T C1 clean package verify -Pcontrib-check,include-grpc This will do everything - build it all, run contrib check, and use all the cores. Thanks On Mon, Apr 4, 2022 at 6:38 AM Mike Thomsen wrote: > Did you run "mvn install" from the root of the project before > attempting to run a full test of the code base? > > On Mon, Apr 4, 2022 at 9:36 AM Phil H wrote: > > > > Hi there, > > > > So I got past this issue by running "mvn test" from the root of the nifi > > tree. After much compiling and testing, I am now getting the attached > > error. I think I am following the contributor guide in having my local > > branch track origin/main (currently "4a90334728: NIFI-9846 Implement > > pagination listing for Azure List processors"). > > > > Is this some environmental issue on my machine somehow? Or does the main > > branch contain compilation errors, and I should be tracking something > else > > for my own branch? As I said in my previous email, I am new to github, > so > > I'm not sure of the conventions there. > > > > Thanks in advance > > Phil > > > > (PS: this occur whethers I use JDK1.8 or JDK11, the two options I have > > installed) > > > > > > > > [ERROR] Failed to execute goal > > org.apache.maven.plugins:maven-compiler-plugin:3.9.0:compile > > (default-compile) on project nifi-h2-database-migrator: Compilation > > failure: Compilation failure: > > [ERROR] > > > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[19,35] > > error: package org.apache.nifi.org.h2.jdbcx does not exist > > [ERROR] > > > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,8] > > error: cannot find symbol > > [ERROR] symbol: class JdbcDataSource > > [ERROR] location: class H2DatabaseMigrator > > [ERROR] > > > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,49] > > error: cannot find symbol > > [ERROR] symbol: class JdbcDataSource > > [ERROR] location: class H2DatabaseMigrator > > > > On Sat, Apr 2, 2022 at 10:20 PM Phil H wrote: > > > > > Hi there, > > > > > > Trying to build the nifi git repo for the first time and I'm getting > the > > > message below > > > > > > I'm used to working in offline development environments, so I'm a bit > lost > > > with this online dependency stuff. I note that > > > https://repository.apache.org/snapshots/org/apache/nifi/ doesn't > contain > > > any similar subdirectories to that in the URL. > > > > > > Help! > > > > > > Regards, > > > Phil > > > > > > [WARNING] The POM for org.apache.nifi:nifi-utils:jar:1.16.0-SNAPSHOT is > > > missing, no dependency information available > > > Downloading from apache.snapshots: > > > > https://repository.apache.org/snapshots/org/apache/nifi/nifi-utils/1.16.0-SNAPSHOT/nifi-utils-1.16.0-SNAPSHOT.jar > > > [INFO] > > > > > > > [INFO] Reactor Summary for nifi-standard-bundle 1.16.0-SNAPSHOT: > > > [INFO] > > > [INFO] nifi-standard-bundle ... SUCCESS > [01:07 > > > min] > > > [INFO] nifi-standard-utils FAILURE [ > > > 3.218 s] > > > [INFO] nifi-standard-processors ... SKIPPED > > > [INFO] nifi-standard-reporting-tasks .. SKIPPED > > > [INFO] nifi-standard-content-viewer ... SKIPPED > > > [INFO] nifi-jolt-transform-json-ui SKIPPED > > > [INFO] nifi-standard-nar .. SKIPPED > > > [INFO] > > > > > > > [INFO] BUILD FAILURE > > > [INFO] > > > > > > > [INFO] Total time: 01:43 min > > > [INFO] Finished at: 2022-04-02T21:58:47+10:00 > > > [INFO] > > > > > > > [ERROR] Failed to execute goal on project nifi-standard-utils: Could > not > > > resolve dependencies for project > > > org.apache.nifi:nifi-standard-utils:jar:1.16.0-SNAPSHOT: Could not find > > > artifact org.apache.nifi:nifi-utils:jar:1.16.0-SNAPSHOT in > apache.snapshots > > > (https://repository.apache.org/snapshots) -> [Help 1] > > > >
Re: Missing maven dependencies when building nifi
Did you run "mvn install" from the root of the project before attempting to run a full test of the code base? On Mon, Apr 4, 2022 at 9:36 AM Phil H wrote: > > Hi there, > > So I got past this issue by running "mvn test" from the root of the nifi > tree. After much compiling and testing, I am now getting the attached > error. I think I am following the contributor guide in having my local > branch track origin/main (currently "4a90334728: NIFI-9846 Implement > pagination listing for Azure List processors"). > > Is this some environmental issue on my machine somehow? Or does the main > branch contain compilation errors, and I should be tracking something else > for my own branch? As I said in my previous email, I am new to github, so > I'm not sure of the conventions there. > > Thanks in advance > Phil > > (PS: this occur whethers I use JDK1.8 or JDK11, the two options I have > installed) > > > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.9.0:compile > (default-compile) on project nifi-h2-database-migrator: Compilation > failure: Compilation failure: > [ERROR] > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[19,35] > error: package org.apache.nifi.org.h2.jdbcx does not exist > [ERROR] > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,8] > error: cannot find symbol > [ERROR] symbol: class JdbcDataSource > [ERROR] location: class H2DatabaseMigrator > [ERROR] > D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,49] > error: cannot find symbol > [ERROR] symbol: class JdbcDataSource > [ERROR] location: class H2DatabaseMigrator > > On Sat, Apr 2, 2022 at 10:20 PM Phil H wrote: > > > Hi there, > > > > Trying to build the nifi git repo for the first time and I'm getting the > > message below > > > > I'm used to working in offline development environments, so I'm a bit lost > > with this online dependency stuff. I note that > > https://repository.apache.org/snapshots/org/apache/nifi/ doesn't contain > > any similar subdirectories to that in the URL. > > > > Help! > > > > Regards, > > Phil > > > > [WARNING] The POM for org.apache.nifi:nifi-utils:jar:1.16.0-SNAPSHOT is > > missing, no dependency information available > > Downloading from apache.snapshots: > > https://repository.apache.org/snapshots/org/apache/nifi/nifi-utils/1.16.0-SNAPSHOT/nifi-utils-1.16.0-SNAPSHOT.jar > > [INFO] > > > > [INFO] Reactor Summary for nifi-standard-bundle 1.16.0-SNAPSHOT: > > [INFO] > > [INFO] nifi-standard-bundle ... SUCCESS [01:07 > > min] > > [INFO] nifi-standard-utils FAILURE [ > > 3.218 s] > > [INFO] nifi-standard-processors ... SKIPPED > > [INFO] nifi-standard-reporting-tasks .. SKIPPED > > [INFO] nifi-standard-content-viewer ... SKIPPED > > [INFO] nifi-jolt-transform-json-ui SKIPPED > > [INFO] nifi-standard-nar .. SKIPPED > > [INFO] > > > > [INFO] BUILD FAILURE > > [INFO] > > > > [INFO] Total time: 01:43 min > > [INFO] Finished at: 2022-04-02T21:58:47+10:00 > > [INFO] > > > > [ERROR] Failed to execute goal on project nifi-standard-utils: Could not > > resolve dependencies for project > > org.apache.nifi:nifi-standard-utils:jar:1.16.0-SNAPSHOT: Could not find > > artifact org.apache.nifi:nifi-utils:jar:1.16.0-SNAPSHOT in apache.snapshots > > (https://repository.apache.org/snapshots) -> [Help 1] > >
Re: Missing maven dependencies when building nifi
Hi there, So I got past this issue by running "mvn test" from the root of the nifi tree. After much compiling and testing, I am now getting the attached error. I think I am following the contributor guide in having my local branch track origin/main (currently "4a90334728: NIFI-9846 Implement pagination listing for Azure List processors"). Is this some environmental issue on my machine somehow? Or does the main branch contain compilation errors, and I should be tracking something else for my own branch? As I said in my previous email, I am new to github, so I'm not sure of the conventions there. Thanks in advance Phil (PS: this occur whethers I use JDK1.8 or JDK11, the two options I have installed) [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.9.0:compile (default-compile) on project nifi-h2-database-migrator: Compilation failure: Compilation failure: [ERROR] D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[19,35] error: package org.apache.nifi.org.h2.jdbcx does not exist [ERROR] D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,8] error: cannot find symbol [ERROR] symbol: class JdbcDataSource [ERROR] location: class H2DatabaseMigrator [ERROR] D:\nifi.github\nifi\nifi-h2\nifi-h2-database-migrator\src\main\java\org\apache\nifi\h2\database\migration\H2DatabaseMigrator.java:[37,49] error: cannot find symbol [ERROR] symbol: class JdbcDataSource [ERROR] location: class H2DatabaseMigrator On Sat, Apr 2, 2022 at 10:20 PM Phil H wrote: > Hi there, > > Trying to build the nifi git repo for the first time and I'm getting the > message below > > I'm used to working in offline development environments, so I'm a bit lost > with this online dependency stuff. I note that > https://repository.apache.org/snapshots/org/apache/nifi/ doesn't contain > any similar subdirectories to that in the URL. > > Help! > > Regards, > Phil > > [WARNING] The POM for org.apache.nifi:nifi-utils:jar:1.16.0-SNAPSHOT is > missing, no dependency information available > Downloading from apache.snapshots: > https://repository.apache.org/snapshots/org/apache/nifi/nifi-utils/1.16.0-SNAPSHOT/nifi-utils-1.16.0-SNAPSHOT.jar > [INFO] > > [INFO] Reactor Summary for nifi-standard-bundle 1.16.0-SNAPSHOT: > [INFO] > [INFO] nifi-standard-bundle ... SUCCESS [01:07 > min] > [INFO] nifi-standard-utils FAILURE [ > 3.218 s] > [INFO] nifi-standard-processors ... SKIPPED > [INFO] nifi-standard-reporting-tasks .. SKIPPED > [INFO] nifi-standard-content-viewer ... SKIPPED > [INFO] nifi-jolt-transform-json-ui SKIPPED > [INFO] nifi-standard-nar .. SKIPPED > [INFO] > > [INFO] BUILD FAILURE > [INFO] > > [INFO] Total time: 01:43 min > [INFO] Finished at: 2022-04-02T21:58:47+10:00 > [INFO] > > [ERROR] Failed to execute goal on project nifi-standard-utils: Could not > resolve dependencies for project > org.apache.nifi:nifi-standard-utils:jar:1.16.0-SNAPSHOT: Could not find > artifact org.apache.nifi:nifi-utils:jar:1.16.0-SNAPSHOT in apache.snapshots > (https://repository.apache.org/snapshots) -> [Help 1] >
[GitHub] [nifi] turcsanyip commented on a diff in pull request #5916: NIFI-9846 Implement pagination listing for Azure List processors
turcsanyip commented on code in PR #5916:
URL: https://github.com/apache/nifi/pull/5916#discussion_r841468038
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage_v12.java:
##
@@ -199,34 +201,44 @@ protected boolean isListingResetNecessary(final
PropertyDescriptor property) {
}
@Override
-protected List performListing(ProcessContext context, Long
minTimestamp, ListingMode listingMode) throws IOException {
-String containerName =
context.getProperty(CONTAINER).evaluateAttributeExpressions().getValue();
-String prefix =
context.getProperty(BLOB_NAME_PREFIX).evaluateAttributeExpressions().getValue();
+protected List performListing(final ProcessContext context,
final Long minTimestamp, final ListingMode listingMode) throws IOException {
+final String containerName =
context.getProperty(CONTAINER).evaluateAttributeExpressions().getValue();
+final String prefix =
context.getProperty(BLOB_NAME_PREFIX).evaluateAttributeExpressions().getValue();
+final long minimumTimestamp = minTimestamp == null ? 0 : minTimestamp;
try {
-List listing = new ArrayList<>();
+final List listing = new ArrayList<>();
-BlobContainerClient containerClient =
storageClient.getBlobContainerClient(containerName);
+final BlobContainerClient containerClient =
storageClient.getBlobContainerClient(containerName);
-ListBlobsOptions options = new ListBlobsOptions()
+final ListBlobsOptions options = new ListBlobsOptions()
.setPrefix(prefix);
-for (BlobItem blob : containerClient.listBlobs(options, null)) {
-BlobItemProperties properties = blob.getProperties();
-
-Builder builder = new Builder()
-.containerName(containerName)
-.blobName(blob.getName())
-.primaryUri(String.format("%s/%s",
containerClient.getBlobContainerUrl(), blob.getName()))
-.etag(properties.getETag())
-.blobType(properties.getBlobType().toString())
-.contentType(properties.getContentType())
-.contentLanguage(properties.getContentLanguage())
-
.lastModifiedTime(properties.getLastModified().toInstant().toEpochMilli())
-.length(properties.getContentLength());
-
-listing.add(builder.build());
-}
+final Iterator> result =
containerClient.listBlobs(options, null).iterableByPage().iterator();
+String continuationToken;
+
+do {
+final PagedResponse pagedResult = result.next();
+continuationToken = pagedResult.getContinuationToken();
Review Comment:
It seems to me that the items can be processed without tracking the
continuation token and using a simple iterator (instead of the paged iterator).
Based on this documentation:
https://docs.microsoft.com/en-us/azure/developer/java/sdk/pagination
> Make it possible to easily iterate over each element in the collection
individually, ignoring any need for manual pagination or tracking of
continuation tokens.
> Regardless of whether you iterate by page or by each item, there's no
difference in performance or the number of calls made to the service.
It is applicable to the v12 Blob and the ADLS processors as they use the new
SDK.
So I think only the timestamp filtering needs to be added here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [nifi] turcsanyip commented on a diff in pull request #5916: NIFI-9846 Implement pagination listing for Azure List processors
turcsanyip commented on code in PR #5916:
URL: https://github.com/apache/nifi/pull/5916#discussion_r841463319
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage.java:
##
@@ -183,41 +185,53 @@ protected String getDefaultTimePrecision() {
final String containerName =
context.getProperty(AzureStorageUtils.CONTAINER).evaluateAttributeExpressions().getValue();
final String prefix =
Optional.ofNullable(context.getProperty(PROP_PREFIX).evaluateAttributeExpressions().getValue()).orElse("");
final List listing = new ArrayList<>();
+final long minimumTimestamp = minTimestamp == null ? 0 : minTimestamp;
+
try {
final CloudBlobClient blobClient =
AzureStorageUtils.createCloudBlobClient(context, getLogger(), null);
final CloudBlobContainer container =
blobClient.getContainerReference(containerName);
final OperationContext operationContext = new OperationContext();
AzureStorageUtils.setProxy(operationContext, context);
-for (final ListBlobItem blob : container.listBlobs(prefix, true,
EnumSet.of(BlobListingDetails.METADATA), null, operationContext)) {
-if (blob instanceof CloudBlob) {
-final CloudBlob cloudBlob = (CloudBlob) blob;
-final BlobProperties properties =
cloudBlob.getProperties();
-final StorageUri uri =
cloudBlob.getSnapshotQualifiedStorageUri();
-
-final Builder builder = new BlobInfo.Builder()
-
.primaryUri(uri.getPrimaryUri().toString())
- .blobName(cloudBlob.getName())
- .containerName(containerName)
-
.contentType(properties.getContentType())
-
.contentLanguage(properties.getContentLanguage())
- .etag(properties.getEtag())
-
.lastModifiedTime(properties.getLastModified().getTime())
- .length(properties.getLength());
-
-if (uri.getSecondaryUri() != null) {
-builder.secondaryUri(uri.getSecondaryUri().toString());
-}
-
-if (blob instanceof CloudBlockBlob) {
-builder.blobType(AzureStorageUtils.BLOCK);
-} else {
-builder.blobType(AzureStorageUtils.PAGE);
+ResultContinuation continuationToken = null;
+
+do {
+final ResultSegment result =
container.listBlobsSegmented(prefix, true,
EnumSet.of(BlobListingDetails.METADATA), null, continuationToken, null,
operationContext);
+continuationToken = result.getContinuationToken();
+
+for (final ListBlobItem blob : result.getResults()) {
+if (blob instanceof CloudBlob) {
+final CloudBlob cloudBlob = (CloudBlob) blob;
+final BlobProperties properties =
cloudBlob.getProperties();
+
+if (properties.getLastModified().getTime() >
minimumTimestamp) {
Review Comment:
`>=` needs to be used instead because new files can be uploaded to Azure
with the same timestamp (second precision on Azure) after the listing.
`AbsractListProcessor` stores the already listed files with the previous max
timesamp and can filter out those files later in the process.
##
nifi-nar-bundles/nifi-azure-bundle/nifi-azure-processors/src/main/java/org/apache/nifi/processors/azure/storage/ListAzureBlobStorage_v12.java:
##
@@ -199,34 +201,44 @@ protected boolean isListingResetNecessary(final
PropertyDescriptor property) {
}
@Override
-protected List performListing(ProcessContext context, Long
minTimestamp, ListingMode listingMode) throws IOException {
-String containerName =
context.getProperty(CONTAINER).evaluateAttributeExpressions().getValue();
-String prefix =
context.getProperty(BLOB_NAME_PREFIX).evaluateAttributeExpressions().getValue();
+protected List performListing(final ProcessContext context,
final Long minTimestamp, final ListingMode listingMode) throws IOException {
+final String containerName =
context.getProperty(CONTAINER).evaluateAttributeExpressions().getValue();
+final String prefix =
context.getProperty(BLOB_NAME_PREFIX).evaluateAttributeExpressions().getValue();
+final long minimumTimestamp = minTimestamp == null ? 0 : minTimestamp;
try {
-List listing = new ArrayList<>();
+final List listing = new ArrayList<>();
-BlobContainerClient containerClient =
