[jira] [Commented] (MINIFICPP-895) Modbus Integration
[ https://issues.apache.org/jira/browse/MINIFICPP-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787070#comment-17787070 ] Martin Zink commented on MINIFICPP-895: --- [~cdutz] any updates on the PLC7C project? From github it seems its still not ready to use, but there are frequent commits to it. > Modbus Integration > -- > > Key: MINIFICPP-895 > URL: https://issues.apache.org/jira/browse/MINIFICPP-895 > Project: Apache NiFi MiNiFi C++ > Issue Type: Epic >Reporter: Jeremy Dyer >Priority: Minor > > Modbus is a de facto industry standard for communicating with electronic > devices. Our primary goal here would be to provide a modbus implementation > that would allow for users to connect to those electronic devices to both > transfer and receive information from them. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[
https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787062#comment-17787062
]
ASF subversion and git services commented on NIFI-12383:
Commit 63364687d8bbe2af2d0031f745ca3c61144f8b4f in nifi's branch
refs/heads/main from Zoltan Kornel Torok
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=63364687d8 ]
NIFI-12383 Replication client should handle accept encoding with lowercase
Signed-off-by: Bence Simon
This closes #8043
> GZipException occur during cluster replication, when the original request
> contains "accept-encoding" header with lowercase
> --
>
> Key: NIFI-12383
> URL: https://issues.apache.org/jira/browse/NIFI-12383
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Zoltán Kornél Török
>Assignee: Zoltán Kornél Török
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I had a three node cluster with knox.
> Time to time an error occured in the nifi logs on this cluster:
> {code}
> 2023-11-15 13:25:51,637 INFO
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator:
> Received a status of 500 from xy:8443 for request PUT
> /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing
> first stage of two-stage commit. The action will not occur. Node explanation:
> An unexpected error has occurred. Please check the logs for additional
> details.
> {code}
> Also sometimes I got "An unexpected error has occurred. Please check the logs
> for additional details." error on the UI too. After some investigation in I
> found the error in the logs:
> {code}
> 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78]
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> java.util.zip.ZipException: Not in GZIP format. Returning Internal Server
> Error response.
> java.util.zip.ZipException: Not in GZIP format
> at
> java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176)
> at
> java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79)
> at
> java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91)
> at
> org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49)
> at
> org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
> at
> org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072)
> at
> org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919)
> at
> org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290)
> at
> org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73)
> {code}
> After many hours of debugging, I found out, that sometimes when I use the
> cluster via knox, some unknown reason the incoming "Accept-Encoding" come
> with all leters lowercase (which is valid, becase HTTP header is case
> insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers).
> However OkHttpReplicationClient assume that the header is always
> "Accept-Encoding"
> (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294
> and
> https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25).
> Because of that, during replication the client not use gzip compression but
> when other node get the requests, the jetty read the original
> "accept-encoding" header and try to uncompress the inputstream, which lead to
> the above error.
> We need to add a few line code to the client to read the header case
> insensitivity
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12383 Replication client should handle accept encoding with lowe… [nifi]
simonbence closed pull request #8043: NIFI-12383 Replication client should handle accept encoding with lowe… URL: https://github.com/apache/nifi/pull/8043 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12383 Replication client should handle accept encoding with lowe… [nifi]
simonbence commented on PR #8043: URL: https://github.com/apache/nifi/pull/8043#issuecomment-1815835313 lgtm -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12385) How to set Provenance Repository Properties in nifi docker image
[ https://issues.apache.org/jira/browse/NIFI-12385?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo updated NIFI-12385: --- Description: Hi, I've got a question: How to set Provenance Repository Properties in Nifi docker image as env variable to overwrite some properties like this: nifi.provenance.repository.max.storage.time=24 hours nifi.provenance.repository.max.storage.size=1 GB I couldn't find anything in sh docker folder ([https://github.com/apache/nifi/tree/main/nifi-docker/dockerhub/sh)] Could you show me some tutorials to configure it? Thanks! was: Hi, I've got a question: How to set Provenance Repository Properties in Nifi docker image as env variable to overwrite some properties like this: nifi.provenance.repository.max.storage.time=24 hours nifi.provenance.repository.max.storage.size=1 GB I couldn't find anything in [sh folder]([https://github.com/apache/nifi/tree/main/nifi-docker/dockerhub/sh)] Could you show me some tutorials to configure it? Thanks! > How to set Provenance Repository Properties in nifi docker image > > > Key: NIFI-12385 > URL: https://issues.apache.org/jira/browse/NIFI-12385 > Project: Apache NiFi > Issue Type: Wish >Reporter: Marcelo >Priority: Minor > > Hi, I've got a question: > How to set Provenance Repository Properties in Nifi docker image as env > variable to overwrite some properties like this: > nifi.provenance.repository.max.storage.time=24 hours > nifi.provenance.repository.max.storage.size=1 GB > I couldn't find anything in sh docker folder > ([https://github.com/apache/nifi/tree/main/nifi-docker/dockerhub/sh)] > Could you show me some tutorials to configure it? > Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12385) How to set Provenance Repository Properties in nifi docker image
Marcelo created NIFI-12385: -- Summary: How to set Provenance Repository Properties in nifi docker image Key: NIFI-12385 URL: https://issues.apache.org/jira/browse/NIFI-12385 Project: Apache NiFi Issue Type: Wish Reporter: Marcelo Hi, I've got a question: How to set Provenance Repository Properties in Nifi docker image as env variable to overwrite some properties like this: nifi.provenance.repository.max.storage.time=24 hours nifi.provenance.repository.max.storage.size=1 GB I couldn't find anything in [sh folder]([https://github.com/apache/nifi/tree/main/nifi-docker/dockerhub/sh)] Could you show me some tutorials to configure it? Thanks! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[
https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786987#comment-17786987
]
David Handermann commented on NIFI-12383:
-
Thanks for summarizing this issue [~taz1988].
Do you know if the {{h2}} protocol was enabled for the HTTP protocols property
in nifi.properties? The HTTP/2 protocol uses binary for headers, and as a
result, the header names can be reported as lowercase when HTTP/2 is negotiated.
> GZipException occur during cluster replication, when the original request
> contains "accept-encoding" header with lowercase
> --
>
> Key: NIFI-12383
> URL: https://issues.apache.org/jira/browse/NIFI-12383
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Zoltán Kornél Török
>Assignee: Zoltán Kornél Török
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I had a three node cluster with knox.
> Time to time an error occured in the nifi logs on this cluster:
> {code}
> 2023-11-15 13:25:51,637 INFO
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator:
> Received a status of 500 from xy:8443 for request PUT
> /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing
> first stage of two-stage commit. The action will not occur. Node explanation:
> An unexpected error has occurred. Please check the logs for additional
> details.
> {code}
> Also sometimes I got "An unexpected error has occurred. Please check the logs
> for additional details." error on the UI too. After some investigation in I
> found the error in the logs:
> {code}
> 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78]
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> java.util.zip.ZipException: Not in GZIP format. Returning Internal Server
> Error response.
> java.util.zip.ZipException: Not in GZIP format
> at
> java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176)
> at
> java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79)
> at
> java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91)
> at
> org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49)
> at
> org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
> at
> org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072)
> at
> org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919)
> at
> org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290)
> at
> org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73)
> {code}
> After many hours of debugging, I found out, that sometimes when I use the
> cluster via knox, some unknown reason the incoming "Accept-Encoding" come
> with all leters lowercase (which is valid, becase HTTP header is case
> insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers).
> However OkHttpReplicationClient assume that the header is always
> "Accept-Encoding"
> (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294
> and
> https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25).
> Because of that, during replication the client not use gzip compression but
> when other node get the requests, the jetty read the original
> "accept-encoding" header and try to uncompress the inputstream, which lead to
> the above error.
> We need to add a few line code to the client to read the header case
> insensitivity
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (NIFI-12384) Upgrade Spring Framework to 6.0.13 for Registry
[ https://issues.apache.org/jira/browse/NIFI-12384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12384: Status: Patch Available (was: Open) > Upgrade Spring Framework to 6.0.13 for Registry > --- > > Key: NIFI-12384 > URL: https://issues.apache.org/jira/browse/NIFI-12384 > Project: Apache NiFi > Issue Type: Improvement > Components: NiFi Registry >Reporter: David Handermann >Assignee: David Handermann >Priority: Major > Fix For: 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Spring Framework 6 represents a major upgrade from version 5 and requires a > number of supporting libraries to be upgraded. These supporting libraries > include the following: > - Spring Security 6 > - Spring Boot 3 > - Jakarta Servlet API 5 > - Jetty 11 > - JAX RS 3 > - Jersey 3 > NiFi Registry should be upgraded independently of NiFi itself to provide an > incremental path forward. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12384 Upgrade Registry to Spring Framework 6 [nifi]
exceptionfactory opened a new pull request, #8044: URL: https://github.com/apache/nifi/pull/8044 # Summary [NIFI-12384](https://issues.apache.org/jira/browse/NIFI-12384) Upgrades NiFi Registry from Spring Framework 5 to 6.0.13 along with supporting libraries. The majority of the changes involve replacing `javax` with `jakarta` in package names to align with major version changes in the Servlet API, JAX RS API, and related programming interfaces. Related changes include upgrading the NiFi Toolkit CLI to the same JAX RS and Jersey versions as Registry to align with packaging requirements from shared dependencies. NiFi Registry changes include adjusting the Spring Security configuration to align with Spring Security 6, and removing the test Jetty Server Customizer that is no longer required for integration tests. These changes are scoped to NiFi Registry using properties in the `nifi-registry` root module, leaving the remaining of the repository on the earlier versions of these libraries. Upgraded libraries for Registry include the following: - Upgraded Spring from 5.3.30 to 6.0.13 - Upgraded Spring Boot from 2.7.16 to 3.1.5 - Upgraded Spring Security from 5.8.7 to 6.1.5 - Upgraded Jetty from 10.0.18 to 11.0.18 - Upgraded Servlet API from 3.1.0 to 5.0.0 - Upgraded JAX-RS from 2.1.1 to 3.1.0 - Upgraded Jersey from 2.41 to 3.1.3 - Upgraded JAXB from 2.3.2 to 4.0.4 - Upgraded Java Validation from 2.0.1 to 3.0.2 - Upgraded Flyway from 8.5.13 to 9.22.3 - Upgraded ASM from 9.3 to 9.6 for Java 21 # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [X] Build completed using `mvn clean install -P contrib-check` - [X] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12384) Upgrade Spring Framework to 6.0.13 for Registry
David Handermann created NIFI-12384: --- Summary: Upgrade Spring Framework to 6.0.13 for Registry Key: NIFI-12384 URL: https://issues.apache.org/jira/browse/NIFI-12384 Project: Apache NiFi Issue Type: Improvement Components: NiFi Registry Reporter: David Handermann Assignee: David Handermann Fix For: 2.latest Spring Framework 6 represents a major upgrade from version 5 and requires a number of supporting libraries to be upgraded. These supporting libraries include the following: - Spring Security 6 - Spring Boot 3 - Jakarta Servlet API 5 - Jetty 11 - JAX RS 3 - Jersey 3 NiFi Registry should be upgraded independently of NiFi itself to provide an incremental path forward. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12368) Copy/Pasting components can incorrectly copy versionedComponentId
[ https://issues.apache.org/jira/browse/NIFI-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12368: Fix Version/s: 2.0.0-M1 1.25.0 Resolution: Fixed Status: Resolved (was: Patch Available) > Copy/Pasting components can incorrectly copy versionedComponentId > - > > Key: NIFI-12368 > URL: https://issues.apache.org/jira/browse/NIFI-12368 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.24.0, 1.23.2 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Fix For: 2.0.0-M1, 1.25.0 > > Time Spent: 40m > Remaining Estimate: 0h > > If you import a flow from registry, or create a PG from file upload, the > components inside the PG will have the versionedComponentId field set. If you > then go inside this PG and copy/paste a processor, it will create a new > processor with the same versionedComponentId which is not correct. However, > if you paste an entire PG it should retain the versionedComponentIds within > the PG. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12368) Copy/Pasting components can incorrectly copy versionedComponentId
[ https://issues.apache.org/jira/browse/NIFI-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12368: Affects Version/s: 1.24.0 (was: 2.0.0-M1) > Copy/Pasting components can incorrectly copy versionedComponentId > - > > Key: NIFI-12368 > URL: https://issues.apache.org/jira/browse/NIFI-12368 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 1.24.0, 1.23.2 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > If you import a flow from registry, or create a PG from file upload, the > components inside the PG will have the versionedComponentId field set. If you > then go inside this PG and copy/paste a processor, it will create a new > processor with the same versionedComponentId which is not correct. However, > if you paste an entire PG it should retain the versionedComponentIds within > the PG. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12381) Fix typo in SplitText.REMOVE_TRAILING_NEWLINES property description
[ https://issues.apache.org/jira/browse/NIFI-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786985#comment-17786985 ] ASF subversion and git services commented on NIFI-12381: Commit 062dd0da1d63f3f549fe09f57708627062967221 in nifi's branch refs/heads/support/nifi-1.x from Ferenc Gerlits [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=062dd0da1d ] NIFI-12381 Fixed CR and LF documentation in SplitText This closes #8041 Signed-off-by: David Handermann (cherry picked from commit ceeb0deb3d703ad0f21a543f90cda82b69815c76) > Fix typo in SplitText.REMOVE_TRAILING_NEWLINES property description > --- > > Key: NIFI-12381 > URL: https://issues.apache.org/jira/browse/NIFI-12381 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Ferenc Gerlits >Assignee: Ferenc Gerlits >Priority: Trivial > Time Spent: 20m > Remaining Estimate: 0h > > We found this typo when porting SplitText to MiNiFi C++, so I'm contributing > it upstream. Every little counts, right? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12380) Update iceberg.version to 1.4.2
[ https://issues.apache.org/jira/browse/NIFI-12380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12380: Affects Version/s: (was: 1.23.2) > Update iceberg.version to 1.4.2 > --- > > Key: NIFI-12380 > URL: https://issues.apache.org/jira/browse/NIFI-12380 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike R >Assignee: Mike R >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Update iceberg.version to 1.4.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12380) Update iceberg.version to 1.4.2
[ https://issues.apache.org/jira/browse/NIFI-12380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-12380. - Fix Version/s: 2.0.0-M1 1.25.0 Resolution: Fixed > Update iceberg.version to 1.4.2 > --- > > Key: NIFI-12380 > URL: https://issues.apache.org/jira/browse/NIFI-12380 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike R >Assignee: Mike R >Priority: Major > Fix For: 2.0.0-M1, 1.25.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Update iceberg.version to 1.4.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12381) Fix typo in SplitText.REMOVE_TRAILING_NEWLINES property description
[ https://issues.apache.org/jira/browse/NIFI-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-12381. - Fix Version/s: 2.0.0-M1 1.25.0 Resolution: Fixed > Fix typo in SplitText.REMOVE_TRAILING_NEWLINES property description > --- > > Key: NIFI-12381 > URL: https://issues.apache.org/jira/browse/NIFI-12381 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Ferenc Gerlits >Assignee: Ferenc Gerlits >Priority: Trivial > Fix For: 2.0.0-M1, 1.25.0 > > Time Spent: 20m > Remaining Estimate: 0h > > We found this typo when porting SplitText to MiNiFi C++, so I'm contributing > it upstream. Every little counts, right? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12380) Update iceberg.version to 1.4.2
[ https://issues.apache.org/jira/browse/NIFI-12380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12380: Priority: Minor (was: Major) > Update iceberg.version to 1.4.2 > --- > > Key: NIFI-12380 > URL: https://issues.apache.org/jira/browse/NIFI-12380 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike R >Assignee: Mike R >Priority: Minor > Fix For: 2.0.0-M1, 1.25.0 > > Time Spent: 20m > Remaining Estimate: 0h > > Update iceberg.version to 1.4.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12380) Update iceberg.version to 1.4.2
[ https://issues.apache.org/jira/browse/NIFI-12380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786986#comment-17786986 ] ASF subversion and git services commented on NIFI-12380: Commit 5f108361f43e69f85379ec013ac3a527d5f93d11 in nifi's branch refs/heads/support/nifi-1.x from mr1716 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=5f108361f4 ] NIFI-12380 Upgraded Iceberg from 1.4.0 to 1.4.2 This closes #8039 Signed-off-by: David Handermann (cherry picked from commit a3983ae248516dff5f6d2914cacbfa9d00281fdf) > Update iceberg.version to 1.4.2 > --- > > Key: NIFI-12380 > URL: https://issues.apache.org/jira/browse/NIFI-12380 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Mike R >Assignee: Mike R >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Update iceberg.version to 1.4.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12368) Copy/Pasting components can incorrectly copy versionedComponentId
[ https://issues.apache.org/jira/browse/NIFI-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786984#comment-17786984 ] ASF subversion and git services commented on NIFI-12368: Commit ac2a5afb20b59f87ebc3bcfaac283ab8dc084f72 in nifi's branch refs/heads/support/nifi-1.x from Bryan Bende [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ac2a5afb20 ] NIFI-12368 Clear versionedComponentId for copied Snippets This closes #8025 Signed-off-by: David Handermann (cherry picked from commit 1cc346b63d0ecea1d0b2e9079a0f31c15f035b6d) > Copy/Pasting components can incorrectly copy versionedComponentId > - > > Key: NIFI-12368 > URL: https://issues.apache.org/jira/browse/NIFI-12368 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 2.0.0-M1, 1.23.2 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > If you import a flow from registry, or create a PG from file upload, the > components inside the PG will have the versionedComponentId field set. If you > then go inside this PG and copy/paste a processor, it will create a new > processor with the same versionedComponentId which is not correct. However, > if you paste an entire PG it should retain the versionedComponentIds within > the PG. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12368) Copy/Pasting components can incorrectly copy versionedComponentId
[ https://issues.apache.org/jira/browse/NIFI-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786980#comment-17786980 ] ASF subversion and git services commented on NIFI-12368: Commit 1cc346b63d0ecea1d0b2e9079a0f31c15f035b6d in nifi's branch refs/heads/main from Bryan Bende [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1cc346b63d ] NIFI-12368 Clear versionedComponentId for copied Snippets This closes #8025 Signed-off-by: David Handermann > Copy/Pasting components can incorrectly copy versionedComponentId > - > > Key: NIFI-12368 > URL: https://issues.apache.org/jira/browse/NIFI-12368 > Project: Apache NiFi > Issue Type: Bug >Affects Versions: 2.0.0-M1, 1.23.2 >Reporter: Bryan Bende >Assignee: Bryan Bende >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > If you import a flow from registry, or create a PG from file upload, the > components inside the PG will have the versionedComponentId field set. If you > then go inside this PG and copy/paste a processor, it will create a new > processor with the same versionedComponentId which is not correct. However, > if you paste an entire PG it should retain the versionedComponentIds within > the PG. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12378) Remove Jython support in NiFi 2.x given real Python support in 2.x
[ https://issues.apache.org/jira/browse/NIFI-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786983#comment-17786983 ] ASF subversion and git services commented on NIFI-12378: Commit 9cfdebc3e875a681dfa170d0478264d6522b1506 in nifi's branch refs/heads/main from Joe Witt [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=9cfdebc3e8 ] NIFI-12378 Removed Jython This closes #8040 Signed-off-by: David Handermann > Remove Jython support in NiFi 2.x given real Python support in 2.x > -- > > Key: NIFI-12378 > URL: https://issues.apache.org/jira/browse/NIFI-12378 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 2.latest > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12381) Fix typo in SplitText.REMOVE_TRAILING_NEWLINES property description
[ https://issues.apache.org/jira/browse/NIFI-12381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786981#comment-17786981 ] ASF subversion and git services commented on NIFI-12381: Commit ceeb0deb3d703ad0f21a543f90cda82b69815c76 in nifi's branch refs/heads/main from Ferenc Gerlits [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=ceeb0deb3d ] NIFI-12381 Fixed CR and LF documentation in SplitText This closes #8041 Signed-off-by: David Handermann > Fix typo in SplitText.REMOVE_TRAILING_NEWLINES property description > --- > > Key: NIFI-12381 > URL: https://issues.apache.org/jira/browse/NIFI-12381 > Project: Apache NiFi > Issue Type: Improvement >Reporter: Ferenc Gerlits >Assignee: Ferenc Gerlits >Priority: Trivial > Time Spent: 20m > Remaining Estimate: 0h > > We found this typo when porting SplitText to MiNiFi C++, so I'm contributing > it upstream. Every little counts, right? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (NIFI-12378) Remove Jython support in NiFi 2.x given real Python support in 2.x
[ https://issues.apache.org/jira/browse/NIFI-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann resolved NIFI-12378. - Fix Version/s: 2.0.0-M1 (was: 2.latest) Resolution: Fixed > Remove Jython support in NiFi 2.x given real Python support in 2.x > -- > > Key: NIFI-12378 > URL: https://issues.apache.org/jira/browse/NIFI-12378 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 2.0.0-M1 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12380) Update iceberg.version to 1.4.2
[ https://issues.apache.org/jira/browse/NIFI-12380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786982#comment-17786982 ] ASF subversion and git services commented on NIFI-12380: Commit a3983ae248516dff5f6d2914cacbfa9d00281fdf in nifi's branch refs/heads/main from mr1716 [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=a3983ae248 ] NIFI-12380 Upgraded Iceberg from 1.4.0 to 1.4.2 This closes #8039 Signed-off-by: David Handermann > Update iceberg.version to 1.4.2 > --- > > Key: NIFI-12380 > URL: https://issues.apache.org/jira/browse/NIFI-12380 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.23.2 >Reporter: Mike R >Assignee: Mike R >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Update iceberg.version to 1.4.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12378) Remove Jython support in NiFi 2.x given real Python support in 2.x
[ https://issues.apache.org/jira/browse/NIFI-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Handermann updated NIFI-12378: Epic Link: NIFI-11169 > Remove Jython support in NiFi 2.x given real Python support in 2.x > -- > > Key: NIFI-12378 > URL: https://issues.apache.org/jira/browse/NIFI-12378 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 2.0.0-M1 > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12380 Update iceberg.version to 1.4.2 [nifi]
exceptionfactory closed pull request #8039: NIFI-12380 Update iceberg.version to 1.4.2 URL: https://github.com/apache/nifi/pull/8039 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12378 remove Jython [nifi]
exceptionfactory closed pull request #8040: NIFI-12378 remove Jython URL: https://github.com/apache/nifi/pull/8040 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12381 Fix typo: \r, \n -> \\r, \\n in string constant [nifi]
exceptionfactory closed pull request #8041: NIFI-12381 Fix typo: \r, \n -> \\r, \\n in string constant URL: https://github.com/apache/nifi/pull/8041 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12368 Clear versionedComponentId on top-level components of popu… [nifi]
exceptionfactory closed pull request #8025: NIFI-12368 Clear versionedComponentId on top-level components of popu… URL: https://github.com/apache/nifi/pull/8025 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[
https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786967#comment-17786967
]
Zoltán Kornél Török commented on NIFI-12383:
I deployed the fix to my cluster, after the change the problem dissapeared
> GZipException occur during cluster replication, when the original request
> contains "accept-encoding" header with lowercase
> --
>
> Key: NIFI-12383
> URL: https://issues.apache.org/jira/browse/NIFI-12383
> Project: Apache NiFi
> Issue Type: Bug
>Reporter: Zoltán Kornél Török
>Assignee: Zoltán Kornél Török
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I had a three node cluster with knox.
> Time to time an error occured in the nifi logs on this cluster:
> {code}
> 2023-11-15 13:25:51,637 INFO
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator:
> Received a status of 500 from xy:8443 for request PUT
> /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing
> first stage of two-stage commit. The action will not occur. Node explanation:
> An unexpected error has occurred. Please check the logs for additional
> details.
> {code}
> Also sometimes I got "An unexpected error has occurred. Please check the logs
> for additional details." error on the UI too. After some investigation in I
> found the error in the logs:
> {code}
> 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78]
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> java.util.zip.ZipException: Not in GZIP format. Returning Internal Server
> Error response.
> java.util.zip.ZipException: Not in GZIP format
> at
> java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176)
> at
> java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79)
> at
> java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91)
> at
> org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49)
> at
> org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
> at
> org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072)
> at
> org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919)
> at
> org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290)
> at
> org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73)
> {code}
> After many hours of debugging, I found out, that sometimes when I use the
> cluster via knox, some unknown reason the incoming "Accept-Encoding" come
> with all leters lowercase (which is valid, becase HTTP header is case
> insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers).
> However OkHttpReplicationClient assume that the header is always
> "Accept-Encoding"
> (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294
> and
> https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25).
> Because of that, during replication the client not use gzip compression but
> when other node get the requests, the jetty read the original
> "accept-encoding" header and try to uncompress the inputstream, which lead to
> the above error.
> We need to add a few line code to the client to read the header case
> insensitivity
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] NIFI-12383 Replication client should handle accept encoding with lowe… [nifi]
taz1988 opened a new pull request, #8043: URL: https://github.com/apache/nifi/pull/8043 …rcase # Summary [NIFI-12383](https://issues.apache.org/jira/browse/NIFI-12383) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-12383) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-12383` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
Zoltán Kornél Török created NIFI-12383:
--
Summary: GZipException occur during cluster replication, when the
original request contains "accept-encoding" header with lowercase
Key: NIFI-12383
URL: https://issues.apache.org/jira/browse/NIFI-12383
Project: Apache NiFi
Issue Type: Bug
Reporter: Zoltán Kornél Török
Assignee: Zoltán Kornél Török
I had a three node cluster with knox.
Time to time an error occured in the nifi logs on this cluster:
{code}
2023-11-15 13:25:51,637 INFO
org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator:
Received a status of 500 from xy:8443 for request PUT
/nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing
first stage of two-stage commit. The action will not occur. Node explanation:
An unexpected error has occurred. Please check the logs for additional details.
{code}
Also sometimes I got "An unexpected error has occurred. Please check the logs
for additional details." error on the UI too. After some investigation in I
found the error in the logs:
{code}
23-11-15 13:40:25,289 ERROR [NiFi Web Server-78]
o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
java.util.zip.ZipException: Not in GZIP format. Returning Internal Server Error
response.
java.util.zip.ZipException: Not in GZIP format
at
java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176)
at
java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79)
at
java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91)
at org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49)
at
org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100)
at
org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
at
org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49)
at
org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
at
org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072)
at
org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919)
at
org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290)
at
org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73)
{code}
After many hours of debugging, I found out, that sometimes when I use the
cluster via knox, some unknown reason the incoming "Accept-Encoding" come with
all leters lowercase (which is valid, becase HTTP header is case insensitive -
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). However
OkHttpReplicationClient assume that the header is always "Accept-Encoding"
(https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294
and
https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25).
Because of that, during replication the client not use gzip compression but
when other node get the requests, the jetty read the original "accept-encoding"
header and try to uncompress the inputstream, which lead to the above error.
We need to add a few line code to the client to read the header case
insensitivity
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] NIFI-12382: Add DatabaseSchemaRegistry service [nifi]
mattyb149 commented on PR #8042: URL: https://github.com/apache/nifi/pull/8042#issuecomment-1815409306 I'll add more documentation around the use case and what this service does, but I didn't see any use for a DB-backed registry that just stores schema text (such as a String column containing an Avro schema) since we have other implementations that basically do that. I can rename the service DatabaseTableSchemaRegistry if that helps make it clearer what this service is for, but the idea is to generate the schema from metadata rather than store and retrieve schemas. This can be used by things like ValidateRecord in order to ensure that records will successfully go into a database with a downstream PutDatabaseRecord for example. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12382) Add DatabaseSchemaRegistry service
[ https://issues.apache.org/jira/browse/NIFI-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-12382: Status: Patch Available (was: In Progress) > Add DatabaseSchemaRegistry service > -- > > Key: NIFI-12382 > URL: https://issues.apache.org/jira/browse/NIFI-12382 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Sometimes it is necessary to get the table definition from a database as a > NiFi Record Schema. This can help ensure that data in the flow adheres to the > table definition rather than relying on type conversions and such. > This case proposes to add a controller service called DatabaseSchemaRegistry > that uses a DBCPService and a table name, fetches the table metadata from the > database, and creates a RecordSchema from it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12382: Add DatabaseSchemaRegistry service [nifi]
mattyb149 opened a new pull request, #8042: URL: https://github.com/apache/nifi/pull/8042 # Summary [NIFI-12382](https://issues.apache.org/jira/browse/NIFI-12382) This PR adds a DatabaseSchemaRegistry controller service which gets the metadata for a specified database table and creates a RecordSchema from it. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-12382) Add DatabaseSchemaRegistry service
[ https://issues.apache.org/jira/browse/NIFI-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess reassigned NIFI-12382: --- Assignee: Matt Burgess > Add DatabaseSchemaRegistry service > -- > > Key: NIFI-12382 > URL: https://issues.apache.org/jira/browse/NIFI-12382 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Assignee: Matt Burgess >Priority: Major > Fix For: 2.latest > > > Sometimes it is necessary to get the table definition from a database as a > NiFi Record Schema. This can help ensure that data in the flow adheres to the > table definition rather than relying on type conversions and such. > This case proposes to add a controller service called DatabaseSchemaRegistry > that uses a DBCPService and a table name, fetches the table metadata from the > database, and creates a RecordSchema from it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (NIFI-12382) Add DatabaseSchemaRegistry service
[ https://issues.apache.org/jira/browse/NIFI-12382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Burgess updated NIFI-12382: Fix Version/s: 2.latest > Add DatabaseSchemaRegistry service > -- > > Key: NIFI-12382 > URL: https://issues.apache.org/jira/browse/NIFI-12382 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions >Reporter: Matt Burgess >Priority: Major > Fix For: 2.latest > > > Sometimes it is necessary to get the table definition from a database as a > NiFi Record Schema. This can help ensure that data in the flow adheres to the > table definition rather than relying on type conversions and such. > This case proposes to add a controller service called DatabaseSchemaRegistry > that uses a DBCPService and a table name, fetches the table metadata from the > database, and creates a RecordSchema from it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12382) Add DatabaseSchemaRegistry service
Matt Burgess created NIFI-12382: --- Summary: Add DatabaseSchemaRegistry service Key: NIFI-12382 URL: https://issues.apache.org/jira/browse/NIFI-12382 Project: Apache NiFi Issue Type: New Feature Components: Extensions Reporter: Matt Burgess Sometimes it is necessary to get the table definition from a database as a NiFi Record Schema. This can help ensure that data in the flow adheres to the table definition rather than relying on type conversions and such. This case proposes to add a controller service called DatabaseSchemaRegistry that uses a DBCPService and a table name, fetches the table metadata from the database, and creates a RecordSchema from it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12368 Clear versionedComponentId on top-level components of popu… [nifi]
bbende commented on code in PR #8025: URL: https://github.com/apache/nifi/pull/8025#discussion_r1396346393 ## nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-web/nifi-web-api/src/main/java/org/apache/nifi/web/util/SnippetUtils.java: ## @@ -404,9 +404,26 @@ private Set getControllerServices(final Map
[jira] [Updated] (NIFI-12378) Remove Jython support in NiFi 2.x given real Python support in 2.x
[ https://issues.apache.org/jira/browse/NIFI-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joe Witt updated NIFI-12378: Fix Version/s: 2.latest > Remove Jython support in NiFi 2.x given real Python support in 2.x > -- > > Key: NIFI-12378 > URL: https://issues.apache.org/jira/browse/NIFI-12378 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Joe Witt >Priority: Major > Fix For: 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12362) Conduct Apache NiFi 1.24.0 Release
[ https://issues.apache.org/jira/browse/NIFI-12362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786897#comment-17786897 ] ASF subversion and git services commented on NIFI-12362: Commit 17167819faa1e78a1176eb254bbb72a4e708f551 in nifi's branch refs/heads/NIFI-12362-RC3 from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=17167819fa ] NIFI-12362-RC3 prepare for next development iteration > Conduct Apache NiFi 1.24.0 Release > -- > > Key: NIFI-12362 > URL: https://issues.apache.org/jira/browse/NIFI-12362 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: Pierre Villard >Priority: Major > Fix For: 1.24.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12362) Conduct Apache NiFi 1.24.0 Release
[ https://issues.apache.org/jira/browse/NIFI-12362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786896#comment-17786896 ] ASF subversion and git services commented on NIFI-12362: Commit 586a1a8789e9c39914f4a6088ac26e22d60eeb91 in nifi's branch refs/heads/NIFI-12362-RC3 from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=586a1a8789 ] NIFI-12362-RC3 prepare release nifi-1.24.0-RC3 > Conduct Apache NiFi 1.24.0 Release > -- > > Key: NIFI-12362 > URL: https://issues.apache.org/jira/browse/NIFI-12362 > Project: Apache NiFi > Issue Type: Task >Reporter: David Handermann >Assignee: Pierre Villard >Priority: Major > Fix For: 1.24.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-11671) JoinEnrichment SQL strategy doesn't allow attributes in join statement
[
https://issues.apache.org/jira/browse/NIFI-11671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786892#comment-17786892
]
Philipp Korniets commented on NIFI-11671:
-
[~mburgess149] - can you please take a look?
> JoinEnrichment SQL strategy doesn't allow attributes in join statement
> --
>
> Key: NIFI-11671
> URL: https://issues.apache.org/jira/browse/NIFI-11671
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core Framework
>Affects Versions: 1.18.0, 1.20.0, 1.23.0
>Reporter: Philipp Korniets
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> We use ForkEnrichement - JoinEnrichment pattern and want to include filtering
> in join SQL. Filter value is coming from FlowFile attribute
> {code:sql}
> ${test} = 'NewValue'
> SELECT original.*, enrichment.*,'${test}'
> FROM original
> LEFT OUTER JOIN enrichment
> ON original.Underlying = enrichment.Underlying
> WHERE enrichment.MyField = '${test}'
> {code}
> However this doesnt work because JoinEnrichment doesnt use
> evaluateAttributeExpressions
> Additionally in version 1.18,1.23 - doesnt allow whole query to be passed as
> attribute.
> !screenshot-1.png|width=692,height=431!
>
> {code:java}
> 2023-06-28 11:07:16,611 ERROR [Timer-Driven Process Thread-7]
> o.a.n.processors.standard.JoinEnrichment
> JoinEnrichment[id=dbe156ac-0187-1000-4477-0183899e0432] Failed to join
> 'original' FlowFile
> StandardFlowFileRecord[uuid=2ab9f6ad-73a5-4763-b25e-fd26c44835e1,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1687948831976-629,
> container=default, section=629], offset=8334082,
> length=600557],offset=0,name=lmr_SY08C41-1_S_514682_20230627.csv,size=600557]
> and 'enrichment' FlowFile
> StandardFlowFileRecord[uuid=e4bb7769-fdce-4dfe-af18-443676103035,claim=StandardContentClaim
> [resourceClaim=StandardResourceClaim[id=1687949723375-631,
> container=default, section=631], offset=5362822,
> length=1999502],offset=0,name=lmr_SY08C41-1_S_514682_20230627.csv,size=1999502];
> routing to failure
> java.sql.SQLException: Error while preparing statement [${instrumentJoinSQL}]
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:224)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:203)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:99)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:178)
> at
> org.apache.nifi.processors.standard.enrichment.SqlJoinCache.createCalciteParameters(SqlJoinCache.java:91)
> at
> org.apache.nifi.processors.standard.enrichment.SqlJoinCache.getCalciteParameters(SqlJoinCache.java:65)
> at
> org.apache.nifi.processors.standard.enrichment.SqlJoinStrategy.join(SqlJoinStrategy.java:49)
> at
> org.apache.nifi.processors.standard.JoinEnrichment.processBin(JoinEnrichment.java:387)
> at
> org.apache.nifi.processor.util.bin.BinFiles.processBins(BinFiles.java:233)
> at
> org.apache.nifi.processors.standard.JoinEnrichment.processBins(JoinEnrichment.java:503)
> at
> org.apache.nifi.processor.util.bin.BinFiles.onTrigger(BinFiles.java:193)
> at
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1354)
> at
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:246)
> at
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:102)
> at org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:750)
> Caused by: java.lang.RuntimeException: parse failed: Encountered "$" at line
> 1, column 1.
> Was expecting one of:
> "ABS" ...
> {code}
> As I understand issue is in following line of code
>
[PR] NIFI-12381 Fix typo: \r, \n -> \\r, \\n in string constant [nifi]
fgerlits opened a new pull request, #8041: URL: https://github.com/apache/nifi/pull/8041 # Summary [NIFI-12381](https://issues.apache.org/jira/browse/NIFI-12381) Fix typo. This is currently broken in the online docs https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.12.1/org.apache.nifi.processors.standard.SplitText/: it reads `(i.e., consists only of and characters)`, should read `(i.e., consists only of \r and \n characters)`. # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (NIFI-12381) Fix typo in SplitText.REMOVE_TRAILING_NEWLINES property description
Ferenc Gerlits created NIFI-12381: - Summary: Fix typo in SplitText.REMOVE_TRAILING_NEWLINES property description Key: NIFI-12381 URL: https://issues.apache.org/jira/browse/NIFI-12381 Project: Apache NiFi Issue Type: Improvement Reporter: Ferenc Gerlits Assignee: Ferenc Gerlits We found this typo when porting SplitText to MiNiFi C++, so I'm contributing it upstream. Every little counts, right? -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12372 MiNiFi C2 Encrypt Flow Configuration Properties when Transferring [nifi]
ferencerdei commented on PR #8028: URL: https://github.com/apache/nifi/pull/8028#issuecomment-1814856878 Thanks for the feature, this really useful. I have only one question. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12372 MiNiFi C2 Encrypt Flow Configuration Properties when Transferring [nifi]
ferencerdei commented on code in PR #8028:
URL: https://github.com/apache/nifi/pull/8028#discussion_r1395655853
##
minifi/minifi-nar-bundles/minifi-framework-bundle/minifi-framework/minifi-framework-core/src/main/java/org/apache/nifi/minifi/c2/command/DefaultUpdateConfigurationStrategy.java:
##
@@ -90,11 +93,12 @@ public boolean update(byte[] rawFlow) {
}
try {
-byte[] enrichedFlowCandidate =
flowEnrichService.enrichFlow(rawFlow);
+byte[] propertyEncryptedRawDataFlow =
flowPropertyEncryptor.encryptSensitiveProperties(rawFlow);
Review Comment:
Shouldn't we enrich the flow first and encrypt properties after? Now the
properties that are set for example in the sslControllerServiceProperties
method won't be encrypted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[PR] NIFI-12378 remove Jython [nifi]
joewitt opened a new pull request, #8040: URL: https://github.com/apache/nifi/pull/8040 # Summary [NIFI-0](https://issues.apache.org/jira/browse/NIFI-0) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] NIFI-12380 Update iceberg.version to 1.4.2 [nifi]
mr1716 opened a new pull request, #8039: URL: https://github.com/apache/nifi/pull/8039 # Summary [NIFI-12380](https://issues.apache.org/jira/browse/NIFI-12380) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [X] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI-12380) issue created ### Pull Request Tracking - [X] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [X] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [X] Pull Request based on current revision of the `main` branch - [X] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (NIFI-12380) Update iceberg.version to 1.4.2
[ https://issues.apache.org/jira/browse/NIFI-12380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mike R reassigned NIFI-12380: - Assignee: Mike R > Update iceberg.version to 1.4.2 > --- > > Key: NIFI-12380 > URL: https://issues.apache.org/jira/browse/NIFI-12380 > Project: Apache NiFi > Issue Type: Improvement >Affects Versions: 1.23.2 >Reporter: Mike R >Assignee: Mike R >Priority: Major > > Update iceberg.version to 1.4.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (NIFI-12380) Update iceberg.version to 1.4.2
Mike R created NIFI-12380: - Summary: Update iceberg.version to 1.4.2 Key: NIFI-12380 URL: https://issues.apache.org/jira/browse/NIFI-12380 Project: Apache NiFi Issue Type: Improvement Affects Versions: 1.23.2 Reporter: Mike R Update iceberg.version to 1.4.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12241 Efficient Parquet Splitting [nifi]
tpalfy commented on code in PR #7893:
URL: https://github.com/apache/nifi/pull/7893#discussion_r1392528848
##
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/processors/parquet/FetchParquet.java:
##
@@ -63,9 +79,35 @@ public class FetchParquet extends AbstractFetchHDFSRecord {
@Override
public HDFSRecordReader createHDFSRecordReader(final ProcessContext
context, final FlowFile flowFile, final Configuration conf, final Path path)
throws IOException {
+final Long offset =
Optional.ofNullable(flowFile.getAttribute(ParquetAttribute.RECORD_OFFSET))
+.map(Long::parseLong)
+.orElse(null);
+
+final Long count =
Optional.ofNullable(flowFile.getAttribute(ParquetAttribute.RECORD_COUNT))
+.map(Long::parseLong)
+.orElse(null);
+
+final long fileStartOffset =
Optional.ofNullable(flowFile.getAttribute(ParquetAttribute.FILE_RANGE_START_OFFSET))
+.map(Long::parseLong)
+.orElse(0L);
+final long fileEndOffset =
Optional.ofNullable(flowFile.getAttribute(ParquetAttribute.FILE_RANGE_END_OFFSET))
+.map(Long::parseLong)
+.orElse(Long.MAX_VALUE);
+
final InputFile inputFile = HadoopInputFile.fromPath(path, conf);
-final ParquetReader.Builder readerBuilder =
AvroParquetReader.builder(inputFile).withConf(conf);
-return new AvroParquetHDFSRecordReader(readerBuilder.build());
+final ParquetReader.Builder readerBuilder =
AvroParquetReader.builder(inputFile)
+.withConf(conf)
+.withFileRange(fileStartOffset, fileEndOffset);
+
+if (offset != null) {
+
readerBuilder.withFilter(FilterCompat.get(OffsetRecordFilter.offset(offset)));
+}
+
+if (count == null) {
+return new AvroParquetHDFSRecordReader(readerBuilder.build());
+} else {
+return new AvroParquetHDFSRecordReader(readerBuilder.build(),
count);
+}
Review Comment:
```suggestion
return new AvroParquetHDFSRecordReader(readerBuilder.build(), count);
```
##
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/main/java/org/apache/nifi/parquet/hadoop/AvroParquetHDFSRecordReader.java:
##
@@ -35,11 +35,18 @@ public class AvroParquetHDFSRecordReader implements
HDFSRecordReader {
private GenericRecord lastRecord;
private RecordSchema recordSchema;
private boolean initialized = false;
+private final Long count;
+private long recordsRead = 0;
private final ParquetReader parquetReader;
public AvroParquetHDFSRecordReader(final ParquetReader
parquetReader) {
+this(parquetReader, null);
+}
+
+public AvroParquetHDFSRecordReader(final ParquetReader
parquetReader, Long count) {
Review Comment:
```suggestion
public AvroParquetHDFSRecordReader(final ParquetReader
parquetReader, final Long count) {
```
##
nifi-nar-bundles/nifi-parquet-bundle/nifi-parquet-processors/src/test/java/org/apache/nifi/parquet/ParquetTestUtils.java:
##
@@ -0,0 +1,95 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.nifi.parquet;
+
+import static java.util.stream.Collectors.toList;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+import java.util.Collection;
+import java.util.Map;
+import java.util.stream.IntStream;
+import org.apache.avro.Schema;
+import org.apache.avro.generic.GenericData;
+import org.apache.avro.generic.GenericRecord;
+import org.apache.commons.io.IOUtils;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.parquet.avro.AvroParquetWriter;
+import org.apache.parquet.hadoop.ParquetWriter;
+import org.apache.parquet.hadoop.util.HadoopOutputFile;
+
+public class ParquetTestUtils {
+
+private static final String SCHEMA_PATH =
"src/test/resources/avro/user.avsc";
+
+public static File createUsersParquetFile(int numUsers) throws IOException
{
+return createUsersParquetFile(IntStream
+
[jira] [Resolved] (NIFI-11037) Add support of StandardProxyConfigurationService to StandardOauth2AccessTokenProvider
[ https://issues.apache.org/jira/browse/NIFI-11037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] crissaegrim resolved NIFI-11037. Resolution: Fixed > Add support of StandardProxyConfigurationService to > StandardOauth2AccessTokenProvider > - > > Key: NIFI-11037 > URL: https://issues.apache.org/jira/browse/NIFI-11037 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Reporter: crissaegrim >Priority: Major > > Currently, `StandardOauth2AccessTokenProvider` does not support > `StandardProxyConfigurationService`. > > I'm aware I can probaly use `java.arg.x=-Dhttp—` proxy flags. But this > breaks S3 supports because of a known issue with `aws-java-sdk`. See > aws/aws-sdk-java issue #2797 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI 12017 add ability to choose to output to single line for base32 base64 contents [nifi]
knguyen1 opened a new pull request, #8038: URL: https://github.com/apache/nifi/pull/8038 # Summary [NIFI-12017](https://issues.apache.org/jira/browse/NIFI-12017) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [x] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12017: feat(processor): add ability to choose to output to single line for base32 base64 contents [nifi]
knguyen1 closed pull request #8036: NIFI-12017: feat(processor): add ability to choose to output to single line for base32 base64 contents URL: https://github.com/apache/nifi/pull/8036 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
martinzink commented on code in PR #1693: URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1395661548 ## extensions/splunk/PutSplunkHTTP.cpp: ## Review Comment: Yeah, the general expressions should still work, maybe we should be more explicit about how expressions are evaluated like nifi does. (but thats a more general problem) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
martinzink commented on code in PR #1693: URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1395661548 ## extensions/splunk/PutSplunkHTTP.cpp: ## Review Comment: Yeah, the general expressions should still work. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
szaszm commented on code in PR #1693: URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1395656597 ## extensions/splunk/PutSplunkHTTP.cpp: ## Review Comment: In that case, we should change the docs to indicate that EL is not supported for those properties. Or do we still evaluate expressions, just without the flow file context? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12379) Refactor nifi-uuid5 to Remove External Dependencies
[ https://issues.apache.org/jira/browse/NIFI-12379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12379: -- Fix Version/s: 2.0.0-M1 (was: 2.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > Refactor nifi-uuid5 to Remove External Dependencies > --- > > Key: NIFI-12379 > URL: https://issues.apache.org/jira/browse/NIFI-12379 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 2.0.0-M1 > > Time Spent: 20m > Remaining Estimate: 0h > > The nifi-uuid5 module provides the Uuid5Util class for generating a UUID > string according to RFC 4122 Section 4.3 with SHA-1 hashing for version 5 > generation. The initial version used external libraries for hexadecimal > encoding and hashing, but these libraries can be removed in favor of standard > JDK HexFormat and MessageDigest classes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12379) Refactor nifi-uuid5 to Remove External Dependencies
[ https://issues.apache.org/jira/browse/NIFI-12379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786711#comment-17786711 ] ASF subversion and git services commented on NIFI-12379: Commit 026222d1573726dfece9a525867f8ab2ab90d7b1 in nifi's branch refs/heads/main from David Handermann [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=026222d157 ] NIFI-12379 Refactored nifi-uuid5 using standard classes - Replaced commons-codec Hex with Java HexFormat - Replaced commons-codec DigestUtils with Java MessageDigest Signed-off-by: Pierre Villard This closes #8035. > Refactor nifi-uuid5 to Remove External Dependencies > --- > > Key: NIFI-12379 > URL: https://issues.apache.org/jira/browse/NIFI-12379 > Project: Apache NiFi > Issue Type: Improvement > Components: Core Framework >Reporter: David Handermann >Assignee: David Handermann >Priority: Minor > Fix For: 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > The nifi-uuid5 module provides the Uuid5Util class for generating a UUID > string according to RFC 4122 Section 4.3 with SHA-1 hashing for version 5 > generation. The initial version used external libraries for hexadecimal > encoding and hashing, but these libraries can be removed in favor of standard > JDK HexFormat and MessageDigest classes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12379 Refactor nifi-uuid5 using standard classes [nifi]
asfgit closed pull request #8035: NIFI-12379 Refactor nifi-uuid5 using standard classes URL: https://github.com/apache/nifi/pull/8035 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12377) Deprecate support for Jython in NiFi 1.x line
[ https://issues.apache.org/jira/browse/NIFI-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12377: -- Fix Version/s: 1.24.0 (was: 1.latest) Resolution: Fixed Status: Resolved (was: Patch Available) > Deprecate support for Jython in NiFi 1.x line > - > > Key: NIFI-12377 > URL: https://issues.apache.org/jira/browse/NIFI-12377 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Pierre Villard >Priority: Major > Fix For: 1.24.0 > > Time Spent: 40m > Remaining Estimate: 0h > > Jython is not being actively maintained. There are a number of vulnerability > reports tied to it. Python 2 is EOL. The NiFi 2.x line has real integration > with real Python and is a better go forward approach. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] NIFI-12377 - Deprecate support for Jython in NiFi 1.x line [nifi]
arpadboda commented on PR #8037: URL: https://github.com/apache/nifi/pull/8037#issuecomment-1814233637 Merged in [44de1f2](https://github.com/apache/nifi/commit/44de1f2d671844157dc87300398382c1d52790c2) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] NIFI-12377 - Deprecate support for Jython in NiFi 1.x line [nifi]
arpadboda closed pull request #8037: NIFI-12377 - Deprecate support for Jython in NiFi 1.x line URL: https://github.com/apache/nifi/pull/8037 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12377) Deprecate support for Jython in NiFi 1.x line
[ https://issues.apache.org/jira/browse/NIFI-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786706#comment-17786706 ] ASF subversion and git services commented on NIFI-12377: Commit 44de1f2d671844157dc87300398382c1d52790c2 in nifi's branch refs/heads/support/nifi-1.x from Pierre Villard [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=44de1f2d67 ] NIFI-12377 - Deprecate support for Jython in NiFi 1.x line Signed-off-by: Arpad Boda This closes #8037 > Deprecate support for Jython in NiFi 1.x line > - > > Key: NIFI-12377 > URL: https://issues.apache.org/jira/browse/NIFI-12377 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Pierre Villard >Priority: Major > Fix For: 1.latest > > Time Spent: 20m > Remaining Estimate: 0h > > Jython is not being actively maintained. There are a number of vulnerability > reports tied to it. Python 2 is EOL. The NiFi 2.x line has real integration > with real Python and is a better go forward approach. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] MINIFICPP-62 Add SplitText processor [nifi-minifi-cpp]
fgerlits commented on code in PR #1682:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1682#discussion_r1395500165
##
extensions/standard-processors/tests/unit/SplitTextTests.cpp:
##
@@ -0,0 +1,860 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "FlowFileRecord.h"
+#include "TestBase.h"
+#include "Catch.h"
+#include "processors/SplitText.h"
+#include "SingleProcessorTestController.h"
+#include "io/BufferStream.h"
+
+namespace org::apache::nifi::minifi::test {
+
+TEST_CASE("Test LineReader with nullptr") {
+ processors::detail::LineReader reader{nullptr};
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with empty stream") {
+ auto stream = std::make_shared();
+ processors::detail::LineReader reader{nullptr};
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with trailing endline") {
+ auto stream = std::make_shared();
+ std::string input = "this is a new line\nand another line\r\nthirdline\n";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size = 19, .endline_size
= 1});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 19, .size = 18,
.endline_size = 2});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 37, .size = 10,
.endline_size = 1});
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader without trailing endlines") {
+ auto stream = std::make_shared();
+ std::string input = "this is a new line\nand another line\r\nthirdline";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size = 19, .endline_size
= 1});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 19, .size = 18,
.endline_size = 2});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 37, .size = 9, .endline_size
= 0});
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with input larger than buffer length") {
+ auto stream = std::make_shared();
+ const auto first_line_size =
static_cast(processors::detail::SPLIT_TEXT_BUFFER_SIZE * 1.5);
+ const auto second_line_size =
static_cast(processors::detail::SPLIT_TEXT_BUFFER_SIZE * 1.7);
+ std::string input = std::string(first_line_size, 'a') + "\n" +
std::string(second_line_size, 'b') + "\n";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size = first_line_size +
1, .endline_size = 1});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = first_line_size +1 , .size =
second_line_size + 1, .endline_size = 1});
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with input of same size as buffer length") {
+ auto stream = std::make_shared();
+ std::string input = std::string(processors::detail::SPLIT_TEXT_BUFFER_SIZE -
1, 'a') + "\n" + std::string(processors::detail::SPLIT_TEXT_BUFFER_SIZE * 2 -
1, 'b') + "\n";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size =
processors::detail::SPLIT_TEXT_BUFFER_SIZE, .endline_size = 1});
+ CHECK(*reader.readNextLine() ==
+processors::detail::LineReader::LineInfo{.offset =
Re: [PR] MINIFICPP-62 Add SplitText processor [nifi-minifi-cpp]
lordgamez commented on code in PR #1682:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1682#discussion_r1395492616
##
extensions/standard-processors/tests/unit/SplitTextTests.cpp:
##
@@ -0,0 +1,860 @@
+/**
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+#include "FlowFileRecord.h"
+#include "TestBase.h"
+#include "Catch.h"
+#include "processors/SplitText.h"
+#include "SingleProcessorTestController.h"
+#include "io/BufferStream.h"
+
+namespace org::apache::nifi::minifi::test {
+
+TEST_CASE("Test LineReader with nullptr") {
+ processors::detail::LineReader reader{nullptr};
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with empty stream") {
+ auto stream = std::make_shared();
+ processors::detail::LineReader reader{nullptr};
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with trailing endline") {
+ auto stream = std::make_shared();
+ std::string input = "this is a new line\nand another line\r\nthirdline\n";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size = 19, .endline_size
= 1});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 19, .size = 18,
.endline_size = 2});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 37, .size = 10,
.endline_size = 1});
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader without trailing endlines") {
+ auto stream = std::make_shared();
+ std::string input = "this is a new line\nand another line\r\nthirdline";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size = 19, .endline_size
= 1});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 19, .size = 18,
.endline_size = 2});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 37, .size = 9, .endline_size
= 0});
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with input larger than buffer length") {
+ auto stream = std::make_shared();
+ const auto first_line_size =
static_cast(processors::detail::SPLIT_TEXT_BUFFER_SIZE * 1.5);
+ const auto second_line_size =
static_cast(processors::detail::SPLIT_TEXT_BUFFER_SIZE * 1.7);
+ std::string input = std::string(first_line_size, 'a') + "\n" +
std::string(second_line_size, 'b') + "\n";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size = first_line_size +
1, .endline_size = 1});
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = first_line_size +1 , .size =
second_line_size + 1, .endline_size = 1});
+ CHECK(reader.readNextLine() == std::nullopt);
+ CHECK(reader.getState() == processors::detail::StreamReadState::EndOfStream);
+}
+
+TEST_CASE("Test LineReader with input of same size as buffer length") {
+ auto stream = std::make_shared();
+ std::string input = std::string(processors::detail::SPLIT_TEXT_BUFFER_SIZE -
1, 'a') + "\n" + std::string(processors::detail::SPLIT_TEXT_BUFFER_SIZE * 2 -
1, 'b') + "\n";
+ stream->write(reinterpret_cast(input.data()), input.size());
+ processors::detail::LineReader reader{stream};
+ CHECK(*reader.readNextLine() ==
processors::detail::LineReader::LineInfo{.offset = 0, .size =
processors::detail::SPLIT_TEXT_BUFFER_SIZE, .endline_size = 1});
+ CHECK(*reader.readNextLine() ==
+processors::detail::LineReader::LineInfo{.offset =
Re: [PR] MINIFICPP-62 Add SplitText processor [nifi-minifi-cpp]
lordgamez commented on code in PR #1682:
URL: https://github.com/apache/nifi-minifi-cpp/pull/1682#discussion_r1395492392
##
extensions/standard-processors/processors/SplitText.h:
##
@@ -0,0 +1,234 @@
+/**
+ * @file SplitText.h
+ * SplitText class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include
+#include
+#include
+#include
+
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/PropertyDefinitionBuilder.h"
+#include "core/PropertyDefinition.h"
+#include "core/PropertyType.h"
+#include "core/RelationshipDefinition.h"
+#include "FlowFileRecord.h"
+#include "utils/Export.h"
+#include "utils/expected.h"
+
+namespace org::apache::nifi::minifi::processors {
+
+struct SplitTextConfiguration {
+ uint64_t line_split_count = 0;
+ std::optional maximum_fragment_size;
+ uint64_t header_line_count = 0;
+ std::optional header_line_marker_characters;
+ bool remove_trailing_new_lines = true;
+};
+
+namespace detail {
+
+constexpr size_t SPLIT_TEXT_BUFFER_SIZE = 8192;
Review Comment:
Good point, updated in fccf152353ae7c25a8d4944b8b0e3add15254984
##
extensions/standard-processors/processors/SplitText.h:
##
@@ -0,0 +1,234 @@
+/**
+ * @file SplitText.h
+ * SplitText class declaration
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+#pragma once
+
+#include
+#include
+#include
+#include
+
+#include "core/Processor.h"
+#include "core/ProcessSession.h"
+#include "core/PropertyDefinitionBuilder.h"
+#include "core/PropertyDefinition.h"
+#include "core/PropertyType.h"
+#include "core/RelationshipDefinition.h"
+#include "FlowFileRecord.h"
+#include "utils/Export.h"
+#include "utils/expected.h"
+
+namespace org::apache::nifi::minifi::processors {
+
+struct SplitTextConfiguration {
+ uint64_t line_split_count = 0;
+ std::optional maximum_fragment_size;
+ uint64_t header_line_count = 0;
+ std::optional header_line_marker_characters;
+ bool remove_trailing_new_lines = true;
+};
+
+namespace detail {
+
+constexpr size_t SPLIT_TEXT_BUFFER_SIZE = 8192;
+
+enum class StreamReadState {
+ Ok,
+ StreamReadError,
+ EndOfStream
+};
+
+class LineReader {
+ public:
+ struct LineInfo {
+uint64_t offset = 0;
+uint64_t size = 0;
+uint8_t endline_size = 0;
+bool matches_starts_with = true;
+
+bool operator==(const LineInfo& line_info) const = default;
+ };
+
+ explicit LineReader(const std::shared_ptr& stream);
+ std::optional readNextLine(const std::optional&
starts_with = std::nullopt);
+ StreamReadState getState() const { return state_; }
+
+ private:
+ uint8_t getEndLineSize(size_t newline_index);
+ void setLastLineInfoAttributes(uint8_t endline_size, const
std::optional& starts_with);
+ bool readNextBuffer();
+ std::optional finalizeLineInfo(uint8_t endline_size,
const std::optional& starts_with);
+
+ size_t buffer_offset_ = 0;
+ uint64_t current_buffer_count_ = 0;
+ size_t last_read_size_ = 0;
+ uint64_t read_size_ = 0;
+ std::array buffer_{};
+ std::shared_ptr stream_;
+ std::optional last_line_info_;
+ StreamReadState state_ = StreamReadState::Ok;
+};
+
+class SplitTextFragmentGenerator {
+ public:
+ struct Fragment {
+uint64_t text_line_count = 0;
+uint64_t processed_line_count = 0;
+uint64_t fragment_size = 0;
+uint64_t fragment_offset = 0;
+uint8_t endline_size = 0;
+ };
+
+ SplitTextFragmentGenerator(const std::shared_ptr& stream,
const
Re: [PR] NIFI-12377 - Deprecate support for Jython in NiFi 1.x line [nifi]
arpadboda commented on PR #8037: URL: https://github.com/apache/nifi/pull/8037#issuecomment-1814161542 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (NIFI-12377) Deprecate support for Jython in NiFi 1.x line
[ https://issues.apache.org/jira/browse/NIFI-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard updated NIFI-12377: -- Status: Patch Available (was: Open) > Deprecate support for Jython in NiFi 1.x line > - > > Key: NIFI-12377 > URL: https://issues.apache.org/jira/browse/NIFI-12377 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Pierre Villard >Priority: Major > Fix For: 1.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Jython is not being actively maintained. There are a number of vulnerability > reports tied to it. Python 2 is EOL. The NiFi 2.x line has real integration > with real Python and is a better go forward approach. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (NIFI-12377) Deprecate support for Jython in NiFi 1.x line
[ https://issues.apache.org/jira/browse/NIFI-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pierre Villard reassigned NIFI-12377: - Assignee: Pierre Villard (was: Joe Witt) > Deprecate support for Jython in NiFi 1.x line > - > > Key: NIFI-12377 > URL: https://issues.apache.org/jira/browse/NIFI-12377 > Project: Apache NiFi > Issue Type: Task >Reporter: Joe Witt >Assignee: Pierre Villard >Priority: Major > Fix For: 1.latest > > Time Spent: 10m > Remaining Estimate: 0h > > Jython is not being actively maintained. There are a number of vulnerability > reports tied to it. Python 2 is EOL. The NiFi 2.x line has real integration > with real Python and is a better go forward approach. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12377 - Deprecate support for Jython in NiFi 1.x line [nifi]
pvillard31 opened a new pull request, #8037: URL: https://github.com/apache/nifi/pull/8037 # Summary [NIFI-12377](https://issues.apache.org/jira/browse/NIFI-12377)- Deprecate support for Jython in NiFi 1.x line # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [ ] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [ ] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [ ] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [ ] Pull Request based on current revision of the `main` branch - [ ] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [ ] Build completed using `mvn clean install -P contrib-check` - [ ] JDK 21 ### Licensing - [ ] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [ ] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (NIFI-12017) EncodeContent: Allow single line output for base64 and base32
[ https://issues.apache.org/jira/browse/NIFI-12017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786684#comment-17786684 ] crissaegrim commented on NIFI-12017: https://github.com/apache/nifi/pull/8036/files > EncodeContent: Allow single line output for base64 and base32 > - > > Key: NIFI-12017 > URL: https://issues.apache.org/jira/browse/NIFI-12017 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.23.2 >Reporter: Umar Hussain >Assignee: crissaegrim >Priority: Minor > Fix For: 1.latest, 2.latest > > Time Spent: 10m > Remaining Estimate: 0h > > This ticket focuses on adding the option in EncodeContent processor for > outputting Base64 and Base32 encoded content in a single line, in addition to > by default splitting output. > This option will only appear when encoding base64 and base32 (will not apply > to hex) and not in any decoding. > It will help in cases where single line encoded content is required, for > which currently we need to use an additional ReplaceText processor to remove > line separator -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] NIFI-12017: feat(processor): add ability to choose to output to single line for base32 base64 contents [nifi]
knguyen1 opened a new pull request, #8036: URL: https://github.com/apache/nifi/pull/8036 # Summary [NIFI-12017](https://issues.apache.org/jira/projects/NIFI/issues/NIFI-12017) # Tracking Please complete the following tracking steps prior to pull request creation. ### Issue Tracking - [x] [Apache NiFi Jira](https://issues.apache.org/jira/browse/NIFI) issue created ### Pull Request Tracking - [x] Pull Request title starts with Apache NiFi Jira issue number, such as `NIFI-0` - [x] Pull Request commit message starts with Apache NiFi Jira issue number, as such `NIFI-0` ### Pull Request Formatting - [x] Pull Request based on current revision of the `main` branch - [x] Pull Request refers to a feature branch with one commit containing changes # Verification Please indicate the verification steps performed prior to pull request creation. ### Build - [x] Build completed using `mvn clean install -P contrib-check` - [x] JDK 21 ### Licensing - [x] New dependencies are compatible with the [Apache License 2.0](https://apache.org/licenses/LICENSE-2.0) according to the [License Policy](https://www.apache.org/legal/resolved.html) - [x] New dependencies are documented in applicable `LICENSE` and `NOTICE` files ### Documentation - [ ] Documentation formatting appears as expected in rendered files -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
martinzink commented on code in PR #1693: URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1395342750 ## extensions/http-curl/processors/InvokeHTTP.cpp: ## Review Comment: Same here > This was intentional, it was badly designed before. The creation of the client is only done when there is no available clients, and they are reused afterhand, so setting up a client based on a flowfile then using it with an other flowfile was incorrect. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] MINIFICPP-1415 Pass references to onTrigger and onSchedule instead of… [nifi-minifi-cpp]
martinzink commented on code in PR #1693: URL: https://github.com/apache/nifi-minifi-cpp/pull/1693#discussion_r1395342638 ## extensions/splunk/PutSplunkHTTP.cpp: ## Review Comment: This was intentional, it was badly designed before. The creation of the client is only done when there is no available clients, and they are reused afterhand, so setting up a client based on a flowfile then using it with an other flowfile was incorrect. ## extensions/http-curl/processors/InvokeHTTP.cpp: ## Review Comment: Same here -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@nifi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
