[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[ https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787300#comment-17787300 ] David Handermann commented on NIFI-12383: - Thanks for the confirmation! > GZipException occur during cluster replication, when the original request > contains "accept-encoding" header with lowercase > -- > > Key: NIFI-12383 > URL: https://issues.apache.org/jira/browse/NIFI-12383 > Project: Apache NiFi > Issue Type: Bug >Reporter: Zoltán Kornél Török >Assignee: Zoltán Kornél Török >Priority: Major > Fix For: 2.0.0-M1, 1.25.0 > > Time Spent: 1h > Remaining Estimate: 0h > > I had a three node cluster with knox. > Time to time an error occured in the nifi logs on this cluster: > {code} > 2023-11-15 13:25:51,637 INFO > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator: > Received a status of 500 from xy:8443 for request PUT > /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing > first stage of two-stage commit. The action will not occur. Node explanation: > An unexpected error has occurred. Please check the logs for additional > details. > {code} > Also sometimes I got "An unexpected error has occurred. Please check the logs > for additional details." error on the UI too. After some investigation in I > found the error in the logs: > {code} > 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.util.zip.ZipException: Not in GZIP format. Returning Internal Server > Error response. > java.util.zip.ZipException: Not in GZIP format > at > java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) > at > org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49) > at > org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072) > at > org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919) > at > org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290) > at > org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73) > {code} > After many hours of debugging, I found out, that sometimes when I use the > cluster via knox, some unknown reason the incoming "Accept-Encoding" come > with all leters lowercase (which is valid, becase HTTP header is case > insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). > However OkHttpReplicationClient assume that the header is always > "Accept-Encoding" > (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294 > and > https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25). > Because of that, during replication the client not use gzip compression but > when other node get the requests, the jetty read the original > "accept-encoding" header and try to uncompress the inputstream, which lead to > the above error. > We need to add a few line code to the client to read the header case > insensitivity -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[ https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787276#comment-17787276 ] Zoltán Kornél Török commented on NIFI-12383: Here is the content of nifi properties (ommit some sensitive values) {code:java} nifi.administrative.yield.duration=30 sec nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares nifi.analytics.connection.model.score.name=rSquared nifi.analytics.connection.model.score.threshold=.90 nifi.analytics.predict.enabled=true nifi.analytics.predict.interval=3 mins nifi.analytics.query.interval=5 mins nifi.authorizer.configuration.file=/var/run/../authorizers.xml nifi.bored.yield.duration=10 millis nifi.cluster.firewall.file= nifi.cluster.flow.election.max.candidates=3 nifi.cluster.flow.election.max.wait.time=1 mins nifi.cluster.is.node=true nifi.cluster.load.balance.comms.timeout=30 sec nifi.cluster.load.balance.connections.per.node=1 nifi.cluster.load.balance.host= nifi.cluster.load.balance.max.thread.count=8 nifi.cluster.load.balance.port=6342 nifi.cluster.node.address=* nifi.cluster.node.connection.timeout=30 sec nifi.cluster.node.event.history.size=25 nifi.cluster.node.max.concurrent.requests=100 nifi.cluster.node.protocol.max.threads=50 nifi.cluster.node.protocol.port=9088 nifi.cluster.node.read.timeout=30 sec nifi.cluster.protocol.heartbeat.interval=5 sec nifi.cluster.protocol.is.secure=true nifi.components.status.repository.buffer.size=1440 nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository nifi.components.status.snapshot.frequency=1 min nifi.content.claim.max.appendable.size=50 KB nifi.content.repository.always.sync=false nifi.content.repository.archive.enabled=true nifi.content.repository.archive.max.retention.period=30 days nifi.content.repository.archive.max.usage.percentage=70% nifi.content.repository.directory.default=* nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository nifi.content.viewer.url=../nifi-content-viewer/ nifi.database.directory==* nifi.documentation.working.directory==* nifi.flow.analysis.background.task.schedule=5 mins nifi.flow.configuration.archive.dir==* nifi.flow.configuration.archive.enabled=true nifi.flow.configuration.archive.max.storage=500 MB nifi.flow.configuration.archive.max.time=30 days nifi.flow.configuration.file=flow.json.gz nifi.flow.configuration.json.file=flow.json.gz nifi.flowcontroller.autoResumeState=true nifi.flowcontroller.graceful.shutdown.period=10 sec nifi.flowfile.repository.always.sync=false nifi.flowfile.repository.checkpoint.interval=2 mins nifi.flowfile.repository.directory==flowfile-repo nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository nifi.flowfile.repository.partitions=256 nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog nifi.flowservice.writedelay.interval=500 ms nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE nifi.initial.admin.identity= nifi.kerberos.krb5.file=/etc/krb5.conf nifi.kerberos.service.keytab.location==b nifi.kerberos.service.principal=n=K nifi.login.identity.provider.configuration.file==/login-identity-providers.xml nifi.monitor.long.running.task.schedule= nifi.monitor.long.running.task.threshold= nifi.provenance.repository.always.sync=false nifi.provenance.repository.buffer.size=10 nifi.provenance.repository.compress.on.rollover=true nifi.provenance.repository.concurrent.merge.threads=2 nifi.provenance.repository.directory.default== nifi.provenance.repository.encryption.key.provider.location= nifi.provenance.repository.encryption.key.provider.password= nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository nifi.provenance.repository.index.shard.size=4 GB nifi.provenance.repository.index.threads=2 nifi.provenance.repository.indexed.attributes= nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship nifi.provenance.repository.max.attribute.length=65536 nifi.provenance.repository.max.storage.size=150 GB nifi.provenance.repository.max.storage.time=30 days nifi.provenance.repository.query.threads=2 nifi.provenance.repository.rollover.size=1 GB nifi.provenance.repository.rollover.time=10 mins nifi.python.extensions.source.directory.default== nifi.python.framework.source.directory== nifi.python.logs.directory== nifi.python.max.processes=100 nifi.python.max.processes.per.extension.type=10 nifi.python.working.directory== nifi.queue.backpressure.count=1 nifi.queue.backpressure.size=1 GB nifi.queue.swap.threshold=2 nifi.remote.contents.cache.expiration=30 secs nifi.remote.input.host== nifi.remote.input.http.enabled=true
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[ https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787274#comment-17787274 ] Zoltán Kornél Török commented on NIFI-12383: Oh I see. It was a nifi-2.0 cluster what I played with where this error occured > GZipException occur during cluster replication, when the original request > contains "accept-encoding" header with lowercase > -- > > Key: NIFI-12383 > URL: https://issues.apache.org/jira/browse/NIFI-12383 > Project: Apache NiFi > Issue Type: Bug >Reporter: Zoltán Kornél Török >Assignee: Zoltán Kornél Török >Priority: Major > Fix For: 2.0.0-M1, 1.25.0 > > Time Spent: 1h > Remaining Estimate: 0h > > I had a three node cluster with knox. > Time to time an error occured in the nifi logs on this cluster: > {code} > 2023-11-15 13:25:51,637 INFO > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator: > Received a status of 500 from xy:8443 for request PUT > /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing > first stage of two-stage commit. The action will not occur. Node explanation: > An unexpected error has occurred. Please check the logs for additional > details. > {code} > Also sometimes I got "An unexpected error has occurred. Please check the logs > for additional details." error on the UI too. After some investigation in I > found the error in the logs: > {code} > 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.util.zip.ZipException: Not in GZIP format. Returning Internal Server > Error response. > java.util.zip.ZipException: Not in GZIP format > at > java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) > at > org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49) > at > org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072) > at > org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919) > at > org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290) > at > org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73) > {code} > After many hours of debugging, I found out, that sometimes when I use the > cluster via knox, some unknown reason the incoming "Accept-Encoding" come > with all leters lowercase (which is valid, becase HTTP header is case > insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). > However OkHttpReplicationClient assume that the header is always > "Accept-Encoding" > (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294 > and > https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25). > Because of that, during replication the client not use gzip compression but > when other node get the requests, the jetty read the original > "accept-encoding" header and try to uncompress the inputstream, which lead to > the above error. > We need to add a few line code to the client to read the header case > insensitivity -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[ https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787241#comment-17787241 ] David Handermann commented on NIFI-12383: - Thanks for the reply [~taz1988]. The {{h2}} protocol is now the default on the main branch, but it was not the default for NiFi 1, so that is why I asked. The change itself is simple enough, so it is not so much of a concern, but it would be useful to know the NiFi version and configured property value if you have that information available for reference. > GZipException occur during cluster replication, when the original request > contains "accept-encoding" header with lowercase > -- > > Key: NIFI-12383 > URL: https://issues.apache.org/jira/browse/NIFI-12383 > Project: Apache NiFi > Issue Type: Bug >Reporter: Zoltán Kornél Török >Assignee: Zoltán Kornél Török >Priority: Major > Fix For: 2.latest > > Time Spent: 1h > Remaining Estimate: 0h > > I had a three node cluster with knox. > Time to time an error occured in the nifi logs on this cluster: > {code} > 2023-11-15 13:25:51,637 INFO > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator: > Received a status of 500 from xy:8443 for request PUT > /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing > first stage of two-stage commit. The action will not occur. Node explanation: > An unexpected error has occurred. Please check the logs for additional > details. > {code} > Also sometimes I got "An unexpected error has occurred. Please check the logs > for additional details." error on the UI too. After some investigation in I > found the error in the logs: > {code} > 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.util.zip.ZipException: Not in GZIP format. Returning Internal Server > Error response. > java.util.zip.ZipException: Not in GZIP format > at > java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) > at > org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49) > at > org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072) > at > org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919) > at > org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290) > at > org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73) > {code} > After many hours of debugging, I found out, that sometimes when I use the > cluster via knox, some unknown reason the incoming "Accept-Encoding" come > with all leters lowercase (which is valid, becase HTTP header is case > insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). > However OkHttpReplicationClient assume that the header is always > "Accept-Encoding" > (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294 > and > https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25). > Because of that, during replication the client not use gzip compression but > when other node get the requests, the jetty read the original > "accept-encoding" header and try to uncompress the inputstream, which lead to > the above error. > We need to add a few line code to the client to read the header case > insensitivity -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[ https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787116#comment-17787116 ] ASF subversion and git services commented on NIFI-12383: Commit 4e38e28d050d4ddcf766860ef820cb79aa8cfbc7 in nifi's branch refs/heads/support/nifi-1.x from Zoltan Kornel Torok [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=4e38e28d05 ] NIFI-12383 Replication client should handle accept encoding with lowercase Signed-off-by: Bence Simon This closes #8043 > GZipException occur during cluster replication, when the original request > contains "accept-encoding" header with lowercase > -- > > Key: NIFI-12383 > URL: https://issues.apache.org/jira/browse/NIFI-12383 > Project: Apache NiFi > Issue Type: Bug >Reporter: Zoltán Kornél Török >Assignee: Zoltán Kornél Török >Priority: Major > Fix For: 2.latest > > Time Spent: 40m > Remaining Estimate: 0h > > I had a three node cluster with knox. > Time to time an error occured in the nifi logs on this cluster: > {code} > 2023-11-15 13:25:51,637 INFO > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator: > Received a status of 500 from xy:8443 for request PUT > /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing > first stage of two-stage commit. The action will not occur. Node explanation: > An unexpected error has occurred. Please check the logs for additional > details. > {code} > Also sometimes I got "An unexpected error has occurred. Please check the logs > for additional details." error on the UI too. After some investigation in I > found the error in the logs: > {code} > 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.util.zip.ZipException: Not in GZIP format. Returning Internal Server > Error response. > java.util.zip.ZipException: Not in GZIP format > at > java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) > at > org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49) > at > org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072) > at > org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919) > at > org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290) > at > org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73) > {code} > After many hours of debugging, I found out, that sometimes when I use the > cluster via knox, some unknown reason the incoming "Accept-Encoding" come > with all leters lowercase (which is valid, becase HTTP header is case > insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). > However OkHttpReplicationClient assume that the header is always > "Accept-Encoding" > (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294 > and > https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25). > Because of that, during replication the client not use gzip compression but > when other node get the requests, the jetty read the original > "accept-encoding" header and try to uncompress the inputstream, which lead to > the above error. > We need to add a few line code to the client to read the header case > insensitivity -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[ https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787089#comment-17787089 ] Zoltán Kornél Török commented on NIFI-12383: [~exceptionfactory], As I see h2 http/1.1 is the default, so I guess that means yes? (I didn't override this value) > GZipException occur during cluster replication, when the original request > contains "accept-encoding" header with lowercase > -- > > Key: NIFI-12383 > URL: https://issues.apache.org/jira/browse/NIFI-12383 > Project: Apache NiFi > Issue Type: Bug >Reporter: Zoltán Kornél Török >Assignee: Zoltán Kornél Török >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > I had a three node cluster with knox. > Time to time an error occured in the nifi logs on this cluster: > {code} > 2023-11-15 13:25:51,637 INFO > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator: > Received a status of 500 from xy:8443 for request PUT > /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing > first stage of two-stage commit. The action will not occur. Node explanation: > An unexpected error has occurred. Please check the logs for additional > details. > {code} > Also sometimes I got "An unexpected error has occurred. Please check the logs > for additional details." error on the UI too. After some investigation in I > found the error in the logs: > {code} > 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.util.zip.ZipException: Not in GZIP format. Returning Internal Server > Error response. > java.util.zip.ZipException: Not in GZIP format > at > java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) > at > org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49) > at > org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072) > at > org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919) > at > org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290) > at > org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73) > {code} > After many hours of debugging, I found out, that sometimes when I use the > cluster via knox, some unknown reason the incoming "Accept-Encoding" come > with all leters lowercase (which is valid, becase HTTP header is case > insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). > However OkHttpReplicationClient assume that the header is always > "Accept-Encoding" > (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294 > and > https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25). > Because of that, during replication the client not use gzip compression but > when other node get the requests, the jetty read the original > "accept-encoding" header and try to uncompress the inputstream, which lead to > the above error. > We need to add a few line code to the client to read the header case > insensitivity -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[ https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17787062#comment-17787062 ] ASF subversion and git services commented on NIFI-12383: Commit 63364687d8bbe2af2d0031f745ca3c61144f8b4f in nifi's branch refs/heads/main from Zoltan Kornel Torok [ https://gitbox.apache.org/repos/asf?p=nifi.git;h=63364687d8 ] NIFI-12383 Replication client should handle accept encoding with lowercase Signed-off-by: Bence Simon This closes #8043 > GZipException occur during cluster replication, when the original request > contains "accept-encoding" header with lowercase > -- > > Key: NIFI-12383 > URL: https://issues.apache.org/jira/browse/NIFI-12383 > Project: Apache NiFi > Issue Type: Bug >Reporter: Zoltán Kornél Török >Assignee: Zoltán Kornél Török >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > I had a three node cluster with knox. > Time to time an error occured in the nifi logs on this cluster: > {code} > 2023-11-15 13:25:51,637 INFO > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator: > Received a status of 500 from xy:8443 for request PUT > /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing > first stage of two-stage commit. The action will not occur. Node explanation: > An unexpected error has occurred. Please check the logs for additional > details. > {code} > Also sometimes I got "An unexpected error has occurred. Please check the logs > for additional details." error on the UI too. After some investigation in I > found the error in the logs: > {code} > 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.util.zip.ZipException: Not in GZIP format. Returning Internal Server > Error response. > java.util.zip.ZipException: Not in GZIP format > at > java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) > at > org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49) > at > org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072) > at > org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919) > at > org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290) > at > org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73) > {code} > After many hours of debugging, I found out, that sometimes when I use the > cluster via knox, some unknown reason the incoming "Accept-Encoding" come > with all leters lowercase (which is valid, becase HTTP header is case > insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). > However OkHttpReplicationClient assume that the header is always > "Accept-Encoding" > (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294 > and > https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25). > Because of that, during replication the client not use gzip compression but > when other node get the requests, the jetty read the original > "accept-encoding" header and try to uncompress the inputstream, which lead to > the above error. > We need to add a few line code to the client to read the header case > insensitivity -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[ https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786987#comment-17786987 ] David Handermann commented on NIFI-12383: - Thanks for summarizing this issue [~taz1988]. Do you know if the {{h2}} protocol was enabled for the HTTP protocols property in nifi.properties? The HTTP/2 protocol uses binary for headers, and as a result, the header names can be reported as lowercase when HTTP/2 is negotiated. > GZipException occur during cluster replication, when the original request > contains "accept-encoding" header with lowercase > -- > > Key: NIFI-12383 > URL: https://issues.apache.org/jira/browse/NIFI-12383 > Project: Apache NiFi > Issue Type: Bug >Reporter: Zoltán Kornél Török >Assignee: Zoltán Kornél Török >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > I had a three node cluster with knox. > Time to time an error occured in the nifi logs on this cluster: > {code} > 2023-11-15 13:25:51,637 INFO > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator: > Received a status of 500 from xy:8443 for request PUT > /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing > first stage of two-stage commit. The action will not occur. Node explanation: > An unexpected error has occurred. Please check the logs for additional > details. > {code} > Also sometimes I got "An unexpected error has occurred. Please check the logs > for additional details." error on the UI too. After some investigation in I > found the error in the logs: > {code} > 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.util.zip.ZipException: Not in GZIP format. Returning Internal Server > Error response. > java.util.zip.ZipException: Not in GZIP format > at > java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) > at > org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49) > at > org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072) > at > org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919) > at > org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290) > at > org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73) > {code} > After many hours of debugging, I found out, that sometimes when I use the > cluster via knox, some unknown reason the incoming "Accept-Encoding" come > with all leters lowercase (which is valid, becase HTTP header is case > insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). > However OkHttpReplicationClient assume that the header is always > "Accept-Encoding" > (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294 > and > https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25). > Because of that, during replication the client not use gzip compression but > when other node get the requests, the jetty read the original > "accept-encoding" header and try to uncompress the inputstream, which lead to > the above error. > We need to add a few line code to the client to read the header case > insensitivity -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (NIFI-12383) GZipException occur during cluster replication, when the original request contains "accept-encoding" header with lowercase
[ https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17786967#comment-17786967 ] Zoltán Kornél Török commented on NIFI-12383: I deployed the fix to my cluster, after the change the problem dissapeared > GZipException occur during cluster replication, when the original request > contains "accept-encoding" header with lowercase > -- > > Key: NIFI-12383 > URL: https://issues.apache.org/jira/browse/NIFI-12383 > Project: Apache NiFi > Issue Type: Bug >Reporter: Zoltán Kornél Török >Assignee: Zoltán Kornél Török >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > I had a three node cluster with knox. > Time to time an error occured in the nifi logs on this cluster: > {code} > 2023-11-15 13:25:51,637 INFO > org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator: > Received a status of 500 from xy:8443 for request PUT > /nifi-api/process-groups/d2cedf64-018b-1000--164a79fc when performing > first stage of two-stage commit. The action will not occur. Node explanation: > An unexpected error has occurred. Please check the logs for additional > details. > {code} > Also sometimes I got "An unexpected error has occurred. Please check the logs > for additional details." error on the UI too. After some investigation in I > found the error in the logs: > {code} > 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78] > o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: > java.util.zip.ZipException: Not in GZIP format. Returning Internal Server > Error response. > java.util.zip.ZipException: Not in GZIP format > at > java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:79) > at > java.base/java.util.zip.GZIPInputStream.(GZIPInputStream.java:91) > at > org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49) > at > org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49) > at > org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132) > at > org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072) > at > org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919) > at > org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290) > at > org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73) > {code} > After many hours of debugging, I found out, that sometimes when I use the > cluster via knox, some unknown reason the incoming "Accept-Encoding" come > with all leters lowercase (which is valid, becase HTTP header is case > insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers). > However OkHttpReplicationClient assume that the header is always > "Accept-Encoding" > (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294 > and > https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25). > Because of that, during replication the client not use gzip compression but > when other node get the requests, the jetty read the original > "accept-encoding" header and try to uncompress the inputstream, which lead to > the above error. > We need to add a few line code to the client to read the header case > insensitivity -- This message was sent by Atlassian Jira (v8.20.10#820010)