[
https://issues.apache.org/jira/browse/NIFI-12383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787276#comment-17787276
]
Zoltán Kornél Török commented on NIFI-12383:
--------------------------------------------
Here is the content of nifi properties (ommit some sensitive values)
{code:java}
nifi.administrative.yield.duration=30 sec
nifi.analytics.connection.model.implementation=org.apache.nifi.controller.status.analytics.models.OrdinaryLeastSquares
nifi.analytics.connection.model.score.name=rSquared
nifi.analytics.connection.model.score.threshold=.90
nifi.analytics.predict.enabled=true
nifi.analytics.predict.interval=3 mins
nifi.analytics.query.interval=5 mins
nifi.authorizer.configuration.file=/var/run/../authorizers.xml
nifi.bored.yield.duration=10 millis
nifi.cluster.firewall.file=
nifi.cluster.flow.election.max.candidates=3
nifi.cluster.flow.election.max.wait.time=1 mins
nifi.cluster.is.node=true
nifi.cluster.load.balance.comms.timeout=30 sec
nifi.cluster.load.balance.connections.per.node=1
nifi.cluster.load.balance.host=
nifi.cluster.load.balance.max.thread.count=8
nifi.cluster.load.balance.port=6342
nifi.cluster.node.address=*****
nifi.cluster.node.connection.timeout=30 sec
nifi.cluster.node.event.history.size=25
nifi.cluster.node.max.concurrent.requests=100
nifi.cluster.node.protocol.max.threads=50
nifi.cluster.node.protocol.port=9088
nifi.cluster.node.read.timeout=30 sec
nifi.cluster.protocol.heartbeat.interval=5 sec
nifi.cluster.protocol.is.secure=true
nifi.components.status.repository.buffer.size=1440
nifi.components.status.repository.implementation=org.apache.nifi.controller.status.history.VolatileComponentStatusRepository
nifi.components.status.snapshot.frequency=1 min
nifi.content.claim.max.appendable.size=50 KB
nifi.content.repository.always.sync=false
nifi.content.repository.archive.enabled=true
nifi.content.repository.archive.max.retention.period=30 days
nifi.content.repository.archive.max.usage.percentage=70%
nifi.content.repository.directory.default=*****
nifi.content.repository.implementation=org.apache.nifi.controller.repository.FileSystemRepository
nifi.content.viewer.url=../nifi-content-viewer/
nifi.database.directory==*****
nifi.documentation.working.directory==*****
nifi.flow.analysis.background.task.schedule=5 mins
nifi.flow.configuration.archive.dir==*****
nifi.flow.configuration.archive.enabled=true
nifi.flow.configuration.archive.max.storage=500 MB
nifi.flow.configuration.archive.max.time=30 days
nifi.flow.configuration.file=****flow.json.gz
nifi.flow.configuration.json.file=****flow.json.gz
nifi.flowcontroller.autoResumeState=true
nifi.flowcontroller.graceful.shutdown.period=10 sec
nifi.flowfile.repository.always.sync=false
nifi.flowfile.repository.checkpoint.interval=2 mins
nifi.flowfile.repository.directory==****flowfile-repo
nifi.flowfile.repository.implementation=org.apache.nifi.controller.repository.WriteAheadFlowFileRepository
nifi.flowfile.repository.partitions=256
nifi.flowfile.repository.wal.implementation=org.apache.nifi.wali.SequentialAccessWriteAheadLog
nifi.flowservice.writedelay.interval=500 ms
nifi.h2.url.append=;LOCK_TIMEOUT=25000;WRITE_DELAY=0;AUTO_SERVER=FALSE
nifi.initial.admin.identity=
nifi.kerberos.krb5.file=/etc/krb5.conf
nifi.kerberos.service.keytab.location==****b
nifi.kerberos.service.principal=n=****K
nifi.login.identity.provider.configuration.file==****/login-identity-providers.xml
nifi.monitor.long.running.task.schedule=
nifi.monitor.long.running.task.threshold=
nifi.provenance.repository.always.sync=false
nifi.provenance.repository.buffer.size=100000
nifi.provenance.repository.compress.on.rollover=true
nifi.provenance.repository.concurrent.merge.threads=2
nifi.provenance.repository.directory.default==****
nifi.provenance.repository.encryption.key.provider.location=
nifi.provenance.repository.encryption.key.provider.password=
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
nifi.provenance.repository.index.shard.size=4 GB
nifi.provenance.repository.index.threads=2
nifi.provenance.repository.indexed.attributes=
nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename,
ProcessorID, Relationship
nifi.provenance.repository.max.attribute.length=65536
nifi.provenance.repository.max.storage.size=150 GB
nifi.provenance.repository.max.storage.time=30 days
nifi.provenance.repository.query.threads=2
nifi.provenance.repository.rollover.size=1 GB
nifi.provenance.repository.rollover.time=10 mins
nifi.python.extensions.source.directory.default==****
nifi.python.framework.source.directory==****
nifi.python.logs.directory==****
nifi.python.max.processes=100
nifi.python.max.processes.per.extension.type=10
nifi.python.working.directory==****
nifi.queue.backpressure.count=10000
nifi.queue.backpressure.size=1 GB
nifi.queue.swap.threshold=20000
nifi.remote.contents.cache.expiration=30 secs
nifi.remote.input.host==****
nifi.remote.input.http.enabled=true
nifi.remote.input.http.transaction.ttl=30 sec
nifi.remote.input.secure=true
nifi.repository.encryption.key.id=
nifi.repository.encryption.key.provider=
nifi.repository.encryption.key.provider.keystore.location=
nifi.repository.encryption.key.provider.keystore.password=
nifi.repository.encryption.protocol.version=
nifi.security.allow.anonymous.authentication=false
nifi.security.autoreload.enabled=false
nifi.security.autoreload.interval=10 secs
nifi.security.identity.mapping.pattern.dn=^CN=(.*?), .+$
nifi.security.identity.mapping.transform.dn=NONE
nifi.security.identity.mapping.value.dn=$1
nifi.security.ocsp.responder.certificate=
nifi.security.ocsp.responder.url=
nifi.security.user.authorizer=ranger-provider
nifi.security.user.knox.audiences=
nifi.security.user.knox.cookieName=hadoop-jwt
nifi.security.user.login.identity.provider=kerberos-provider
nifi.security.user.oidc.client.id=
nifi.security.user.oidc.client.secret=
nifi.security.user.oidc.connect.timeout=5 secs
nifi.security.user.oidc.discovery.url=
nifi.security.user.oidc.preferred.jwsalgorithm=
nifi.security.user.oidc.read.timeout=5 secs
nifi.sensitive.props.additional.keys=
nifi.sensitive.props.algorithm=NIFI_PBKDF2_AES_GCM_256
nifi.sensitive.props.key.protected=aes/gcm/256
nifi.sensitive.props.provider=BC
nifi.state.management.provider.cluster=zk-provider
nifi.state.management.provider.local=local-provider
nifi.swap.manager.implementation=org.apache.nifi.controller.FileSystemSwapManager
nifi.ui.autorefresh.interval=30 sec
nifi.ui.banner.text=
nifi.ui.knox.url=****
nifi.web.http.host=
nifi.web.http.network.interface.default=
nifi.web.http.port=
nifi.web.https.ciphersuites.exclude=
nifi.web.https.ciphersuites.include=
nifi.web.https.host=****
nifi.web.https.network.interface.default=
nifi.web.https.port=8443
nifi.web.jetty.threads=200
nifi.web.max.access.token.requests.per.second=25
nifi.web.max.header.size=16 KB
nifi.web.proxy.context.path=****
nifi.web.proxy.globals.path=
nifi.web.proxy.host=
nifi.web.request.log.format=%{client}a - %u %t "%r" %s %O "%{Referer}i"
"%{User-Agent}i"
nifi.zookeeper.auth.type=
nifi.zookeeper.client.secure=true
nifi.zookeeper.connect.string=****
nifi.zookeeper.connect.timeout=60 secs
nifi.zookeeper.jute.maxbuffer=1048575
nifi.zookeeper.kerberos.removeHostFromPrincipal=
nifi.zookeeper.kerberos.removeRealmFromPrincipal=
nifi.zookeeper.root.node=/BASE
{code}
It was a cluster with 3 node and a knox gateway is used to reach nifi
> GZipException occur during cluster replication, when the original request
> contains "accept-encoding" header with lowercase
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-12383
> URL: https://issues.apache.org/jira/browse/NIFI-12383
> Project: Apache NiFi
> Issue Type: Bug
> Reporter: Zoltán Kornél Török
> Assignee: Zoltán Kornél Török
> Priority: Major
> Fix For: 2.0.0-M1, 1.25.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> I had a three node cluster with knox.
> Time to time an error occured in the nifi logs on this cluster:
> {code}
> 2023-11-15 13:25:51,637 INFO
> org.apache.nifi.cluster.coordination.http.replication.ThreadPoolRequestReplicator:
> Received a status of 500 from xy:8443 for request PUT
> /nifi-api/process-groups/d2cedf64-018b-1000-0000-0000164a79fc when performing
> first stage of two-stage commit. The action will not occur. Node explanation:
> An unexpected error has occurred. Please check the logs for additional
> details.
> {code}
> Also sometimes I got "An unexpected error has occurred. Please check the logs
> for additional details." error on the UI too. After some investigation in I
> found the error in the logs:
> {code}
> 23-11-15 13:40:25,289 ERROR [NiFi Web Server-78]
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred:
> java.util.zip.ZipException: Not in GZIP format. Returning Internal Server
> Error response.
> java.util.zip.ZipException: Not in GZIP format
> at
> java.base/java.util.zip.GZIPInputStream.readHeader(GZIPInputStream.java:176)
> at
> java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:79)
> at
> java.base/java.util.zip.GZIPInputStream.<init>(GZIPInputStream.java:91)
> at
> org.glassfish.jersey.message.GZipEncoder.decode(GZipEncoder.java:49)
> at
> org.glassfish.jersey.spi.ContentEncoder.aroundReadFrom(ContentEncoder.java:100)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
> at
> org.glassfish.jersey.server.internal.MappableExceptionWrapperInterceptor.aroundReadFrom(MappableExceptionWrapperInterceptor.java:49)
> at
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor.proceed(ReaderInterceptorExecutor.java:132)
> at
> org.glassfish.jersey.message.internal.MessageBodyFactory.readFrom(MessageBodyFactory.java:1072)
> at
> org.glassfish.jersey.message.internal.InboundMessageContext.readEntity(InboundMessageContext.java:919)
> at
> org.glassfish.jersey.server.ContainerRequest.readEntity(ContainerRequest.java:290)
> at
> org.glassfish.jersey.server.internal.inject.EntityParamValueParamProvider$EntityValueSupplier.apply(EntityParamValueParamProvider.java:73)
> {code}
> After many hours of debugging, I found out, that sometimes when I use the
> cluster via knox, some unknown reason the incoming "Accept-Encoding" come
> with all leters lowercase (which is valid, becase HTTP header is case
> insensitive - https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers).
> However OkHttpReplicationClient assume that the header is always
> "Accept-Encoding"
> (https://github.com/apache/nifi/blob/main/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-framework-cluster/src/main/java/org/apache/nifi/cluster/coordination/http/replication/okhttp/OkHttpReplicationClient.java#L294
> and
> https://github.com/apache/nifi/blob/main/nifi-commons/nifi-site-to-site-client/src/main/java/org/apache/nifi/remote/protocol/http/HttpHeaders.java#L25).
> Because of that, during replication the client not use gzip compression but
> when other node get the requests, the jetty read the original
> "accept-encoding" header and try to uncompress the inputstream, which lead to
> the above error.
> We need to add a few line code to the client to read the header case
> insensitivity
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
