Re: Persisting/loading NIFI flows for Docker deployment

[Endre Kovacs]

Hi,
Why not use NiFi Registry to deploy / version the flows?
Best regards,
Endre

Sent from ProtonMail mobile

 Original Message 
On Apr 8, 2020, 4:27 PM, Kevin Telford wrote:

> Hi all – I have a two part question.
>
> I’d like to run NiFi inside a container in order to deploy to various
> environments. As far as I can tell, the flow.xml.gz file is the main
> “source” if you will, for a NiFi data flow.
>
> Q1) Is the flow.xml.gz file the “source” of a NiFi data flow, and if so, is
> it best practice to copy it to a new env in order to “deploy” a prebuilt
> flow? Or how best is this handled?
>
> Given that Q1 is true, my challenge then becomes somewhat Docker-specific…
>
> Situation:
>
> - In the Dockerfile we unzip the NiFi source (L62
> )
> and then create Docker volumes (L75
> 
> specifically for the conf dir). Once the container starts all the normal
> NiFi startup things happen, and /opt/nifi/nifi-current/conf/flow.xml.gz
> created.
>
> Complication:
>
> - In order to persist flow.xml.gz outside of the container, I would
> normally mount the /opt/nifi/nifi-current/conf directory, however in this
> case I cannot mount it on initialization because that will overwrite conf
> config files with whatever directory I bind it to (Docker container
> isolation ensures host -> container file precedence).
> - I could mount to a running container, but this is less ideal due to
> the various ways a container can be deployed.
> - I could copy manually from the running container, but this is less
> ideal as it’s on demand, and not always persisting latest.
>
> Resolution:
>
> - I believe instead, we would ideally create a few flow config specific
> env vars and use them to update our nifi.properties (via
> https://github.com/apache/nifi/blob/master/nifi-docker/dockerhub/sh/start.sh),
> i.e. NIFI_FLOW_CONFIG_FILE_LOCATION, NIFI_FLOW_CONFIG_ARCHIVE_ENABLED,
> NIFI_FLOW_CONFIG_ARCHIVE_DIR and so on for all nifi.flow.configuration
> props.
>
> Q2) Would the above proposal be ideal? (add a few env vars to start.sh) –
> if so, happy to add a PR for the code and doc change. Or have others solved
> this a different way?
>
> Best,
>
> Kevin

Re: hello I wanna ask somthing about apache minifi

[Endre Kovacs]

Hi,
To the AFAIK minifi does not yet support java 11.
Please install java 8 and try with that.

Best regards,
Endre

Sent from ProtonMail mobile

 Original Message 
On Mar 26, 2020, 2:21 AM, 박재영 wrote:

> I use nifi and minifi.
>
> about 2 weeks ago, i had to use minifi on RaspberryPi 4.
> So, I install minifi and minifi toolkit. but when i used 
> ${minifi-toolkit-0.5.0_HOME}/bin/config.sh with transform
> it didn't work.
>
> The error message is this.
> Error : Unable to initialize main class 
> org.apache.nifi.minifi.toolkit.configuration.Configmain
> Caused by: java.lang.NoClassDefFoundError : javax/xml/bind/JAXBException
>
> Have I should install Nifi at RaspberryPi4?
>
> Please let me know to how to solve this problem.
>
> ps. OS is a Raspbian GNU/Linux 10 and java version is 11.0.6
>
> [http://www.penta.co.kr/sign/bar.jpg]
>
> 박재영사원 / Big Data팀
> Park Jaeyoung / Big Data Team
>
> Mobile : 010.6322.2066 | email : qorghaos...@penta.co.kr | DIR : 02.769.9834
>
> [http://www.penta.co.kr/sign/footer_bd.jpg]

Re: Add users in NiFi

[Endre Kovacs]

Hi,

It is hard to know what the problem is since no configuration was provided, so 
my best guess is that you have not configured NiFi to be secure, and as such, 
only anonymous access is supported in an unsecured NiFi. To enable login (user 
access), please configure NiFi with one of the available authentication 
mechanisms:

https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#user_authentication

Best regards,
Endre


Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Monday, March 9, 2020 4:28 PM, Carlos Eduardo Ochoa Ceballos 
 wrote:

> Hi there!!!
>
> I'm trying create users to my NiFi but it's impossible to do it. I got my 
> NiFi on Ubuntu and documentation says about UI and I have no that option.
>
> How can I solve this?
>
> Hope your help.
>
> Regards,

Re: 1.11.3 trust store error

[Endre Kovacs]

Hi Nathan,

There is already a ticket about this: 
https://issues.apache.org/jira/browse/NIFI-7219

Best regards,
Endre

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Wednesday, March 4, 2020 5:25 PM, Nathan Gough  wrote:

> I've opened https://issues.apache.org/jira/browse/NIFI-7223 to track and
> I'm working on a fix for this.
>
> Nathan
>
> On Tue, Mar 3, 2020 at 6:17 PM Nathan Gough thena...@gmail.com wrote:
>
> > Hi Joe,
> > Just to confirm here - was the nifi.security.keyPasswd not defined at all
> > in your nifi.properties? Did you have to add the property and give it the
> > correct value? Or was it in the nifi.properties file but blank? Or were the
> > keyPasswd and keystorePasswd different values?
> > Thanks,
> > Nathan
> > On Tue, Mar 3, 2020 at 3:38 PM Joe Gresock jgres...@gmail.com wrote:
> >
> > > Yep, setting the nifi.security.keyPasswd to the same as
> > > nifi.security.keystorePasswd fixed it. Thanks for the insight, Endre!
> > > On Tue, Mar 3, 2020 at 2:01 PM Joe Witt joe.w...@gmail.com wrote:
> > >
> > > > relevant change I believe is here:
> > >
> > > https://github.com/apache/nifi/commit/46d3b6b0dc28f04da124be7685f82bec52e88775
> > >
> > > > and
> > > > is from https://issues.apache.org/jira/browse/NIFI-6927
> > > > It looks to me like this was fixing an improper naming/usage issue
> > > > that
> > > > has been present but if so we probably should have addressed not in this
> > > > bug fix line. Will defer to Troy/Andy for more context and next steps
> > > > On Tue, Mar 3, 2020 at 5:53 AM Joe Witt joe.w...@gmail.com wrote:
> > > >
> > > > > If accurateWe need to look into whether this was a mistake and
> > > > > fix it
> > > >
> > > > > if so. And we need to reflect this in the migration guide
> > > > > On Tue, Mar 3, 2020 at 4:40 AM Ryan Ward ryan.wa...@gmail.com
> > > > > wrote:
> > > >
> > > > > > Endre - thanks that was it
> > > > > > On Tue, Mar 3, 2020 at 6:50 AM Endre Kovacs
> > > > > > andrewsmit...@protonmail.com.invalid wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > One additional thing:
> > > > > > > we encountered something strange as well:
> > > > > > > on 1.11.2 clustered, kerberized: request replication worked well.
> > > > > > > on 1.11.3 clustered, kerberized: request replication did not work,
> > > > > > > unless
> > > > > > > you specify, and set
> > > > > > > nifi.security.keyPasswd
> > > > > > > to the very same password as the
> > > > > > > nifi.security.keystorePasswd
> > > > > > > For us this resolved the issue.
> > > > > > > Best regards,
> > > > > > > Endre
> > > > > > > Sent with ProtonMail Secure Email.
> > > > > > > ‐‐‐ Original Message ‐‐‐
> > > > > > > On Tuesday, March 3, 2020 12:40 PM, Ryan Ward <
> > > > > > > ryan.wa...@gmail.com>
> > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Joe - Did you resolve your issue? If so I am wondering what
> > > > > > > > the
> > > > > > > > fix
> > > >
> > > > > > > was as I'm seeing the same error on my cluster.
> > > > > > >
> > > > > > > > On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs <
> > > > > > > > andrewsmit...@protonmail.com.invalid> wrote:
> > > > > > > >
> > > > > > > > > Hi Joe,
> > > > > > > > >
> > > > > > > > > 1.  Have you tried connecting/debugging with openssl? From one
> > > > > > > > > pod
> > > > > > > > > to
> > > > > > > > >
> > > >
> > > > > > > the other:
> > > > > > >
> > > > > > > > > (openssl s_client -debug -CAfile
> > > > > > > > >
> > > > > > >
> > > > > > > ca-bundle-signing-node-certificates.crt -cert my-client-cert.crt
> > > > > > > -c

Re: 1.11.3 trust store error

[Endre Kovacs]

Hi,

One additional thing:

we encountered something strange as well:

on 1.11.2 clustered, kerberized: request replication worked well.

on 1.11.3 clustered, kerberized: request replication did not work, unless you 
specify, and set
nifi.security.keyPasswd

to the very same password as the

nifi.security.keystorePasswd

For us this resolved the issue.

Best regards,
Endre

Sent with [ProtonMail](https://protonmail.com) Secure Email.

‐‐‐ Original Message ‐‐‐
On Tuesday, March 3, 2020 12:40 PM, Ryan Ward  wrote:

> Hi Joe - Did you resolve your issue? If so I am wondering what the fix was as 
> I'm seeing the same error on my cluster.
>
> On Thu, Feb 27, 2020 at 3:13 AM Endre Kovacs 
>  wrote:
>
>> Hi Joe,
>>
>> 1.  Have you tried connecting/debugging with openssl? From one pod to the 
>> other:
>> (openssl s_client -debug -CAfile ca-bundle-signing-node-certificates.crt 
>> -cert my-client-cert.crt -connect 
>> nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)
>>
>> 2. certs can also be verified by:
>>  openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt
>>
>> 3.  Can you check if no intermediary CAs are missing from the nodes 
>> truststore?
>>
>> 4.  This exception is coming from inter-node communication (replication of 
>> request from one node to the other). This means that it is unrelated to 
>> external user's authentication by client certificate. The question is: is 
>> your inter node communication secured by the trusted root CA (that you are 
>> sure that the CA cert is present in the trust store) or is it secured by 
>> selfsigned CA (which's CA may be lacking from your truststore)?
>>
>> 5.  `nifi.security.needClientAuth` is not part of NiFi properties any more. 
>> If SSL is turned on, and no `nifi.security.user.login.identity.provider` is 
>> set, then client cert based auth is the default. But supplying this property 
>> have no detrimental effect anyhow.
>>
>> Best regards,
>> Endre
>>
>> Sent with ProtonMail Secure Email.
>>
>> ‐‐‐ Original Message ‐‐‐
>> On Wednesday, February 26, 2020 6:22 PM, Joe Gresock jgres...@gmail.com 
>> wrote:
>>
>>> Were there any changes with how the trust store is used in 1.11.3? I had a
>>> 1.11.0 deployment working with the following settings, but when I deployed
>>> 1.11.3, the cluster can't seem to replicate requests to itself:
>>> nifi.remote.input.host=
>>> nifi.remote.input.secure=true
>>> nifi.remote.input.socket.port=32440
>>> nifi.remote.input.http.enabled=true
>>> nifi.cluster.protocol.is.secure=true
>>> nifi.cluster.is.node=true
>>> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
>>> nifi.cluster.node.protocol.port=6007
>>> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
>>> nifi.web.https.port=8443
>>> nifi.security.keystore=./conf/keystore.jks
>>> nifi.security.keystoreType=jks
>>> nifi.security.keystorePasswd=
>>> nifi.security.keyPasswd=
>>> nifi.security.truststore=./conf/truststore.jks
>>> nifi.security.truststoreType=jks
>>> nifi.security.truststorePasswd=
>>> nifi.security.needClientAuth=true
>>> A trusted client cert that worked against the old cluster is getting the
>>> same trust error (PKIX path building failed). I've verified that the
>>> client cert was issued by an issuer that is definitely in the
>>> ./conf/truststore.jks as a trustedCertEntry.
>>> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
>>> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
>>> javax.net.ssl.SSLHandshakeException:
>>> sun.security.validator.ValidatorException: PKIX path building failed:
>>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>>> valid certification path to r
>>> equested target
>>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>>> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
>>> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
>>> at
>>> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
>>> at
>>> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
>>> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
>>> at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
>>> at
>>> sun.security.ssl.SSLSocketImpl.readRecord(

Re: 1.11.3 trust store error

[Endre Kovacs]

Hi Joe,

1.  Have you tried connecting/debugging with openssl? From one pod to the other:
(openssl s_client -debug -CAfile ca-bundle-signing-node-certificates.crt 
-cert my-client-cert.crt -connect 
nifi-3.nifi-headless.lizardspock.svc.cluster.local:6007)


2. certs can also be verified by:
 openssl verify -verbose -CAfile ca-bundle.crt my-client-cert.crt

3.  Can you check if no intermediary CAs are missing from the nodes truststore?

4.  This exception is coming from inter-node communication (replication of 
request from one node to the other). This means that it is unrelated to 
external user's authentication by client certificate. The question is: is your 
inter node communication secured by the trusted root CA (that you are sure that 
the CA cert is present in the trust store) or is it secured by selfsigned CA 
(which's CA may be lacking from your truststore)?

5.  `nifi.security.needClientAuth` is not part of NiFi properties any more. If 
SSL is turned on, and no `nifi.security.user.login.identity.provider` is set, 
then client cert based auth is the default. But supplying this property have no 
detrimental effect anyhow.

Best regards,
Endre


Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Wednesday, February 26, 2020 6:22 PM, Joe Gresock jgres...@gmail.com 
wrote:


> Were there any changes with how the trust store is used in 1.11.3? I had a
> 1.11.0 deployment working with the following settings, but when I deployed
> 1.11.3, the cluster can't seem to replicate requests to itself:
> nifi.remote.input.host=
> nifi.remote.input.secure=true
> nifi.remote.input.socket.port=32440
> nifi.remote.input.http.enabled=true
> nifi.cluster.protocol.is.secure=true
> nifi.cluster.is.node=true
> nifi.cluster.node.address=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> nifi.cluster.node.protocol.port=6007
> nifi.web.https.host=nifi-3.nifi-headless.lizardspock.svc.cluster.local
> nifi.web.https.port=8443
> nifi.security.keystore=./conf/keystore.jks
> nifi.security.keystoreType=jks
> nifi.security.keystorePasswd=
> nifi.security.keyPasswd=
> nifi.security.truststore=./conf/truststore.jks
> nifi.security.truststoreType=jks
> nifi.security.truststorePasswd=
> nifi.security.needClientAuth=true
> A trusted client cert that worked against the old cluster is getting the
> same trust error (PKIX path building failed). I've verified that the
> client cert was issued by an issuer that is definitely in the
> ./conf/truststore.jks as a trustedCertEntry.
> 2020-02-26 17:11:09,573 WARN [Replicate Request Thread-7]
> o.a.n.c.c.h.r.ThreadPoolRequestReplicator
> javax.net.ssl.SSLHandshakeException:
> sun.security.validator.ValidatorException: PKIX path building failed:
> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
> valid certification path to r
> equested target
> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
> at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
> at
> sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
> at
> sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
> at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
> at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
> at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
> at
> okhttp3.internal.connection.RealConnection.connectTls(RealConnection.java:302)
> at
> okhttp3.internal.connection.RealConnection.establishProtocol(RealConnection.java:270)
> at
> okhttp3.internal.connection.RealConnection.connect(RealConnection.java:162)
> at
> okhttp3.internal.connection.StreamAllocation.findConnection(StreamAllocation.java:257)
> at
> okhttp3.internal.connection.StreamAllocation.findHealthyConnection(StreamAllocation.java:135)
> at
> okhttp3.internal.connection.StreamAllocation.newStream(StreamAllocation.java:114)
> at
> okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:42)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:121)
> at
> okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
> at
> okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:147)
> at
> 

Re: New developer - workstation laptop processing power

[Endre Kovacs]

Hi Eric,

My machine is a:

2.6 GHz Intel Core i7, 16 GB Mac Book Pro.

To compile the whole project it takes about 10-14 minutes.
(the more powerful your CPU is and the more cores you have, will drive this 
down when compiling with -T 2.0C flag.)

To run in standalone mode (non-clustered) NiFi can run basically on any machine.

The speed of processing data with it will be faster if you have faster CPU, and 
fast / multiple SSDs.

If you plan to run NiFi clustered on your machine (either through 
docker-compose or in multiple virtual machines), then you will need more ram 
and cores that you can assign to the docker engine / the virtual machines.

My machine fits the job well for standalone mode and compile. (However i am not 
running NiFi in clustered mode on this machine).

So it really depends on what you intend to do.
Best regards,
Endre

Sent with [ProtonMail](https://protonmail.com) Secure Email.

‐‐‐ Original Message ‐‐‐
On Friday, February 21, 2020 10:32 AM, Eric Butts buttse...@hotmail.com wrote:

> Hi
> I’m a new “learning” Nifi developer needing recommendations on how much horse 
> power in a laptop workstation do I need ?
> All of my developer learning will be done on workstation. No access to 
> server-side storage systems
> Thanks in advance for the recommendations

Re: Hi please add me to this group

[Endre Kovacs]

Hi Chetan,

To sign up, please see:

https://nifi.apache.org/mailing_lists.html

Endre



Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Friday, February 21, 2020 9:17 AM, Kumtakar, Chetan 
 wrote:

> Hi all,
>
> I am a Nifi dev and would like to utilize this forum to post my findings and 
> get benefit of the archived /new features which will help me.
>
> Regards
> Chetan

Re: Running Nifi on OpenShift

[Endre Kovacs]

Hi,

If to make NiFi work on K8S is a beast, then to make it work on Openshift, is a 
category-5 Kaiju [1][2].

This is definitely not a few days task for a single person.

Why not run NiFi just in docker (docker-compose)? Or on bare metal?

Best regards,
Endre


[1] https://en.wikipedia.org/wiki/Kaiju
[2] https://en.wikipedia.org/wiki/Pacific_Rim_(film)

Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Wednesday, February 12, 2020 8:14 PM, Fill, Natalia  
wrote:

> Public
>
> Hi,
> I am trying to run Nifi pod on OpenShift for several days now and 
> unfortunately unsuccessfully.
>
> The error that I am getting persistently is
> replacing target file /opt/nifi/nifi-current/conf/nifi.properties
> sed: couldn't open temporary file /opt/nifi/nifi-current/conf/sedXGg2lo: 
> Permission denied
>
> I have tried several things to resolve the issue:
> My images are downloaded from https://hub.docker.com/r/apache/nifi
>
> 1.First I run 1.10.0 image which resulted in error above
>
>
>
> 2. Upgraded to 1.11.1 image, the error still persist
>
> 3. Tried wrapping the above images in my own image with following 
> modifications to docker file (used various paths to chmod opt/ opt/nifi), 
> still the same error
>
> FROM xxxRegistry/apache-nifi:1.11.1
> USER root
> RUN chmod -R 777 /opt
> USER 1000
>
> 4. Mounted volume opt/nifi, but this resulted in nifi properties file not 
> being found, so removed volume as it overwrites Nifi paths
>
> 5. Involved OpenShift administrators to create privileged account for nifi 
> and altered my yml to use that account (SUPPLEMENTAL_GROUP is what all our 
> pods run under and sn_nif was created specially to resolve this case)
>
> securityContext:
> supplementalGroups:
>
> -   ${SUPPLEMENTAL_GROUP}
> serviceAccount: sn-nif
> serviceAccountName: sn-nif
>
>
> 6.Removed securityContext to ensure serviceAccount is used
>
>
>
> Can someone please suggest how to resolve this issue. Otherwise I will have 
> to give up on Nifi as I don't have any more time on this project to spend on 
> Nifi config.
>
> Thank you
>
> Natalia
>
> Natalia Fill
> Analyst Software Developer
> Legal and General Investment Management
> One Coleman Street, London, EC2R 5AA
> 020 3124 3430
> www.lgim.com
> This e-mail (and any attachments) may contain privileged and/or confidential 
> information. If you are not the intended recipient please do not disclose, 
> copy, distribute, disseminate or take any action in reliance on it. If you 
> have received this message in error please reply and tell us and then delete 
> it. Should you wish to communicate with us by e-mail we cannot guarantee the 
> security of any data outside our own computer systems.
>
> Any information contained in this message may be subject to applicable terms 
> and conditions and must not be construed as giving investment advice within 
> or outside the United Kingdom or Republic of Ireland.
>
> Telephone Conversations may be recorded for your protection and to ensure 
> quality of service
>
> Legal & General Investment Management Limited (no 2091894), LGIM Real Assets 
> (Operator) Limited (no 05522016), LGIM (International) Limited (no 7716001) 
> Legal & General Unit Trust Managers (no 1009418), GO ETF Solutions LLP 
> (OC329482) and LGIM Corporate Director Limited (no 7105051) are authorised 
> and regulated by the Financial Conduct Authority. All are registered in 
> England & Wales with a registered office at One Coleman Street, London, EC2R 
> 5AA
>
> Legal & General Assurance (Pensions Management) Limited (no 1006112) is 
> authorised by the Prudential Regulation Authority and regulated by the 
> Financial Conduct Authority and the Prudential Regulation Authority. It is 
> registered in England & Wales with a registered office at One Coleman Street, 
> London, EC2R 5AA.
>
> Legal & General Property Limited (no 2091897) is authorised and regulated by 
> the Financial Conduct Authority for insurance mediation activities. It is 
> registered in England & Wales with a registered office at One Coleman Street, 
> London, EC2R 5AA.
>
> LGIM Managers (Europe) Limited is authorised and regulated by the Central 
> Bank of Ireland (C173733). It is registered in the Republic of Ireland (no 
> 609677) with a registered office at 33/34 Sir John Rogerson's Quay, Dublin 2, 
> D02 XK09.
>
> Legal & General Group PLC, Registered Office One Coleman Street, London, EC2R 
> 5AA.
>
> Registered in England no: 1417162
>
>  This email has come from the internet and has been scanned for all 
> viruses and potentially offensive content by Messagelabs on behalf of Legal & 
> General 

Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream

[Endre Kovacs]

The more i think about this, the more i believe that the PV had to be 
provisioned as some kind of network attached storage, and the exception had to 
be happening because of intermittent network issue for this NAS.




Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Monday, February 10, 2020 8:41 PM, Endre Kovacs 
 wrote:

> Hi,
>
> What cloud are you running this on?
> Kubernetes version?
>
> Can you share the storageClass you use for creating the repo PVCs?
>
> what is the topology of the cluster? Muli zone?
>
> Best regards,
> Endre
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐ Original Message ‐‐‐
> On Monday, February 10, 2020 7:41 PM, Shawn Weeks swe...@weeksconsulting.us 
> wrote:
>
> > The more I look at the error the more it looks like the container couldn't 
> > access the persistence volume for some period of time. Can you share your 
> > yaml configuration minus sensitive stuff and what environment your 
> > deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc.
> > Thanks
> > Shawn
> > On 2/10/20, 12:27 PM, "Shawn Weeks" swe...@weeksconsulting.us wrote:
> > I've been testing out NiFi on K8s quite a bit lately and might be able to 
> > help. What are you using for your persistent volume, this kinda sounds like 
> > NiFi can't always access data in the content_repository folder?
> > Thanks
> > Shawn
> > On 2/10/20, 11:53 AM, "Joe Witt" joe.w...@gmail.com wrote:
> > ...sent a little too quick. Also seems like if it is related to your k8s
> > process that the issue would occur on older nifi releases too. So that
> > might be an interesting test for you.
> > On Mon, Feb 10, 2020 at 9:52 AM Joe Witt joe.w...@gmail.com wrote:
> >
> > > Joe
> >
> > >
> >
> > > If you can replicate this in a non k8s environment that would 
> > likely make
> >
> > > resolution easier.  Far too many variables/questions at this 
> > stage.
> >
> > >
> >
> > > Thanks
> >
> > >
> >
> > > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock  
> > wrote:
> >
> > >
> >
> > >> I don't know if there is something wrong with my Kubernetes-ized
> >
> > >> deployment
> >
> > >> of NiFi, but I'm seeing the following error crop up a lot in my 
> > NiFi
> >
> > >> 1.11.1
> >
> > >> deployment.  I can see the flow file attributes in "List Queue" 
> > for flow
> >
> > >> files affected by this problem, but the content claim is missing.
> >
> > >>
> >
> > >> It's not easily reproducible, but I've been running data 
> > constantly
> >
> > >> through
> >
> > >> the cluster for a few days, and every 2-4 hours or so, this 
> > problem
> >
> > >> appears
> >
> > >> in one of my processors (it's not always the same processors).  
> > The error
> >
> > >> only goes away when I drain the relevant queue, but obviously 
> > that's not
> >
> > >> an
> >
> > >> option in production.
> >
> > >>
> >
> > >> Any thoughts on what might be causing intermittent truncations 
> > in the
> >
> > >> content_repository?
> >
> > >>
> >
> > >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6]
> >
> > >> o.a.nifi.processors.standard.ReplaceText
> >
> > >> ReplaceText[id=1abd1d49-0170-1000--da59ecb2]
> >
> > >> ReplaceText[id=1abd1d49-0170-1000--da59ecb2] fa
> >
> > >> iled to process session due to
> >
> > >> org.apache.nifi.processor.exception.FlowFileAccessException: 
> > Could not
> >
> > >> read
> >
> > >> from
> >
> > >>
> >
> > >> 
> > StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> >
> > >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> >
> > >> container=default, section=465], offset=0,
> >
> > >>
> >
> > >> 
> > 

Re: NiFi 1.11.1: EOFException in FlowFileAccessInputStream

[Endre Kovacs]

Hi,

What cloud are you running this on?
Kubernetes version?

Can you share the storageClass you use for creating the repo PVCs?

what is the topology of the cluster? Muli zone?

Best regards,
Endre


Sent with ProtonMail Secure Email.

‐‐‐ Original Message ‐‐‐
On Monday, February 10, 2020 7:41 PM, Shawn Weeks  
wrote:

> The more I look at the error the more it looks like the container couldn't 
> access the persistence volume for some period of time. Can you share your 
> yaml configuration minus sensitive stuff and what environment your deploying 
> K8s on, aka Azure, ECS, Rancher, Openshift, etc.
>
> Thanks
> Shawn
>
> On 2/10/20, 12:27 PM, "Shawn Weeks" swe...@weeksconsulting.us wrote:
>
> I've been testing out NiFi on K8s quite a bit lately and might be able to 
> help. What are you using for your persistent volume, this kinda sounds like 
> NiFi can't always access data in the content_repository folder?
>
> Thanks
> Shawn
>
> On 2/10/20, 11:53 AM, "Joe Witt" joe.w...@gmail.com wrote:
>
> ...sent a little too quick. Also seems like if it is related to your k8s
> process that the issue would occur on older nifi releases too. So that
> might be an interesting test for you.
>
> On Mon, Feb 10, 2020 at 9:52 AM Joe Witt joe.w...@gmail.com wrote:
>
> > Joe
>
> >
>
> > If you can replicate this in a non k8s environment that would 
> likely make
>
> > resolution easier.  Far too many variables/questions at this stage.
>
> >
>
> > Thanks
>
> >
>
> > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock  
> wrote:
>
> >
>
> >> I don't know if there is something wrong with my Kubernetes-ized
>
> >> deployment
>
> >> of NiFi, but I'm seeing the following error crop up a lot in my 
> NiFi
>
> >> 1.11.1
>
> >> deployment.  I can see the flow file attributes in "List Queue" 
> for flow
>
> >> files affected by this problem, but the content claim is missing.
>
> >>
>
> >> It's not easily reproducible, but I've been running data constantly
>
> >> through
>
> >> the cluster for a few days, and every 2-4 hours or so, this problem
>
> >> appears
>
> >> in one of my processors (it's not always the same processors).  
> The error
>
> >> only goes away when I drain the relevant queue, but obviously 
> that's not
>
> >> an
>
> >> option in production.
>
> >>
>
> >> Any thoughts on what might be causing intermittent truncations in 
> the
>
> >> content_repository?
>
> >>
>
> >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process Thread-6]
>
> >> o.a.nifi.processors.standard.ReplaceText
>
> >> ReplaceText[id=1abd1d49-0170-1000--da59ecb2]
>
> >> ReplaceText[id=1abd1d49-0170-1000--da59ecb2] fa
>
> >> iled to process session due to
>
> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could 
> not
>
> >> read
>
> >> from
>
> >>
>
> >> 
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>
> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>
> >> container=default, section=465], offset=0,
>
> >>
>
> >> 
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
>
> >> Processor Administratively Yielded for 1 sec:
>
> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could 
> not
>
> >> read
>
> >> from
>
> >>
>
> >> 
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>
> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>
> >> container=default, section=465], offset=0,
>
> >>
>
> >> 
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>
> >> org.apache.nifi.processor.exception.FlowFileAccessException: Could 
> not
>
> >> read
>
> >> from
>
> >>
>
> >> 
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
>
> >> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
>
> >> container=default, section=465], offset=0,
>
> >>
>
> >> 
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
>
> >> at
>
> >> org.apache.nifi.controller.repository.io
>
> >> 
> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
>
> >> at
>
> >> 
> org.apache.nifi.stream.io.StreamUtils.fillBuffer(StreamUtils.java:89)
>
> >>