Sure, sorry for the delay.
It's an AWS EBS persistent volume. Here are the PV and storage class
objects. It's single-zone (us-east-1d):
apiVersion: v1
kind: PersistentVolume
metadata:
annotations:
kubernetes.io/createdby: aws-ebs-dynamic-provisioner
pv.kubernetes.io/bound-by-controller: "yes"
pv.kubernetes.io/provisioned-by: kubernetes.io/aws-ebs
labels:
failure-domain.beta.kubernetes.io/region: us-east-1
failure-domain.beta.kubernetes.io/zone: us-east-1d
name: pvc-abb1e679-523d-4ef7-a42e-cb42d8b8da0d
spec:
accessModes:
- ReadWriteOnce
awsElasticBlockStore:
fsType: ext4
volumeID: aws://us-east-1d/vol-<instanceId> # Redacting some details
capacity:
storage: 150Gi
claimRef:
apiVersion: v1
kind: PersistentVolumeClaim
name: content-repository-nifi-4
namespace: default
resourceVersion: "7190127"
uid: abb1e679-523d-4ef7-a42e-cb42d8b8da0d
mountOptions:
- debug
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/region
operator: In
values:
- us-east-1
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- us-east-1d
persistentVolumeReclaimPolicy: Retain
storageClassName: gp2
volumeMode: Filesystem
---
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"allowVolumeExpansion":true,"apiVersion":"storage.k8s.io/v1
","kind":"StorageClass","metadata":{"annotations":{"
storageclass.kubernetes.io/is-default-class
":"true"},"labels":{"addonmanager.kubernetes.io.mode":"EnsureExists"},"name":"gp2"},"mountOptions":["debug"],"parameters":{"type":"gp2"},"provisioner":"
kubernetes.io/aws-ebs
","reclaimPolicy":"Retain","volumeBindingMode":"WaitForFirstConsumer"}
storageclass.kubernetes.io/is-default-class: "true"
creationTimestamp: "2020-01-03T12:30:49Z"
labels:
addonmanager.kubernetes.io.mode: EnsureExists
name: gp2
resourceVersion: "126035"
selfLink: /apis/storage.k8s.io/v1/storageclasses/gp2
uid: b91b9233-cd1b-4850-b0e1-bb984c4a31c5
allowVolumeExpansion: true
mountOptions:
- debug
parameters:
type: gp2
provisioner: kubernetes.io/aws-ebs
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer
On Tue, Feb 11, 2020 at 9:20 AM Endre Kovacs
<[email protected]> wrote:
> The more i think about this, the more i believe that the PV had to be
> provisioned as some kind of network attached storage, and the exception had
> to be happening because of intermittent network issue for this NAS.
>
>
>
>
> Sent with ProtonMail Secure Email.
>
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> On Monday, February 10, 2020 8:41 PM, Endre Kovacs <
> [email protected]> wrote:
>
> > Hi,
> >
> > What cloud are you running this on?
> > Kubernetes version?
> >
> > Can you share the storageClass you use for creating the repo PVCs?
> >
> > what is the topology of the cluster? Muli zone?
> >
> > Best regards,
> > Endre
> >
> > Sent with ProtonMail Secure Email.
> >
> > ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
> > On Monday, February 10, 2020 7:41 PM, Shawn Weeks
> [email protected] wrote:
> >
> > > The more I look at the error the more it looks like the container
> couldn't access the persistence volume for some period of time. Can you
> share your yaml configuration minus sensitive stuff and what environment
> your deploying K8s on, aka Azure, ECS, Rancher, Openshift, etc.
> > > Thanks
> > > Shawn
> > > On 2/10/20, 12:27 PM, "Shawn Weeks" [email protected] wrote:
> > > I've been testing out NiFi on K8s quite a bit lately and might be able
> to help. What are you using for your persistent volume, this kinda sounds
> like NiFi can't always access data in the content_repository folder?
> > > Thanks
> > > Shawn
> > > On 2/10/20, 11:53 AM, "Joe Witt" [email protected] wrote:
> > > ...sent a little too quick. Also seems like if it is related to your
> k8s
> > > process that the issue would occur on older nifi releases too. So that
> > > might be an interesting test for you.
> > > On Mon, Feb 10, 2020 at 9:52 AM Joe Witt [email protected] wrote:
> > >
> > > > Joe
> > >
> > > >
> > >
> > > > If you can replicate this in a non k8s environment that
> would likely make
> > >
> > > > resolution easier. Far too many variables/questions at this
> stage.
> > >
> > > >
> > >
> > > > Thanks
> > >
> > > >
> > >
> > > > On Mon, Feb 10, 2020 at 8:15 AM Joe Gresock <
> [email protected]> wrote:
> > >
> > > >
> > >
> > > >> I don't know if there is something wrong with my
> Kubernetes-ized
> > >
> > > >> deployment
> > >
> > > >> of NiFi, but I'm seeing the following error crop up a lot
> in my NiFi
> > >
> > > >> 1.11.1
> > >
> > > >> deployment. I can see the flow file attributes in "List
> Queue" for flow
> > >
> > > >> files affected by this problem, but the content claim is
> missing.
> > >
> > > >>
> > >
> > > >> It's not easily reproducible, but I've been running data
> constantly
> > >
> > > >> through
> > >
> > > >> the cluster for a few days, and every 2-4 hours or so, this
> problem
> > >
> > > >> appears
> > >
> > > >> in one of my processors (it's not always the same
> processors). The error
> > >
> > > >> only goes away when I drain the relevant queue, but
> obviously that's not
> > >
> > > >> an
> > >
> > > >> option in production.
> > >
> > > >>
> > >
> > > >> Any thoughts on what might be causing intermittent
> truncations in the
> > >
> > > >> content_repository?
> > >
> > > >>
> > >
> > > >> 2020-02-10 10:38:57,205 ERROR [Timer-Driven Process
> Thread-6]
> > >
> > > >> o.a.nifi.processors.standard.ReplaceText
> > >
> > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2]
> > >
> > > >> ReplaceText[id=1abd1d49-0170-1000-ffff-ffffda59ecb2] fa
> > >
> > > >> iled to process session due to
> > >
> > > >>
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> > >
> > > >> read
> > >
> > > >> from
> > >
> > > >>
> > >
> > > >>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> > >
> > > >>
> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> > >
> > > >> container=default, section=465], offset=0,
> > >
> > > >>
> > >
> > > >>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289];
> > >
> > > >> Processor Administratively Yielded for 1 sec:
> > >
> > > >>
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> > >
> > > >> read
> > >
> > > >> from
> > >
> > > >>
> > >
> > > >>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> > >
> > > >>
> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> > >
> > > >> container=default, section=465], offset=0,
> > >
> > > >>
> > >
> > > >>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
> > >
> > > >>
> org.apache.nifi.processor.exception.FlowFileAccessException: Could not
> > >
> > > >> read
> > >
> > > >> from
> > >
> > > >>
> > >
> > > >>
> StandardFlowFileRecord[uuid=b0ca5649-ca8f-4a99-88df-70e1143c5f1e,claim=StandardContentClaim
> > >
> > > >>
> [resourceClaim=StandardResourceClaim[id=1581318452649-212433,
> > >
> > > >> container=default, section=465], offset=0,
> > >
> > > >>
> > >
> > > >>
> length=6401249],offset=2199110,name=90e47e27f806d46b2e190b462db929441993d166-ED,size=318289]
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:93)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .TaskTerminationInputStream.read(TaskTerminationInputStream.java:68)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.stream.io
> .StreamUtils.fillBuffer(StreamUtils.java:89)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.processors.standard.ReplaceText$RegexReplace$2.process(ReplaceText.java:532)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2324)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2292)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.repository.BatchingSessionFactory$HighThroughputSession.read(BatchingSessionFactory.java:192)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.processors.standard.ReplaceText$RegexReplace.replace(ReplaceText.java:529)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.processors.standard.ReplaceText.onTrigger(ReplaceText.java:316)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1176)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.tasks.ConnectableTask.invoke(ConnectableTask.java:213)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:117)
> > >
> > > >> at
> org.apache.nifi.engine.FlowEngine$2.run(FlowEngine.java:110)
> > >
> > > >> at
> > >
> > > >>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > >
> > > >> at
> > >
> > > >>
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >
> > > >> at
> > >
> > > >>
> > >
> > > >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >
> > > >> at java.lang.Thread.run(Thread.java:748)
> > >
> > > >> Caused by: java.io.EOFException: null
> > >
> > > >> at org.apache.nifi.stream.io
> > >
> > > >> .StreamUtils.skip(StreamUtils.java:270)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .ContentClaimInputStream.formDelegate(ContentClaimInputStream.java:155)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .ContentClaimInputStream.getDelegate(ContentClaimInputStream.java:51)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .ContentClaimInputStream.read(ContentClaimInputStream.java:89)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >> .LimitedInputStream.read(LimitedInputStream.java:86)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .DisableOnCloseInputStream.read(DisableOnCloseInputStream.java:49)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.stream.io
> > >
> > > >>
> .ByteCountingInputStream.read(ByteCountingInputStream.java:52)
> > >
> > > >> at
> java.io.FilterInputStream.read(FilterInputStream.java:133)
> > >
> > > >> at
> > >
> > > >> org.apache.nifi.controller.repository.io
> > >
> > > >>
> .FlowFileAccessInputStream.read(FlowFileAccessInputStream.java:82)
> > >
> > > >> ... 20 common frames omitted
> > >
> > > >>
> > >
> > > >>
> > >
> > > >> Joe Gresock
> > >
> > > >>
> > >
> > > >
> > >
>
>
>
--
Be on your guard; stand firm in the faith; be courageous; be strong. Do
everything in love. -*1 Corinthians 16:13-14*
