Re: VolatileContentRepository removal
Hey Matthieu,
If using a RAM disk, I would recommend trying 1.16 and also setting
“nifi.content.claim.max.appendable.size” in nifi.properties to “1 byte”. This
will help to ensure that you’re eliminating data from the repository as quickly
as possible. Additionally, I would recommend that you configure
“nifi.flowfile.repository.checkpoint.interval” to a very small value, such as
“5 secs” or “10 secs” as this will also help in eliminating data from the
Content Repo as quickly as possible. And you’ll want to ensure that you have
“nifi.content.repository.archive.enabled” set to “false”.
Thanks
-Mark
On Mar 31, 2022, at 8:46 AM, Matthieu Ré
mailto:re.matth...@gmail.com>> wrote:
Hi Mike, David,
Thanks for your answers and your clarity !
To sum up our use case, our team has set up two cloud-based NiFi ("stateful")
clusters. The second one deals with a big amount of record-based data, to drag
data from sources (Kafka, files and databases) to other systems (OpenSearch and
internal tools) performing a lot of costly transformations (in the last 5
minutes for instance, my prod instance indicates 300G of read, 170G of write,
within 400 processors). This cluster deals with data we can recover from a
process, so in this use case we are data-loss tolerant. The 12 NiFi nodes of
the cluster (as well as a 3-nodes ZK cluster, and 1 NiFi Registry) are running
on private-cloud VM instances. Their operations on disks are not as performant
as it could be on physical machines, and are restricted to a certain amount of
IOps that we can easily reach with disk-based Repositories ; that is the main
reason why we tried the VolatileContentRepository. Since 1.13.1, we use a
version of NiFi that we build with our custom bundles and the fix that was
linked in NIFI-8760 and for now it is working fine for us. We had some memory
leaks on custom processors but after correction and limitation on queue sizes
we never encountered OOM on the Volatile again.
Don't hesitate to ask for more information or precision on the use case, or for
any advice !
About the OOM on RAM-disk approach, the last time we tried it was on 1.14.0. In
the next few weeks we will try to migrate to 1.16.0, I'll be glad to
investigate if we still experience OOMs and report it if we do. And if we
don't, it could be a great solution for us to replace the
VolatileContentRepository. I have to add that I just discovered the NiFi
Stateless engine and especially the ExecuteStateless processor and the
repository you mentioned, with some refacto we could reduce the number of
queues in between ExecuteStateless processors that would induce FlowFiles to be
written on disks. We will investigate if this is enough for our nodes to be
below the threshold of authorized IOps per volume.
So thanks to you I understand that the Volatile has some weaknesses and could
have great alternatives. If all these alternatives fail, I would be glad to
investigate for 1.17 the possibility to promote the ByteArrayContentRepository
to the main framework.
Thank you!
Matthieu
Le mer. 30 mars 2022 à 15:18, David Handermann
mailto:exceptionfact...@apache.org>> a écrit :
Hi Matthieu,
Thanks for raising this question for discussion. Other maintainers may be able
to provide additional background, but part of the reason for removing the
VolatileContentRepository implementation was that there were some more
fundamental problems with the implementation. Although various framework
updates included patching the implementation along the way, the repository was
not maintained on a regular basis, which resulted in it being broken for
several releases.
As Mike said, it would be helpful to share more details about your use case,
and also to hear more about whether you still experience memory issues with the
file system repository in current releases.
On a related note, NiFi Stateless includes a new in-memory content repository
named ByteArrayContentRepository [1]. It is currently packaged in the NiFi
Stateless bundle, but it might be possible to consider promoting it to the
framework level, if there is value in a non-persistent content repository going
forward.
Regards,
David Handermann
[1]
https://github.com/apache/nifi/blob/main/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/repository/ByteArrayContentRepository.java
On Wed, Mar 30, 2022 at 7:45 AM Mike Thomsen
mailto:mikerthom...@gmail.com>> wrote:
We've been moving away from supporting it for a while, and I think it
comes down to a lot of both factors when you consider the time
involved in getting good patches and reviewing them. That said, until
1.17 is released, I think there's room for community members like you
and your team to work with us on fixing the gaps that made a strong
case for removing it.
I think I saw in your ticket that you provided patches through Jira.
My recommendation wou
Re: VolatileContentRepository removal
Hi Mike, David,
Thanks for your answers and your clarity !
To sum up our use case, our team has set up two cloud-based NiFi
("stateful") clusters. The second one deals with a big amount of
record-based data, to drag data from sources (Kafka, files and databases)
to other systems (OpenSearch and internal tools) performing a lot of costly
transformations (in the last 5 minutes for instance, my prod instance
indicates 300G of read, 170G of write, within 400 processors). This cluster
deals with data we can recover from a process, so in this use case we are
data-loss tolerant. The 12 NiFi nodes of the cluster (as well as a 3-nodes
ZK cluster, and 1 NiFi Registry) are running on private-cloud VM instances.
Their operations on disks are not as performant as it could be on physical
machines, and are restricted to a certain amount of IOps that we can easily
reach with disk-based Repositories ; that is the main reason why we tried
the VolatileContentRepository. Since 1.13.1, we use a version of NiFi that
we build with our custom bundles and the fix that was linked in NIFI-8760
and for now it is working fine for us. We had some memory leaks on custom
processors but after correction and limitation on queue sizes we never
encountered OOM on the Volatile again.
Don't hesitate to ask for more information or precision on the use case, or
for any advice !
About the OOM on RAM-disk approach, the last time we tried it was on
1.14.0. In the next few weeks we will try to migrate to 1.16.0, I'll be
glad to investigate if we still experience OOMs and report it if we do. And
if we don't, it could be a great solution for us to replace the
VolatileContentRepository. I have to add that I just discovered the NiFi
Stateless engine and especially the ExecuteStateless processor and the
repository you mentioned, with some refacto we could reduce the number of
queues in between ExecuteStateless processors that would induce FlowFiles
to be written on disks. We will investigate if this is enough for our nodes
to be below the threshold of authorized IOps per volume.
So thanks to you I understand that the Volatile has some weaknesses and
could have great alternatives. If all these alternatives fail, I would be
glad to investigate for 1.17 the possibility to promote the
ByteArrayContentRepository to the main framework.
Thank you!
Matthieu
Le mer. 30 mars 2022 à 15:18, David Handermann
a écrit :
> Hi Matthieu,
>
> Thanks for raising this question for discussion. Other maintainers may be
> able to provide additional background, but part of the reason for removing
> the VolatileContentRepository implementation was that there were some more
> fundamental problems with the implementation. Although various framework
> updates included patching the implementation along the way, the repository
> was not maintained on a regular basis, which resulted in it being broken
> for several releases.
>
> As Mike said, it would be helpful to share more details about your use
> case, and also to hear more about whether you still experience memory
> issues with the file system repository in current releases.
>
> On a related note, NiFi Stateless includes a new in-memory content
> repository named ByteArrayContentRepository [1]. It is currently packaged
> in the NiFi Stateless bundle, but it might be possible to consider
> promoting it to the framework level, if there is value in a non-persistent
> content repository going forward.
>
> Regards,
> David Handermann
>
> [1]
> https://github.com/apache/nifi/blob/main/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/repository/ByteArrayContentRepository.java
>
> On Wed, Mar 30, 2022 at 7:45 AM Mike Thomsen
> wrote:
>
>> We've been moving away from supporting it for a while, and I think it
>> comes down to a lot of both factors when you consider the time
>> involved in getting good patches and reviewing them. That said, until
>> 1.17 is released, I think there's room for community members like you
>> and your team to work with us on fixing the gaps that made a strong
>> case for removing it.
>>
>> I think I saw in your ticket that you provided patches through Jira.
>> My recommendation would be to do a feature branch that reverts the
>> removal, applies your patches and submit it as a PR on GitHub. Then
>> request a review. Obviously, there's no guarantees there because it's
>> based on folks' time and energy to do a review, but that would be the
>> right process at least to move your request forward.
>>
>> In the long run, I think it would be a lot better for you to share
>> your use case with us and to see if there's a better route ahead for
>> your team and NiFi. Sounds like an interesting use case, so it would
>
Re: VolatileContentRepository removal
Hi Matthieu, Thanks for raising this question for discussion. Other maintainers may be able to provide additional background, but part of the reason for removing the VolatileContentRepository implementation was that there were some more fundamental problems with the implementation. Although various framework updates included patching the implementation along the way, the repository was not maintained on a regular basis, which resulted in it being broken for several releases. As Mike said, it would be helpful to share more details about your use case, and also to hear more about whether you still experience memory issues with the file system repository in current releases. On a related note, NiFi Stateless includes a new in-memory content repository named ByteArrayContentRepository [1]. It is currently packaged in the NiFi Stateless bundle, but it might be possible to consider promoting it to the framework level, if there is value in a non-persistent content repository going forward. Regards, David Handermann [1] https://github.com/apache/nifi/blob/main/nifi-stateless/nifi-stateless-bundle/nifi-stateless-engine/src/main/java/org/apache/nifi/stateless/repository/ByteArrayContentRepository.java On Wed, Mar 30, 2022 at 7:45 AM Mike Thomsen wrote: > We've been moving away from supporting it for a while, and I think it > comes down to a lot of both factors when you consider the time > involved in getting good patches and reviewing them. That said, until > 1.17 is released, I think there's room for community members like you > and your team to work with us on fixing the gaps that made a strong > case for removing it. > > I think I saw in your ticket that you provided patches through Jira. > My recommendation would be to do a feature branch that reverts the > removal, applies your patches and submit it as a PR on GitHub. Then > request a review. Obviously, there's no guarantees there because it's > based on folks' time and energy to do a review, but that would be the > right process at least to move your request forward. > > In the long run, I think it would be a lot better for you to share > your use case with us and to see if there's a better route ahead for > your team and NiFi. Sounds like an interesting use case, so it would > be good to get those requirements on the table since most users aren't > operating with those constraints. > > Thanks, > > Mike > > On Tue, Mar 29, 2022 at 12:20 PM Matthieu Ré > wrote: > > > > Hi everyone, > > > > We wanted to talk about this ticket > https://issues.apache.org/jira/browse/NIFI-8760 and the > VolatileContentRepository... I understood that we weren't many to still use > this Repository, but in our use case with a very limited cloud environment > with strict IOps regulations, it fitted perfectly and we managed several To > of data per day efficiently. > > > > We tested other repositories, even a FileSystemContentRepo with RAM > based disk that did not match the case since we experimented numerous OOMs > with the same amount of RAM mounted. > > > > I provided a patch to fix it, that should be applied after 1.13.0 and a > refactor of Claims handling, waiting for a discussion about it. Now I read > that it should disappear in 1.17.0 :( > > > > Is it due to a technical limitation for further features ? Or is it too > costly to maintain it ? > > > > Thanks! Regards, > > Matthieu >
Re: VolatileContentRepository removal
We've been moving away from supporting it for a while, and I think it comes down to a lot of both factors when you consider the time involved in getting good patches and reviewing them. That said, until 1.17 is released, I think there's room for community members like you and your team to work with us on fixing the gaps that made a strong case for removing it. I think I saw in your ticket that you provided patches through Jira. My recommendation would be to do a feature branch that reverts the removal, applies your patches and submit it as a PR on GitHub. Then request a review. Obviously, there's no guarantees there because it's based on folks' time and energy to do a review, but that would be the right process at least to move your request forward. In the long run, I think it would be a lot better for you to share your use case with us and to see if there's a better route ahead for your team and NiFi. Sounds like an interesting use case, so it would be good to get those requirements on the table since most users aren't operating with those constraints. Thanks, Mike On Tue, Mar 29, 2022 at 12:20 PM Matthieu Ré wrote: > > Hi everyone, > > We wanted to talk about this ticket > https://issues.apache.org/jira/browse/NIFI-8760 and the > VolatileContentRepository... I understood that we weren't many to still use > this Repository, but in our use case with a very limited cloud environment > with strict IOps regulations, it fitted perfectly and we managed several To > of data per day efficiently. > > We tested other repositories, even a FileSystemContentRepo with RAM based > disk that did not match the case since we experimented numerous OOMs with the > same amount of RAM mounted. > > I provided a patch to fix it, that should be applied after 1.13.0 and a > refactor of Claims handling, waiting for a discussion about it. Now I read > that it should disappear in 1.17.0 :( > > Is it due to a technical limitation for further features ? Or is it too > costly to maintain it ? > > Thanks! Regards, > Matthieu
VolatileContentRepository removal
Hi everyone, We wanted to talk about this ticket https://issues.apache.org/jira/browse/NIFI-8760 and the VolatileContentRepository... I understood that we weren't many to still use this Repository, but in our use case with a very limited cloud environment with strict IOps regulations, it fitted perfectly and we managed several To of data per day efficiently. We tested other repositories, even a FileSystemContentRepo with RAM based disk that did not match the case since we experimented numerous OOMs with the same amount of RAM mounted. I provided a patch to fix it, that should be applied after 1.13.0 and a refactor of Claims handling, waiting for a discussion about it. Now I read that it should disappear in 1.17.0 :( Is it due to a technical limitation for further features ? Or is it too costly to maintain it ? Thanks! Regards, Matthieu
Re: [NiFi-8760] Processors fail to process flowfiles with VolatileContentRepository
Thank you very much for your answers ! That's a surprise ! The VolatileContentRepository seemed to answer perfectly our need to treat a big amount of data with low resources and especially low I/O on mounted disks, with non critical data and potential data loss authorized. I just tried your solution @Mark mounting a tmpfs and FileSystemRepository (on 1.11.4), but it seems like for the same amount of data and same RAM space used, the VolatileContentRepository used a constant <5% of space, while the FileSystemRepository was using a very unstable amount of space, frequently running out of space. (I must add that we don't store any archive, nifi.content.repository.archive.enabled=false). Maybe am I missing a configuration that consumes a lot of space with the FileSystemRepository ? Stateless NiFi sounds very interesting ! Just had a look at pvillard's demo (https://github.com/pvillard31/nifi-stateless-demo) and the framework's readme ( https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-framework-bundle/nifi-framework/nifi-stateless), but do you have any more resources about it ? I would like to understand a little bit more what differs from the standard framework and how it can fit our use case. Have a nice day, Matthieu Le ven. 23 juil. 2021 à 17:20, Joe Witt a écrit : > It seems like any use case that we previously thought VolatileContentRepo > would be good for now we'd say Stateless NiFi is a dramatically better > approach. > > We need to doc this better but the capability is there now for sure. > > On Fri, Jul 23, 2021 at 8:13 AM Mark Payne wrote: > >> Matthieu, >> >> I would highly recommend against using VolatileContentRepository. You’re >> the first one I’ve heard of using it in a few years. Typically, the >> FileSystemRepository is sufficient. If you truly want to run with the >> content in RAM I would recommend creating a RAM Disk and pointing the >> FileSystemRepository to that. >> >> Thanks >> -Mark >> >> >> On Jul 21, 2021, at 10:31 AM, Matthieu Ré wrote: >> >> Hi Chris, thank you for your quick response >> >> I tried the flow with 1.13.2 and 1.13.1, and 1.14.0 just before the first >> RC and it still had the problem, so I am not sure if this is related to the >> session handling you pointed out, that has been fixed in 1.13.2 >> >> Le mer. 21 juil. 2021 à 16:22, Chris Sampson >> a écrit : >> >>> 1.13.1 was known to have problems with session handling - see the >>> Release Note "lowlights" for 1.13.1 [1] >>> >>> It is recommended to upgrade to version 1.13.2 (or the latest 1.14.0). >>> If you can't upgrade then 1.13.0 would be better than 1.13.1. >>> >>> >>> [1] >>> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1 >>> >>> --- >>> *Chris Sampson* >>> IT Consultant >>> chris.samp...@naimuri.com >>> <https://www.naimuri.com/> >>> >>> >>> On Wed, 21 Jul 2021 at 15:14, Matthieu Ré wrote: >>> >>>> Hi all, >>>> >>>> Currently using NiFi 1.11.4, we face a blocking issue trying to switch >>>> to NiFi 1.13.1+ due to the VolatileContentRepository : some processors we >>>> use (and probably others that we didn't try) were not able to process >>>> flowfiles, such as MargeRecord, QueryRecord or SplitJson (logs are in the >>>> Jira >>>> ticket NiFi-8760 <https://issues.apache.org/jira/browse/NIFI-8760>). >>>> >>>> I wanted to know if any of you guys are able to reproduce the issue, >>>> and if this is not a misconfiguration from our side. The nifi.properties >>>> and flow.xml.gz used are available in the ticket. If I am not missing >>>> anything, we could identify that the issue could come from this commit >>>> <https://github.com/apache/nifi/commit/528fce2407d092d4ced1a58fcc14d0bc6e660b89> >>>> since >>>> it appeared with the 1.13.1 and the flow is working fine with 1.13.0. >>>> >>>> Open to contribute as much as I can if you confirm that this is not due >>>> to a misconfiguration.. >>>> >>>> Thanks ! >>>> Matthieu >>>> >>> >> >> -- >> >> Matthieu RÉ >> Data Scientist - Machine Learning Engineer - Dassault Systèmes >> >> ENSIIE, M2 AIC (Université Paris-Saclay) >> >> Tel: 0631609755 >> >> Email: re.matth...@gmail.com >> >> >>
Re: [NiFi-8760] Processors fail to process flowfiles with VolatileContentRepository
It seems like any use case that we previously thought VolatileContentRepo would be good for now we'd say Stateless NiFi is a dramatically better approach. We need to doc this better but the capability is there now for sure. On Fri, Jul 23, 2021 at 8:13 AM Mark Payne wrote: > Matthieu, > > I would highly recommend against using VolatileContentRepository. You’re > the first one I’ve heard of using it in a few years. Typically, the > FileSystemRepository is sufficient. If you truly want to run with the > content in RAM I would recommend creating a RAM Disk and pointing the > FileSystemRepository to that. > > Thanks > -Mark > > > On Jul 21, 2021, at 10:31 AM, Matthieu Ré wrote: > > Hi Chris, thank you for your quick response > > I tried the flow with 1.13.2 and 1.13.1, and 1.14.0 just before the first > RC and it still had the problem, so I am not sure if this is related to the > session handling you pointed out, that has been fixed in 1.13.2 > > Le mer. 21 juil. 2021 à 16:22, Chris Sampson > a écrit : > >> 1.13.1 was known to have problems with session handling - see the Release >> Note "lowlights" for 1.13.1 [1] >> >> It is recommended to upgrade to version 1.13.2 (or the latest 1.14.0). If >> you can't upgrade then 1.13.0 would be better than 1.13.1. >> >> >> [1] >> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1 >> >> --- >> *Chris Sampson* >> IT Consultant >> chris.samp...@naimuri.com >> <https://www.naimuri.com/> >> >> >> On Wed, 21 Jul 2021 at 15:14, Matthieu Ré wrote: >> >>> Hi all, >>> >>> Currently using NiFi 1.11.4, we face a blocking issue trying to switch >>> to NiFi 1.13.1+ due to the VolatileContentRepository : some processors we >>> use (and probably others that we didn't try) were not able to process >>> flowfiles, such as MargeRecord, QueryRecord or SplitJson (logs are in the >>> Jira >>> ticket NiFi-8760 <https://issues.apache.org/jira/browse/NIFI-8760>). >>> >>> I wanted to know if any of you guys are able to reproduce the issue, and >>> if this is not a misconfiguration from our side. The nifi.properties and >>> flow.xml.gz used are available in the ticket. If I am not missing anything, >>> we could identify that the issue could come from this commit >>> <https://github.com/apache/nifi/commit/528fce2407d092d4ced1a58fcc14d0bc6e660b89> >>> since >>> it appeared with the 1.13.1 and the flow is working fine with 1.13.0. >>> >>> Open to contribute as much as I can if you confirm that this is not due >>> to a misconfiguration.. >>> >>> Thanks ! >>> Matthieu >>> >> > > -- > > Matthieu RÉ > Data Scientist - Machine Learning Engineer - Dassault Systèmes > > ENSIIE, M2 AIC (Université Paris-Saclay) > > Tel: 0631609755 > > Email: re.matth...@gmail.com > > >
Re: [NiFi-8760] Processors fail to process flowfiles with VolatileContentRepository
Matthieu, I would highly recommend against using VolatileContentRepository. You’re the first one I’ve heard of using it in a few years. Typically, the FileSystemRepository is sufficient. If you truly want to run with the content in RAM I would recommend creating a RAM Disk and pointing the FileSystemRepository to that. Thanks -Mark On Jul 21, 2021, at 10:31 AM, Matthieu Ré mailto:re.matth...@gmail.com>> wrote: Hi Chris, thank you for your quick response I tried the flow with 1.13.2 and 1.13.1, and 1.14.0 just before the first RC and it still had the problem, so I am not sure if this is related to the session handling you pointed out, that has been fixed in 1.13.2 Le mer. 21 juil. 2021 à 16:22, Chris Sampson mailto:chris.samp...@naimuri.com>> a écrit : 1.13.1 was known to have problems with session handling - see the Release Note "lowlights" for 1.13.1 [1] It is recommended to upgrade to version 1.13.2 (or the latest 1.14.0). If you can't upgrade then 1.13.0 would be better than 1.13.1. [1] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1 --- Chris Sampson IT Consultant chris.samp...@naimuri.com<mailto:chris.samp...@naimuri.com> [https://docs.google.com/uc?export=download&id=1oPtzd0P7DqtuzpjiTRAa6h6coFitpqom&revid=0B9aXwC5rMc6lVlZ2OWpUaVlFVmUwTlZBdjQ0KzAxb1dZS2hJPQ]<https://www.naimuri.com/> On Wed, 21 Jul 2021 at 15:14, Matthieu Ré mailto:re.matth...@gmail.com>> wrote: Hi all, Currently using NiFi 1.11.4, we face a blocking issue trying to switch to NiFi 1.13.1+ due to the VolatileContentRepository : some processors we use (and probably others that we didn't try) were not able to process flowfiles, such as MargeRecord, QueryRecord or SplitJson (logs are in the Jira ticket NiFi-8760<https://issues.apache.org/jira/browse/NIFI-8760>). I wanted to know if any of you guys are able to reproduce the issue, and if this is not a misconfiguration from our side. The nifi.properties and flow.xml.gz used are available in the ticket. If I am not missing anything, we could identify that the issue could come from this commit<https://github.com/apache/nifi/commit/528fce2407d092d4ced1a58fcc14d0bc6e660b89> since it appeared with the 1.13.1 and the flow is working fine with 1.13.0. Open to contribute as much as I can if you confirm that this is not due to a misconfiguration.. Thanks ! Matthieu -- Matthieu RÉ Data Scientist - Machine Learning Engineer - Dassault Systèmes ENSIIE, M2 AIC (Université Paris-Saclay) Tel: 0631609755 Email: re.matth...@gmail.com<mailto:re.matth...@gmail.com>
Re: [NiFi-8760] Processors fail to process flowfiles with VolatileContentRepository
Hi Chris, thank you for your quick response I tried the flow with 1.13.2 and 1.13.1, and 1.14.0 just before the first RC and it still had the problem, so I am not sure if this is related to the session handling you pointed out, that has been fixed in 1.13.2 Le mer. 21 juil. 2021 à 16:22, Chris Sampson a écrit : > 1.13.1 was known to have problems with session handling - see the Release > Note "lowlights" for 1.13.1 [1] > > It is recommended to upgrade to version 1.13.2 (or the latest 1.14.0). If > you can't upgrade then 1.13.0 would be better than 1.13.1. > > > [1] > https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1 > > --- > *Chris Sampson* > IT Consultant > chris.samp...@naimuri.com > <https://www.naimuri.com/> > > > On Wed, 21 Jul 2021 at 15:14, Matthieu Ré wrote: > >> Hi all, >> >> Currently using NiFi 1.11.4, we face a blocking issue trying to switch to >> NiFi 1.13.1+ due to the VolatileContentRepository : some processors we use >> (and probably others that we didn't try) were not able to process >> flowfiles, such as MargeRecord, QueryRecord or SplitJson (logs are in the >> Jira >> ticket NiFi-8760 <https://issues.apache.org/jira/browse/NIFI-8760>). >> >> I wanted to know if any of you guys are able to reproduce the issue, and >> if this is not a misconfiguration from our side. The nifi.properties and >> flow.xml.gz used are available in the ticket. If I am not missing anything, >> we could identify that the issue could come from this commit >> <https://github.com/apache/nifi/commit/528fce2407d092d4ced1a58fcc14d0bc6e660b89> >> since >> it appeared with the 1.13.1 and the flow is working fine with 1.13.0. >> >> Open to contribute as much as I can if you confirm that this is not due >> to a misconfiguration.. >> >> Thanks ! >> Matthieu >> > -- Matthieu RÉ Data Scientist - Machine Learning Engineer - Dassault Systèmes ENSIIE, M2 AIC (Université Paris-Saclay) Tel: 0631609755 Email: re.matth...@gmail.com
Re: [NiFi-8760] Processors fail to process flowfiles with VolatileContentRepository
1.13.1 was known to have problems with session handling - see the Release Note "lowlights" for 1.13.1 [1] It is recommended to upgrade to version 1.13.2 (or the latest 1.14.0). If you can't upgrade then 1.13.0 would be better than 1.13.1. [1] https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.13.1 --- *Chris Sampson* IT Consultant chris.samp...@naimuri.com <https://www.naimuri.com/> On Wed, 21 Jul 2021 at 15:14, Matthieu Ré wrote: > Hi all, > > Currently using NiFi 1.11.4, we face a blocking issue trying to switch to > NiFi 1.13.1+ due to the VolatileContentRepository : some processors we use > (and probably others that we didn't try) were not able to process > flowfiles, such as MargeRecord, QueryRecord or SplitJson (logs are in the Jira > ticket NiFi-8760 <https://issues.apache.org/jira/browse/NIFI-8760>). > > I wanted to know if any of you guys are able to reproduce the issue, and > if this is not a misconfiguration from our side. The nifi.properties and > flow.xml.gz used are available in the ticket. If I am not missing anything, > we could identify that the issue could come from this commit > <https://github.com/apache/nifi/commit/528fce2407d092d4ced1a58fcc14d0bc6e660b89> > since > it appeared with the 1.13.1 and the flow is working fine with 1.13.0. > > Open to contribute as much as I can if you confirm that this is not due to > a misconfiguration.. > > Thanks ! > Matthieu >
[NiFi-8760] Processors fail to process flowfiles with VolatileContentRepository
Hi all, Currently using NiFi 1.11.4, we face a blocking issue trying to switch to NiFi 1.13.1+ due to the VolatileContentRepository : some processors we use (and probably others that we didn't try) were not able to process flowfiles, such as MargeRecord, QueryRecord or SplitJson (logs are in the Jira ticket NiFi-8760 <https://issues.apache.org/jira/browse/NIFI-8760>). I wanted to know if any of you guys are able to reproduce the issue, and if this is not a misconfiguration from our side. The nifi.properties and flow.xml.gz used are available in the ticket. If I am not missing anything, we could identify that the issue could come from this commit <https://github.com/apache/nifi/commit/528fce2407d092d4ced1a58fcc14d0bc6e660b89> since it appeared with the 1.13.1 and the flow is working fine with 1.13.0. Open to contribute as much as I can if you confirm that this is not due to a misconfiguration.. Thanks ! Matthieu
Re: Bug reports about working with VolatileContentRepository and VolatileFlowFileRepository
Thanks Mark,
I have posted these issues in Jira.
https://issues.apache.org/jira/projects/NIFI/issues/NIFI-7388?filter=allissues
Your suggestion is very constructive, we have switched back to
WriteAheadFlowFileRepository and FileSystemRepository temporarily, looking
forward to a new version which can fix these issues.
We are running NiFi-1.11.4. The environment is jdk8_141, docker 18.04.0-ce and
Kubernetes v1.13.3.
We doubt that it may be related to CodeCache in JVM, so we have increase
CodeCacheSize to 256MB, using CodeCacheFlushing. We will continue to pay
attention to this issue and gather a thread dump when stuck occurs according to
your suggestions.
ZhangXinchen
On 2020/04/22 14:51:51, Mark Payne wrote:
> Hello,
>
> Thanks for reporting the issues with the Volatile Content & FlowFile
> Repositories. These definitely sounds like bugs. Do you mind filing a Jira
> [1] for these?
> If you’d like to store everything in memory, though, my recommendation would
> honestly be to use a RAM disk rather than the volatile repositories, since
> the standard repositories are much more widely used and therefore extremely
> well tested, while the volatile implementations are much less so.
>
> In terms of the last issue, in which NiFi becomes stuck after a week: what
> version of NiFi are you running? I would recommend gathering a thread dump
> (bin/nifi.sh dump dump1.txt) when this occurs and providing the thread dump
> so that it can be analyzed to determine what’s happening.
>
> Thanks
> -Mark
>
>
> [1] https://issues.apache.org/jira/projects/NIFI
>
>
>
> On Apr 21, 2020, at 11:35 PM, abcfd abcdd
> mailto:officek7...@gmail.com>> wrote:
>
> Hola，
>
> Our team has worked with NiFi for over one year. Our scenario is dealing with
> 3-5 billion data using NiFi, we found that WriteAheadFlowFileRepository and
> FileSystemRepository cannot meet command，so we put data which need to be
> consumed in tmpfs and choose VolatileFlowFileRepository and
> VolatileContentRepository to reduce I/O costs and avoid WAL, because in our
> scenario, the data can be thrown away when backpressure occurs or NiFi
> restarted.
>
> But, we find three problems working with VolatileFlowFileRepository and
> VolatileContentRepository.
> 1. VolatileContentRepository
> when maxSize = 100MB and blockSize = 2KB, there should be 51200 "slots". If
> we write one kb by one kb, 102400 one kb should be written in, but when
> writing 51201th one kb, "java.io.IOException: Content Repository is out of
> space" occurs. Here's the Junit Test I write.
>
> @Test
> public void test() throws IOException {
> System.setProperty(NiFiProperties.PROPERTIES_FILE_PATH,
> TestVolatileContentRepository.class.getResource("/conf/nifi.properties").getFile());
> final Map addProps = new HashMap<>();
> addProps.put(VolatileContentRepository.BLOCK_SIZE_PROPERTY, "2 KB");
> final NiFiProperties nifiProps =
> NiFiProperties.createBasicNiFiProperties(null, addProps);
> final VolatileContentRepository contentRepo = new
> VolatileContentRepository(nifiProps);
> contentRepo.initialize(claimManager);
> // can write 100 * 1024 /1 = 102400, but after 51201, blocks exhausted
> for (int idx =0; idx < 51201; ++idx) {
> final ContentClaim claim = contentRepo.create(true);
> try (final OutputStream out = contentRepo.write(claim)){
> final byte[] oneK = new byte[1024];
> Arrays.fill(oneK, (byte) 55);
>
> out.write(oneK);
> }
> }
> }
>
> 2. VolatileFlowFileRepository
> When the backpressure occurs, FileSystemSwapManager will swap out FlowFiles
> to disk whenever swapQueue size exceeds 1, there's no problem in
> swap-out process BUT in swap-in process, VolatileFlowFileRepository does not
> "acknowledge" the FlowFiles which has been swap out when
> FileSystemSwapManager swaps in FlowFiles from the disk and logs the warning
> information "Cannot swap in FlowFiles from location..." because the
> implementation of "isValidSwapLocationSuffix" in VolatileFlowFileRepository
> is always FALSE.
> And the queue is still like FULL when checking the NiFi frontend, the
> upstream processor is STUCKED, maybe FileSystemSwapManager "thinks" these
> FlowFiles are still not consumed.
>
> 3. we found that NiFi cannot live more than a week even if we use
> WriteAheadFlowFileRepository and FileSystemRepository. NiFi stucked, didn't
> process any data and there was no output in nifi-app.log. We restart NiFi and
> it is back to normal, but we didn't know what happened.
>
> Muchas Gracias
>
>
Re: Bug reports about working with VolatileContentRepository and VolatileFlowFileRepository
Hello,
Thanks for reporting the issues with the Volatile Content & FlowFile
Repositories. These definitely sounds like bugs. Do you mind filing a Jira [1]
for these?
If you’d like to store everything in memory, though, my recommendation would
honestly be to use a RAM disk rather than the volatile repositories, since the
standard repositories are much more widely used and therefore extremely well
tested, while the volatile implementations are much less so.
In terms of the last issue, in which NiFi becomes stuck after a week: what
version of NiFi are you running? I would recommend gathering a thread dump
(bin/nifi.sh dump dump1.txt) when this occurs and providing the thread dump so
that it can be analyzed to determine what’s happening.
Thanks
-Mark
[1] https://issues.apache.org/jira/projects/NIFI
On Apr 21, 2020, at 11:35 PM, abcfd abcdd
mailto:officek7...@gmail.com>> wrote:
Hola，
Our team has worked with NiFi for over one year. Our scenario is dealing with
3-5 billion data using NiFi, we found that WriteAheadFlowFileRepository and
FileSystemRepository cannot meet command，so we put data which need to be
consumed in tmpfs and choose VolatileFlowFileRepository and
VolatileContentRepository to reduce I/O costs and avoid WAL, because in our
scenario, the data can be thrown away when backpressure occurs or NiFi
restarted.
But, we find three problems working with VolatileFlowFileRepository and
VolatileContentRepository.
1. VolatileContentRepository
when maxSize = 100MB and blockSize = 2KB, there should be 51200 "slots". If we
write one kb by one kb, 102400 one kb should be written in, but when writing
51201th one kb, "java.io.IOException: Content Repository is out of space"
occurs. Here's the Junit Test I write.
@Test
public void test() throws IOException {
System.setProperty(NiFiProperties.PROPERTIES_FILE_PATH,
TestVolatileContentRepository.class.getResource("/conf/nifi.properties").getFile());
final Map addProps = new HashMap<>();
addProps.put(VolatileContentRepository.BLOCK_SIZE_PROPERTY, "2 KB");
final NiFiProperties nifiProps =
NiFiProperties.createBasicNiFiProperties(null, addProps);
final VolatileContentRepository contentRepo = new
VolatileContentRepository(nifiProps);
contentRepo.initialize(claimManager);
// can write 100 * 1024 /1 = 102400, but after 51201, blocks exhausted
for (int idx =0; idx < 51201; ++idx) {
final ContentClaim claim = contentRepo.create(true);
try (final OutputStream out = contentRepo.write(claim)){
final byte[] oneK = new byte[1024];
Arrays.fill(oneK, (byte) 55);
out.write(oneK);
}
}
}
2. VolatileFlowFileRepository
When the backpressure occurs, FileSystemSwapManager will swap out FlowFiles to
disk whenever swapQueue size exceeds 1, there's no problem in swap-out
process BUT in swap-in process, VolatileFlowFileRepository does not
"acknowledge" the FlowFiles which has been swap out when FileSystemSwapManager
swaps in FlowFiles from the disk and logs the warning information "Cannot swap
in FlowFiles from location..." because the implementation of
"isValidSwapLocationSuffix" in VolatileFlowFileRepository is always FALSE.
And the queue is still like FULL when checking the NiFi frontend, the upstream
processor is STUCKED, maybe FileSystemSwapManager "thinks" these FlowFiles are
still not consumed.
3. we found that NiFi cannot live more than a week even if we use
WriteAheadFlowFileRepository and FileSystemRepository. NiFi stucked, didn't
process any data and there was no output in nifi-app.log. We restart NiFi and
it is back to normal, but we didn't know what happened.
Muchas Gracias
Bug reports about working with VolatileContentRepository and VolatileFlowFileRepository
Hola，
Our team has worked with NiFi for over one year. Our scenario is dealing
with 3-5 billion data using NiFi, we found that
WriteAheadFlowFileRepository and FileSystemRepository cannot meet
command，so we put data which need to be consumed in tmpfs and choose
VolatileFlowFileRepository and VolatileContentRepository to reduce I/O
costs and avoid WAL, because in our scenario, the data can be thrown away
when backpressure occurs or NiFi restarted.
But, we find three problems working with VolatileFlowFileRepository and
VolatileContentRepository.
1. VolatileContentRepository
when maxSize = 100MB and blockSize = 2KB, there should be 51200 "slots". If
we write one kb by one kb, 102400 one kb should be written in, but when
writing 51201th one kb, "java.io.IOException: Content Repository is out of
space" occurs. Here's the Junit Test I write.
@Test
public void test() throws IOException {
System.setProperty(NiFiProperties.PROPERTIES_FILE_PATH,
TestVolatileContentRepository.class.getResource("/conf/nifi.properties").getFile());
final Map addProps = new HashMap<>();
addProps.put(VolatileContentRepository.BLOCK_SIZE_PROPERTY, "2 KB");
final NiFiProperties nifiProps =
NiFiProperties.createBasicNiFiProperties(null, addProps);
final VolatileContentRepository contentRepo = new
VolatileContentRepository(nifiProps);
contentRepo.initialize(claimManager);
// can write 100 * 1024 /1 = 102400, but after 51201, blocks exhausted
for (int idx =0; idx < 51201; ++idx) {
final ContentClaim claim = contentRepo.create(true);
try (final OutputStream out = contentRepo.write(claim)){
final byte[] oneK = new byte[1024];
Arrays.fill(oneK, (byte) 55);
out.write(oneK);
}
}
}
2. VolatileFlowFileRepository
When the backpressure occurs, FileSystemSwapManager will swap out FlowFiles
to disk whenever swapQueue size exceeds 1, there's no problem in
swap-out process BUT in swap-in process, VolatileFlowFileRepository does
not "acknowledge" the FlowFiles which has been swap out when
FileSystemSwapManager swaps in FlowFiles from the disk and logs the warning
information "Cannot swap in FlowFiles from location..." because the
implementation of "isValidSwapLocationSuffix" in VolatileFlowFileRepository
is always FALSE.
And the queue is still like FULL when checking the NiFi frontend, the
upstream processor is STUCKED, maybe FileSystemSwapManager "thinks" these
FlowFiles are still not consumed.
3. we found that NiFi cannot live more than a week even if we use
WriteAheadFlowFileRepository and FileSystemRepository. NiFi stucked,
didn't process any data and there was no output in nifi-app.log. We
restart NiFi and it is back to normal, but we didn't know what happened.
Muchas Gracias
Re: VolatileContentRepository
Hi Brian, The question is about the content volatile repository. If you are only using the provenance volatile repository (not the content one), then only the provenance data would be lost in case of NiFi shutdown, you're right. Pierre 2017-08-10 10:03 GMT+02:00 BD International : > Pierre, > > Just a clarification on what you said below the potential data loss on > nifi shutdown would only be on provenance information right? > > Or would the data loss affect other repos? > > Thanks > > Brian > > On 10 Aug 2017 8:57 am, "Pierre Villard" > wrote: > > Hi Margus, > > I believe that your memory settings are enough. Giving more memory will > likely increase duration of garbage collections and won't increase your > performances. Others on this mailing list will probably be able to give > better recommendations on this one. Also, keep in mind that volatile > repositories can cause data loss in case of NiFi shutdown. > > I wanted to answer your question to check if you tried to increase the > number of concurrent tasks in your SplitJson processor (if you have enough > resources, that will likely improve the throughput), but also, if you > increased the run duration of the processor? That's something that can make > a huge difference on the performances if you don't care about latency. > > Also, it depends of what your are doing with your data, but did you > consider the new records oriented processors? If your use case can fit with > the record processors, that will certainly improve the overall performances > of your workflow. > > Thanks! > Pierre > > > 2017-08-10 7:50 GMT+02:00 Margus Roo : > >> Hi >> >> I am playing with nifi performance using one nifi node. >> >> At the moment I think the bottleneck in my flow is SplitJson processor >> who can work with 2 000 000 items per 5 minutes (downstrem queues are not >> full and queue before SplitJson is constantly full). >> >> I tried to change as much repos to volatile but if I change content repo >> to volatile then speed degrees a lot 2 000 000 to 5000 or smth. >> >> Befor I set content repo to volatile I increased volatile content repo >> max size: >> >> nifi.volatile.content.repository.max.size=12GB >> >> Do I need increase JVM setting that content repo can live inside JVM? >> >> At the moment I have: >> >> # JVM memory settings >> java.arg.2=-Xms2048m >> java.arg.3=-Xmx46384m >> >> >> -- >> Margus (margusja) Roohttp://margus.roo.ee >> skype: margusjahttps://www.facebook.com/allan.tuuring+372 51 48 780 >> <+372%20514%208780> >> >> > >
Re: VolatileContentRepository
Pierre, Just a clarification on what you said below the potential data loss on nifi shutdown would only be on provenance information right? Or would the data loss affect other repos? Thanks Brian On 10 Aug 2017 8:57 am, "Pierre Villard" wrote: Hi Margus, I believe that your memory settings are enough. Giving more memory will likely increase duration of garbage collections and won't increase your performances. Others on this mailing list will probably be able to give better recommendations on this one. Also, keep in mind that volatile repositories can cause data loss in case of NiFi shutdown. I wanted to answer your question to check if you tried to increase the number of concurrent tasks in your SplitJson processor (if you have enough resources, that will likely improve the throughput), but also, if you increased the run duration of the processor? That's something that can make a huge difference on the performances if you don't care about latency. Also, it depends of what your are doing with your data, but did you consider the new records oriented processors? If your use case can fit with the record processors, that will certainly improve the overall performances of your workflow. Thanks! Pierre 2017-08-10 7:50 GMT+02:00 Margus Roo : > Hi > > I am playing with nifi performance using one nifi node. > > At the moment I think the bottleneck in my flow is SplitJson processor who > can work with 2 000 000 items per 5 minutes (downstrem queues are not full > and queue before SplitJson is constantly full). > > I tried to change as much repos to volatile but if I change content repo > to volatile then speed degrees a lot 2 000 000 to 5000 or smth. > > Befor I set content repo to volatile I increased volatile content repo max > size: > > nifi.volatile.content.repository.max.size=12GB > > Do I need increase JVM setting that content repo can live inside JVM? > > At the moment I have: > > # JVM memory settings > java.arg.2=-Xms2048m > java.arg.3=-Xmx46384m > > > -- > Margus (margusja) Roohttp://margus.roo.ee > skype: margusjahttps://www.facebook.com/allan.tuuring+372 51 48 780 > <+372%20514%208780> > >
Re: VolatileContentRepository
Hi Margus, I believe that your memory settings are enough. Giving more memory will likely increase duration of garbage collections and won't increase your performances. Others on this mailing list will probably be able to give better recommendations on this one. Also, keep in mind that volatile repositories can cause data loss in case of NiFi shutdown. I wanted to answer your question to check if you tried to increase the number of concurrent tasks in your SplitJson processor (if you have enough resources, that will likely improve the throughput), but also, if you increased the run duration of the processor? That's something that can make a huge difference on the performances if you don't care about latency. Also, it depends of what your are doing with your data, but did you consider the new records oriented processors? If your use case can fit with the record processors, that will certainly improve the overall performances of your workflow. Thanks! Pierre 2017-08-10 7:50 GMT+02:00 Margus Roo : > Hi > > I am playing with nifi performance using one nifi node. > > At the moment I think the bottleneck in my flow is SplitJson processor who > can work with 2 000 000 items per 5 minutes (downstrem queues are not full > and queue before SplitJson is constantly full). > > I tried to change as much repos to volatile but if I change content repo > to volatile then speed degrees a lot 2 000 000 to 5000 or smth. > > Befor I set content repo to volatile I increased volatile content repo max > size: > > nifi.volatile.content.repository.max.size=12GB > > Do I need increase JVM setting that content repo can live inside JVM? > > At the moment I have: > > # JVM memory settings > java.arg.2=-Xms2048m > java.arg.3=-Xmx46384m > > > -- > Margus (margusja) Roohttp://margus.roo.ee > skype: margusjahttps://www.facebook.com/allan.tuuring+372 51 48 780 > <+372%20514%208780> > >
VolatileContentRepository
Hi I am playing with nifi performance using one nifi node. At the moment I think the bottleneck in my flow is SplitJson processor who can work with 2 000 000 items per 5 minutes (downstrem queues are not full and queue before SplitJson is constantly full). I tried to change as much repos to volatile but if I change content repo to volatile then speed degrees a lot 2 000 000 to 5000 or smth. Befor I set content repo to volatile I increased volatile content repo max size: nifi.volatile.content.repository.max.size=12GB Do I need increase JVM setting that content repo can live inside JVM? At the moment I have: # JVM memory settings java.arg.2=-Xms2048m java.arg.3=-Xmx46384m -- Margus (margusja) Roo http://margus.roo.ee skype: margusja https://www.facebook.com/allan.tuuring +372 51 48 780
