Frontend for Nifi support / mailing list

Hi, Is there a procedure for Nifi users to get nabble posting access to the Nifi mailing list ? I can post (obviously) via email, but would prefer to use the nabble web UI. Both myself and a colleague of mine have requested access via the nabble UI but we fail to get a response. Or are there

How does Nifi ingest large files?

Hi, I'm figuring out how does Nifi ingest large files: does it split them into chunks or is it a massive load?Can you please, explain the behavior? Kind regards, Monica -- *Monica Franceschini* Solution Architecture Manager *Big Data Competence Center Engineering Group* Corso Stati Uniti

Re: Frontend for Nifi support / mailing list

List has been created on nabble, it will be empty until messages start coming in. I don't yet see a way to backfill with history. I'll get this link on https://nifi.apache.org/mailing_lists.html in a bit. http://apache-nifi-users-list.2361937.n4.nabble.com/ On Thu, Oct 27, 2016 at 6:59 AM, Tony

Error instantiating template on cluster: The specified observer identifier already exists.

Hello, I am running into a problem moving some fairly large flows (40-50 processors) from a NiFi 1.0.0 standalone instance to an 3-node NiFi 1.0.0 cluster. I saved the flows in the standalone instance as templates and uploaded the templates to the cluster instance. When I instantiate the

Re: Frontend for Nifi support / mailing list

I will look into setting it up today On Oct 27, 2016 5:04 AM, "Davy De Waele" wrote: > Hi, > > Is there a procedure for Nifi users to get nabble posting access to the > Nifi mailing list ? > I can post (obviously) via email, but would prefer to use the nabble web > UI. > >

Re: How does Nifi ingest large files?

Bryan, If I have the content repo implementation set to org.apache.nifi.controller.repository.VolatileContentRepository, it will stream the content in memory, correct? On Thu, Oct 27, 2016 at 6:22 AM, Bryan Bende wrote: > Monica, > > Are you asking what does NiFi do when it

Re: How does Nifi ingest large files?

That is correct. Thanks Joe On Thu, Oct 27, 2016 at 11:55 AM, Jeremy Farbota wrote: > Bryan, > > If I have the content repo implementation set to > org.apache.nifi.controller.repository.VolatileContentRepository, it will > stream the content in memory, correct? > > On Thu,

Re: How does Nifi ingest large files?

I should add that if you're comfortable with that sort of volatile behavior a better path to consider is to setup a RAM-Disk and just run a persistent content repository on that. It will survive process restarts, give better memory/heap behavior (by a lot), but you'll lose data on system

Re: IllegalStateException

Conrad, Am looking into this now. Could you please file a JIRA. Given the error it is showing there is without any doubt a coding error. Thanks Joe On Thu, Oct 27, 2016 at 9:35 AM, Conrad Crampton wrote: > Hi, > > I’m getting this exception thrown by most of my

data flow from one s3 bucket to another

Has anyone implemented data copy from one s3 bucket to another. i would greatly appreciate if you can share with me your sample processors configuration. Thanks Rai

Re: nifi is running out of memory

Looking at this line [1] makes me think the FetchS3 processor is properly streaming the bytes directly to the content repository. Looking at the screenshot showing nothing out of the ListS3 processor makes me think the bucket has so many things in it that the processor or associated library isn't

Re: nifi is running out of memory

Hello, Are you running with all of the default settings? If so you would probably want to try increasing the memory settings in conf/bootstrap.conf. They default to 512mb, you may want to try bumping it up to 1024mb. -Bryan On Thu, Oct 27, 2016 at 5:46 PM, Gop Krr wrote:

Re: nifi is running out of memory

moving dev to bcc Yes I believe the issue here is that FetchS3 doesn't do chunked transfers and so is loading all into memory. I've not verified this in the code yet but it seems quite likely. Krish if you can verify that going with a larger heap gets you in the game can you please file a JIRA.

Re: Frontend for Nifi support / mailing list

Oh ... I was under the impression that there already was a nabble listing for nifi : http://apache-nifi.1125220.n5.nabble.com/Apache-NiFi-f1.subapps.html On Thu, Oct 27, 2016 at 1:36 PM, Tony Kurc wrote: > List has been created on nabble, it will be empty until messages

Re: How does Nifi ingest large files?

Thank you Bryan, yes that's what I meant and it makes sense to me. Only a further question: is this stream parallelized if needed on the (hypothetical) Nifi cluster or it is managed by a single node? Cheers Monica

Re: How does Nifi ingest large files?

Monica, Are you asking what does NiFi do when it picks up a large file from the filesystem using a processor like GetFile? If so, it will stream the content of that file into NiFi's content repository, and create a FlowFile pointing to that content. As far as NiFi is concerned the content is

Re: How does Nifi ingest large files?

In the case of a GetFile processor it is managed by a single node since the file being picked up is on the local filesystem of one of the nodes. There are other approaches to parallelize work... If you had a shared network location you can use ListFile + FetchFile in a certain way so that one

Re: How does Nifi ingest large files?

Indeed. I went ahead and configured my dev cluster to use RAM-disk for content and flowfile repositories and turned back on FileSystemRepository and WriteAheadFlowFileRepository respectively. As long as the content/provenance archive is off, I'm good wrt compliance. The performance seems great so