Re: Fwd: GetKafka Processor Issue

Indus Sorry for the slow response. Is there anyone that can look into this? Thanks Joe On Oct 8, 2015 8:37 PM, "indus well" wrote: > Hello NiFi Team: > > This issue keeps happening. Please see the attached log file for the full > stack dump. > > Thanks, > > Indus > > On Sun, Oct 4, 2015 at 11:

Fwd: GetKafka Processor Issue

Hello NiFi Team: This issue keeps happening. Please see the attached log file for the full stack dump. Thanks, Indus On Sun, Oct 4, 2015 at 11:31 PM, indus well wrote: > Thanks, Joe. Not sure what happened, but it appeared to be working when I > turned both on. I will keep monitoring the flow

Re: S3 processor with Proxy option - Feature request

Thanks Joe, I will let my engineering team know. On 10/8/15, 2:51 PM, "Joe Witt" wrote: >Chakri, > >That sounds great. Are you interested in contributing back that >modification that meets the need? > >We have a contributor guide that should be of help and we're happy to >help otherwise. > >Tha

Re: S3 processor with Proxy option - Feature request

Chakri, That sounds great. Are you interested in contributing back that modification that meets the need? We have a contributor guide that should be of help and we're happy to help otherwise. Thanks Joe On Thu, Oct 8, 2015 at 5:40 PM, Chakrader Dewaragatla wrote: > Nifi Team – It would be hel

S3 processor with Proxy option - Feature request

Nifi Team – It would be helpful to have Proxy options in S3 processors (Fetch/Put). We modified nifi aws module to support Proxy as a work around and it works like charm. Thanks, -Chakri The information contained in this transmission may contain privileged and c

Re: Nifi & Spark receiver performance configuration

Aurelien, The way that swapping works in NiFi is that when the number of FlowFiles in a particular queue builds up past a certain point, NiFi will write those files to disk and drop them from the Java heap in order to avoid running out of heap space. Then, when the number of FlowFiles in the que

Re: FetchS3Object processor

Thanks Joe, I was about to request a feature like this (ListS3Objects). From: Joe Skora mailto:jsk...@gmail.com>> Reply-To: "users@nifi.apache.org" mailto:users@nifi.apache.org>> Date: Thursday, October 8, 2015 at 11:39 AM To: "users@nifi.apache.org

Re: FetchS3Object processor

There is already a ticket (NIFI-840 ) in the hopper to create a ListS3Objects processor that can track bucket contents and trigger FetchS3Object. In the meantime, if you want it triggered automatically (as opposed to generate flow file) you may be ab

Re: FetchS3Object processor

Thanks Aldrin. GenerateFlowFile helped to trigger. Thanks, -Chakri From: Aldrin Piri mailto:aldrinp...@gmail.com>> Reply-To: "users@nifi.apache.org" mailto:users@nifi.apache.org>> Date: Thursday, October 8, 2015 at 11:22 AM To: "users@nifi.apache.org

Re: FetchS3Object processor

Chakri, The FetchS3Object processor does not act as a source directly and is instead driven by incoming FlowFiles. The use case was that one could utilize Expression Language to interact with buckets and paths in a dynamic fashion. Alternatively, if you are looking to just grab a specific file,

Re: logging GetFile processes?

"what would be the flow to write the filename and date to a text file?" You can use ReplaceText to write out the data in whatever text oriented format you like (CSV, JSON, etc..). Then it is a matter of getting that object to Mongo. I don't know Mongo well enough myself to know the best options

Re: FetchS3Object processor

Thanks Joe. I have S3fetch and putfile processors setup, files are not fetching despite files exist on s3. Am I missing anything ? Does S3fetch processor need any additional processor to feed data ? Thanks, -Chakri From: Joe Skora mailto:jsk...@gmail.com>> Reply-To: "users@nifi.apache.org

Re: logging GetFile processes?

Joe, it appears that the version I'm using doesn't currently support putMongo. So until I can fix that, what would be the flow to write the filename and date to a text file? ExecuteScript? v/r, Ron Sawyer On Thu, 10/8/15, Joe Witt wrote: Subjec

Re: logging GetFile processes?

Yep, that's it, thank you!!! v/r, Ron Sawyer On Thu, 10/8/15, Joe Witt wrote: Subject: Re: logging GetFile processes? To: users@nifi.apache.org Date: Thursday, October 8, 2015, 1:04 PM Ron, Yep makes sense.  We'll try to put together a fl

Re: Provenance repo questions

Oleg, The experience around the NiFi provenance events today is around a "nifi cluster". When you query that cluster it is asking all nodes the question. In each node it is true you can configure multiple storage points for provenance events. The reason you would do this is if you want to distr

Re: logging GetFile processes?

Ron, Yep makes sense. We'll try to put together a flow template for you to check out. Basic gist is - > ReplaceText -> PutMongo In the ReplaceText processor you can use the expression language to make new content which would be a JSON document containing the filename and entry time and whatev

Re: logging GetFile processes?

Would prefer to log them in a mongodb collection, just the file name and date processed is all. I know this is being logged in the app log I think but need something that is publicly viewable so others can know files are in fact being picked up and processed. v/r, Ron Sawyer --

Provenance repo questions

Guys Reading Admin Guide I see that there can be multiple provenance repos. Few questions I have about it: 1. What would be the use case where I want to do that? 2. Would it still maintain a "single repo" view for access/query or I would access each repo individually? 3. In cluster mode, does ea

Re: logging GetFile processes?

Ron, How do you want to keep the file names? In a file on disk? In a database? David. On 10/8/15, 12:38 PM, "Ron Sawyer" wrote: >Hopefully this isn't asking too much, but I have several getFile processes >running and would like to log the fil

logging GetFile processes?

Hopefully this isn't asking too much, but I have several getFile processes running and would like to log the file names and times they process into a mongo collection to easily track what and when files have been handled, can someone tell how to do this or point me in a direction? Thanks. v/

Re: Need help in nifi- flume processor

> If you plan to use NiFi for the long term, it might be worth investing in > converting your custom Flume components to NiFi processors. We can help you > get started if you need any guidance going that route. +1. Running Flume sources/sinks is meant as a transition step. It's really useful if

RE: Nifi & Spark receiver performance configuration

Hello. I'm testing on a VM 8vCPU (E5606 2.13Ghz) / 16Go. I just have a GenerateFLowFIle which send data to an output port for Spark. Here, the performance is very good, I can generate a huge number of flow files. My spark job is configured as local[4], and use 3 receivers. It just doing a si

Re: Nifi & Spark receiver performance configuration

Hello, When you say you were unhappy with the performance, can you give some more information about what was not performing well? Was the NiFi Spark Receiver not pulling messages in fast enough and they were queuing up in NiFi? Was NiFi not producing messages as fast as you expected? What kind of

Re: Need help in nifi- flume processor

Hi Parul, It is possible to deploy a custom Flume source/sink to NiFi, but due to the way the Flume processors load the classes for the sources and sinks, the jar you deploy to the lib directory also needs to include the other dependencies your source/sink needs (or they each need to individually

Re: Nifi 0.3.0 on java-1.8.0 (S3 processor error)

Chakri, Great, thanks for the update! It looks like this is a known issue with the version of the AWS SDK that we are using. The bug ticket is available at: https://github.com/aws/aws-sdk-java/issues/444 I have submitted a bug ticket against NiF

Nifi & Spark receiver performance configuration

Hello. I’m doing some experimentations on Apache Nifi to see where we can use it. One idea is to use nifi to feed a spark cluster. So I’m doing some simple test (GenerateFlowFile => spark output port and a simple word count on spark side. I was pretty unhappy with the performance out of t

Re: FetchS3Object processor

Chakri, The Amazon docs explain here that S3 doesn't support pathes within a bucket, but you can embed path-like naming into the "Object Key" parameter to FetchS3Object. For your example, you would use bucket="s3://" and key="/