Re: ExecuteXQuery only returns single item when there are multiple matching items in FlowFile

Rick, Quick initial response... I think the behavior makes sense because if it did just match all the items with desired titles you're left with blocks of valid XML but as a whole an invalid XML document. What I believe is the expected behavior then is to use SplitXML to split out the item

Re: NiFi Inspector

Totally on the same page Rick. See here [1] for a listing of all our current feature proposals and here [2] for the one specific to what you just mentioned. Let us know if you think we should add more to that. It is awesome you're already at that point to be thinking of things like that. Keep

Re: Determine which Nifi processor eating CPU usage

Elli, In the majority of cases what Mark suggested will help you pinpoint the offending processors. If it is not clear which are the offending processes/extension then it can require some fairly involved digging. For such things I have found the following articles provide good examples of how

Re: Suggestions for good approach to ETL strategy

Thanks for jumping in Lee! Mark, This is a great writeup. We should turn this into a blog w/full explanation and template. Great use case and you just gave us a perfect user perspective/explanation of how you're thinking of it. We will make that happen quickly.

Re: Nifi - InvokeHTTP - retrying on 3xx with Location header issue

m > that would be awesome. > > [1] > http://www.mkyong.com/java/java-httpurlconnection-follow-redirect-example/ > [2] https://issues.apache.org/jira/browse/NIFI-1086 > > Joe > - - - - - - > Joseph Percivall > linkedin.com/in/Percivall > e: joeperciv...@yahoo.com > > &g

Re: Managing flows

Darren, A single NiFi instance (on one node or a cluster of 10+) can handle *many* different flows. Thanks Joe On Wed, Nov 11, 2015 at 10:00 AM, Darren Govoni wrote: > Mark, >Thanks for the tips. Appreciate it. > > So when I run nifi on a single server. It is

Re: High CPU usage in FileSystemRepository.java

sorry - i see now :-) Thanks for the analysis. Will dig in. Joe On Fri, Nov 13, 2015 at 12:28 AM, Joe Witt <joe.w...@gmail.com> wrote: > Adam, > > Are you on a recent master build? > > Thanks > Joe > > On Fri, Nov 13, 2015 at 12:27 AM, Adam Lamar <adamond...@

Re: Nifi Migration

Naveen You can save the flow as a template and import that. It today has limitations such as environmental mapping is too limited and you should know it won't copy sensitive properties. Another option is to copy the flow.xml.gz from the current setup into production. We need to do more though

Re: Queue wasn't emptying

t; > Thanks > -Mark > > > [1] https://issues.apache.org/jira/browse/NIFI-1155 > > > On Nov 12, 2015, at 2:32 AM, Christopher Hamm <em...@christopherhamm.com> > wrote: > > where does it dump? I cant get mine to expire fast enough. It is dev test I > want to b

Re: Why does PutFile create directories for you but PutHDFS does not?

Right - PutHDFS definitely creates directories on the fly if necessary (based on the path attribute of the flowfile) and if it has permissions to do so. Thanks Joe On Thu, Nov 12, 2015 at 8:34 AM, Bryan Bende wrote: > I think PutHDFS always creates them so it isn't an option

Re: processor for validation

Naveen The approach Matt suggested will help. Perhaps a more complete approach would be to be able to apply a true schema validation to JSON documents. I did a quick search for apache friendly Java libraries and found this [1]. Perhaps we should have a processor that lets users validate JSON

Re: GetSQS causes high CPU usage

Adam, Just wanted to follow up on this. Have you had any better results and should we put a JIRA in behind what you're seeing? Thanks Joe On Tue, Oct 20, 2015 at 7:58 PM, Adam Lamar wrote: > Adam, > > Thanks for the reply! > > Amazon supports (and recommends) long

Re: Question on EvaluateJSONPath

Chris, Let me frame it differently and lets see if that does what you'd like. Input is a CSV file. Transform is to filter out CSV entries which do not match desired criteria. Output is a JSON file. If that is correct how about this flow: SplitText (1 line at time) RouteText (to find matching

Re: Using InvokeHTTP to send GET but without content

Mark, What you're trying to with regard to auditing is exactly what the ReportingTask extension is designed to support. It has access to the provenance events which include for each event the flow file attributes. If you built a ReportingTask which pulls whatever attributes you want and formats

Re: Queue wasn't emptying

Elli, Can you share your flow configuration that we could possibly use to replicate this? Perhaps turn this into a JIRA and attach that. Also anytime you see something that appears 'stuck' please try to get a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread we'll see it there

Re: logging GetFile processes?

Ron, Yep makes sense. We'll try to put together a flow template for you to check out. Basic gist is - > ReplaceText -> PutMongo In the ReplaceText processor you can use the expression language to make new content which would be a JSON document containing the filename and entry time and

Re: logging GetFile processes?

a text file? ExecuteScript? > > v/r, > Ron Sawyer > > -------- > On Thu, 10/8/15, Joe Witt <joe.w...@gmail.com> wrote: > > Subject: Re: logging GetFile processes? > To: users@nifi.apache.org > Date: Thursday, October 8,

Re: StoreInKiteDataset help

Chris, Are you seeing errors in NiFi or in HDP? If you're seeing errors in NiFi can you please send us the logs? Thanks Joe On Thu, Oct 15, 2015 at 3:02 PM, Christopher Wilson wrote: > Has anyone gotten Kite to work on HDP? I'd wanted to do this very thing but > am

Re: S3 processor with Proxy option - Feature request

Chakri, That sounds great. Are you interested in contributing back that modification that meets the need? We have a contributor guide that should be of help and we're happy to help otherwise. Thanks Joe On Thu, Oct 8, 2015 at 5:40 PM, Chakrader Dewaragatla

Re: Nifi cluster setup - Safe mode error

...on startup one is chosen automatically. But there are scenarios where as an admin you have to manually tell it how to behave. We agree you should not have to and there are some tickets and a feature proposal covering our plans to solve this. On Oct 11, 2015 8:20 AM, "Corey Flowers"

Re: Custom classpath (using MapR-FS in lieu of HDFS)

. Same system should work with multiple providers. Thanks Joe On Mon, Oct 12, 2015 at 10:16 AM, Andre F de Miranda <af...@fucs.org> wrote: > Joe, > > Epic! > > Exactly what I was looking for. (Also, apologies for getting you to > LMGTFY... :-) ) > > Cheers > >

Re: Custom classpath (using MapR-FS in lieu of HDFS)

Hello Andre, Found on Github here is someone who already got NiFi working with MapR and documented their approach. https://github.com/bbukacek/nifi-hadoop-libraries-bundle Thanks Joe On Mon, Oct 12, 2015 at 9:46 AM, Oleg Zhurakousky wrote: > May be I am missing

Re: Fwd: GetKafka Processor Issue

t; dump for you if it happens again. >> >> On Sun, Oct 4, 2015 at 9:34 PM, Joe Witt <joe.w...@gmail.com> wrote: >> >>> Indus, >>> >>> Can you please have them both running to get it back into the bad >>> state then run /bin/nifi.sh dump

Re: site to site setup - Remore instance not configured

Do we have a JIRA out for making site to site docs better with setup guides/etc..? This has always been tricky... On Tue, Oct 13, 2015 at 5:06 PM, Chakrader Dewaragatla wrote: > Thanks Matthew it worked, no errors this time. > As noted below, we would like to

Re: getmongo processor exception

Subhash, I don't think enough folks are familiar here to help. Please feel free to file a JIRA for that mongo processor to add support for this option. Thanks Joe On Tue, Nov 17, 2015 at 3:19 AM, Subhash Parise wrote: > Hi NiFi experts, > > Do any one know how to

Re: Database Lookup Approach

Indus, Let's project out one order of magnitude. That puts you at just a bit more than 500 lookups per second. You will want to consider the properties of the data itself and the properties of the database. This helps you decide on the appropriate caching and querying strategy. If the data

Re: Nifi doesn't listen on port 8080 in some Linux distributions

Chris, Also please take a look in the 'logs/nifi-app.log' and 'logs/nifi-bootstrap.log' to see if there is anything interesting there. Thanks Joe On Thu, Aug 27, 2015 at 11:53 AM, Corey Flowers cflow...@onyxpoint.com wrote: Check your firewalls. If you have a running instance and can't open

Re: Cron syntax

Alan nailed it. Quartz supported the finer resolution and some folks found that useful. It would be totally fair to ask that given 5 fields we just add a 6th with an implied 0 (for example). A good JIRA if you feel that is helpful. On Wed, Sep 9, 2015 at 8:01 AM, Alan Jackoway

Re: Hacker News

Yeah John thanks for pointing this out. That resulted in the apache nifi site having 30,000+ visits yesterday. Wow! Some good questions worth responding too but I think this is the right forum so probably will hold off until the questions occur here on users@nao. Thanks Joe On Wed, Sep 9,

Re: Run process after getting a file

Broken pipe: java.io.IOException: Broken pipe >> >> >> >> I'm using GetFile->ExecutingStreamProcess do i miss something? i run the >> jar from a terminal and works fine. >> >> Thanks >> >> >> >> 2015-09-25 19:01 GMT-05:0

Re: GetKafka Processor Issue

Indus, Can you please have them both running to get it back into the bad state then run /bin/nifi.sh dump This will write out the stack dump to the application log. Can you send us this? The stack trace could be quite telling. Thanks Joe On Sun, Oct 4, 2015 at 10:30 PM, indus well

Re: Feeding & Consuming data to & from Nifi

Krish, Yes. If it is two different times that the same 'concept of an object' goes through NiFi then there will be two provenance trails. You can add attributes to be indexed though in NiFi by adjusting the appropriate properties for provenance indexing in nifi.properties. With this you can

Re: ExtractText Capturing Groups Limit (1-40)

Adam, We really should just allow it to be configurable. We tend to be conservative on such cases but you should have control. Interested in filing a JIRA? Thanks Joe On Wed, Sep 23, 2015 at 10:01 AM, Adam Williams wrote: > I've been playing with the ExtractText

Re: Run process after getting a file

command path = java command args = -jar;${filename} That might do the trick. Thanks Joe On Fri, Sep 25, 2015 at 7:59 PM, Rick Braddy wrote: > Expression language in ExecuteStreamProcess could be your friend here to use > the filename attribute to set the process arguments.

Re: GetHTTP Usage with Dynamic Filenames

Indus, We should probably have another JIRA to have the InvokeHTTP processor capture response headers and store them as flow file attributes or at least as a single attribute with the headers on it - something. However, the core need you have is totally doable. I'd recommend the following flow:

Re: Generate flowfiles from flowfile content

David, I think if i read your case correctly this should be supported really well. The flow would be something like: GetSQS -> SplitJson -> EvaluateJsonPath -> FetchS3Object In SplitJSON you'll break apart the original object into smaller valid JSON objects. In evaluate JsonPath you'll

Re: Generate flowfiles from flowfile content

Bryan - you may be right that ExtractText will be the right play once splitjson is done doing its thing. Perhaps either will work. Maybe we can show either or. If the schema is fairly well known i'm thinking extract json would be the winner. thanks Joe On Wed, Sep 23, 2015 at 4:04 PM, Bryan

Re: connector "bends"

Charlie, You can double click on the line and it will create a bend point for you. You can also double click on that bend point to remove it. Not the first time someone has asked and i don't see it in the docs. Anybody know if this is in the docs somewhere? Thanks Joe On Tue, Dec 1, 2015 at

Re: Content replacement

digit month, group2 is date and group1 > is ddMMM > > > > > > > > *From:* Joe Witt [mailto:joe.w...@gmail.com] > *Sent:* Thursday, December 03, 2015 7:00 PM > *To:* users@nifi.apache.org > *Subject:* Re: Content replacement > > > > Your use case

Re: Wait from multiple inputs before ending the flow

Louis-Etienne, My initial thought is your idea with MergeContent is the right one. However, the issue there is not just the combining of the data but the 'what does merging truly mean in that case'. So it is a bit undefined what the next step will be. Merge the content? If so, how? What is

Re: The EndPoints of GetTwitter

Is there any reason we'd not just let the user enter an override URL to pull from? On Tue, Dec 8, 2015 at 9:02 PM, Bob Zhao wrote: > Thanks for the quick response, Aldrin! > > I know we are not NSA. But there are at least 2 3rd parties who provide the > Firehose

Re: Recovering from heap memory exceeded

Chris, I think there are some important details missing. In the case of auto resume being false on restart there is no opportunity for those processors to be stuck as they'll never receive a thread. Can you share a thread dump? Feel free to create a JIRA for what you're seeing and attach the

Re: SplitText doesn't seem to work with backpressure

Chris, This is a really common approach and it is backpressure that makes it work and be so successful. So definitely want to help you get to the bottom of it. Can you perhaps make a template of your config and share that so we can see what might be happening? Thanks Joe On Fri, Dec 4, 2015

Re: Mirroring nifi lists on gmane

Is anyone aware of any reasons not to do this? On Fri, Dec 4, 2015 at 8:59 AM, Nigel Jones wrote: > Would it be ok if I asked gmane to mirror these nifi lists on the gmane nntp > server? > > I personally find it a little easier to work through & catch up with >

Re: Content replacement

Your use case description was excellent and helped expose a powerful and simple feature enhancement. https://issues.apache.org/jira/browse/NIFI-1249 Thanks Joe On Thu, Dec 3, 2015 at 9:21 PM, Joe Witt <joe.w...@gmail.com> wrote: > Chandu, > > In building a template to s

Re: NoSuchElementException

adding to Oleg's comment we should have this in a formal nifi release very soon. In fact the great NIFI-655 push we've been holding on just went down so think we're going to kick out an RC very soon. Thanks Joe On Tue, Dec 1, 2015 at 11:45 AM, Oleg Zhurakousky

Re: DistCp from Amazon S3 to HDFS

Hello So we have FetchS3 and PutHDFS and a series of interesting in between processes to help. So that would get you most of the way there. How to get the listing/know what to pull from S3? That part I'm not sure about. This would make for a great example/template for us to post (as would the

[ANNOUNCE] Apache NiFi 0.4.0 Release

Hello, The Apache NiFi team would like to announce the release of Apache NiFi 0.4.0. Apache NiFi is an easy to use, powerful, and reliable system to process and distribute data. Apache NiFi was made for dataflow. It supports highly configurable directed graphs of data routing, transformation,

Re: DistCp from Amazon S3 to HDFS

;> case that I've used with S3 is to setup S3 to notify >>> when an object arrives via SQS. Then have GetSQS get that notification >>> and then pull the data via FetchS3Object. >>> So you could fairly easily setup a GetSQS -> EvaluateJSONPath -> >>> FetchS

Re: Nifi Kite-dataset URI issue

In addition to Alan's question to understand what the actual error is please note that when i try your config I get complains about the target dataset uri. If i have for instance dataset:hive:default/ctest1 As shown in your screenshot, the validation error I see is: 'Target dataset URI'

Re: Data Ingestion forLarge Source Files and Masking

ile is not processing. > Then I randomly changed the content of the file and it was immediately > processed. > > My question is what is the new file selection criteria for "ListFile" ? Can > I change it only to file name ? > > Thanks in advance. > > -Obaid > > &g

Re: Data Ingestion forLarge Source Files and Masking

date(as current one) unique file names, checksum etc. > > Thanks again man. > -Obaid > > > On Monday, January 4, 2016, Joe Witt <joe.w...@gmail.com> wrote: >> >> Hello Obaid, >> >> The default behavior of the ListFile processor is to keep track of

Re: Data Ingestion forLarge Source Files and Masking

in when my >> entire workflow completes. So that thus could be an reference for other :). >> >> -Obaid >> >> On Monday, January 4, 2016, Joe Witt <joe.w...@gmail.com> wrote: >>> >>> Obaid, >>> >>> You make a great point. >>

Re: Upgrade 0.3.0 to 0.4.1

Hello Chakri, Take a look here https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi And here https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance Thanks Joe On Mon, Jan 4, 2016 at 8:02 PM, Chakrader Dewaragatla wrote: > Hi –

Re: Testing a nifi flow via junit

Vincent, As Aldrin mentioned there is an effort underway to help with integration testing. But I'm not sure yet if it will help with what you're looking to do from a performance baseline perspective. I totally get Mark's cautionary comments but given your response it does seem like there is

Re: NIFI connecting to Activemq

Christopher, Is there any error/feedback showing up in the UI or in the logs? Thanks Joe On Wed, Jan 6, 2016 at 5:19 PM, Christopher Hamm wrote: > What am I doing wrong with hooking up my activemq jms get template? I put > stuff into the activeMQ and NIFI wont get

Re: Data Ingestion forLarge Source Files and Masking

Hello Obaid, At 6 TB/day and average size of 2-3GB per dataset you're looking at a sustained rate of 70+MB/s and a pretty low transaction rate. So well within a good range to work with on a single system. 'I's there any way to by pass writing flow files on disk or directly pass those files to

Re: Database Lookup Approach

Joe. Yes, the referenced lookup data does not change frequently so > your built-in caching strategy sounds great. Would you expand on how to > implement the built-in caching mechanism as well as refreshing the cache in > NiFi please? > > Thanks, > > Indus > > On Sat, Nov 14, 2015 at 12

Re: NIFI connecting to Activemq

ser’s should never see NPE), so that can be filed. What I am trying to >> figure out is the condition that triggers it. May be if you shut down NIFI, >> delete all the logs and restart so you can get a fresh data. . . >> >> Cheers >> Oleg >> >> On Jan 9, 2016,

Re: How to validate records in Hadoop using NiFi?

Hello Sudeep, "Which NiFi processor can I use to split each record (separated by a new line character)" For this the SplitText processor is rather helpful if you want to split each line. I recommend you do two SplitText processors in a chain where one splits on every 1000 lines for example

Re: Nifi Kite-dataset URI issue

; ·URI Validates just fine in NIfi >> >> >> >> >> >> >> >> >> >> ·Hadoop Configuration files: >> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/lib/hadoop/etc/hadoop/hdfs-site.xml,/opt/cloudera/parcels/CDH-5.4.0

Re: Mirroring nifi lists on gmane

to many other indexers in general). In my searches Nabble and >> Markmail come up much more often and more pleasant to use. >> >> Nabble can be allowed to do 2-way posts and is also embeddable, nice light >> theme. >> >> Nabble (forum-like) http://apache-nifi.

Re: Nifi Kite-dataset URI issue

ucleus-rdbms-3.2.9.jar >> hadoop-mapreduce-client-common-2.6.0-cdh5.4.2.jar >> hadoop-mapreduce-client-core-2.6.0-cdh5.4.2.jar >> hive-exec-1.1.0-cdh5.4.2.jar >> hive-metastore-1.1.0-cdh5.4.2.jar >> hive-shims-1.1.0-cdh5.4.2.jar >> jdo-api-3.0.1.jar >&

Re: Nifi download archives - 0.3.0

Hello Chakri I updated the downloads page to provide a convenient link to the release archives. Thanks Chris for the pointer to that! Joe On Mon, Dec 28, 2015 at 6:36 PM, Chakrader Dewaragatla wrote: > Thanks Chris. > > From: Chris Lim

Re: Multiple Content Repositories

Chris, It is something which occurs automatically and behind the scenes. Under normal circumstances there will be many FlowFiles written to the same content claim they'll just each have different offsets. It is more aligned with how disks work in terms of efficiently writing data, efficiently

Re: Output port and 2 way SSL

Hello, Ports on the root group automatically become remote group ports. Input ports can receive input from external sources and output ports can be used by external sources to pull data. So long as those external sources are using the site-to-site client lib you should be good. A longer

Is it possible to create instances of a processor dynamically based on HTTP response?

[posting on behalf of Kacem as there seems to be some issue we've not sorted with his mailing attempts] Hello everyone, So my case is as the following, i'm trying to retrieve JSON files from a social network (Facebook), and based on some values in the file of the HTTP response i want to invoke a

Re: queued files

Charlie, The fact that this is confusing is something we agree should be more clear and we will improve. We're tackling it based on what is mentioned here [1]. [1] https://cwiki.apache.org/confluence/display/NIFI/Interactive+Queue+Management Thanks Joe On Thu, Nov 19, 2015 at 10:30 PM, Corey

Re: ExecuteProcess processor cannot be stopped/started

Chris, If you run into a case of a stuck thread we'd love to see the stack trace. You can generate one by running 'bin/nifi.sh dump' and sending us the logs. I believe it is bootstrap.log specifically. If you have a processor which immediately gets stuck after startup where it takes a thread

Re: Viewing flow file contents results in error

aining millions of lines. I guess that might be a limitation I'll have > to find a workaround for. I've managed to recover it but I only noticed this > trying to troubleshoot why the system stopped running the jobs properly. > > On Tue, 24 Nov 2015 at 1:06 AM, Joe Witt <joe.w...

Re: ExecuteProcess processor cannot be stopped/started

you run "bin/nifi.sh dump" it is often easier to specify a > filename there > such as "bin/nifi.sh dump thread-dump.txt" so that it writes the thread dump > to the > filename specified, rather than writing it to the logs. > > Thanks > -Mark >

Re: NiFi processor for Redis

Sudeep, Hello. At this time there are no apache nifi redis processors to push/pull data with Redis that I am aware of. Something you might be interested in contributing or contributing to? Thanks Joe On Wed, Jan 13, 2016 at 12:13 AM, sudeep mishra wrote: > Hi, > >

Re: how to sort json array in dataflow

Hello Roland I don't believe we have out of the box support for that but it would make for likely a well scoped custom processor. You mention support for sort syntax. Can you share what that might look like? Thanks Joe On Sun, Jan 10, 2016 at 9:45 PM, 彭光裕 wrote: > Hi,

Re: Best way to generate a new filename

You could generate a unique filename with that extension by using the following in Update Attribute Add property: property name: filename property value: ${uuid}.json You can also set a directory structure with time stamping this way as well. For instance set another attribute on that

Re: Failure when running a workflow created from a template from another NiFi version.

Keith, Can you collect more information on this from the nifi-app.log? Not in valid state should be associated with some exceptions and such that ideally will be in the logs. Thanks Joe On Wed, Jun 8, 2016 at 1:00 PM, Keith Lim wrote: > Any help on this before while I am

Re: NiFi - Flow Backup Strategy

Tom, I suspect some others might add some additional info here but to achieve the back-up of the configuration such that you could quickly restore you'd generally just need to save the conf directory content. The flow configuration should port well to another node including the sensitive property

Re: Release Dates

Hello Your request for dates on the release notes seems fair to me. I went ahead and added them there assuming nobody has any objection. We generally try to follow a rough 6-8 release cycle. It will be a little longer or a littler shorter sometimes. Right now for example we're trying to close

Re: How to parse GetTwitter output?

Pat, GetTwitter does produce single tweet outputs and the example you provided I believe is actually a single tweet. Try pasting that into a json formatter and I think you'll find it is a single tweet. 'created_at' is used on several entity types in twitter is appears so that isn't telling you

Re: PutKafka won't Put! "Failed while waiting for acks from Kafka"

Pat Could you please make sure we are seeing the full stacktrace from the app log? If we could get a full log that would he even better. Thanks Joe On Jun 11, 2016 1:17 PM, "Pat Trainor" wrote: > Andrew, > > Thanks for writing. > > *Nifi:* > nifi-0.6.1 > *Kafka:* >

Re: GetKafka exception

Not bothering. This is good info we should document On May 27, 2016 11:47 AM, "Igor Kravzov" wrote: > Guys, I found the problem. It was my fault. > Kafka broker configured as fully qualified domain name. > What happens is when you call zookeper it returns FQDN but

Re: About get data in loop and transform dbf file

Hello Shoping I am not aware of any existing processors in nifi that deal with dbf directly. You could certainly a custom processor in place or use the ExecuteScript processors(s) to extend it for that. The pattern you describe though is very doable. Perhaps this library could be a good basis

Re: How to access the counter and provenance Info

Hello Kumiko The ProcessSession API exposes to you the ability to adjust counters [1]. You would simply call that in your on trigger method with something like session.adjustCounter("mycounter",1L,true); [1]

Re: OutOfMemoryError from ListSFTP

Huagen, You can possibly avoid it by giving NiFi a large heap size. The downside is that there is a good library I am aware of which will let us handle large remote listing as nicely as we'd like. It does tend to be a 'send small request', 'wait for a large reply', 'then process large reply'

Re: Custom Controller Service

...i wonder if we've done the same ourselves or given similar advice to others. This feels like something the framework should make available as an expected lifecycle point. I could see it making sense for such cases that we would allow 'OnScheduled' to be on controller services and let the user

Re: Will Nifi eventually become Hortonified?

Pat, I'll respond wearing my Apache member and Apache NiFi PMC hats. No. NiFi is of the Apache Software Foundation and thus cannot be bought. That said to the spirit of your question it is fair to ask whether this community is diverse in its representation of members and who they are

Re: Nifi cluster features - Questions

Chakri, Would love to hear what you've learned and how that differed from the docs themselves. Site-to-site has proven difficult to setup so we're clearly not there yet in having the right operator/admin experience. Thanks Joe On Sun, Jan 10, 2016 at 3:41 PM, Chakrader Dewaragatla

Re: One-time Processors

Lars, First, welcome to NiFi. So you'll initiate the flow from NiFi through some manual trigger and that part makes sense/sounds straightforward. But then the question is 'once the flow is done' I'd like it to shut down. What are the processors/things that begin the flow? We've recently had

Re: Data Ingestion forLarge Source Files and Masking

t; > Now I made as below: > java.arg.2=-Xms5120m > java.arg.3=-Xmx10240m > > latest jstate & iostate output are attached. > To me it is still slow, no significant improvements. > > -Obaid > > On Thu, Jan 14, 2016 at 12:41 PM, Joe Witt <joe.w...@gmail.com> wrote: >&g

Re: NIFI connecting to Activemq

> > On Jan 9, 2016 8:25 PM, "Joe Witt" <joe.w...@gmail.com> wrote: >> >> Chris, >> >> Thanks for sending the screenshot. The NullPointerException is >> trivially reproduced using your settings. I didn't even need a JMS >> server to cause it.

Re: Data Ingestion forLarge Source Files and Masking

t; Hi Joe, > > Please find attached jstat & iostat output. > > So far it seems to me that it is CPU bound. However, your eyes are better > tan mine :). > > -Obaid > > On Thu, Jan 14, 2016 at 11:51 AM, Joe Witt <joe.w...@gmail.com> wrote: >> >> Hello >>

Re: MergeContent: Correlation Attribute Name syntax for matching syslog events

lt. > > I wonder if this is linked to > https://issues.apache.org/jira/browse/NIFI-1438 ? > > > > > > On Mon, Feb 8, 2016 at 12:55 AM, Joe Witt <joe.w...@gmail.com> wrote: > >> Andre >> >> I believe until this next release which is 0.5.0 merge

Re: NiFi logging

There is also the provenance data itself which captures this and there is the LogAttributes processor. Multiple options to get the information about the flow out. Thanks Joe On Thu, Feb 11, 2016 at 10:34 AM, Andrew Grande wrote: > Take a look at ReportingTask, this

Re: ListS3 processor?

Kyle Let us know if that doesn't get you what you need. We have a decent set of templates but I didn't see one that demonstrates interaction with amazon services. Thanks Joe On Jan 30, 2016 12:56 PM, "Joey Frazee" wrote: > Kyle, > > I think you can do what you want

Re: ListS3 processor?

e details: > https://adamlamar.github.io/2016-01-30-monitoring-an-s3-bucket-in-apache-nifi/ > > Hope that helps, > Adam > > > On 1/30/16 9:29 PM, Joe Witt wrote: >> >> Kyle, >> >> The ideal case for communicating how to do this would be both a >> template and a

Re: Split header from CSV and use for JSON

Ian Makes sense. Happy to help put together a template to see if we can get you closer. Working on some 050 release items at the moment but will flag this to come back Thanks Joe On Fri, Jan 29, 2016 at 8:43 AM, Hellstroem Christian (AE/PJ-ASC) wrote: > Hi,

Re: Nifi 0.50 and GetKafka Issues

Sorry to clarify it working against all three of these at once: - Kafka 0.8.1.1 - Kafka 0.8.2.1 - Kafka 0.9.0.1 Thanks Joe On Mon, Feb 22, 2016 at 4:12 PM, Joe Witt <joe.w...@gmail.com> wrote: > All just as a point of reference we now have a live system running > that is o

Re: Nifi 0.50 and GetKafka Issues

and my kafka > processor started reading my 0.8.2.1 stream. Not elegant but glad it worked. > > > Respectfully, > > Kyle Burke | Data Science Engineer > IgnitionOne - Marketing Technology. Simplified. > Office: 1545 Peachtree St NE, Suite 500 | Atlanta, GA | 30309 > Direct: 40

Re: How to combine data from multiple processors?

Sudeep A funnel simply allows you to take multiple connections and combine them into a single connection. Think of it as merging queues into a single queue. This is useful when you want to control prioritization, simplify connection logic to some downstream component, etc.. For the case you're

Re: Authentication error - access denied exporting template

We are working to have an RC for it any day. We're very close so today is feasible. Thanks Joe On Mon, Feb 22, 2016 at 10:18 AM, Conrad Crampton wrote: > Matt, > Thanks for the update. I trawled the user list archive for anything similar, > didn’t think to check

Re: Processor Signaling

Frank, Absolutely. Have seen several of these of various types of the years. It usually works best if the processor itself calls out/polls some endpoint to get the latest instruction set. This approach tends to scale better, means clustered systems work smoothly as well, and it is nice from a

  1   2   3   4   5   6   7   8   9   10   >