Re: PublishAMQP had been working, now failing

2017-03-20 Thread Oleg Zhurakousky
ered with my upgraded version of NiFi. On Mon, Mar 20, 2017 at 12:14 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: James Basically my next question would be if admin-wise the new user had been given the correct permissions as I beli

Re: PublishAMQP had been working, now failing

2017-03-20 Thread Oleg Zhurakousky
MQP. Thanks very much for looking more closely at this. -Jim On Mon, Mar 20, 2017 at 11:55 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: James, while I am looking couple of questions. 1. Did you actual RabbitMQ version has changed si

Re: PublishAMQP had been working, now failing

2017-03-20 Thread Oleg Zhurakousky
help. I am very interested in why I seem to be getting this bad header message now. -Jim On Mon, Mar 20, 2017 at 10:00 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: James, any chance you can provide stack trace from the logs? Cheers Oleg

Re: PublishAMQP had been working, now failing

2017-03-20 Thread Oleg Zhurakousky
James, any chance you can provide stack trace from the logs? Cheers Oleg > On Mar 20, 2017, at 15:34, James McMahon wrote: > > Good morning. I am using a NiFi 0.7 code base. I recently upgraded to that > from a 0.6.x version. I'm limited to that 0.7.x baseline now. I

Re: Automating updates in process groups through templates

2017-03-09 Thread Oleg Zhurakousky
Jim I see where you’re going with this and I do believe that it is a valuable enterprise feature. Unfortunately, template XML files are not the bootstrap source of the flow such as flow.gz file. They are only used for save/move/migrate/etc., which means any modifications to them are not going

Re: ExecuteStreamCommand Zombie

2017-03-02 Thread Oleg Zhurakousky
Olav The fact that the processor does not detect the dead process that was initiated by it does appear to be a bug. Would you mind raising JIRA - https://issues.apache.org/jira/browse/NIFI so we can take a look. Cheers Oleg On Mar 1, 2017, at 7:29 PM, Olav Jordens

Re: Deadletter

2017-03-01 Thread Oleg Zhurakousky
have implemented to deal with failures. - Nick On Wed, Mar 1, 2017 at 12:29 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Nick Since you’ve already designed Process Group (PG) that is specific to failed flow files, I am not sure I

Re: Deadletter

2017-03-01 Thread Oleg Zhurakousky
Nick Since you’ve already designed Process Group (PG) that is specific to failed flow files, I am not sure I understand your last question “. . .How do you manage failure relationships?. . .”. I am assuming that within your global flow all failure relationships are sent to this PG, which

Re: Expression language and UTC millis

2017-02-22 Thread Oleg Zhurakousky
Adam, I think if my math is correct, what you seeing is correct since 22T23:01:23 + 7 = 23T06:01:23 Am i missing something? Cheers Oleg On Feb 22, 2017, at 6:42 PM, Adam Lamar > wrote: Hi, I recently noticed some issues with time parsing in

Re: Need to migrate the NiFi 1.0.0 nar file to NiFi 1.1.1

2017-02-16 Thread Oleg Zhurakousky
Manojkumar The short answer is “yes" it is possible since NAR by definition is a self-sustaining (resources, class path etc) package. However, I am a bit confused about the UI changes you are mentioning and can’t seem to find a correlation to NAR in general. Are you saying you have changed

Re: How to send the success status to GetFile processor?

2017-02-03 Thread Oleg Zhurakousky
don't know file name which is stored inside directory . ${absolute.path}\${filename} Note: while using GetFile processor doesn't have upstream connections but i need processor which fetch file inside directory using directory name without give filename. Many thanks On Thu, Feb 2, 2017 at 7

Re: How to send the success status to GetFile processor?

2017-02-02 Thread Oleg Zhurakousky
Prabhu Not sure I fully understand. While indeed GetFile does not allow for an incoming connection, it does allow for your use case to happen indirectly by monitoring a predefined directory. So, one PG finishes and produces a file that is being put into a directory monitored by another PG’s

Re: Dynamically set the RabbitMQ routing key

2017-01-25 Thread Oleg Zhurakousky
Brian You are correct, at the moment it is somewhat static doe to the fact that for the routing key the ‘expressionLanguageSupported’ defaults to false. We can definitely make that change which would allow you to use NiFi Expression Language to make it dynamic. Please raise a JIRA ticket -

Re: How to print flow

2017-01-19 Thread Oleg Zhurakousky
Alessio Outside of screen shot I am not sure you have many options, at least at the moment. Printing something like a flow is more complicated that it may seem at first, due to formatting issues. Landscape or Portrait, the paper size, etc., and what if the flow doesn’t fit? Should it get

Re: Migrating NiFi templates/canvas

2017-01-13 Thread Oleg Zhurakousky
Panos What version of NiFi are you using? The issue with the NiFi templates diffs has actually been addressed (see https://issues.apache.org/jira/browse/NIFI-826). So as of NiFi 1.0 you can export templates in a deterministic way and then execute diff on them to see only what was changed. It

Re: SFTP and Data Integrity

2017-01-05 Thread Oleg Zhurakousky
Nicolas I don’t believe there is. A possible workaround may be to use ExecuteScript processor, but I’ll be honest I have not tried. That said, what you are describing seems to be rather valuable feature and if the underlying API (or other libraries) allows for it then we can definitely make

Re: Building nifi locally on my mac

2016-12-29 Thread Oleg Zhurakousky
Actually it may be related to the known JDK issue that has been reported several times. Does it look something like this (see below)? If so then please upgrade your JDK to something newer. Something above 1.8.0_65 Cheers Oleg . . .

Re: File causes unexpected stoppage in play

2016-12-14 Thread Oleg Zhurakousky
to that process let me try that. Thank you. -Jim On Wed, Dec 14, 2016 at 2:57 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: James Not sure what kind of liberties you have with your environment, but here is what you can do outside o

Re: File causes unexpected stoppage in play

2016-12-14 Thread Oleg Zhurakousky
rrectly. Let me ask you this: is there any way I can build this into my code baseline - either through a code mod and rebuild or as a custom plugin feature specific to my flow? Thanks very much for your help. On Wed, Dec 14, 2016 at 10:59 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com&l

Re: File causes unexpected stoppage in play

2016-12-14 Thread Oleg Zhurakousky
James Could you also let us know what version of NiFi you are using? The issue with properly handling InvalidPathException was fixed in NiFi 0.7.0 as part of https://issues.apache.org/jira/browse/NIFI-920 It essentially has this catch block: } catch (final ProcessException |

Re: NiFi PlublishAMQP using cert CN as username

2016-12-10 Thread Oleg Zhurakousky
Brian Thank you for detailed explanation. I don't believe you're doing anything wrong. We just need do add the feature you describe (pulling credentials from certificate). Would you mind creating JIRA ticket and if at all possible attach the sample code that demonstrates exactly what you're

Re: Nothing in rabbitMQ queue from Publish AMQP processor

2016-12-08 Thread Oleg Zhurakousky
JIRA tickets and in the meanwhile will battle to bring our rMQ out of the dark ages. Thank you very much once again for your help investigating this behavior Oleg. On Thu, Dec 8, 2016 at 3:08 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Ja

Re: Nothing in rabbitMQ queue from Publish AMQP processor

2016-12-08 Thread Oleg Zhurakousky
d E. While I won't be able to upgrade my NiFi baseline because of other organizational dependencies, I can certainly eliminate the problems due to the rMQ and Erlang versions. Does this seem like a reasonable approach in your opinion? On Thu, Dec 8, 2016 at 1:45 PM, Oleg Zhurakousky

Re: Nothing in rabbitMQ queue from Publish AMQP processor

2016-12-08 Thread Oleg Zhurakousky
On Dec 8, 2016, at 1:04 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Got it, let me dig in, i got time now On Dec 8, 2016, at 12:46 PM, James McMahon <jsmcmah...@gmail.com<mailto:jsmcmah...@gmail.com>> wrote: N

Re: Nothing in rabbitMQ queue from Publish AMQP processor

2016-12-08 Thread Oleg Zhurakousky
ange by NiFi. On Thu, Dec 8, 2016 at 12:41 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: James My apology , but I am not sure I am clear on the second part of your explanation. It appears you are saying that when you bound the Exch

Re: Nothing in rabbitMQ queue from Publish AMQP processor

2016-12-08 Thread Oleg Zhurakousky
a Connection that is shown with State of blocking. and that is how it remains indefinitely. Nothing shows up in the exchange or in the queue. Jim On Thu, Dec 8, 2016 at 8:26 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Fair enough

Re: Nothing in rabbitMQ queue from Publish AMQP processor

2016-12-08 Thread Oleg Zhurakousky
s to me how / if that needs to be done. I'll dig for that. I am using NiFi 0.6.1. Cheers and thanks again, Jim On Thu, Dec 8, 2016 at 7:46 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: James I have not used AMQP in a while (although

Re: Nothing in rabbitMQ queue from Publish AMQP processor

2016-12-08 Thread Oleg Zhurakousky
ng the exchange, but I thought it best to ask. I see nothing on the management console that alludes to enabling the exchange or the queue. Jim On Thu, Dec 8, 2016 at 7:24 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: James Aside from

Re: Nothing in rabbitMQ queue from Publish AMQP processor

2016-12-08 Thread Oleg Zhurakousky
James Aside from “closing” failure message, do you see any other errors or stack traces in the logs? Meanwhile I’ll try to dig more with what I have? Oleg On Dec 8, 2016, at 7:13 AM, James McMahon > wrote: Hello I am trying to use a Publish

Re: How to integrate a custom protocol with Apache Nifi

2016-12-07 Thread Oleg Zhurakousky
or Libraries in my application code? or can I just build a HTTP Server and tell Nifi JVM that "hey here is my Application (it runs on specific IP and port) and it uses HTTP". On Wed, Dec 7, 2016 at 12:00 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonwork

Re: How to integrate a custom protocol with Apache Nifi

2016-12-07 Thread Oleg Zhurakousky
with my HTTP based server? On Wed, Dec 7, 2016 at 4:55 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Kant There are couple of questions here so, let me try one at the time. 1. “why Nifi would act as a server?”. Well, NiFi is a runtime e

Re: How to integrate a custom protocol with Apache Nifi

2016-12-07 Thread Oleg Zhurakousky
ave Java client library as well. > > http://nsq.io/clients/tcp_protocol_spec.html > > > On Tue, Dec 6, 2016 at 5:14 PM, Oleg Zhurakousky > <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: >> >> Hi Kant >> >> What you’re try

Re: How to integrate a custom protocol with Apache Nifi

2016-12-06 Thread Oleg Zhurakousky
Hi Kant What you’re trying to accomplish is definitely possible, however more information may be needed from you. For example, the way I understand your statement about “integration with many systems” is something like JMS, Kafka, TCP, FTP etc…. If that is the case such integration is

Re: Delay Processor

2016-11-15 Thread Oleg Zhurakousky
I am +1 on this as I’ve seen many cases in the field where it is applicable and as you mention exponential back off is one of the common one. That said, I am wondering if that has to be a processor at all? Actually let me answer my own question. There are definitely cases where it has to be a

Re: Nifi- PutEmail processor issue

2016-11-15 Thread Oleg Zhurakousky
Sravani Would you be able to provide a full stack trace of the connection exception. Also, while I assume you are providing the correct connection properties (i.e., host,port etc) I would still recommend to check the they are correct, but in any event the full stack trace would definitely help

Re: NPE MergeContent processor

2016-11-11 Thread Oleg Zhurakousky
ses we > should figure out what happened and make it not possible. We must > always work to eliminate the possibility of corruption causing events > and work to recover well in the face of corruption... > > On Fri, Nov 11, 2016 at 2:57 PM, Oleg Zhurakousky > <ozhurakou.

Re: NPE MergeContent processor

2016-11-11 Thread Oleg Zhurakousky
ro’ to filenames (for reading in Spark subsequently) 11. Save to HDFS Does this help at all? If you need anything else just shout. Regards Conrad additional out of shot • compression level : 1 • Keep Path : false From: Oleg Zhurakousky <ozhurakou...@hortonworks.c

Re: NPE MergeContent processor

2016-11-10 Thread Oleg Zhurakousky
Conrad Any chance you an provide a bit more info about your flow? I was able to find a condition when something like this can happen, but it would have to be with some legacy NiFi distribution, so it’s a bit puzzling, but i really want o see if we can close the loop on this. In any event I

Re: Does NiFi ConsumeJMS Processor supports OpenJMS?

2016-11-01 Thread Oleg Zhurakousky
currently do not have support. Once again, please raise the JIRA if you believe that it’s important to add such feature. Cheers Oleg On Nov 1, 2016, at 9:29 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Nirmal While ConsumeJMS w

Re: Create NiFi Templates

2016-09-28 Thread Oleg Zhurakousky
an I programmatically create a template during > development time?' > > We don't have any direct public API's to accomplish this that I am > aware of though it is an interesting idea. > > Can you help direct us to where you're more interested right now so we > can help mos

Re: Best Practice for backing up NiFi Flows

2016-09-14 Thread Oleg Zhurakousky
Just to add to what James has already stated, templates also have been polished quite a bit to be more deterministic and source control friendly so they are an ideal artifact to be versioned and kept in source control repo. Cheers Oleg On Sep 14, 2016, at 3:02 PM, James Wing

Re: Force Stop a Processor

2016-07-22 Thread Oleg Zhurakousky
Also, if you care to share the code for you processor we’ll be happy to look and help you find the problem. As Joe pointed out we make every possible attempt to stop it, but if your custom code is blocking and does not react to interrupts then there is not much we can do stoping short of

Re: export from Teradata

2016-07-20 Thread Oleg Zhurakousky
Just to add to what Mark said, perhaps you can raise a JIRA (as new Feature) to add an optional property to the processor config where such property would point to an additional class path (to support your case and many others). We are doing it with few processors already. Cheers Oleg On Jul

Re: PutKafka won't Put! "Failed while waiting for acks from Kafka"

2016-06-11 Thread Oleg Zhurakousky
PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Also, the NPE was a bug that was fixed a while back and will be available on O.7. Any chance you can build from 0.x branch? Cheers Oleg Sent from my iPhone On Jun 11, 2016, at 16:58,

Re: PutKafka won't Put! "Failed while waiting for acks from Kafka"

2016-06-11 Thread Oleg Zhurakousky
Also, the NPE was a bug that was fixed a while back and will be available on O.7. Any chance you can build from 0.x branch? Cheers Oleg Sent from my iPhone On Jun 11, 2016, at 16:58, Juan Sequeiros > wrote: Hi Pat, Looks to me that on your

Re: Custom processor is failing for concurrency

2016-06-09 Thread Oleg Zhurakousky
ings.put(matcher, value); } } The exception is coming from the line where it is looping over the entryset: for (Map.Entry<CharSequence, String> entry : mappings.entrySet()) { Right after that loop it calls mappings.clear() so if the RequestFactory is shared by multiple threads (which I think it is

Re: NiFi GetKafka Processor is illegal, contains a character other than ASCII alphanumerics, '.', '_' and '-'

2016-06-08 Thread Oleg Zhurakousky
I am not sure I understand the question Sent from my iPhone > On Jun 8, 2016, at 18:23, Igor Kravzov wrote: > > Guys, what can be wrong?

Re: Excessive logging

2016-06-05 Thread Oleg Zhurakousky
Huagen, it appears you have DEBUG level logging enabled. Oleg On Jun 4, 2016, at 21:27, Huagen peng > wrote: Hi, I got excessive logging from my NiFi instance. I suddenly see logging like the following going into nidi-bootstrap.log very

Re: Custom processor is failing for concurrency

2016-06-03 Thread Oleg Zhurakousky
Kumiko It appears that the current state of the source you linked in is not in sync with what is in the stack trace. Perhaps you have made some code modifications (e.g., line 218 is an empty line in code while it has a pointer in the star trace). In any event, from what I can see the error is

Re: Merge multiple flowfiles

2016-06-03 Thread Oleg Zhurakousky
Huagen, I also want to apologize for my spell-checker butchering your name ;) Cheers Oleg On Jun 3, 2016, at 8:03 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Huge Just to close the loop on this one, I also wanted to point o

Re: Merge multiple flowfiles

2016-06-03 Thread Oleg Zhurakousky
Huge Just to close the loop on this one, I also wanted to point out this JIRA https://issues.apache.org/jira/browse/NIFI-1926 for general purpose aggregation processor which indeed would support multiple connections, configurable aggregation, release and correlation strategies. It would be

Re: GetKafka exception

2016-05-26 Thread Oleg Zhurakousky
Igor, you may want to go they Kafka mailing list. I've seen this several times, and all I know it has to do with broker I'll look in more details once back on line Oleg Sent from my iPhone On May 26, 2016, at 17:20, Igor Kravzov > wrote:

Re: PutKafka warning - why?

2016-05-26 Thread Oleg Zhurakousky
Source) [na:1.8.0_91] at java.lang.Thread.run(Unknown Source) [na:1.8.0_91] On Thu, May 26, 2016 at 4:28 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: That is because the processor is already in the process of being stopped and you pro

Re: PutKafka warning - why?

2016-05-26 Thread Oleg Zhurakousky
That is because the processor is already in the process of being stopped and you probably tried to start it again. That said, the fact that you see it is probably a bug since you should not be able to see ‘start’ action until processor is fully stopped. Could you please share more details as to

New Kafka API support (0.9+)

2016-05-03 Thread Oleg Zhurakousky
As some of you know, we are in the process of providing a set of new Processors that uses the new Kafka API (0.9+). This is a new NAR and will live for a while along side with the old Kafka NAR. The new Processors are called PublishKafka and ConsumeKafka (specifically to emphasize the new

Re: Help Wanted

2016-05-03 Thread Oleg Zhurakousky
ps://issues.apache.org/jira/browse/NIFI-1837?filter=12331874 > > As a community is there a preference given to JIRA's with Patch or GH PR's > or are they all treated with the same priority? > > Thanks, > Andrew > > On Tue, May 3, 2016 at 11:12 AM, Oleg Zhurakousky &l

Help Wanted

2016-05-03 Thread Oleg Zhurakousky
Guys I’d like to use this opportunity to address all members of the NiFi community hence this email is sent to both mailing lists (dev/users) While somewhat skeptical when I started 6 month ago, I have to admit that now I am very excited to observe the growth and adaption of the Apache NiFi

Re: Nifi 0.6.1 PutKafka error

2016-05-02 Thread Oleg Zhurakousky
Ralf While we are indeed working on it at the moment, please do file an enhancement request with all the details. It’s always good for tracking purposes. Cheers Oleg > On May 2, 2016, at 10:25 AM, Perko, Ralph J wrote: > > Thanks for the update. Is there any discussion

Re: Default termination for relationships

2016-04-24 Thread Oleg Zhurakousky
Manish That is an interesting suggestion. I personally don't see issues with it and generally do believe it would improve user experience, but interested what others think. Cheers Oleg Sent from my iPhone On Apr 24, 2016, at 08:46, Manish Gupta 8

Re: NiFi Supports for Kerberized Kafka Cluster

2016-04-07 Thread Oleg Zhurakousky
Yep, just watch the JIRA for PR Sent from my iPhone On Apr 7, 2016, at 12:35, indus well <indusw...@gmail.com<mailto:indusw...@gmail.com>> wrote: Thank you, Oleg. I'll be testing it out once it is available next week. Regards, Indus On Thu, Apr 7, 2016 at 10:48 AM, Oleg

Re: NiFi Supports for Kerberized Kafka Cluster

2016-04-07 Thread Oleg Zhurakousky
Not at the moment unless you are using HDP distribution of Kafka broker 0.8 which has Kerberos support while ASF version of 0.8 Kafka doesn't. That said, we are working on new pair of Kafka processors that will rely on Kafka 0.9 API and broker where such support is provided. It is slated for

Re: Having on processor block while another one is running

2016-03-30 Thread Oleg Zhurakousky
29, 2016, at 9:26 AM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Vincent I do have a suggestion for you but need a bit more time to craft my response. Give me till tonight EST. Cheers Oleg On Mar 29, 2016, at 8:55 AM, Vin

Re: Having on processor block while another one is running

2016-03-28 Thread Oleg Zhurakousky
Vincent This sounds more like an architectural question and even outside of NiFi in order to achieve that especially in the distributed environment one would need some kind of coordination component. And while we can think of variety of way to accomplish that I am not entirely convinced that

Re: Has anyone this error signature in PutKafka

2016-03-23 Thread Oleg Zhurakousky
back seems to correspond to this line in PutKafka.java > > final byte[] value = new byte[(int) flowFile.getSize()]; > > > Can the flow file size be negative? > > > > On 3/23/16, 6:24 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com> wrote

Re: To help fulfill NIFI-513...

2016-03-23 Thread Oleg Zhurakousky
Russell First, thank you for taking initiative! Indeed we need to bring that JIRA to a closure. Having said that there is actually a link in our Contributor Guide on interactive debugging

Re: Nifi 0.50 and GetKafka Issues

2016-03-15 Thread Oleg Zhurakousky
ew-Kafka-0-8-2-0-1-kafka1-3-1-p0-9-Parcel-What-are-the-changes/td-p/30506 http://archive.cloudera.com/kafka/parcels/1.3.1/ Michael On Tue, Mar 15, 2016 at 12:01 PM, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Michael What is KAFKA-0.8.

Re: Nifi 0.50 and GetKafka Issues

2016-02-21 Thread Oleg Zhurakousky
.1 Kafka NAR still work in 0.5.x? If the NAR doesn't depend on any other NARs I would think it would still work, and could be a work around for those that need to stay on Kafka 0.8.2. On Sunday, February 21, 2016, Oleg Zhurakousky <ozhurakou...@hortonworks.com<mailto:ozhurakou...@horto

Re: Nifi 0.50 and GetKafka Issues

2016-02-21 Thread Oleg Zhurakousky
est <josh_w...@bose.com<mailto:josh_w...@bose.com>> Bose Corporation On Sun, 2016-02-21 at 15:02 +, Oleg Zhurakousky wrote: Josh Also, keep in mind that there are incompatible property names in Kafka between the 0.7 and 0.8 releases. One of the change that wen

Re: Nifi 0.50 and GetKafka Issues

2016-02-21 Thread Oleg Zhurakousky
on this property it’s value was completely ignored with 0.8 client libraries (you could actually see the WARN message to that effect) and now it is not ignored, so take a look and see if tinkering with its value changes something. Cheers Oleg On Feb 20, 2016, at 6:47 PM, Oleg Zhurakousky

Re: Nifi 0.50 and GetKafka Issues

2016-02-20 Thread Oleg Zhurakousky
016-02-20 at 15:46 -0500, Joe Witt wrote: And also what version of Kafka are you using? On Feb 20, 2016 3:37 PM, "Oleg Zhurakousky" <ozhurakou...@hortonworks.com<mailto:ozhurakou...@hortonworks.com>> wrote: Josh Any chance to attache the app-log or relevant stack trace?

Re: Nifi 0.50 and GetKafka Issues

2016-02-20 Thread Oleg Zhurakousky
Josh Any chance to attache the app-log or relevant stack trace? Thanks Oleg On Feb 20, 2016, at 3:30 PM, West, Joshua > wrote: Hi folks, I've upgraded from Nifi 0.4.1 to 0.5.0 and I am no longer able to use the GetKafka processor. I'm seeing

Re: Version Control on NiFi flow.xml

2016-02-17 Thread Oleg Zhurakousky
Jeff, what you are describing is in works and actively discussed https://cwiki.apache.org/confluence/display/NIFI/Extension+Registry and https://cwiki.apache.org/confluence/display/NIFI/Component+documentation+improvements The last one may not directly speaks to the “ExtensionRegistry”, but if

Re: Running Processors Synchronously

2016-02-11 Thread Oleg Zhurakousky
Obaid Thanks for reaching out! Currently it is not possible to wire a flow the way you describe. Wha you are asking is a true Event Driven Consumer paradigm which been discussed internally a lot lately, so it would be very interesting to get your perspective as to why do you believe it is

Re: NiFi processor for Redis

2016-01-13 Thread Oleg Zhurakousky
Suddep Also, when you say push/retrieve do you mean publish/subscribe or put/get since Redis is both storage and messaging system. As Joe mentioned we don’t have any at the moment but would be nice to start prioritizing. Cheers Oleg > On Jan 13, 2016, at 12:41 AM, Joe Witt

Re: NIFI connecting to Activemq

2016-01-09 Thread Oleg Zhurakousky
Chris Are you sure you are providing the correct logs? I can’t see a single mention of JMS nor any stack traces which would definitely be there given what you see in UI. Also, the fact that you see NPE is definitely a bug that we have to fix (user’s should never see NPE), so that can be filed.

Re: Testing a nifi flow via junit

2016-01-09 Thread Oleg Zhurakousky
This is definitely possible and been done. What makes it difficult at times is to have all required NiFi dependencies in the process space of a given test. I've actually proposed a separate module for these types of 'headless' flow tests. It actually helped me to discover some of the bugs as

Re: NoSuchElementException

2015-12-01 Thread Oleg Zhurakousky
Douglass Just looked at the code and would like to ask you if you could try it with our latest snapshot. This was a bug that was fixed. Thanks Oleg On Dec 1, 2015, at 11:23 AM, Douglas Doughty > wrote: Hi All, I’ve enjoyed working with nifi,

Re: High CPU usage in FileSystemRepository.java

2015-11-19 Thread Oleg Zhurakousky
Adam, thanks for doing all the research and pointing out where the problem is. It is definitely a bug. Joe I’ve raised the https://issues.apache.org/jira/browse/NIFI-1200 Cheers Oleg On Nov 19, 2015, at 1:18 PM, Joe Percivall > wrote:

Re: Replicate flow files to multiple processors

2015-11-11 Thread Oleg Zhurakousky
I am still a bit confused with the problem that is being solved here. “replication” implies some type of redundancy allowing processing that failed “here” to be resumed “there”. What I am reading here is more about "content based routing” - (route to their respective workflows based on their

Re: Replicate flow files to multiple processors

2015-11-10 Thread Oleg Zhurakousky
Multiple processors or multiple instance of the same processor? Could you also elaborate on your use case a bit more, simply because their may be several ways of accomplishing your goal and to pick the best understanding of the underlying problem would help. Thanks Oleg On Nov 10, 2015, at

Re: Determine which Nifi processor eating CPU usage

2015-10-26 Thread Oleg Zhurakousky
Unfortunately I can’t seem to even see how it would be possible for NiFi to tell you that since your custom Processors are running within the same JVM as NiFi. Having said that the 800% tells me that you probably have some processor with custom thread pool where each thread is spinning in the

Re: StoreInKiteDataset help

2015-10-16 Thread Oleg Zhurakousky
Chris Could you elaborate on your use case a bit more? Specifically about where is the source of data you want to pump into hive (e.g., Streaming, bulk file load etc.) Cheers Oleg On Oct 16, 2015, at 8:56 AM, Christopher Wilson > wrote: Joe,

Re: Provenance doesn't work with FetchS3Object

2015-10-15 Thread Oleg Zhurakousky
Ben I don’t think it needs an incoming FlowFile. It is a scheduled component and will retrieve contents based on how you configure scheduling. Have you tried it without incoming FlowFiles? Cheers Oleg On Oct 15, 2015, at 3:38 PM, Ben Meng >

Re: Custom classpath (using MapR-FS in lieu of HDFS)

2015-10-12 Thread Oleg Zhurakousky
May be I am missing certain details, but are you asking about custom bundle (NAR) or using existing NiFi bundles which would somehow have to delegate to some custom JARs? I am thinking that if you are using custom client, you may need a custom bundle. Oleg > On Oct 12, 2015, at 9:29 AM, Andre