Re: Processor logic

Hi Uwe, I believe a lot of this is covered in the Developer Guide [1]. Specifically, there are discussions of various processor patterns, including Split Content [2], and a section on Cohesion and Usability [3], which states: > In order to avoid these issues, and make Processors more reusable,

Re: Kafkaesque Output port

Aldrin, Another point for consideration is the scope of this information. Core NiFi flows and those components are very much about data whereas those IPs may not necessarily be data for consumption, per se, but context that governs how the data flow is operating. In this case, there is a diffe

Processor logic

Hello, I have a little bit of a hard time to design processors correctly. I find it difficult to decide if a processor should e.g. process a single line from a flow file or process also flow files with multiples lines of data (e.g. in the case of CSV files). Another point is the handling of hea

Re: All Partitions have been blacklisted due to failures when attempting to update. If the Write-Ahead Log is able to perform a checkpoint, this issue may resolve itself. Otherwise, manual interventio

Would it be possible to configure EvaluateJsonPath to place the selected JSON fragment in the flowfile content instead of an attribute? Or to break the selection across multiple attributes rather than one big one? Thanks, James On Thu, Mar 16, 2017 at 11:57 AM, srini wrote: > Hi James, > > Ye

Re: All Partitions have been blacklisted due to failures when attempting to update. If the Write-Ahead Log is able to perform a checkpoint, this issue may resolve itself. Otherwise, manual interventio

Hi James, Yes, EvaluateJsonPath is creating attributes exceeding 64 KB. What should I do to avoid this? thanks Srini -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/All-Partitions-have-been-blacklisted-due-to-failures-when-attempting-to-update-If-the-Writ

Re: Need help in JSON genarate automation

Anshuman, You can use the GenerateFlowFile processor to generate arbitrary amounts of binary or text data at any interval you want. You can copy some static JSON template into the processor properties as default content (Lorem ipsum, if you will), and then use a follow-on ReplaceText to perform

Re: All Partitions have been blacklisted due to failures when attempting to update. If the Write-Ahead Log is able to perform a checkpoint, this issue may resolve itself. Otherwise, manual interventio

Srini, The error message "FlowFile Repository failed to update" matches a known issue where NiFi has trouble persisting attributes larger than 64 KB ( https://issues.apache.org/jira/browse/NIFI-3389). Is it possible that your EvaluateJsonPath is creating attributes exceeding 64 KB? From your err

All Partitions have been blacklisted due to failures when attempting to update. If the Write-Ahead Log is able to perform a checkpoint, this issue may resolve itself. Otherwise, manual intervention wi

Hi, We have single nifi instance, and this is our production environment. Yesterday I increased from 512m to 4096m in bootstrap.conf for both Xms and Xmx, and restarted nifi. Then I see this error [1]. Then I deleted the folder ../nifi-1.1.0/flowfile_repository and restarted the nifi. Then everyth

Re: [VOTE] Release Apache NiFi nifi-nar-maven-plugin-1.2.0

+1 Release this package as nifi-nar-maven-plugin-1.2.0 Went through the release helper, built NiFi with the new plugin, ran NiFi to make sure it didn't explode (it didn't). On Tue, Mar 14, 2017 at 9:21 AM, Bryan Bende wrote: > Hello, > > I am pleased to be calling this vote for the source rele

Re: When should MergeContent stop and proceed to next processor?

Ok, can you please set the “Correlation Identifier” to “fragment.identifier”? That is what I was trying to explain in the previous email. Cheers Oleg > On Mar 16, 2017, at 11:06 AM, srini wrote: > > Hi Oleg, > > Here is MergetContent screenshot. My flowfiles don't give any clue about > what re

Re: When should MergeContent stop and proceed to next processor?

Hi Oleg, Here is MergetContent screenshot. My flowfiles don't give any clue about what record it belongs to. I have an attribute called recordId which distinguishes each record. But I shouldn't add recordId in the flowfiles to be merged.

Re: Old Data Provenance. Even though there is new, it is not showing.

Srini, Looking at the screenshot, it appears that you have a search applied. The criteria have resulted in more than the maximum supported number of results (1000). If you want more recent results, try updating the search criteria to more closely hone in on the desired timeframe. The sorting that

Re: Kafkaesque Output port

Interesting points. Certainly agree with the difference between the two classes as well as where output ports are now. Whether or not there is an extension of the output port or a whole new component, the shared references/data set is a common one. There are a lot of options out there that provi

Re: Regarding changes in Apache Nifi

We’ve built custom, UI abstractions on top of Nifi using the REST API’s exclusively. The API documentation is very good, and as people have suggested, we became very effective at using the browser Developer console to understand the REST calls in the native UI so we could better understand how

Re: Regarding changes in Apache Nifi

Hi Sunil, Everything that the NiFi UI does is performed via a REST API [1]. You could write your own front end that utilizes that API to perform operations. There are also processors that contribute their own UI [2] so you could potentially go down that road if you wanted. Thanks, Bryan [1] ht

Re: Kafkaesque Output port

Simon, Thank you for your comments... I was aware of the use of Kafka and alternatives but I think that limitations aside (e. g. dificulty of transfering files over kafka, broken provenance chain, etc) many would refrain from using yet another piece of infra to reach the 1-n clients. This is spec

Re: Kafkaesque Output port

Just wanted to throw out a couple of other ideas... I ran into a similar situation and ended up creating a web-service at the core (HandleHttpRequest/HandleHttpResponse) where the edge instances could poll for the latest instructions [1][2]. This works well when theres basically one new piece of i

Re: Kafkaesque Output port

There are two different classes of queues really and you can't mix them semantically. The pubsub model * where ordering isn't guaranteed, * messages may appear at least once but can be duplicates * messages need to be explicitly deleted or aged * messages may or may not be persisted The ev

Re: Kafkaesque Output port

Hey Andre, Interesting scenario and certainly can understand the need for such functionality. As a bit of background, my mind traditionally goes to custom controller services used for referencing datasets typically served up via some service. This means we don't get the Site to Site goodness and

Re: When should MergeContent stop and proceed to next processor?

Hi Is there any chance you can share your processor’s configuration? I am curious as to what are you using as “Correlation Attribute Name” in the MergeContent processor. Basically this attribute allows to distinguish groups of flow files so, since you have SplitJson as an upstream processor fee

Re: Regarding changes in Apache Nifi

Forwarding this to dev@nifi.a.o If I understood the question here, the ask is for a white-labeled Nifi that could be customized per deployment !! On Thu, Mar 16, 2017 at 8:23 AM, Sunil Neurgaonkar < sunil.neurgaon...@techprimelab.com> wrote: > Hey Suneel, > > I am a developer @ Techprimelab So

Kafkaesque Output port

dev, I recently created a demo environment where two remote MiNiFi instances (m1 and m2) were sending diverse range of security telemetry (suspicious email attachments, syslog streams, individual session honeypot logs, merged honeypot session logs, etc) from edge to DC via S2S Input ports Once so

Re: [VOTE] Release Apache NiFi nifi-nar-maven-plugin-1.2.0

+1 Release this package as nifi-nar-maven-plugin-1.2.0 Verified checksums and git commit id. Built NiFi using new nar plugin was successful. Created simple NiFi flow, confirmed it works as expected. On Thu, Mar 16, 2017 at 3:31 AM, Scott Aslan wrote: > Built sample NAR, built NiFi with new NAR a

Need help in JSON genarate automation

Hello all, Trust you are doing great great! One quick question, I need some help for simulating a testing pipeline - I want to create an automatic pipeline where there should be continuous flow of JSON messages/ records that I would publish onto Kafka topic and later consume the same for further