Attributes To JSON question

Hi I am using the attributesToJSON processor to create a JSON flowfile which I then send to Elastic, I have noticed that some of my attributes which ar numbers such as file size always come out of the attributesToJSON processor as string values (ie with double quotes around them), therefore when

Merging the unique attributes of 2 flowfiles

Hi I have a group of log files coming in via http listener, up to 30 logs  per transaction, of which I only need the values that are in 2 of those log files per transaction. After using some RouteOnContents I end up with the two log flowfiles I want. In my current flow I am using a MergeContent

Counting the number of occurrences of a phrase

Good morning devs I have a flowfile which contains a space separated log file, within this file is a section such as:- FileSelectors:[{fileselector:},{fileselector:},{fileselector:}] I would like to create an attribute which is the count the number of occurrences of the word fileselector and idea

Re: Record Readers and Writers

-scripting-nar/1.11.4/org.apache.nifi.record.script.ScriptedReader/index.html [6] https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-scripting-nar/1.11.4/org.apache.nifi.record.script.ScriptedRecordSetWriter/index.html Andy LoPresto alopre...@apache.org alopresto.apa...@gmail.com

Record Readers and Writers

Hi I want to use the ConvertRecord Processor with it's underlying Record Readers and Writers to convert files from XML or JSON to a bespoke format and probably vice versa.I have looked at the Readers/Writers currently provided and decided that I can use the XML/JSON ones provided but I will need

Re: Advanced UI button on Update Attributes

te nifi-processor-configuration file, too? nifi-update-attribute-ui/src/main/webapp/META-INF/nifi-processor-configuration Thanks, Koji On Thu, Jan 3, 2019 at 7:29 PM DAVID SMITH wrote: > > Hi > I am looking to create a bespoke processor based very loosely on the Update At

Advanced UI button on Update Attributes

Hi I am looking to create a bespoke processor based very loosely on the Update Attribute processor set, I have taken a copy of the update attribute bundle and changed the name of the bundle and its components to nifi-test-build-processor, nifi-test-build-model etc, I have also amended the pom fi

Re: Help with loading a file into a cache

"WHI8765", "WHT8765" ] }} ZA103:{"Aircraft Type":"Wessex", "Lifed Items":{ "port engine":"RR-P7654", "starboard engine":"RR-S7654","gearboxes":[ "WHM7654", "WHI7654", "WHT765

Help with loading a file into a cache

Hi Devs I am running a NiFi 1.8 cluster, each node has 128Gb of Ram. I need to load the contents of a file of which is around 5Gb in size  into a Key/Value cache. The file I want to load is produced by another company so the format it comes in is not negotiable. The file contains thousands of lin

Re: Accumulo processors

and mentioned to me that he might be up for a code review on the Accumulo processors. On Sun, May 27, 2018 at 4:38 PM DAVID SMITH wrote: > Mike > Thanks for the suggestion I will certainly have a look at the HBase code, > I'm not really familiar with Accumulo, I am currently readi

Re: Accumulo processors

;t help you with the Accumulo APIs, but it could provide useful strategies for how to identify and assign labels from user input. On Sun, May 27, 2018 at 3:14 PM DAVID SMITH wrote: > Mark > Thanks for the link, I have downloaded the code from Github, it will be a > good basis to start with.

Re: Accumulo processors

Mark Thanks for the link, I have downloaded the code from Github, it will be a good basis to start with. Many thanksDave On Saturday, 26 May 2018, 20:22, Mark Payne wrote: Hi Dave, I do have a branch in Github with the work that I had done: https://github.com/apache/nifi/tree/NIFI-81

Help & advice needed on application.js in Processors

Hi I am trying to create a processor that is partially based on the UpdateAttribute processor 0.7.3.I have cloned the UpdateAttribute source, renamed the processor and I have started by trying to amend the Advanced configuration UI. I have found that I can change labels for fields in the Advanc

Re: Setting up an SSL context in Minifi 0.3.0

possible for your instance exhibiting problems.  Please be mindful to strip out any sensitive information. [1] https://issues.apache.org/jira/browse/MINIFI-408 [2] https://issues.apache.org/jira/browse/MINIFI-403 Thanks, Aldrin On Thu, Jan 18, 2018 at 10:56 AM, DAVID SMITH wrote: > Hi >

Setting up an SSL context in Minifi 0.3.0

Hi I put up a post a little while ago, because I was having trouble getting an SSL context working in Minifi 0.2.0Aldrin advised to wait until 0.3.0 came out as some dev work had been done in this area. I have trying it again today and still cannot get an SSL context to work in a ListenHTTP proc

Problems running Minifi with an SSL Context

Hi In my work dev environment I have a VM that is running NiFi v1.3, I am using this to create a flow template for a Minifi (v0.2.0 Java ) instance (initially on the same VM). My flow has a listenHttp which I am listening on port 1 using an sslContext which I have set up in NiFi, I zip the f

Re: Syslog processing from cisco switches to Splunk

u, Oct 19, 2017 at 7:31 AM, DAVID SMITH > wrote: > >> Hi >> We are trying to do something which on the face of it seems fairly simple >> but will not work.We have a cisco switch which is producing syslogs, >> normally we use zoneranger to send them to Splunk and the rec

Syslog processing from cisco switches to Splunk

Hi We are trying to do something which on the face of it seems fairly simple but will not work.We have a cisco switch which is producing syslogs, normally we use zoneranger to send them to Splunk and the records are shown.However we want to do a bit of content routing, so we are using NiFi 0.7.3

MiNiFi & SSL Context

Hi I tried the Java version of MiNiFi yesterday and was very impressed, I do however have a question. What is the best way to set up and SSL context in MiNiFi so that I can do HTTPS rather than HTTP? Many thanksDave

Help with Flowfile Content to Attribute

Hi I have a scenario where I have a flowfile which is come in and I want to email the contents to a recipient. I am currently using the putSmtp processor which works fine except that I have to email the flowfile as an attachment, so when the recipient gets the email they have to open an attachme

Re: Is there a ticket for XMPP processors

contributed anything toward it that I am aware of. Thanks On Mon, May 8, 2017 at 3:15 PM, DAVID SMITH wrote: > Hi > Is there still a ticket open for someone to design/code a set of XMPP > processors, as I cannot find one in Jira, could someone please send a link or > ticket number pl

Is there a ticket for XMPP processors

Hi Is there still a ticket open for someone to design/code a set of XMPP processors, as I cannot find one in Jira, could someone please send a link or ticket number please? Many thanksDave

Re: Making the server hostname available in NiFi

Thanks for the replies, I was using ${hostname()}, I found the reason it wasn't working was that the NiFi instance was running in a Docker container. When I tried it on a non Docker container instance it worked fine. Dave On Thursday, 30 March 2017, 0:43, Mark Payne wrote: Dave, You

Deconstructing SMTP /Mime emails

Hi I have a scenario where I use listenSMTP to listen for emails, and then I use ExtractEmailHeaders and ExtractEmailAttachments to deconstruct these emails into their constituent parts, however, I don't seem to be able to get the body text of the email as either an attribute or a FlowFile. Can

Help with back porting Email bundle from NiFi 1.1.1 to NiFi 0.7.2

Hi At work we are in need of a set of email/smtp processors, our system currently uses NiFi 0.7.2  on Centos 7 and Java 8. We can't upgrade the NiFi version at the present time.I found a set of processors which will do the job we require in NiFi 1.1.1, I have tried running the 1.1.1 nar bundle

Help with ExtractText processor and Mime format files

Fri, 13 Jan 2017 10:26:12 -0400 (EDT) From: David Smith To: John Doe Subject: =?iso-8859-1?Q?Test Email?= Message-ID: MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="-123456789-987654321-958746372=:6982"   This message is in MIME format.  The first part should be plain

NiFi Elastic Clustering

Hi We are looking to set up some NiFi Clusters, however, we would like them to scale elastically without having to restart the NCM as we add or remove nodes. >From NiFi 0.7.1 onwards is this possible or is this a feature that is being >looked at or should we be looking at our own solution? Many t

ExecuteScript and flowfiles

Hi I have a question from a colleague who is using nifi 0.5.1, he has some files coming in which he needs to break up into 3 constituent parts. He has a python script which should do this but what he wants to know is can he either 1)   send all three parts down one relationship as three separat

Re: Problems with links on the Contributors Guide

figuration, if I remember correctly there are two links pointing to the configuration file but only one is valid. Did you manage to add the checkstyle conf? Pierre 2016-02-23 16:02 GMT+01:00 DAVID SMITH : > Hi > As I am looking to submit code I looked on the Contributors Guide at the > C

Problems with links on the Contributors Guide

Hi As I am looking to submit code I looked on the Contributors Guide at the Code Style section.I am using Eclipse Mars and I loaded the Checksytyle plugin, I downloaded the eclipse-formatter.xml from the link, but it will not load into the Check Configuration panel in Eclipse, I get the followin

Re: Extending the NiFi API

to ensure it is shaped in a way that helps you move forward.  Great time to have these discussions. Thanks Joe On Tue, Feb 23, 2016 at 8:55 AM, DAVID SMITH wrote: > Hi > At work we run many instances of NiFi and other associated data applications, > with more instances of NiFi being pla

Extending the NiFi API

Hi At work we run many instances of NiFi and other associated data applications, with more instances of NiFi being planned it is becoming impracticable to use the UI's to add, delete users or amend their roles. We have developed an in house tool which controls these associated applications via t

Re: [GitHub] nifi pull request: NIFI-865 added initial support for AMQP publish...

so what is in > them that is specific to QPID? The reason why I am asking (and you can see it > from discussion on JIRA) is that with this effort we are aiming for > processors that are compliment with specific protocol regardless of the > broker implementation used, so it must be ne

Re: [GitHub] nifi pull request: NIFI-865 added initial support for AMQP publish...

Hi Guys || This email has been sent from a virus-free computer protected by Avast. www.avast.com | As you may remember I have developed some processors that publish/subscribe to AMQP brokers, but I was having problems writing Junit tests for these processors. I was interested to see that

Testing processor logging message in Junit tests

Hi Does anyone know how to test for logging messages generated by my processors in my Juint tests for that processor?I am trying to determine whilst testing my processor if I have entered an exception, where I would just normally log a warning or an Error but necessarily end up on the failure re

Re: XML Questions

Joe Thanks for the reply, that is pretty much what I thought about the flowfile attributes. I'll have to have a think about that one to see if I can work out a way around it. With the merging, we will get multiple xml files which will be related by an attribute, I want to combine them into one

XML Questions

Hi I have two questions which I think may require bespoke processors. 1)  I have extracted a set of  values from xml tags and made them into flowfile attributes, how a can I add these flowfile attributes to one or more flowfiles? 2)  I have multiple xml files which I need to merge into one xml

Re: LogAttribute - Sending that output to a custom logger?

Adam I have uploaded to github the bundle as I described earlier, for evaluation and possible future inclusion into the NiFi codebase, I would appreciate user feedback. The github link is :    helicopterman22/nifi-whatsflowing-bundle |   | |   | |   |   |   |   |   | | helicopterman22/nifi-whats

Re: NIFI RabbitMQ Processor

Chris I have tried running these processors and I have a couple of  observations: 1)   You may want to include a failure relationship, I found that when I was trying to connect to the broker which was not local server and i couldn't connect the processor was constantly yielding. 2)    You may w

Bulk data ingest processor

Hi Has anyone or is anyone creating a bulk data ingest processor. We are looking into the feasibility of using NiFi to do our bulk data pickups from many locations and possibly multiple directories from each location. My initial thoughts were to use a processor which could run multiple SFTP ses

Re: NIFI RabbitMQ Processor

Mark Thanks very much for the advice, that sounds a bit easier do you have an example where you have done this sort of thing that I could have a look at? Many thanksDave On Friday, 2 October 2015, 6:39, DAVID SMITH wrote: Hi Chris I have produced a set of processors which work

Re: NIFI RabbitMQ Processor

Hi Chris I have produced a set of processors which work with Java and C++ brokers (but not RabbitMQ), but like you I haven't raised a ticket yet because I am unsure how to mock an AMQP broker. I would certainly be interested in seeing how you would do this. Many thanks Dave Sent from Yahoo! Mai

[DISCUSS] Job Code entry feature for graph changes

Hi  I was doing a demonstration of NiFi to a new audience at work yesterday, the audience were most impressed and could see many uses, however two major questions arose. Audit and Accounting is a major thing where I work, and we would like to track who made changes and why.   I think I have seen

Re: Advanced UI on a Processor

has your processors type specified. Let me know if that helps. Matt Sent from my iPhone > On Sep 12, 2015, at 3:59 PM, DAVID SMITH wrote: > > Hi > I have to create a processor that dedupes  some values held in an flowfile > attribute against a set of 'criteria' t

Advanced UI on a Processor

Hi  I have to create a processor that dedupes  some values held in an flowfile attribute against a set of 'criteria' that may vary over time. I then need to update/create a new attribute based on the algorithm and then route on this attribute. I have a prototype of this processor working but I c

Re: Update Attribute Advanced rules not persisting

at 4:20 PM, DAVID SMITH wrote: Hi I have an instance of nifi-0.2.1 running in Centos 6.6, I have used an Update Attribute processor to set some rules, I can save the rules ad the processor runs as expected. However, if I go back into the processor configuration the rules have not persisted.Do you

Update Attribute Advanced rules not persisting

Hi I have an instance of nifi-0.2.1 running in Centos 6.6, I have used an Update Attribute processor to set some rules, I can save the rules ad the processor runs as expected. However, if I go back into the processor configuration the rules have not persisted.Do you know about this? Dave

AMQP Processor Github link

Please find below a link to my nifi-amqp-processor project, I would welcome any feedback and also any help with creating Junit tests, particularly how to create a mock broker/queue. Dave helicopterman22/nifi_amqp_processors |   | |   | |   |   |   |   |   | | helicopterman22/nifi_amqp_proces

Re: Writing to a flowfile

Mark Thanks very much for all of your help, that works really well, I have also taken on board your other comments and implemented them on my home version.  I will use it all at work tomorrow. As you may have seen on a post I made in July, I have taken the put & get JMS processors and made a mod

Re: Writing to a flowfile

Mark Thanks for your help. I have used the snippet of code you sent and it works although I am fairly sure I haven't implemented it correctly, I have had to put all of my code in the OnTrigger method, instead of in the the callback.I also need to change the filename attribute of the parsed flowf

Writing to a flowfile

Hi I'm writing a processor which parses a file, I want the parsed file to go to relationship parsed, and the original file to go to relationship original, if the parse was ok. If the parse fails I want the original file to go to relationship failure. I have an inner class which contains a call

Re: Help required with my custom controller service

All That certainly cured the problem in my case. Dave Sent from Yahoo! Mail on Android

Re: Instantiating a Controller Service in a Junit test

Mark Thanks for the information, it works a treat. Dave On Monday, 10 August 2015, 1:12, Mark Payne wrote: David, Yes, you'll also need to set the controller service in your processor. Sorry, I forgot to mention that. So after the call to runner.enableControllerService(), and bef

Re: Instantiating a Controller Service in a Junit test

Mark Thanks for the reply, I have changed my test as you suggested, see below: @Test     public void checkCache() throws InitializationException, IOException{         final TestRunner runner = TestRunners.newTestRunner(CacheTester.class);         final StandardCacheService cacheService = new Standa

Instantiating a Controller Service in a Junit test

Hi I have written a simple Cache Controller Service, this Controller Service has a property which if populated allows the cache to be populated when it is intialized.  I have also written a simple processor that allows me to utilize the Controller Service and checks some of the preloaded values

Help with Junit tests for AMQP processors -update

Hi I have put my initial code version on github.com, at the following url: helicopterman22/nifi_amqp_processors I have not started to write any junit tests yet. I need some guidance on how to test the connection code to an AMQP broker, I assume I have to create a mock broker with a mock queue.

Advice on testing AMQP processor

Hi I have created a set of processors which get/put to an AMQP broker based on the org.apache get/put JMS processors.My boss would like me to contribute them back to org.apache, however I feel I need to create some robust Junit user tests for the processors, but I need some advice on how to crea

Help required with my custom controller service

Hi I have been trying to write a simple Cache Controller Service that uses a Map to hold key/value pairs. I am using nifi-0.1.0-incubating. I have created an Interface called CacheService which extends ControllerService.public interface CacheService extends ControllerService { I have created a