Re: SplitText Usage - how to output my individual files?

Mark, thank you so much for your reply. It is very much clear now and I had suspected the mgmt toolbar and in particular the Data Provenance area is very informative regarding the issue. Thank you again! -- View this message in context:

SplitText Usage - how to output my individual files?

Hi, I am not sure whether I am terribly missing the point, but I have a simple dataflow (CSV2JSON) that does the following: GetFile (1000 line-cvs file) SplitText (one line per file) Extract and ReplaceText in order to extract the content and construct the JSON structure UpdateAttribute in which

Re: Apache NiFi 0.7.0 Release date ?

Joe, I was actually going to post the very same question when I have found this one. I am personally interested in the following features: JSON-to-JSON Schema Converter Editor ; Transform JOLT Processor

Re: Help with ReplaceTextWithMapping processor: multi-column mappings

Hi Andy, apologies for the delay in replying. I have come back to the task and tried to write a simple Groovy script in order to make the substitutions coming from the mapping files. In order to have something working, I have modified the mapping files, so that each of the incoming Json values

Re: Apache NiFi expression language

Andy, as always thank you for your reply (and apologies for delay of my response too). Just to clarify, I have come across this issue while using the processor ReplaceTextWithMapping. I shared a related post with the community:

Apache NiFi expression language

Hi, I am interested to understand whether in NiFi it is possible to write regular expressions that only match the first occurrence of a pattern. I do not seem to be able to make this work and wonder whether it is just something not possible at all. Here is an example of what I mean:

Re: Issues with Regex used with ReplaceTextWithMapping: where am I going wrong?

Hi all, I went through a number tests until the moment when I have realised that the problem seems to be dealing with phrase sentences with spaces, e.g. "Text Again". Although the final version of my regex was still correct, as follows: [A-Z][A-Za-z]*\s[A-Z][A-Za-z]* The pattern was not picked,

Issues with Regex used with ReplaceTextWithMapping: where am I going wrong?

Hi, I have come across an issue with my regex associated with ReplaceTextWithMapping. I have attached input and mapping files together with a NiFi template. My Json looks like this: {"field1": "someText", "field2": "Text Again", "field3": "Text Again"} I am trying to match the

Re: Formatting issues with Json: what is the best approach in NiFi?

Andy, thank you so much for this, it sounds like a very interesting idea. To clarify, are you suggesting to modify the existing Json validator so that it can read the invalid Json? Will I then be able to pass it successfully to EvaluateJsonPath? Have I understood it correctly? Thank you, I.

Re: Formatting issues with Json: what is the best approach in NiFi?

Andy, thanks for sharing this with me, I was not aware of it. Unfortunately, for a number of reasons, I will need to use 0.5.1 for the time being, so I was looking at existing processors. I have had a go with the combination EvaluateJsonPath + UpdateAttribute and although I am not a bit more

Re: Formatting issues with Json: what is the best approach in NiFi?

Hi, I have had a look at the available processors and it seems I can do the following: GetFile (with my json already created) ->EvaluateJsonPath -> UpdateAttribute ->PutFile So, lets get started with simply adding the missing '"", I am under the impression that I can use something like this:

Formatting issues with Json: what is the best approach in NiFi?

Hi, I have a quite complex dataflow and half way through I am returned with a Json file that looks like this: {"field1": "D", "field2": "12345", "field3": "myText", "field4": , "field5": "B2", "field6": "B", "field7": 74664", "field8": 2, "field9": [something."2334", something."9973"],

Re: Help with ReplaceTextWithMapping processor: multi-column mappings

Andy, all, thank you so much for your help. I have already started using ExecuteScript quite heavily, in particular for data cleaning and wrangling between processors. I was only concern that being experimental, it was not particularly recommended to use in complex dataflow like the one I am

Re: Help with ReplaceTextWithMapping processor: multi-column mappings

Joe, thanks for your reply. I wonder whether a typo in my example might have made what I would like to achieve slightly confusing. My mapping file looks like this (note value in bold as opposed to "2" as included in my first post": Header1;Header2;Header3 A;some text;*A2 * Am I correct in

ReplaceTextWithMapping mapping question

Hi, is it possible to match the same field value in your input file to 2 or more different mapping values from your mapping file? Does it only work in a 1-1 fashion? Thank you -- View this message in context:

ReplaceTextWithMapping Usage

Hi, I have started looking into the ReplaceTextWithMapping process because I want to produce a Json file with some look-ups coming for externa files. I have looked at this: https://community.hortonworks.com/questions/10005/how-are-mapping-files-formatted-for-the-nifi-repla.html , but still I am

Re: Apache NiFi and support for Regex

Thanks Mark, appreciated. No reason really, I was just curious to know whether NiFi provides regex support towards syntaxes other than Java. Glad the use of ExecuteScript was actually a good idea for non Java regex. Thanks, -- View this message in context:

Re: Apache NiFi and support for Regex

I guess that if you want to use Python or other flavors, you can actually right a script and running it from within ExecuteScript, is that a fair statement? -- View this message in context:

Apache NiFi and support for Regex

Hi, I have been using regex to match attributes in the flowfiles and from the documentation it looks like that Apache NiFi offers full support for Java syntax regex. Does it also support other 'flavors'? Thanks -- View this message in context:

Re: Using the template cvs to json to convert dsv files into json format: a few questions

Hi, I have made some progress and specifically modified the regex in ExtractText, as follows: (.*?);(.*?);(\d+.*) The same value is used as SearchValue in my ReplaceText and the Replacement Value is as before = { "testField" : "${dsv.1}" } This time, I am outputting the right value, but I

Re: Using the template cvs to json to convert dsv files into json format: a few questions

With the view of ultimately sharing a template for converting dsv into json (this is the least I can do with all the help received from this amazing community), I have been working my way trough the steps for the dataflow and still running into some troubles and not sure why. Here is my

Re: Using the template cvs to json to convert dsv files into json format: a few questions

Just to add on my previous comment. I have just used ConvertCharacterSet from ASCII to UTF-8 and I am getting the same result. The issue seems to be how ¦ are interpreted, I am not sure why. -- View this message in context:

Re: Using the template cvs to json to convert dsv files into json format: a few questions

Bryan, thank you so much. I will then try a combination of ReplaceText and MergeContent and work out whether I need any additional manipulation after that. I am kinda struggling with ReplaceText, once again, I am using in the search value (\r\n) and the replacement value empty. Furthermore, my

Re: Using the template cvs to json to convert dsv files into json format: a few questions

Hi Bryan, as always you are a star! Thank you so much for your help. It does start making much more sense now, I cannot thank you enough, this flow is really something challenging considering my current knowledge of NiFi. So, in my case: 1 ) I have replaced GenerateFlowFile with GetFile (which

ReplaceText Usage

Hi, I am trying to make use of ReplaceText in order to match each each line of delimited values (.dsv files) whose format is: DD-MM- HH.MM.SS and replace it with -MM-DD HH:MM:SS+. I am not sure I have really understood how to use this out-of-the-box processor and I looked at various

Using the template cvs to json to convert dsv files into json format: a few questions

Hi, I am presented with the task of converting dsv files into json format. The resulting json format will consist of a number of "joins" between the data contained in these files. The steps I have identified so far are the following: - Read the data from a location; - Apply some basic

Re: Identifier Attribute for PutElasticSearch

Hi Bryan, thank you so much for this. Once again, your replies are very valuable to me, it does now make more sense. Thank you -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Identifier-Attribute-for-PutElasticSearch-tp9683p9710.html Sent from the Apache

Testing a custom processor: connectivity issues

Hi, I have put together a custom processor that sends a SOAP Post HTTP request and writes the response to a flowFile. My issue with testing such processor is that, for connectivity issues, I cannot access to the Post HTTP Url, which is one of my PropertyDescriptor. I am not sure how to overcome

Re: Is my custom processor doing too many things? OnTrigger question

Oleg, thanks for your help and patience, much appreciated. No, I would actually like to stop the flow if I am catching or at least I thought it was what I was doing with session.transfer(flowFile, FAILURE). Is that incorrect? -- View this message in context:

Re: Is my custom processor doing too many things? OnTrigger question

Thanks Oleg, would this be what I want? public void onTrigger(final ProcessContext context, final ProcessSession session) throws ProcessException { FlowFile flowFile = session.get(); if (flowFile == null) return; final String userId =

Re: Is my custom processor doing too many things? OnTrigger question

Oleg, this is actually a great idea, I will follow it for certain (a good compromise, too). I have an additional question on the onTrigger method for my Post Http. Among the main NiFi components, my custom processor contains a number of separate methods, in particular one that send a Post HTTP

Re: Is my custom processor doing too many things? OnTrigger question

Thank you Joe and thank you for understanding the struggle of a newbie, not many communities are so welcoming and inclusive like Apache NiFi Developer List! :) Yes, you are right posting to a URL endpoint to get a response and put that on Kafka is all I want, do you have any existing out of the

Is my custom processor doing too many things? OnTrigger question

Hi, I have some fundamental questions on a custom Processor I am working on. First of all, I think I am trying to do too many things in one custom processor (correct me if I am wrong :) as 1) I am sending a Post HTTP request 2) Posting the JSON response to Kafka. In my custom processor I have all

Re: Multiple nar/custom processors: advisable directory structure

Yes, but eventually I have resorted to copy and pasted the existing nar and processor folders and changing references in the pom.xml and in the resources folder. Thank you! -- View this message in context:

Re: Multiple nar/custom processors: advisable directory structure

Oleg, thanks for your reply. No, in this case it is not strictly related to my first processor so I felt myself it should go in a separate NAR. I am probably still unsure on how to generate it. For my first one, I have created an empty folder and then run mvn archetype:generate, then after filling

Multiple nar/custom processors: advisable directory structure

Hi, currently, I have one custom processor + test in a similar folder structure in my IDE (IntelliJ): -CustomProcessors -nifi-myprocessor-nar -nifi-myprocessor -src -main -java MyProcessor.java -test -MyProcessorTest.java I am

Re: catch commit error in OnTrigger to diversify session behaviour

Matt, thanks for your reply, but I am not sure I have actually understood what you mean. In my load method I have the following: try{ transaction.commit(); }catch (TitanException ex) { System.out.println("This is a failure message"); transaction.rollback(); } How are you

Re: Error setting up environment for Custom Processor

Thanks Bryan, over the weekend I had another look at the issue and yes I managed to get it work changing it to the latest NiFi 0.6.0. Thank you for your help, I. -- View this message in context:

Error setting up environment for Custom Processor

Hi, I have followed this set of instructions in order to build custom NiFi Processor: https://community.hortonworks.com/articles/4318/build-custom-nifi-processor.html However, when running mvn install or mvn

Re: Import Kafka messages into Titan

Bryan, thank you so much, this is exactly what I was looking for. Thank you! -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Import-Kafka-messages-into-Titan-tp8647p8702.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Import Kafka messages into Titan

Thanks once again Matt, but I wonder whether we can make it even easier. GetKafka -> Custom Processor that will use the GraphSON Reader lib (https://github.com/tinkerpop/blueprints/wiki/GraphSON-Reader-and-Writer-Library) -> Custom PutTitan Processor that will insert the graph into Titan. Does it

Re: Import Kafka messages into Titan

Matt, thank you for this this is brilliant. So, as it is I am thinking that I would like to use the following: GetKafka -> EvaluateJsonPath -> ExecuteScript+Groovy Script My questions are two: 1) How do I import the Titan-compliant file into Titan? I guess I can modify the script and load it

Re: Import Kafka messages into Titan

Simon, thanks for this. This sounds very reasonable. I have a very naive question on top my initial now, I am afraid. If I end up using 4 standard processors (GetKafka -> EvaluateJsonPath -> AttributesToJson -> PutElastic) from a Java application, how do I bundle them? Thanks indeed! -- View

Import Kafka messages into Titan

Hi, I am very new to NiFi and I have the following flow: Consume Messages from Kafka based on a particular topic (JSON format) ->Transform JSON format into some Titan-compliant format -> put them into Titan/ElasticSearch on AWS I have done researching and I believe I can set use the standard