unsubscribe

unsubscribe _ Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc pas etre diffuses, exploites ou copies sans

RE: Nifi kafka ser/deser problem

Hi Mark, It works like a charm! Thank you very very much …. ☺ I understand better now the ser/deser and registries …..( confluent ,HWX or built in) I was thinking wrongly that it was not necessary to write my own ser/deser code ( java Stream Application- Confluent Side) Phil Tanks again

RE: Nifi kafka ser/deser problem

Hello Andrew Yes ..I trace the call to confluent registry and I see the get on the subject ( Confluent registry side) And I also see the binary content on a consumer shell console (kafka-console-consumer ) --- room room1147.100,3.246▒NF My question was much mor

RE: Nifi kafka ser/deser problem

Hello Bryan, Schema Write Strategy : Confluent Schema Registry Reference Schema Access strategy : Use 'Schema Name' Property Schema registry : ConfluentSchemaRegistry Schema Name : topic7 Schema Text : ${avro.schema} Compression Format : NONE Thanks Phil -Original Message- From: B

RE: Nifi kafka ser/deser problem

Hello Joe This json content {"type":"room","id":"room11","attributes":{"position":"47.100,3.246","surface":223,"norme":"NF"}} is the input of the PublishKafkaRecord_0_10 processor. This processor have among others the following properties set : Record Reader : JsonTreeReader --> pointing

Nifi kafka ser/deser problem

Hello, My SW environment --> Nifi 1.4 compiled from sources , on Ubuntu 14.04 and a running Confluent 3.3 Platform (running Confluent Registry and kafkaStreams ) I have a process group with a PublishKafkaRecord_0_10 that serialize a record ( with 3 fields) And another process group with

RE: appending \n to a flow

Thanks Mark .. I understand the difference of records and flow of records and how to manage them with the “Entire text “ you mention ☺ From: Mark Payne [mailto:marka...@hotmail.com] Sent: jeudi 20 avril 2017 15:48 To: users@nifi.apache.org Subject: Re: appending \n to a flow Phil, For your c

RE: appending \n to a flow

☺ Thanks a lot James it works nicely now …… From: James McMahon [mailto:jsmcmah...@gmail.com] Sent: jeudi 20 avril 2017 15:36 To: users@nifi.apache.org Subject: Re: appending \n to a flow Can you try something like this Phil: ${uuid:append('\n')} Jim On Thu, Apr 20, 2017 at 9:25 AM, mailto:p

appending \n to a flow

Hello , I want to add a "\n "at the end of my flow ( it's a record in fact) before sending the record to a Put HDFS processor in append mode I tried to put the ReplaceText Processor in the flow with the following config Search Value (?s)(^.*$) Replacement Value \\n Charac

tranform a variable JSON list of properties with JoltTransformJSON

Hello , My flow input csv files with a variable # of fields ( value1, value2 , valueN) and I infer the types with InferAvro schema ( please see below) : { "type": "record", "name": "pgcsvavro", "doc": "Schema generated by Kite", "fields": [ { "name": "Value1", "type": "long", "doc":

RE: REST provenance/lineage API

Thanks , very simple at the end ☺ ! Last question , If I can .. Initially I have a process group named PG1 with some processors. I want to Execute the DFlow located in PG1 and then with and external program call via REST lineage/provenance API to get the complete report about the flow. M

RE: REST provenance/lineage API

Hello matt Almost perfect ! I call nifi REST api with the following JSON body in the post ( just indicating the flow uuid) -- { "lineage":{ "request":{"lineageRequestType":"FLOWFILE","uuid":"2a5d427f-085e-4993-bdc6-abf5709df7fc" } } } -- And results are records like this bu

REST provenance/lineage API

Hello Matt, I have the same problem with the provenance/lineage REST call curl 'http://localhost:8080/nifi-api/provenance/lineage/.. very difficult for me (sorry ☹ ) to find the right json body to send in the POST call . if you can give an example again Philippe Best regards

RE: REST provenance api search options

Thanks a lot Matt It works nicely ! From: Matt Gilman [mailto:matt.c.gil...@gmail.com] Sent: mardi 15 novembre 2016 15:08 To: users@nifi.apache.org Subject: Re: REST provenance api search options Philippe, Here's an example command for initiating a provenance search: curl 'http://localhost:8080

REST APIs provenance-events API

Hello, My SW context : nifi 1.0.0/ubuntu I would like to know if both following REST API calls : GET /provenance-events/{id}/content/input GET /provenance-events/{id}/content/output Are related to getting in the response the flowfile content before and after an event happened on a processor . M

REST provenance api search options

Hello, My SW context : nifi 1.0.0/ubuntu I am trying to use the provenance search options . I have the Id of my processor ie (ProcessorID ) but it’s not very clear for me how to fill the searchableFields. Is something as the following to be right ? and in this case where do i put the Proce

RE: logging all transformed flowfiles

Hello Andrew, thx. I saw the S2S Provenance reporting task but I do not understand how to use it …. ☺ May I implement my own reporting task , or I should use the one already there in nifi ? https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-site-to-site-reporting-bundle/nifi-si

RE: logging all transformed flowfiles

Hello Manish Thx for the very helpful answer , but I was thinking that this functional perimeter ( ie logging, storing transformations of data, data lineage ) was built in Nifi and available through REST API ... Or internal calls ... The point is that I am not ready to hook devoted logging pr

logging all transformed flowfiles

Hello, My SW context : standalone NiFi 1.0.0 My Problem : I would like to log all the different transformations applied to an initial file ( input) up to exiting the DF ( output) : If imagine this simple DF : File1 (in) --> Processor1 --> flow1 --> Processor2 --> flow2 --> File2 (out) I would

DF design question

Hello I have this simple Use Case to implement ( but it's not so clear for me which processors to put in the chain :)) : I have JSON file with records identified by one type property {.. "type": " smartphone"}, { . "type" : "PC" } , { "type": "tablet"} ..

RE: nifi processor to parse+update the current json on the fly

Hello thx Thad , joe , all For different answers. I understand how to know…. Philippe Best regards De : Thad Guidry [mailto:thadgui...@gmail.com] Envoyé : jeudi 7 avril 2016 17:32 À : users@nifi.apache.org Objet : Re: nifi processor to parse+update the current json on the fly Philippe, I would e

nifi processor to parse+update the current json on the fly

Hello I have a DF with a processor owning a json document that need to be transformed before sent to an ElasticSearch processor I have something like this $.element.attributes[0].value --> not to be changed $.element.attributes[0].type--> not to be changed ... $.element.attributes[10].value ->

RE: problem with putElasticsearch processor

Hello thx arpit It works nicely with port 9300 :) Thx philippe Hey Philippe Couple of things we can do to see what might be going on. 1. Check the port being used by put elastic search. Port 9200 is usually the default http port and 9300 is the default transport port. In put elastic search

problem with putElasticsearch processor

Hello My context nifi 0.6.0 on Ubuntu 14.0 my small use case is : 'sending JSON notifications arriving on http to an ElasticSearch instance ' So I started to develop a DataFlow with 1 handleHttprequest sending in parallel to 2 processors -to a Putfile ( perfect I see the JSON notifs on

GetMail processor

Hi, I would like to know if a GetEmail processor is available somewhere or planned ☺ . I have seen PutEmail but not the dual processor in the help The goal is to automatically and regularly processes incoming mail , transforms the content and index the transformed content with solr phil Best re

processor stats and count

Hello, 1) I would like to know if it is possible to get the stats from 1 processor running and send them ( bytes read , written, transferred in last 5min,... etc ...) To an external consumer outside of Nifi for example a kafka broker 2) I had a glance to counters returned her

RE: control nifi from REST API

Thanks a lot Matt ☺ It works nicely … ‘. I would love any contributions or collaborations on this project if you are interested.’ Sorry I am not an expert with gradle + groovy but If I can help … Philippe Best regards De : Matthew Burgess [mailto:mattyb...@gmail.com] Envoyé : vendredi 22

control nifi from REST API

Hello I would like to control Nifi by program an implement this: 1. find the right template in the templates repository --> result T1 2. instantiate the Template (T1) 3. set properties on different processors contained in the topology T1 4. start T1 5. stop T1

RE: control nifi processors with REST (ie without GUI)

Hello Matt Thanks for the article .. exactly in the scope ☺ I will check my clientId philippe De : Matt Gilman [mailto:matt.c.gil...@gmail.com] Envoyé : mardi 19 janvier 2016 20:21 À : users@nifi.apache.org Objet : Re: control nifi processors with REST (ie without GUI) Philippe, This article [1

RE: control nifi processors with REST (ie without GUI)

Hello thanks everybody for your answers The request is a put request from a Jax rs client java ……….. response = webResource.type("application/json").put(ClientResponse.class, returnedJSONProcessor1Updated); with returnedJSONProcessor1Updated ( see the gist pls) https://gist.github.com/pgib

control nifi processors with REST (ie without GUI)

Hello, My nifi version : nifi 0.4.1 I try to control a nifi topology with REST API (java jaxRS client ). I have a Nifi group of 2 processors and I can retrieve the json config for each of them . perfect ! But when I modify the json config ( Input Directory for my getfile processor inst

RE: Is nifi a good fit for this use case?

Hello Joe Thanks for these clear explanations Another great feature available from NIfi comparing with storm is : ( if I understand well :-) ) - The possibility to stop processors , then add some processors in the middle of the topology and then restart the workflow .. It can be qualified

RE: Nifi connection queue monitoring.

Hello I would like to know if a basic or complex nifi workflow of processors can be built just with REST API calls ie by-passing the GUI. More generally if the functional perimeter offered by REST API calls covers the same perimeter than the one offered by the GUI ? Phil Thanks De : Chakrade