Re: configuration variables?

What's about level higher - common variables for several process groups? Thanks. On Mon, Aug 6, 2018 at 7:06 PM, Charlie Meyer < charlie.me...@civitaslearning.com> wrote: > You might want to take a look at utilizing the variable registry for this: >

saving record into different tables?

Hi, I have a json record that contains array "mylst": { "id": 0, "name": "Root", "mylist": [{ "id": 10, "info": "2am-3am" }, { "id": 11, "info": "3AM-4AM" }, { "id": 12, "info": "4am-5am" }] } I have to save root data into one

Re: Nifi RecordReader for Json?

ht work: > > > > {"type":"record","name":"record0","fields":[ > > {"name": "stats", "type" : > > > {"type":"record","name":"statsRecord","field

parsing json array in ExecuteScript?

I have json array attribute: mylist :[{"id":10,"info":"2am-3am"},{"id":11,"info":"3AM-4AM"},{"id":12,"info":"4am-5am"}] How can I parse it to name/value pairs in groovy script? Thanks,

Jolt doesn't work?

I have a json record that contains array "mylst": { "id": 0, "name": "Root", "mylist": [{ "id": 10, "info": "2am-3am" }, { "id": 11, "info": "3AM-4AM" } ] } I tested Jolt spec: [ { "operation": "shift", "spec": {

Re: how to copy root fields in Jolt transform?

ed and puts it in > the output at the same location. > > Regards, > Matt > > On Tue, Sep 4, 2018 at 11:39 AM l vic wrote: > > > > Hi, > > I want to "flatten" the following object: > > { > > "id": 0, > > "name&

PutSQL with multiple upserts?

I need to save data with "root" object to be saved into "root" table and array "mylist" of contained objects to be saved into "mylist" table: { "id": 3, "name": "ROOT", "mylist": [{ "id": 10, "info": "2am-3am" }, { "id": 11, "info": "3AM-4AM"

Re: saving record into different tables?

ray/RecordSet, then use > PutDatabaseRecord to process it. Then you don't need a "split" or "get > into attribute" or any other operation that isn't in line with your > business logic. > > Regards, > Matt > > On Fri, Aug 31, 2018 at 7:34 PM l vic wrote

how to copy root fields in Jolt transform?

Hi, I want to "flatten" the following object: { "id": 0, "name": "Root", "mylist": [{ "id": 10, "info": "2am-3am" }, { "id": 11, "info": "3AM-4AM" }, { "id": 12, "info": "4am-5am" }] } I figured how to "flatten" array, but "root"

JsonSplit array into individual records?

I have array: "mylist": [{ "id": 10, "info": "2am-3am" }, { "id": 11, "info": "3AM-4AM" }, { "id": 12, "info": "4am-5am" }] How can i use combination of SplitJson / JsonPath to split it into multiple flows per array element? Thanks, V

Nifi RecordReader for Json?

I need to save two different json messages according to json schemas available for each to different relational database tables. I saw this blog: https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi with example using CSVRecordReader for csv->json transformation. but what would be

Re: upload template with REST problem

request as a curl command, etc. This should make it much easier for you >> to build the API requests you want. >> >> >> Andy LoPresto >> alopre...@apache.org >> *alopresto.apa...@gmail.com * >> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D E

Re: how to merge attributes?

s, the FlowFile for "Result" carries all > attributes copied that the incoming one has. You just need to connect > the "Result" relationship to ReplaceTest. "Original" can be > auto-terminated in this case. > > Thanks, > Koji > On Wed, Oct 24, 2018

how to merge attributes?

I have to create sql query from results of rest call (InvokeHttp ) and original request parameters... Any idea how i can merge attributes from both in single flowfile for passing it to "ReplaceText"? If I pass both "Original" and "Result" from "InvokeHttp" to "ReplaceText" I end up with one

Re: schedule execution from relational database?

uld handle other parts of it > so you wouldn't need to update a single row in an external database > table? > > Regards, > Matt > > On Mon, Oct 29, 2018 at 12:36 PM l vic wrote: > > > > What if have only one row and update the values in it? Will QDT fetch > update

Re: schedule execution from relational database?

ile to > trigger ExecuteStreamCommand to schedule a cron job? Or do you mean > scheduling a processor to run in NiFi? Or something else? > On Mon, Oct 29, 2018 at 3:58 PM l vic wrote: > > > > QDT works, eg it can detect change in MaximumValue column but how can I > use it t

Re: schedule execution from relational database?

ule > every X sec/min/hours/days, etc, or at given time (at minute 30 of each > hour every day except for Saturday). > If you don't know what your requirements for data availability and latency > are, you could start with something like every "5 mins". And then adjust as > ne

flowfile duplication?

Hello, I need to create 2 independent paths of execution for the same data ( message consumed from kafka topic) One path would save it to one db table, another would be used to perform some computations on attributes and save it to another table. I wonder if DuplicateFlowFile can be used for

schedule execution from relational database?

Hi, i have "event_time" field in SQLite database that means epoch time for triggering of external event. What processor(s) can i use to implement schedule monitoring/ execution based on change in "event_time" value? Thanks,

Re: schedule execution from relational database?

can send > >> downstream to perhaps ExecuteSQL in order to actually fetch the rows. > >> GTF is often used in place of QDT if you'll be fetching a large number > >> of rows in each statement, as you can distribute the SQL flow files > >> among the nodes in a clu

Re: schedule execution from relational database?

ter, to do the fetch in parallel. > > Regards, > Matt > > On Mon, Oct 29, 2018 at 11:13 AM l vic wrote: > > > > Hi, > > i have "event_time" field in SQLite database that means epoch time for > triggering of external event. What processor(s) can i use to implement > schedule monitoring/ execution based on change in "event_time" value? > > Thanks, >

Re: [EXT] ExecuteSQL: convertToAvroStream failure with SQlite integer

m, though, and will probably result in a bug report. > > > > > > *From: *l vic > *Reply-To: *"users@nifi.apache.org" > *Date: *Friday, November 9, 2018 at 6:52 AM > *To: *"users@nifi.apache.org" > *Subject: *Re: [EXT] ExecuteSQL: convertToAvroStream fail

ExecuteSQL: convertToAvroStream failure with SQlite integer

Hi, I am trying to use ExecuteSQL to get "epoch time" value from SQLite table: select start_date from sched where start_time is defined as INTEGER If the start_date = 1536548297955 i see the following exception: failed to process due to org.apache.avro.file.DataFileWriter$AppendWriteException:

Getting a timestamp for today at midnight?

I have to retrieve "today's" records in ExecuteSQL, eg. with the query using timestamp "ts" column in table where ts type is "epoch" time as long integer : select * from mytable where ts >= midnight-timestamp-value Any idea how i can use "now()" function to get today's timestamp value at midnight?

Re: [EXT] ExecuteSQL: convertToAvroStream failure with SQlite integer

unsigned ints. 1.8.0 also improved the error > message so that it should show the type of the object that was passed into > the unresolvable union. > > > > https://github.com/apache/nifi/pull/3032 > > > > > > *From: *l vic > *Reply-To: *"users@nifi.apache.

Re: Nifi RecordReader for Json?

> - > Sivaprasanna > > On Thu, 30 Aug 2018 at 2:21 PM, l vic wrote: > >> I need to save two different json messages according to json schemas >> available for each to different relational database tables. >> I saw this blog: >> https://blogs.apache.org/nifi/e

Re: Nifi RecordReader for Json?

2 AM Otto Fowler wrote: > The record readers are services, that processors use. > When you use a *Record* processor, you will have to select a Reader and a > Writer Service, or create one ( which you can do through the UI ). > https://blogs.apache.org/nifi/entry/record-oriented-data-with-nifi >

Re: saving record into different tables?

ith > evaluatejson path, and at the end route with RouteOnAttribute. > > > > *Aurélien* > > > > *From:* l vic [mailto:lvic4...@gmail.com] > *Sent:* samedi 1 septembre 2018 01:34 > *To:* users@nifi.apache.org > *Subject:* saving record into different tables? &g

upload template with REST problem

Hi, I have template for the processing group "test-template.xml" with id=f2ae3d20-e105-37a1-- that I am trying to upload from my mac curl -iv -F template=@test-template.xml -X POST http://10.10.100.105:8080/nifi-api/f2ae3d20-e105-37a1--/templates/upload I see

Re: upload template with REST problem

> id pathstring The process group id. > > [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html > On Thu, Sep 20, 2018 at 12:49 PM l vic wrote: > > > > yes, the "id" must actually be "groupId", then it works... As usual, >

Re: upload template with REST problem

p id. > > [1] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html > On Thu, Sep 20, 2018 at 12:49 PM l vic wrote: > > > > yes, the "id" must actually be "groupId", then it works... As usual, > Nifi documentation is vague or misleading >

PutSQL jdbc error

I have PutSQL on host 10.10.105.214 trying to execute query on phoenix query server on the host 10.10.105.202 and i see the following error: Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot create JDBC driver of class 'org.apache.phoenix.jdbc.PhoenixDriver' for connect URL

Re: upload template with REST problem

yes, the "id" must actually be "groupId", then it works... As usual, Nifi documentation is vague or misleading On Thu, Sep 20, 2018 at 12:38 PM Bryan Bende wrote: > That means there is no process group with the id you specified. > On Thu, Sep 20, 2018 at 12:36 PM

Re: upload template with REST problem

http://10.10.100.105:8080/nifi-api/process-groups/f2ae3d20-e105-37a1--/templates/upload > On Wed, Sep 19, 2018 at 4:17 PM l vic wrote: > > > > Hi, > > I have template for the processing group "test-template.xml" with > > id=f2ae3d20-e105-37a

UpdateAttributes use problems

Hello, I am trying to use UpdateAttributes remove 'value_filter" and "name" attributes from flowfile and add timestamp attribute, so I specified "value_filter | name" in "DeleteAttributesExpression" and added attribute "ts" with value '${now():toNumber()}'... No changes in resulting flowfile and

Re: Getting a timestamp for today at midnight?

, which > essentially results in > all hours/mins/seconds/milliseconds getting dropped. > > Thanks > -Mark > > > > On Nov 14, 2018, at 9:01 AM, l vic wrote: > > > > I have to retrieve "today's" records in ExecuteSQL, eg. with the query > using times

InvokeHTTP failure to schedule fro CRON?

Hello, I am trying to perform "get" request over SSL from InvokeHTTP nifi-1.5.0-RC1; I configured SSL by the means of a StandardSSLContextService with jks certificate (see attached) When I try to

Re: InvokeHTTP failure to schedule fro CRON?

rts, > the password, and the list command. For example: > > keytool -storepass changeit -keystore > /Library/Java/JavaVirtualMachines/jdk1.8.0_192.jdk/Contents/Home/jre/lib/security/cacerts > -list > > - Jeff > > On Fri, Dec 21, 2018 at 2:55 PM l vic wrote: > >&g

Re: InvokeHTTP failure to schedule fro CRON?

; Path: $JAVA_HOME/jre/lib/security/cacerts (i.e. > /Library/Java/JavaVirtualMachines/jdk1.8.0_101.jdk/Contents/Home/jre/lib/security/cacerts) > Password: changeit (default) > Type: JKS > > > Andy LoPresto > alopre...@apache.org > *alopresto.apa...@gmail.com * > PGP Fingerprint: 70EC

Re: setting topic name dynamically in PutKafka?

ant to do it for each flowfile. > > > > On Tue, Nov 20, 2018 at 10:09 AM l vic wrote: > >> > >> Hi, > >> I have to set topic name property of PutKafka processors from an > attribute of the incoming flowfile...Is there any possible way of doing > this? > >> Thank you, > >> -V >

Re: UpdateAttributes use problems

one does not affect the other. If you > want to manipulate your JSON content, look at the JoltTransformJSON > processor: > https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.8.0/org.apache.nifi.processors.standard.JoltTransformJSON/index.html > &g

setting topic name dynamically in PutKafka?

Hi, I have to set topic name property of PutKafka processors from an attribute of the incoming flowfile...Is there any possible way of doing this? Thank you, -V

Re: UpdateAttributes use problems

h wrote: > I set up an UpdateAttribute processor with the same configuration as the > screenshot you sent and it worked for me. > > > > What version of NiFi are you using? > > > > *From: *l vic > *Reply-To: * > *Date: *Friday, November 16, 2018 at 1:53 PM

Re: UpdateAttributes use problems

1.7.1 On Fri, Nov 16, 2018 at 4:27 PM Nathan Gough wrote: > I set up an UpdateAttribute processor with the same configuration as the > screenshot you sent and it worked for me. > > > > What version of NiFi are you using? > > > > *From: *l vic > *Reply-To: * &g

merging with default values?

I have to merge results of 3 different queries into one flowfile The problem happens if some of the queries result in empty resultset. For example, suppose i have 3 incoming connections for MergeContent: query1->flowfile1/attribute1, query2->flowfile2/attribute2, query3->flowfile3/attribute3

Re: merging with default values?

Solutions Architect > Progenity, Inc. > > > > *From:* l vic [mailto:lvic4...@gmail.com] > *Sent:* Thursday, January 10, 2019 2:18 PM > *To:* users@nifi.apache.org > *Subject:* merging with default values? > > > > I have to merge results of 3 different queries into one flo

problem with merging of failed response

I have to merge results of "original", REST call and sql query responses into one flowfile/json The problem happens if REST call, or sql query result in empty resultset. For example, suppose i have 3 incoming connections for Merge: original->flowfile1/attribute1, REST->flowfile2/attribute2,

Re: PutKafka use with large quantity of data?

on and optional back-off. > > Is this what you have in mind? > > Thanks > > On Thu, Apr 4, 2019 at 10:34 AM l vic wrote: > >> I have to ingest large (200,000 messages) data set into Kafka topic as >> quickly as possible without overrunning topic... Right now I just

PutKafka use with large quantity of data?

I have to ingest large (200,000 messages) data set into Kafka topic as quickly as possible without overrunning topic... Right now I just use rate limiter to do it but can be there some better "adaptive" way to do it? Thank you... -V

Re: PutKafka use with large quantity of data?

led description of what > you're doing, observing with nifi and the system and what you expect it to > be doing. > > Thanks > > On Thu, Apr 4, 2019 at 10:56 AM l vic wrote: > >> No, actually what happens is - NiFi stops responding ( if I use it >> without rate contol) &

Re: PutKafka use with large quantity of data?

system and what you expect it to > be doing. > > Thanks > > On Thu, Apr 4, 2019 at 10:56 AM l vic wrote: > >> No, actually what happens is - NiFi stops responding ( if I use it >> without rate contol) >> >> >> On Thu, Apr 4, 2019 at 10:42 AM Joe

jolt transform spec ?

I have json message that contains another json message in textual form: { "one": "one". "two":2, "MSG": "{\"src\":\"my source\",\"priority\":\"0\"}" } What would be transform spec to get "contained" message in json ? { "src": "my source", "priority": "0" } I've tried the following spec: [ {

jolt transform

I have json message that contains another json message in textual form: { "one": "one". "two":2, "MSG": "{\"src\":\"my source\",\"priority\":\"0\"}" } What would be transform spec to get "contained" message in json ? { "src": "my source", "priority": "0" } Thank you

Re: jolt transform spec ?

ssage'. > Then use ReplaceText with 'Replacement Value' as > "${message:unescapeJson()}" to update FlowFile content with the normal > JSON representation. > > Thanks, > Koji > > On Thu, Mar 7, 2019 at 5:18 AM l vic wrote: > > > > I have json message that con

Re: sensitive variable values ?

I don't see how to do it for service from NiFi REST api... Any suggestions? Thank you > >

sensitive variable values ?

Is there a mechanism in Nifi to use sensitive variable values? For example: would it be possible to store/set password value in StandardRestrictedSSLContextService without exposing it? I am looking for the way to set password in StandardRestrictedSSLContextService from the automated deployment

Re: PutSQL benchmarking ?

scribe your flow (processors, e.g.)? > > Thanks, > Matt > > On Wed, Feb 6, 2019 at 3:41 PM l vic wrote: > > > > Hi Matt, > > No, I don't do record splitting, data looks like { > "attr1":"val1",...[{}]} > > where "parent&quo

PutSQL benchmarking ?

I have performance issues with PutSQL i my flow... Is there some way to benchmark time required to write certain number of records to table from GenerateFlowFile? Thank you,

Re: PutSQL benchmarking ?

few through PDR, and check the > provenance events for fields such as durationMillis or calculations > like (timestampMills - lineageStart). > > Regards, > Matt > > On Wed, Feb 6, 2019 at 2:07 PM l vic wrote: > > > > I have performance issues with PutSQL i my flow.

Re: PutSQL benchmarking ?

ap that needs filling. I'm hopefully wrapping up some > stuff shortly, and would like to take a crack at upsert for PDR. > > Regards, > Matt > > On Thu, Feb 7, 2019 at 12:54 PM l vic wrote: > > > > Sorry, I realize i do indeed perform record splitting, the problem with

source build error?

Any idea what the problem is? nifi-standard-processors: *Could not resolve dependencies for project org.apache.nifi:nifi-standard-processors:jar:1.6.0: Failure to find com.martiansoftware:macnificent:jar:0.2.0*

Re: [EXT] Unexpected char exception in InvokeHTTP

ASCII space, I wonder if that’s what’s causing your issue? > > > > Cheers, > > > > Michael J. Vincent > > Lead Network Systems Engineer | The MITRE Corporation | Network Technology > & Security (T864) | +1 (781) 271-8381 > > > > *From:* l vic >

Unexpected char exception in InvokeHTTP

I am using InvokeHTTP with StandardRestrictedSSLContextService 1.6.0 to perform REST call with Proxy Type = https On some systems it just works, on other systems i have the following error: InvokeHTTP[id=d3f4b6c9-1204-3e1a-1373-431715543d24] Routing to Failure due to exception: Unexpected char

Re: [EXT] Unexpected char exception in InvokeHTTP

nce 0x20 is an ASCII space, I wonder if that’s what’s causing your issue? > > > > Cheers, > > > > Michael J. Vincent > > Lead Network Systems Engineer | The MITRE Corporation | Network Technology > & Security (T864) | +1 (781) 271-8381 > > > >

Re: Unexpected char exception in InvokeHTTP

hat HTTP Headers > are allowed to have spaces. So you'll want to check your configuration of > the InvokeHTTP processor and see > why it's trying to send that header. > > On Jan 30, 2019, at 12:01 PM, l vic wrote: > > I am using InvokeHTTP with StandardRestrictedSSLContextService 1.

expression failure in URL concatenation?

I am using processor group variable as base part of my URL: REST_URL=http://localhost:8080/nifi-api I am trying to append second part of URL in InvokeHTTP regardless if REST_URL ends with '/', or not so that concatenation of " http://localhost:8080/nifi-api/;, or "http://localhost:8080/nifi-api;

Re: expression failure in URL concatenation?

value of the REST_URL variable ends with a slash... > > Thanks > -Mark > > > On Jan 31, 2019, at 4:42 PM, l vic wrote: > > I am using processor group variable as base part of my URL: > REST_URL=http://localhost:8080/nifi-api > > I am trying to append second part of

Re: expression failure in URL concatenation?

No On Thu, Jan 31, 2019 at 4:48 PM James Srinivasan wrote: > Out of interest, does the URL containing the double slash work? > > On Thu, 31 Jan 2019, 21:42 l vic >> I am using processor group variable as base part of my URL: >> REST_URL=http://localhost:8080/nifi-

ISOChronology error in InvokeHTTP?

Have this error in ver. 1.5 of InvokeHTTP (see below), any idea how to fix it? WARN [Timer-Driven Process Thread-8] o.a.n.c.t.ContinuallyRunProcessorTask Administratively Yielding InvokeHTTP[id=2691affc-0034-3c06-3a1f-5a38cb1fc455] due to uncaught Exception: java.lang.NoClassDefFoundError: Could

merging flowfile content and values?

Hello, I have flow file with “root attributes” such as id=‘9b892935-c0e7-42ba-958b-191138c71b39’, name=‘appliance1’ and json array of user account info as content: [{“username”: “one”, “create_date":1550500579036, “role”: “tech lead”}, {“username”: “two”, “create_date":1550500579100, “role”:

problem with setting of Time Duration property from group variable

I am trying to set Time Duration property in ControlRate processor from processor group variable. I have INTERVAL variable set to 20 and I tried to set Time Duration as ${INTERVAL} mins ${INTERVAL:toNumber() mins} I each case i have error like the following: Time duration validated against

ReplaceText removes second '\' character?

I have ReplaceText 1.7.0 performing substitution with windows registry path string value: windows_path: " "C:\\Users\\Administrator\\AppData\\Local\\Microsoft\\OneDrive\\19.070.0410.0007\\ck\\Controls\\Styles\\Flat\\qtquickextrasflatplugin.dll" SearchValue: (?s)(^.*$) ReplacementValue:

jolt array name to attribute?

i have input json such as the following example: { "Address": [ { "Street": "Washington St", "ZIP": "03420", "unit": "1" } ] } How can i use Jolt transform to add array name as attribute value to array members as following result: { *"type": "Address"*, "Street": "Washington St", "ZIP":

Re: jolt array name to attribute?

P": "03420", "unit": "1", "type": "HomeAddress" } ], "BusinessAddress": [ { "Street": "PO Box 22", "ZIP": "03420", "unit": "none", "type": "BusinessAddress" } ] } On Wed, Jan 15, 2020

adding original attribute values to InvokeHttp response?

How can I add original attribute values into InvokeHttp response context (Json)? Thank you,