Re: JoltTransformJSON error
Matt / Yolanda,
Thanks for taking the reply. I just found time to work on this. I removed
the array brackets from the spec and changed it to 'shift', but this is the
only output I am getting as of now.
{
"shift": "Update"
}
I am not getting the entire tree as in the app-demo. Is there something
else I need to change?
Thanks,
Karthik R.
On Wed, Oct 5, 2016 at 7:43 PM, Yolanda Davis
wrote:
> Matt is absolutely correct, your specification is actually in the form of
> Chain spec, where you could have one or more operations defined. In your
> case you have just one operation in your chain so you can try changing the
> drop down to Chain or edit your your specification to only include the
> portion that is in braces next to the "spec" variable as Matt suggested and
> leave the specification type as Shift. Either method should fix the issue.
>
> Please let us know how it goes!
>
> Yolanda
>
> Sent from my iPhone
>
> On Oct 5, 2016, at 7:40 PM, Matt Burgess wrote:
>
> I'm not near my computer but my knee-jerk reaction is that all the
> jolt-app-demo transforms are actually Chain transforms, some (like your
> example) with a single transform inside (like a Shift). Try removing the
> array brackets if you're selecting a Shift transform, or choose Chain and
> keep them in.
>
> Yolanda knows MUCH more about this than I do, so if that doesn't fix your
> problem, she likely has the solution (or will identify any bug). No
> pressure Yolanda ;)
>
> Regards,
> Matt
>
> On Oct 5, 2016, at 7:13 PM, Karthik Ramakrishnan <
> karthik.ramakrishna...@gmail.com> wrote:
>
> Hi Yolanda,
>
> Thanks for taking this up.
>
> I am running *NiFi 1.0.0* in my local. I have attached the screen from
> both the Processor's Advanced UI and the Jolt Transform Demo website.
>
> If I click Transform - There is no change in the UI. It doesn't even throw
> and error. When I see the processor in the main UI, it is giving the error
> which I had mentioned. Appreciate any help possible. TIA
>
> Regards,
> Karthik
>
>
> On Wed, Oct 5, 2016 at 5:33 PM, Yolanda Davis
> wrote:
>
>> Hi Karthik,
>>
>> What version of NiFi are you using? Also do you mind sharing the
>> specification that you are attempting to use? If you could provide a
>> picture/snapshot of the processor's configuration using the Advance UI that
>> would help me to try to recreate the problem or provide a solution.
>>
>> Thanks,
>>
>> Yolanda
>>
>> On Wed, Oct 5, 2016 at 4:29 PM, Karthik Ramakrishnan <
>> karthik.ramakrishna...@gmail.com> wrote:
>>
>>> Hello -
>>>
>>> I am using the JoltTransformJSON processor for a Jolt transformation. I
>>> am getting an error:
>>>
>>> *" is invalid because Specification is not valid for the given
>>> transformation*
>>>
>>> I am using a "Shift" DSL and my spec is working while testing in -
>>> http://jolt-demo.appspot.com/
>>>
>>> Can anyone throw some light on this? This is the only blocker that is
>>> hindering my flow as of now.
>>>
>>> Thanks!!!
>>>
>>>
>>> --
>>> Thanks,
>>> *Karthik Ramakrishnan*
>>> *Intern - Data Services*
>>> *Copart Inc.*
>>> *Contact : +1 (469) 951-8854 <%2B1%20%28469%29%20951-8854>*
>>>
>>>
>>
>>
>> --
>> --
>> yolanda.m.da...@gmail.com
>> @YolandaMDavis
>>
>>
>
>
> --
> Thanks,
> *Karthik Ramakrishnan*
> *Contact : +1 (469) 951-8854 <%2B1%20%28469%29%20951-8854>*
>
>
>
>
>
>
--
Thanks,
*Karthik Ramakrishnan*
*Contact : +1 (469) 951-8854*
Re: JoltTransformJSON error
Hi Yolanda, Thanks for taking this up. I am running *NiFi 1.0.0* in my local. I have attached the screen from both the Processor's Advanced UI and the Jolt Transform Demo website. If I click Transform - There is no change in the UI. It doesn't even throw and error. When I see the processor in the main UI, it is giving the error which I had mentioned. Appreciate any help possible. TIA Regards, Karthik On Wed, Oct 5, 2016 at 5:33 PM, Yolanda Davis wrote: > Hi Karthik, > > What version of NiFi are you using? Also do you mind sharing the > specification that you are attempting to use? If you could provide a > picture/snapshot of the processor's configuration using the Advance UI that > would help me to try to recreate the problem or provide a solution. > > Thanks, > > Yolanda > > On Wed, Oct 5, 2016 at 4:29 PM, Karthik Ramakrishnan < > karthik.ramakrishna...@gmail.com> wrote: > >> Hello - >> >> I am using the JoltTransformJSON processor for a Jolt transformation. I >> am getting an error: >> >> *" is invalid because Specification is not valid for the given >> transformation* >> >> I am using a "Shift" DSL and my spec is working while testing in - >> http://jolt-demo.appspot.com/ >> >> Can anyone throw some light on this? This is the only blocker that is >> hindering my flow as of now. >> >> Thanks!!! >> >> >> -- >> Thanks, >> *Karthik Ramakrishnan* >> *Intern - Data Services* >> *Copart Inc.* >> *Contact : +1 (469) 951-8854 <%2B1%20%28469%29%20951-8854>* >> >> > > > -- > -- > yolanda.m.da...@gmail.com > @YolandaMDavis > > -- Thanks, *Karthik Ramakrishnan* *Contact : +1 (469) 951-8854*
JoltTransformJSON error
Hello - I am using the JoltTransformJSON processor for a Jolt transformation. I am getting an error: *" is invalid because Specification is not valid for the given transformation* I am using a "Shift" DSL and my spec is working while testing in - http://jolt-demo.appspot.com/ Can anyone throw some light on this? This is the only blocker that is hindering my flow as of now. Thanks!!! -- Thanks, *Karthik Ramakrishnan* *Intern - Data Services* *Copart Inc.* *Contact : +1 (469) 951-8854*
Re: Generating SQL from JSON
Thanks Bryan. My bad. I missed this big time. Regards, Karthik On Wed, Sep 21, 2016 at 2:26 PM, Bryan Bende wrote: > Hello, > > Have you taken a look at ConvertJSONToSQL? > > https://nifi.apache.org/docs/nifi-docs/components/org. > apache.nifi.processors.standard.ConvertJSONToSQL/index.html > > -Bryan > > > On Wed, Sep 21, 2016 at 2:55 PM, Karthik Ramakrishnan < > karthik.ramakrishna...@gmail.com> wrote: > >> Hello -, >> >> I wish to know if there are any processors that do this? I have seen most >> of them and none seem to fit the purpose well, or it seems so to me. I have >> a JSON and it contains all the data information for the row and I would >> extract all the attributes and then make a SQL statement out of it and that >> would be executed by PutSQL. Is this possible or should I go for a custom >> processor? >> >> -- >> Thanks, >> *Karthik Ramakrishnan* >> *Contact : +1 (469) 951-8854 <%2B1%20%28469%29%20951-8854>* >> >> > -- Thanks, *Karthik Ramakrishnan* *Contact : +1 (469) 951-8854*
Generating SQL from JSON
Hello -, I wish to know if there are any processors that do this? I have seen most of them and none seem to fit the purpose well, or it seems so to me. I have a JSON and it contains all the data information for the row and I would extract all the attributes and then make a SQL statement out of it and that would be executed by PutSQL. Is this possible or should I go for a custom processor? -- Thanks, *Karthik Ramakrishnan* *Contact : +1 (469) 951-8854*
Re: Does NiFi support multiple queries
Thank you people!! Great help. On Wed, Sep 21, 2016 at 7:19 AM, Peter Wicks (pwicks) wrote: > Karthik, > > > > PutSQL will handle both styles, and for multiple tables, without issue. > > > > Internally it creates a separate SQL Batch for each distinct SQL statement > in the queue and then executes these batches separately. Feel free to mix > as many Inserts/Updates as you wish for as many tables as you wish. > > > > Thanks, > > Peter > > > > *From:* Karthik Ramakrishnan [mailto:karthik.ramakrishna...@gmail.com] > *Sent:* Tuesday, September 20, 2016 9:57 PM > *To:* users@nifi.apache.org > *Subject:* Does NiFi support multiple queries > > > > Hello - > > > > I was wondering if NiFi can support multiple queries in the same PutSQL > processor. For example, if an attribute is set to 'update' - will PutSQL > run the defined update query and next time when it is an 'insert' - it runs > the insert query. Or should we go ahead and add two separate processors and > make a decision on the RouteAttributes processor? Any thoughts would be > welcome!! > > > > TIA!! > > > > -- > > Thanks, > > *Karthik Ramakrishnan* > > *Data Services Intern* > > *Copart Inc.* > > *Contact : +1 (469) 951-8854 <%2B1%20%28469%29%20951-8854>* > > > -- Thanks, *Karthik Ramakrishnan* *Contact : +1 (469) 951-8854*
Does NiFi support multiple queries
Hello - I was wondering if NiFi can support multiple queries in the same PutSQL processor. For example, if an attribute is set to 'update' - will PutSQL run the defined update query and next time when it is an 'insert' - it runs the insert query. Or should we go ahead and add two separate processors and make a decision on the RouteAttributes processor? Any thoughts would be welcome!! TIA!! -- Thanks, *Karthik Ramakrishnan* *Data Services Intern* *Copart Inc.* *Contact : +1 (469) 951-8854*
Re: ETL processors for NiFi
Hey Mark, This would help me a lot in building the flow. Thanks a bunch!! Warm Regards, Karthik R. On Mon, Sep 19, 2016 at 3:11 PM, Mark Payne wrote: > Hi Karthik, > > I think what you want to be using here is PutSQL, rather than ExecuteSQL. > ExecuteSQL > is designed to perform a SELECT statement, whereas PutSQL would update a > database. > > PutSQL expects the incoming FlowFile to contain the SQL to execute. > So you could use ReplaceText with the 'Replacement Strategy' set to > "Always Replace" > and set the Replacement Value to whatever SQL you want to use. The > ReplaceText processor > supports the Expression Language, so you can reference whatever FlowFile > attributes you would > like to formulate the message. > > How to get information from the Kafka message into your FlowFile > attributes depends on the type of > data that you are pulling. If it is JSON, for instance, you could use > EvaluateJsonPath to extract out > particular pieces of information into FlowFile attributes. > > Does this help? > > Thanks > -Mark > > On Sep 19, 2016, at 3:58 PM, Karthik Ramakrishnan < > karthik.ramakrishna...@gmail.com> wrote: > > Hello All, > > I am currently developing an ETL pipeline and I am using Apache NiFi for > this task. I would like to know if there is any processor that help with > the following use case: I am consuming from Kafka and based on some > attributes in the message then I am updating the database with the data. I > have a GetKafka for getting the messages and a ExecuteSQL processor for > updating the database. I am stuck in finding a processor that reads the > Kafka message and checks the attribute and then sends the data values to > the executeSQL processor. Can someone help on this? > > Thanks!! > > -- > Thanks, > *Karthik Ramakrishnan* > *Data Services Intern* > *Copart Inc.* > *Dallas, TX* > *Contact : +1 (469) 951-8854 <%2B1%20%28469%29%20951-8854>* > > > -- Thanks, *Karthik Ramakrishnan* *Contact : +1 (469) 951-8854*
ETL processors for NiFi
Hello All, I am currently developing an ETL pipeline and I am using Apache NiFi for this task. I would like to know if there is any processor that help with the following use case: I am consuming from Kafka and based on some attributes in the message then I am updating the database with the data. I have a GetKafka for getting the messages and a ExecuteSQL processor for updating the database. I am stuck in finding a processor that reads the Kafka message and checks the attribute and then sends the data values to the executeSQL processor. Can someone help on this? Thanks!! -- Thanks, *Karthik Ramakrishnan* *Data Services Intern* *Copart Inc.* *Dallas, TX* *Contact : +1 (469) 951-8854*
