Re: ETL processors for NiFi

[Karthik Ramakrishnan]

Hey Mark,

This would help me a lot in building the flow. Thanks a bunch!!

Warm Regards,
Karthik R.

On Mon, Sep 19, 2016 at 3:11 PM, Mark Payne  wrote:

> Hi Karthik,
>
> I think what you want to be using here is PutSQL, rather than ExecuteSQL.
> ExecuteSQL
> is designed to perform a SELECT statement, whereas PutSQL would update a
> database.
>
> PutSQL expects the incoming FlowFile to contain the SQL to execute.
> So you could use ReplaceText with the 'Replacement Strategy' set to
> "Always Replace"
> and set the Replacement Value to whatever SQL you want to use. The
> ReplaceText processor
> supports the Expression Language, so you can reference whatever FlowFile
> attributes you would
> like to formulate the message.
>
> How to get information from the Kafka message into your FlowFile
> attributes depends on the type of
> data that you are pulling. If it is JSON, for instance, you could use
> EvaluateJsonPath to extract out
> particular pieces of information into FlowFile attributes.
>
> Does this help?
>
> Thanks
> -Mark
>
> On Sep 19, 2016, at 3:58 PM, Karthik Ramakrishnan <
> karthik.ramakrishna...@gmail.com> wrote:
>
> Hello All,
>
> I am currently developing an ETL pipeline and I am using Apache NiFi for
> this task. I would like to know if there is any processor that help with
> the following use case: I am consuming from Kafka and based on some
> attributes in the message then I am updating the database with the data. I
> have a GetKafka for getting the messages and a ExecuteSQL processor for
> updating the database.  I am stuck in finding a processor that reads the
> Kafka message and checks the attribute and then sends the data values to
> the executeSQL processor. Can someone help on this?
>
> Thanks!!
>
> --
> Thanks,
> *Karthik Ramakrishnan*
> *Data Services Intern*
> *Copart Inc.*
> *Dallas, TX*
> *Contact : +1 (469) 951-8854 <%2B1%20%28469%29%20951-8854>*
>
>
>


-- 
Thanks,
*Karthik Ramakrishnan*
*Contact : +1 (469) 951-8854*

Re: ETL processors for NiFi

[Mark Payne]

Hi Karthik,

I think what you want to be using here is PutSQL, rather than ExecuteSQL. 
ExecuteSQL
is designed to perform a SELECT statement, whereas PutSQL would update a 
database.

PutSQL expects the incoming FlowFile to contain the SQL to execute.
So you could use ReplaceText with the 'Replacement Strategy' set to "Always 
Replace"
and set the Replacement Value to whatever SQL you want to use. The ReplaceText 
processor
supports the Expression Language, so you can reference whatever FlowFile 
attributes you would
like to formulate the message.

How to get information from the Kafka message into your FlowFile attributes 
depends on the type of
data that you are pulling. If it is JSON, for instance, you could use 
EvaluateJsonPath to extract out
particular pieces of information into FlowFile attributes.

Does this help?

Thanks
-Mark

> On Sep 19, 2016, at 3:58 PM, Karthik Ramakrishnan 
>  wrote:
> 
> Hello All,
> 
> I am currently developing an ETL pipeline and I am using Apache NiFi for this 
> task. I would like to know if there is any processor that help with the 
> following use case: I am consuming from Kafka and based on some attributes in 
> the message then I am updating the database with the data. I have a GetKafka 
> for getting the messages and a ExecuteSQL processor for updating the 
> database.  I am stuck in finding a processor that reads the Kafka message and 
> checks the attribute and then sends the data values to the executeSQL 
> processor. Can someone help on this? 
> 
> Thanks!!
> 
> -- 
> Thanks,
> Karthik Ramakrishnan
> Data Services Intern
> Copart Inc.
> Dallas, TX
> Contact : +1 (469) 951-8854
>