I meant to say that in ReplaceText you would need use "Always Replace" with
mode of "Entire Text".
On Tue, Apr 9, 2019 at 10:38 AM Matt Burgess <mattyb...@apache.org> wrote:
> Since you know you just want to overwrite the contents with HiveQL, you
> could use ExecuteScript with the following Groovy script:
>
> def flowFile = session.get()
> if(!flowFile) return
> try {
> def path = flowFile.getAttribute('absolute.hdfs.path')
> flowFile = session.write(flowFile, {outStream ->
> outStream.write("LOAD DATA INPATH '$path' INTO TABLE myTable".bytes)
> } as OutputStreamCallback)
> session.transfer(flowFile, REL_SUCCESS)
> } catch(e) {
> log.error('Error generating HiveQL', e)
> session.transfer(flowFile, REL_FAILURE)
> }
>
> Or better yet, you can paste that into InvokeScriptedProcessor using a
> template [1] I created to allow users to port their ExecuteScript body to
> the faster InvokeScriptedProcessor. It might be nice to have a
> SetFlowFileContent processor for this use case, if ReplaceText does load
> the incoming content into memory.
>
> Regards,
> Matt
>
> [1]
> https://funnifi.blogspot.com/2017/06/invokescriptedprocessor-template-faster.html
>
>
>
> On Tue, Apr 9, 2019 at 9:49 AM DEHAY Aurelien <aurelien.de...@faurecia.com>
> wrote:
>
>> Hello.
>>
>>
>>
>> We would like, in a flow to:
>>
>> 1) Read data from kafka
>>
>> 2) Merge records
>>
>> 3) PutParquet on HDFS
>>
>> 4) Load data in Hive using LOAD DATA hiveql.
>>
>>
>>
>> The first 3 steps are no problem, it’s working fine.
>>
>>
>>
>> But we wonder what is the best way to launch the hiveql command. We
>> planned to use the PutHiveQL processor, but it needs the command in the
>> flowfile content.
>>
>>
>>
>> Using generateflowfile would be nice, but we can’t generate the event to
>> trigger the generateflowfile, preventing too to use the wait/notify.
>>
>>
>>
>> What is the best way? I suppose replacetext or so would be too “heavy”
>> has it requires to load the message in memory?
>>
>>
>>
>> Thanks for any pointer/idea.
>>
>>
>>
>>
>>
>>
>> *Aurélien DEHAY *Big Data Architect
>> +33 616 815 441
>>
>> aurelien.de...@faurecia.com
>>
>> 23/27 avenue des Champs Pierreux
>> 92735 Nanterre Cedex – France
>>
>> [image: Faurecia_inspiring_mobility_logo-RVB_150]
>>
>>
>>
>> This electronic transmission (and any attachments thereto) is intended
>> solely for the use of the addressee(s). It may contain confidential or
>> legally privileged information. If you are not the intended recipient of
>> this message, you must delete it immediately and notify the sender. Any
>> unauthorized use or disclosure of this message is strictly prohibited.
>> Faurecia does not guarantee the integrity of this transmission and shall
>> therefore never be liable if the message is altered or falsified nor for
>> any virus, interception or damage to your system.
>>
>
