Thanks Shanker, But I do not see options in putparquet for s3 bucket/credential, I am assuming I need to push this to a local store and then add the puts3object processor on top of that ?
Also, as record reader in putparquet I am using the JsonTreeReader with defaults (InferSchema) and I get the error "Failed to write due to can't redefine: org.apache.nifi.addresstype: Some files however do get written. Are the default settings good or am I missing something ? -Dweep On Fri, Jul 5, 2019 at 11:43 PM Andrew Grande <[email protected]> wrote: > Interestingly enough, the ORC processor in NiFi can just use defaults if > hadoop configs aren't provided, no additional config steps required. Is it > something which can be improved for PutParquet maybe? > > Andrew > > > > On Fri, Jul 5, 2019, 4:18 AM Shanker Sneh <[email protected]> > wrote: > >> Hello Dweep, >> >> In putparquet processor you can set the attribute '*Hadoop Configuration >> Resources*' to a *core-site.xml* file whose content can be somewhat like >> below: >> >> <configuration> >> >> <property> >> >> <name>fs.defaultFS</name> >> >> <value>file:///reservoir-dl</value> >> >> </property> >> >> </configuration> >> >> Here the file:///reservoir-dl could be your path where in-transit parquet >> files have be written to -- before being pushed to S3. >> More importantly, you *do not* need Hadoop to be installed. You can just >> place the core-site.xml file on your NiFi nodes and get started. >> >> >> On Fri, Jul 5, 2019 at 1:54 PM Dweep Sharma <[email protected]> >> wrote: >> >>> Hi, >>> >>> I have been trying to move some JSON data to S3 in Parquet format. >>> >>> From Kafka to s3 is straight forward but I cannot seem to find the right >>> processor to convert JSON to parquet and move it to s3. >>> >>> putparquet does not take s3 bucket or credentials and requires hadoop to >>> be installed. >>> >>> Can someone please share a blog or steps to achieve this. Thanks in >>> advance >>> >>> -Dweep >>> >>> >>> >>> >>> *::DISCLAIMER::----------------------------------------------------------------------------------------------------------------------------------------------------The >>> contents of this e-mail and any attachments are confidential and intended >>> for the named recipient(s) only.E-mail transmission is not guaranteed to be >>> secure or error-free as information could be intercepted, corrupted,lost, >>> destroyed, arrive late or incomplete, or may contain viruses in >>> transmission. The e mail and its contents(with or without referred errors) >>> shall therefore not attach any liability on the originator or redBus.com. >>> Views or opinions, if any, presented in this email are solely those of the >>> author and may not necessarily reflect the views or opinions of redBus.com. >>> Any form of reproduction, dissemination, copying, disclosure, >>> modification,distribution and / or publication of this message without the >>> prior written consent of authorized representative of redbus. >>> <http://redbus.in/>com is strictly prohibited. If you have received this >>> email in error please delete it and notify the sender immediately.Before >>> opening any email and/or attachments, please check them for viruses and >>> other defects.* >> >> >> >> -- >> Best, >> Sneh >> > -- *::DISCLAIMER:: ---------------------------------------------------------------------------------------------------------------------------------------------------- The contents of this e-mail and any attachments are confidential and intended for the named recipient(s) only.E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted,lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents(with or without referred errors) shall therefore not attach any liability on the originator or redBus.com. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of redBus.com. Any form of reproduction, dissemination, copying, disclosure, modification,distribution and / or publication of this message without the prior written consent of authorized representative of redbus. <http://redbus.in/>com is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately.Before opening any email and/or attachments, please check them for viruses and other defects.*
