Hello Dweep,
In putparquet processor you can set the attribute '*Hadoop Configuration
Resources*' to a *core-site.xml* file whose content can be somewhat like
below:
<configuration>
<property>
<name>fs.defaultFS</name>
<value>file:///reservoir-dl</value>
</property>
</configuration>
Here the file:///reservoir-dl could be your path where in-transit parquet
files have be written to -- before being pushed to S3.
More importantly, you *do not* need Hadoop to be installed. You can just
place the core-site.xml file on your NiFi nodes and get started.
On Fri, Jul 5, 2019 at 1:54 PM Dweep Sharma <[email protected]> wrote:
> Hi,
>
> I have been trying to move some JSON data to S3 in Parquet format.
>
> From Kafka to s3 is straight forward but I cannot seem to find the right
> processor to convert JSON to parquet and move it to s3.
>
> putparquet does not take s3 bucket or credentials and requires hadoop to
> be installed.
>
> Can someone please share a blog or steps to achieve this. Thanks in advance
>
> -Dweep
>
>
>
>
> *::DISCLAIMER::----------------------------------------------------------------------------------------------------------------------------------------------------The
> contents of this e-mail and any attachments are confidential and intended
> for the named recipient(s) only.E-mail transmission is not guaranteed to be
> secure or error-free as information could be intercepted, corrupted,lost,
> destroyed, arrive late or incomplete, or may contain viruses in
> transmission. The e mail and its contents(with or without referred errors)
> shall therefore not attach any liability on the originator or redBus.com.
> Views or opinions, if any, presented in this email are solely those of the
> author and may not necessarily reflect the views or opinions of redBus.com.
> Any form of reproduction, dissemination, copying, disclosure,
> modification,distribution and / or publication of this message without the
> prior written consent of authorized representative of redbus.
> <http://redbus.in/>com is strictly prohibited. If you have received this
> email in error please delete it and notify the sender immediately.Before
> opening any email and/or attachments, please check them for viruses and
> other defects.*
--
Best,
Sneh
