On 1 Dec 2017, at 3:44, VinShar wrote:
yes this was my understanding also but then i found that Spark's
DataFrame
does has a method which appends to Parquet (
df.write.parquet(destName,
mode="append")). below is an article that throws some light on this. i
was
wondering if there is a way to
yes this was my understanding also but then i found that Spark's DataFrame
does has a method which appends to Parquet ( df.write.parquet(destName,
mode="append")). below is an article that throws some light on this. i was
wondering if there is a way to achieve the same through NiFi.
Hello,
As far as I know there is not an option in Parquet to append due to
the way it's internal format works.
The ParquetFileWriter has a mode which only has CREATE and OVERWRITE:
https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop
Hi,
Is there any way to use PutParquet to append to an existing parquet file? i
know that i can create a Kite DataSet and write parques to it but i am
looking for an alternate to Spark's DataFrame.write.parquet (destination,
mode="overwrite")
Regards,
Vinay
--
Sent from:
