Re: Does dataframe write append mode work with text format

[Xiao Li]

Hi, Yash,

It should work.

 val df = spark.range(1, 5)
   .select('id + 1 as 'p1, 'id + 2 as 'p2, 'id + 3 as 'p3, 'id + 4 as 'p4,
'id + 5 as 'p5, 'id as 'b)
   .selectExpr("p1", "p2", "p3", "p4", "p5", "CAST(b AS STRING) AS
s").coalesce(1)

 df.write.partitionBy("p1", "p2", "p3", "p4",
"p5").text(dir.getCanonicalPath)
 val newDF = spark.read.text(dir.getCanonicalPath)
 newDF.show()

 df.write.partitionBy("p1", "p2", "p3", "p4", "p5")
   .mode(SaveMode.Append).text(dir.getCanonicalPath)
 val newDF2 = spark.read.text(dir.getCanonicalPath)
 newDF2.show()

I tried it. It works well.

Thanks,

Xiao Li

2016-06-18 8:57 GMT-07:00 Yash Sharma <yash...@gmail.com>:

> Hi All,
> I have been using the parquet append mode for write which works just
> fine. Just wanted to check if the same is supported for plain text format.
> The below code blows up with error saying the file already exists.
>
>
>
> {code}
> userEventsDF.write.mode("append").partitionBy("year", "month",
> "date").text(outputDir)
> or,
> userEventsDF.write.mode("append").partitionBy("year", "month",
> "date").format("text").save(outputDir)
> {code}
>