Hi Subash,
Short answer: It’s effectively random.
Longer answer: In general the DataFrameWriter expects to be receiving data
from multiple partitions. Let’s say you were writing to ORC instead of text.
In this case, even when you specify the output path, the writer creates a
directory at the
Hi,
While saving in Spark2 as text file - I see encoded/hash value attached in
the part files along with part number. I am curious to know what is that
value is about ?
Example:
ds.write.save(SaveMode.Overwrite).option("compression","gzip").text(path)
Produces,
