This partially answers the question:

On 10/04/2016 03:10 PM, Samy Dindane wrote:

I have the following schema:


I'd like to achieve either of these:

1) Use the timestamp field to partition by year, month and day.
This looks weird though, as Spark wouldn't magically know how to load the data 
back since the year, month and day columns don't exist in the schema.

2) If 1) is not possible, partition data by date.year, date.month and
`df.write.partitionBy('date.year')` does not work, since the `date.year` column 
does not exist in the schema.

If 2) isn't possible either, I'll just move year, month and day to the root of 
the schema, which I don't like as it bloats it.

Do you know if any of these is possible?

Thank you,


To unsubscribe e-mail:

To unsubscribe e-mail:

Reply via email to