Re: Converting timezones in Spark

2017-01-31 Thread Don Drake
So, to follow up on this. A few lessons learned, when you print a timestamp, it will only show the date/time in your current timezone, regardless of any conversions you applied to it. The trick is to convert it (cast) to a Long, and then the Java8 java.time.* functions can translate to any

Converting timezones in Spark

2017-01-27 Thread Don Drake
I'm reading CSV with a timestamp clearly identified in the UTC timezone, and I need to store this in a parquet format and eventually read it back and convert to different timezones as needed. Sounds straightforward, but this involves some crazy function calls and I'm seeing strange results as I