Re: Sorting each partitions and writing to CSVs

2017-01-24 Thread Ivan Gozali
For those interested, after digging further, I was able to consistently reproduce the issue with a synthetic dataset. My findings are documented here: https://gist.github.com/igozali/d327a85646abe7ab10c2ae479bed431f -- Regards, Ivan Gozali Lecida Email: i...@lecida.com On Wed, Jan 18, 2017

Sorting each partitions and writing to CSVs

2017-01-18 Thread Ivan Gozali
tially a bug in Spark. I'm using Spark 2.0.2 with Python 2.7.12. Any advice would be very much appreciated! -- Regards, Ivan Gozali