Re: Spark S3

Hi Selvam, Is your 35GB parquet file split up into multiple S3 objects or just one big Parquet file? If its just one big file then I believe only one executor will be able to work on it until some job action partitions the data into smaller chunks. On 11 October 2016 at 06:03, Selvam Raman

Re: IDE for python

I use Pycharm and it works a treat. The big advantage I find is that I can use the same command shortcuts that I do when developing with IntelliJ IDEA when doing Scala or Java. On 27 June 2017 at 23:29, ayan guha wrote: > Depends on the need. For data exploration, i use