- Try doing less in each transformation
- Try using different data structures within the transformations
- Try not caching anything to free up more memory
On Wed, May 25, 2016 at 1:32 AM, pseudo oduesp
wrote:
> hi guys ,
> -i get this errors with pyspark 1.5.0 under cloudera CDH 5.5 (yarn)
>
>
hi guys ,
-i get this errors with pyspark 1.5.0 under cloudera CDH 5.5 (yarn)
-i use yarn to deploy job on cluster.
-i use hive context and parquet file to save my data.
limit container 16 GB
number of executor i tested befor it s 12 GB (executor memory)
-i tested to increase number of partition