Issue: Skew on Dataframes while Joining the dataset

[KhajaAsmath Mohammed]

Hi,

I am facing an issue with join operation on dataframe. My job is running
for very long time( > 2 hrs ) without any result. can someone help me on
how to resolve.

I tried re-partition with 13 but no luck.


val results_dataframe = sqlContext.sql("select gt.*,ct.* from
PredictTempTable pt,ClusterTempTable ct,GamificationTempTable gt where
gt.vin=pt.vin and pt.cluster=ct.cluster")
//val results_dataframe_partitioned=results_dataframe.coalesce(numPartitions)
val results_dataframe_partitioned=results_dataframe.repartition(13)

[image: Inline image 1]

Thanks,
Asmath