Re: Issue: Skew on Dataframes while Joining the dataset

That kind of issue SparkUI and DAG visualization always helpful. https://databricks.com/blog/2015/06/22/understanding-your-spark-application-through-visualization.html Regards, Vaquar khan On Fri, Dec 16, 2016 at 11:10 AM, Vikas K. wrote: > Unsubscribe. > > On Fri,

Re: Issue: Skew on Dataframes while Joining the dataset

Unsubscribe. On Fri, Dec 16, 2016 at 9:21 PM, KhajaAsmath Mohammed < mdkhajaasm...@gmail.com> wrote: > Hi, > > I am facing an issue with join operation on dataframe. My job is running > for very long time( > 2 hrs ) without any result. can someone help me on > how to resolve. > > I tried

Re: Issue: Skew on Dataframes while Joining the dataset

Hi, I am able to resolve this issue. Culprit was the SQL query adding one more join returned records in less time. Thanks, Asmath On Fri, Dec 16, 2016 at 9:51 AM, KhajaAsmath Mohammed < mdkhajaasm...@gmail.com> wrote: > Hi, > > I am facing an issue with join operation on dataframe. My job is

Issue: Skew on Dataframes while Joining the dataset

Hi, I am facing an issue with join operation on dataframe. My job is running for very long time( > 2 hrs ) without any result. can someone help me on how to resolve. I tried re-partition with 13 but no luck. val results_dataframe = sqlContext.sql("select gt.*,ct.* from PredictTempTable