RE: Spark SQL joins taking too long

2016-01-27 Thread Cheng, Hao
”); … And then doing the join. From: Raghu Ganti [mailto:raghuki...@gmail.com] Sent: Thursday, January 28, 2016 3:06 AM To: Ted Yu; Дмитро Попович Cc: user Subject: Re: Spark SQL joins taking too long The problem is with the way Spark query plan is being created, IMO, what was happening before

Re: Spark SQL joins taking too long

2016-01-27 Thread Raghu Ganti
n(10).registerTempTable(“counties”); > … > And then doing the join. > > > From: Raghu Ganti [mailto:raghuki...@gmail.com] > Sent: Thursday, January 28, 2016 3:06 AM > To: Ted Yu; Дмитро Попович > Cc: user > Subject: Re: Spark SQL joins taking too long > > The pr

Re: Spark SQL joins taking too long

2016-01-27 Thread Raghu Ganti
ocation)"); >> long count = joinDf.count(); //gives me the correct answer of 8 >> >> Any suggestions on what I can do to optimize and debug this piece of code? >> >> Regards, >> Raghu >> >> >> >> -- >> View this message in context: >&g

Spark SQL joins taking too long

2016-01-26 Thread raghukiran
//gives me the correct answer of 8 Any suggestions on what I can do to optimize and debug this piece of code? Regards, Raghu -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-SQL-joins-taking-too-long-tp26078.html Sent from the Apache Spark User

Re: Spark SQL joins taking too long

2016-01-26 Thread Ted Yu
//gives me the correct answer of 8 > > Any suggestions on what I can do to optimize and debug this piece of code? > > Regards, > Raghu > > > > -- > View this message in context: > http://apache-spark-user-

Re: Spark SQL joins taking too long

2016-01-26 Thread Raghu Ganti
;> SomeUDF('c.shape as string', h.location)"); >> long count = joinDf.count(); //gives me the correct answer of 8 >> >> Any suggestions on what I can do to optimize and debug this piece of code? >> >> Regards, >