RE: Spark SQL joins taking too long

2016-01-27 Thread Cheng, Hao
”); … And then doing the join. From: Raghu Ganti [mailto:raghuki...@gmail.com] Sent: Thursday, January 28, 2016 3:06 AM To: Ted Yu; Дмитро Попович Cc: user Subject: Re: Spark SQL joins taking too long The problem is with the way Spark query plan is being created, IMO, what was happening before

Re: Spark SQL joins taking too long

2016-01-27 Thread Raghu Ganti
n(10).registerTempTable(“counties”); > … > And then doing the join. > > > From: Raghu Ganti [mailto:raghuki...@gmail.com] > Sent: Thursday, January 28, 2016 3:06 AM > To: Ted Yu; Дмитро Попович > Cc: user > Subject: Re: Spark SQL joins taking too long > > The pr

Re: Spark SQL joins taking too long

2016-01-27 Thread Raghu Ganti
The problem is with the way Spark query plan is being created, IMO, what was happening before is that the order of the tables mattered and when the larger table is given first, it took a very long time (~53mins to complete). I changed the order of the tables with the smaller one first (including

Re: Spark SQL joins taking too long

2016-01-26 Thread Ted Yu
What's the type of shape column ? Can you disclose what SomeUDF does (by showing the code) ? Cheers On Tue, Jan 26, 2016 at 12:41 PM, raghukiran wrote: > Hi, > > I create two tables, one counties with just one row (it actually has 2k > rows, but I used only one) and

Re: Spark SQL joins taking too long

2016-01-26 Thread Raghu Ganti
Yes, the SomeUDF is Contains, shape is a UDT that maps a custom geometry type to sql binary type. Custom geometry type is a Java class. Please let me know if you need further info. Regards Raghu > On Jan 26, 2016, at 17:13, Ted Yu wrote: > > What's the type of shape