Re: broadcast join in SparkSQL requires analyze table noscan

[Michael Armbrust]

>
> My question is that is "NOSCAN" option a must? If I execute "ANALYZE TABLE
>  compute statistics" command in Hive shell, is the statistics
> going to be used by SparkSQL to decide broadcast join?


Yes, spark SQL will only accept the simple no scan version.  However, as
long as the sizeInBytes statistic is present, we will use it.

Re: broadcast join in SparkSQL requires analyze table noscan

[Lan Jiang]

Michael,

Thanks for the reply.

On Wed, Feb 10, 2016 at 11:44 AM, Michael Armbrust 
wrote:

> My question is that is "NOSCAN" option a must? If I execute "ANALYZE TABLE
>>  compute statistics" command in Hive shell, is the statistics
>> going to be used by SparkSQL to decide broadcast join?
>
>
> Yes, spark SQL will only accept the simple no scan version.  However, as
> long as the sizeInBytes statistic is present, we will use it.
>
>