Hi,
After taking a look at the code, I found out the problem:
As spark will use broadcastNestedLoopJoin to treat nonequality condition.
And one of my dataframe(df1) is created from an existing RDD(logicalRDD),
so it uses defaultSizeInBytes * length to estimate the size. The other
dataframe(df2) that I use is created from hive table(about 1G). Therefore
spark think df1 is larger than df2, although df1 is very small. As a
result, spark try to do df2.collect(), which causes the error.
Hope this could be helpful
Cheers
Gen
On Mon, Aug 10, 2015 at 11:29 PM, gen tang <gen.tan...@gmail.com> wrote:
> Hi,
>
> I am sorry to bother again.
> When I do join as follow:
> df = sqlContext.sql("selet a.someItem, b.someItem from a full outer join b
> on condition1 *or* condition2")
> df.first()
>
> The program failed at the result size is bigger than
> spark.driver.maxResultSize.
> It is really strange, as one record is no way bigger than 1G.
> When I do join on just one condition or equity condition, there will be no
> problem.
>
> Could anyone help me, please?
>
> Thanks a lot in advance.
>
> Cheers
> Gen
>
>
> On Sun, Aug 9, 2015 at 9:08 PM, gen tang <gen.tan...@gmail.com> wrote:
>
>> Hi,
>>
>> I might have a stupid question about sparksql's implementation of join on
>> not equality conditions, for instance condition1 or condition2.
>>
>> In fact, Hive doesn't support such join, as it is very difficult to
>> express such conditions as a map/reduce job. However, sparksql supports
>> such operation. So I would like to know how spark implement it.
>>
>> As I observe such join runs very slow, I guess that spark implement it by
>> doing filter on the top of cartesian product. Is it true?
>>
>> Thanks in advance for your help.
>>
>> Cheers
>> Gen
>>
>>
>>
>
