Re: Subtract implementation using broadcast

if i wanted to pimp DataFrame to add subtract and intersect myself with a physical operator, without needing to modify spark directly, is that currently possible/intended? or will i run into the private[spark] issue? On Fri, Nov 27, 2015 at 7:36 PM, Reynold Xin wrote: > We

Re: Subtract implementation using broadcast

We need to first implement subtract and intersect in Spark SQL natively first (i.e. add physical operator for them rather than using RDD.subtract/intersect). Then it should be pretty easy to do that, given it is just about injecting the right exchange operators. > On Nov 27, 2015, at 11:19

Subtract implementation using broadcast

Hi, I have seen massive gains with the broadcast hint for joins with DataFrames, and I was wondering if we have thought about allowing the broadcast hint for the implementation of subtract and intersect. Right now, when I try it, it says that there is no plan for the broadcast hint. Justin