if i wanted to pimp DataFrame to add subtract and intersect myself with a physical operator, without needing to modify spark directly, is that currently possible/intended? or will i run into the private[spark] issue?
On Fri, Nov 27, 2015 at 7:36 PM, Reynold Xin <r...@databricks.com> wrote: > We need to first implement subtract and intersect in Spark SQL natively > first (i.e. add physical operator for them rather than using > RDD.subtract/intersect). > > Then it should be pretty easy to do that, given it is just about injecting > the right exchange operators. > > > > > On Nov 27, 2015, at 11:19 PM, Justin Uang <justin.u...@gmail.com> wrote: > > > > Hi, > > > > I have seen massive gains with the broadcast hint for joins with > DataFrames, and I was wondering if we have thought about allowing the > broadcast hint for the implementation of subtract and intersect. > > > > Right now, when I try it, it says that there is no plan for the > broadcast hint. > > > > Justin > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
