Re: Optimizer rule ConvertToLocalRelation causes expressions to be eager-evaluated in Planning phase

2018-06-10 Thread Xiao Li
t;> >>> >>> >>> On Fri, Jun 8, 2018 at 3:44 PM, Reynold Xin wrote: >>> >>>> But from the user's perspective, optimization is not run right? So it >>>> is still lazy. >>>> >>>> >>>> On Fri, Jun 8, 2018 at 12:

Re: Optimizer rule ConvertToLocalRelation causes expressions to be eager-evaluated in Planning phase

2018-06-08 Thread Li Jin
;> But from the user's perspective, optimization is not run right? So it is >>> still lazy. >>> >>> >>> On Fri, Jun 8, 2018 at 12:35 PM Li Jin wrote: >>> >>>> Hi All, >>>> >>>> Sorry for the long email title. I am a bit

Re: Optimizer rule ConvertToLocalRelation causes expressions to be eager-evaluated in Planning phase

2018-06-08 Thread Li Jin
on is not run right? So it is > still lazy. > > > On Fri, Jun 8, 2018 at 12:35 PM Li Jin wrote: > >> Hi All, >> >> Sorry for the long email title. I am a bit surprised to find that the >> current optimizer rule "ConvertToLocalRelation" causes expre

Re: Optimizer rule ConvertToLocalRelation causes expressions to be eager-evaluated in Planning phase

2018-06-08 Thread Reynold Xin
But from the user's perspective, optimization is not run right? So it is still lazy. On Fri, Jun 8, 2018 at 12:35 PM Li Jin wrote: > Hi All, > > Sorry for the long email title. I am a bit surprised to find that the > current optimizer rule "ConvertToLocalRelation&quo

Optimizer rule ConvertToLocalRelation causes expressions to be eager-evaluated in Planning phase

2018-06-08 Thread Li Jin
Hi All, Sorry for the long email title. I am a bit surprised to find that the current optimizer rule "ConvertToLocalRelation" causes expressions to be eager-evaluated in planning phase, this can be demonstrated with the following code: scala> val myUDF = udf((x: String) => { pri