Ok, but at least we can have a separate binary comparison called DeepEqualTo as an Expression which at least makes the performance issues explicit and also have a way to achieve the equality on maps. In our case, the maps are very small. And this restriction completely reduces the expressibility of such use cases in spark.
On Mon, Sep 26, 2016 at 5:56 PM, Takeshi Yamamuro <linguin....@gmail.com> wrote: > yea, for all I know, there is no reasonable way to implement fast and > efficient equality checks on ArrayBasedMapData (See also: > https://github.com/apache/spark/pull/13847). > > > On Mon, Sep 26, 2016 at 9:04 PM, Lakshmi Rajagopalan <laks...@indix.com> > wrote: > >> If optimization is the problem, can we use precomputed hashes? >> >> >> On Mon, Sep 26, 2016 at 4:50 PM, Lakshmi Rajagopalan <laks...@indix.com> >> wrote: >> >>> Can you please help me understand why the MapType shouldn't be part of >>> equality tests? Practically, if we are using json line formats, the ideal >>> equals is every key should map to exactly the same value in both the maps >>> Which also hold true in Aesthetic case where a MapType can be thought of as >>> a function (Extensional definition). >>> >>> On Mon, Sep 26, 2016 at 3:43 PM, Takeshi Yamamuro <linguin....@gmail.com >>> > wrote: >>> >>>> Hi, >>>> >>>> Have you check this jira? >>>> https://issues.apache.org/jira/browse/SPARK-9415 >>>> >>>> // maropu >>>> >>>> On Mon, Sep 26, 2016 at 7:09 PM, Lakshmi Rajagopalan <laks...@indix.com >>>> > wrote: >>>> >>>>> Hi, >>>>> >>>>> We wanted to extend the existing '===' on Column to support deep >>>>> equals on Maps. >>>>> >>>>> >>>>> Currently it checks for == which does referential checks for maps. >>>>> >>>>> https://github.com/apache/spark/blob/master/sql/catalyst/src >>>>> /main/scala/org/apache/spark/sql/catalyst/expressions/predic >>>>> ates.scala#L420 >>>>> >>>>> Is there any possible work around for this? I have a patch in my mind >>>>> to fix Equals for MapType also. Any thoughts? >>>>> >>>>> Thanks, >>>>> Lakshmi Rajagopalan >>>>> >>>> >>>> >>>> >>>> -- >>>> --- >>>> Takeshi Yamamuro >>>> >>> >>> >> > > > -- > --- > Takeshi Yamamuro >
