Github user eyalfa commented on the issue:
https://github.com/apache/spark/pull/16043
@hvanhovel, currently in the process of refactoring,luckily I've introduced
some tests in the initial sprint so I've cought my mistakes and refreshed
my mind around the assumptions this code makes.
Regarding nulls,I'm using coalesce to add the first positive match after
undetermined ones, when one of the undetermined values is null,coalesce
will override it with the positive match's value.
This was handled in the normalize method of class ClassifiefKeys,part of
the refactoring involves removing this class and simplifying the decision
making around matches.
Will push later today after running scalastyle on my branch.
On Jan 27, 2017 18:40, "Herman van Hovell" <[email protected]> wrote:
> *@hvanhovell* commented on this pull request.
> ------------------------------
>
> In sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/
> optimizer/ComplexTypes.scala <https://github.com/apache/spark/pull/16043>:
>
> > + }
> + }
> + val res = res1.normalize( requestedKey )
> + res
> + }
> +
> + override def apply(plan: LogicalPlan): LogicalPlan = {
> + plan.transformExpressionsUp {
> + // attempt to unfold 'constant' key extraction,
> + // this enables other optimizations to take place.
> + case gmv @ GetMapValue(cm @ CreateMap(elems), key) =>
> + val kvs = cm.keys.zip(cm.values)
> + val classifiedEntries = classifyEntries(kvs, key)
> + classifiedEntries match {
> + case ClassifiedEntries(Seq(), _, None) => Literal.create(null,
gmv.dataType)
> + case ClassifiedEntries(`elems`, _, None) => gmv
>
> It is fine, it just needs a little clarification.
>
> â
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/spark/pull/16043>, or mute the thread
>
<https://github.com/notifications/unsubscribe-auth/ABFFOV2D--1Wea5bmhkZVNwjmAZXxrtLks5rWh3lgaJpZM4K-Obo>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
