viirya commented on pull request #29807: URL: https://github.com/apache/spark/pull/29807#issuecomment-695811872
> > @dongjoon-hyun Thanks for pinging me. Hmm, this is actually related to what we are working on [SPARK-32943](https://issues.apache.org/jira/browse/SPARK-32943). We should not do it at physical plan level. We plan to tackle this kind of issue at optimizer. > > For this case, seem we still need to handle Physical plan level > > ``` > > == Optimized Logical Plan == > Project [c1#1, CASE WHEN (cast(expensive_col_6#6 as int) = 1) THEN a WHEN (cast(expensive_col_6#6 as int) = 2) THEN b END AS s_type#0] > +- Filter expensive_col_6#6 IN (1,2) > +- Project [c1#1, get_json_object(s#2, $.a) AS expensive_col_6#6] > +- HiveTableRelation `default`.`t`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#1, s#2], [P1#3], Statistics(sizeInBytes=8.0 EiB) > > == Physical Plan == > Project [c1#1, CASE WHEN (cast(get_json_object(s#2, $.a) AS expensive_col_6#6 as int) = 1) THEN a WHEN (cast(get_json_object(s#2, $.a) AS expensive_col_6#6 as int) = 2) THEN b END AS s_type#0] > +- Filter get_json_object(s#2, $.a) AS expensive_col_6#6 IN (1,2) > +- Scan hive default.t [c1#1, s#2], HiveTableRelation `default`.`t`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [c1#1, s#2], [P1#3], Statistics(sizeInBytes=8.0 EiB) > ``` That is one issue our ongoing work SPARK-32943 wants to fix. The problem here involves not just one, but some issues. There are some complicated issue we need to address. Current approach to fix it in physical plan is too hacky, As I see it. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
