[
https://issues.apache.org/jira/browse/SPARK-16807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15400532#comment-15400532
]
Kazuaki Ishizaki commented on SPARK-16807:
------------------------------------------
Interesting if we can ensure {{x - y}} is not {{infinite}} or {{NaN}}. Since I
am not familiar with SQL, I do not know that how we can ensure this condition
in Spark SQL.
In general, this generated code seems to take care {{filter_value6}} is
{{infinite}} or {{NaN}}. It would be good to read
[abs(float)|https://docs.oracle.com/javase/7/docs/api/java/lang/Math.html#abs(float)]
and
[nanSafeCompareFloats|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/util/Utils.scala#L1615].
> Optimize some ABS() statements
> ------------------------------
>
> Key: SPARK-16807
> URL: https://issues.apache.org/jira/browse/SPARK-16807
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Reporter: Sylvain Zimmer
> Priority: Minor
>
> I'm not a Catalyst expert, but I think some use cases for the ABS() function
> could generate simpler code.
> This is the code generated when doing something like {{ABS(x - y) > 0}} or
> {{ABS(x - y) = 0}} in Spark SQL:
> {code}
> /* 267 */ float filter_value6 = -1.0f;
> /* 268 */ filter_value6 = agg_value27 - agg_value32;
> /* 269 */ float filter_value5 = -1.0f;
> /* 270 */ filter_value5 = (float)(java.lang.Math.abs(filter_value6));
> /* 271 */
> /* 272 */ boolean filter_value4 = false;
> /* 273 */ filter_value4 =
> org.apache.spark.util.Utils.nanSafeCompareFloats(filter_value5, 0.0f) > 0;
> /* 274 */ if (!filter_value4) continue;
> {code}
> Maybe it could all be simplified to something like this?
> {code}
> filter_value4 = (agg_value27 != agg_value32)
> {code}
> (Of course you could write {{x != y}} directly in the SQL query, but the
> {{0}} in my example could be a configurable threshold, not something you can
> hardcode)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
