Bruce Robbins created SPARK-41535:
-------------------------------------
Summary: InterpretedUnsafeProjection and
InterpretedMutableProjection can corrupt unsafe buffer when used with calendar
interval data
Key: SPARK-41535
URL: https://issues.apache.org/jira/browse/SPARK-41535
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 3.4.0
Reporter: Bruce Robbins
This returns the wrong answer:
{noformat}
set spark.sql.codegen.wholeStage=false;
set spark.sql.codegen.factoryMode=NO_CODEGEN;
select first(col1), last(col2) from values
(make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0))
as data(col1, col2);
+---------------+---------------+
|first(col1) |last(col2) |
+---------------+---------------+
|16 years 2 days|16 years 2 days|
+---------------+---------------+
{noformat}
In the above case, {{TungstenAggregationIterator}} uses
{{InterpretedUnsafeProjection}} to create the aggregation buffer and then
initializes all the fields to null. {{InterpretedUnsafeProjection}} incorrectly
calls {{UnsafeRowWriter#setNullAt}}, rather than {{unsafeRowWriter#write}}, for
the two calendar interval fields. As a result, the writer never allocates
memory from the variable length region for the two decimals, and the pointers
in the fixed region get left as zero. Later, when
{{InterpretedMutableProjection}} attempts to update the first field,
{{UnsafeRow#setInterval}} picks up the zero pointer and stores interval data on
top of the null-tracking bit set. The call to UnsafeRow#setInterval for the
second field also stomps the null-tracking bit set. Later updates to the
null-tracking bit set (e.g., calls to setNotNullAt) further corrupt the
interval data, turning {{interval 7 years 2 days}} into {{interval 16 years 2
days}}.
Even if you fix the above bug to {{InterpretedUnsafeProjection}} so that the
buffer is created correctly, {{InterpretedMutableProjection}} has a similar bug
to SPARK-41395, except this time for calendar interval data:
{noformat}
set spark.sql.codegen.wholeStage=false;
set spark.sql.codegen.factoryMode=NO_CODEGEN;
select first(col1), last(col2), max(col3) from values
(null, null, 1),
(make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0), 3)
as data(col1, col2, col3);
+---------------+---------------+---------+
|first(col1) |last(col2) |max(col3)|
+---------------+---------------+---------+
|16 years 2 days|16 years 2 days|3 |
+---------------+---------------+---------+
{noformat}
These two bugs could get exercised during codegen fallback. Take for example
this case where I forced codegen to fail for the Greatest expression:
{noformat}
spark-sql> select first(col1), last(col2), max(col3) from values
(null, null, 1),
(make_interval(0, 0, 0, 7, 0, 0, 0), make_interval(17, 0, 0, 2, 0, 0, 0), 3)
as data(col1, col2, col3);
22/12/15 13:06:23 ERROR CodeGenerator: failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 70,
Column 1: ';' expected instead of 'if'
...
22/12/15 13:06:24 WARN MutableProjection: Expr codegen error and falling back
to interpreter mode
java.util.concurrent.ExecutionException:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 78,
Column 1: failed to compile: org.codehaus.commons.compiler.CompileException:
File 'generated.java', Line 78, Column 1: ';' expected instead of 'boolean'
...
16 years 2 days 16 years 2 days 3
Time taken: 5.852 seconds, Fetched 1 row(s)
spark-sql>
{noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
