Jubin Soni created SPARK-59818:
----------------------------------
Summary: approx_top_k_estimate and approx_top_k_combine throw
NullPointerException on a NULL sketch state
Key: SPARK-59818
URL: https://issues.apache.org/jira/browse/SPARK-59818
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.0
Reporter: Jubin Soni
h2. Summary
The sketch-state argument of \{{approx_top_k_estimate}} and
\{{approx_top_k_combine}} is an ordinary nullable input column, but neither
function guards against a \{{NULL}} value. Both dereference the evaluated
`state` unconditionally and fail with a \{{NullPointerException}}.
Analysis does not prevent this: \{{checkStateFieldAndType}} validates only the
_schema_ of the state struct, not its nullness. Therefore, a correctly-typed
nullable struct column passes analysis and then fails at optimization or
execution time.
A \{{NULL}} sketch can arise naturally from an outer join, a UNION with a
missing side, or a partially populated column.
h2. 1) approx_top_k_estimate
{code:sql}
SELECT approx_top_k_estimate(s, 5) FROM (
SELECT CAST(NULL AS STRUCT<sketch: BINARY, maxItemsTracked: INT,
itemDataType: INT, itemDataTypeDDL: STRING>) AS s
);
{code}
The NPE is raised inside \{{ConstantFolding}} and is reported to the user as an
\{{INTERNAL_ERROR}}:
{code}
[INTERNAL_ERROR] The Spark SQL phase optimization failed with an internal error.
You hit a bug in Spark or the Spark plugins you use. SQLSTATE: XX000
Caused by: java.lang.NullPointerException: Cannot invoke
"org.apache.spark.sql.catalyst.InternalRow.getBinary(int)" because "stateEval"
is null
at ApproxTopKEstimate.eval(ApproxTopKExpressions.scala:113)
at
org.apache.spark.sql.catalyst.optimizer.ConstantFolding$.tryFold(expressions.scala:65)
{code}
There are two distinct problems here:
* {\{ApproxTopKEstimate}} overrides \{{BinaryExpression.eval}} directly
instead of using \{{nullSafeEval}}, so the inherited null handling is bypassed.
* It declares \{{override def nullable: Boolean = false}} even though the
result is \{{NULL}} whenever the state is \{{NULL}}. The reported output schema
is therefore incorrect: it declares \{{nullable=false}} for an expression that
can return \{{NULL}}. This can also allow the optimizer to reason incorrectly
about the expression.
h2. 2) approx_top_k_combine
{code:sql}
SELECT approx_top_k_estimate(approx_top_k_combine(s), 5) FROM (
SELECT CAST(NULL AS STRUCT<sketch: BINARY, maxItemsTracked: INT,
itemDataType: INT, itemDataTypeDDL: STRING>) AS s
);
{code}
{\{ApproxTopKCombine.update}} has the same unguarded dereference. This one
fails at execution time on the executors and aborts the job:
{code}
org.apache.spark.SparkException: Job aborted due to stage failure:
Task 0 in stage 0.0 failed 1 times ...
Caused by: java.lang.NullPointerException
at ApproxTopKCombine.update(ApproxTopKAggregates.scala:908)
at TypedImperativeAggregate.update(interfaces.scala:583)
{code}
h2. Expected behaviour
Consistent with SQL semantics for a nullable input column:
* {\{approx_top_k_estimate(NULL, k)}} should return \{{NULL}}, and the
expression should report itself as nullable when its state argument is nullable.
* {\{approx_top_k_combine}} should skip \{{NULL}} sketches, as other
aggregates ignore \{{NULL}} inputs.
This is different from the existing \{{APPROX_TOP_K_NULL_ARG}} error, which is
raised for \{{NULL}} _foldable configuration_ arguments (\{{k}},
\{{maxItemsTracked}}).
That check, \{{ApproxTopK.checkExpressionNotNull}}, calls \{{expr.eval()}} with
no input row, so it applies only to literals and is not applicable to a per-row
data column.
h2. Relationship to earlier tickets
* SPARK-53960 ("Let approx_top_k_accumulate/combine/estimate handle NULLs")
added handling for NULLs _within_ the accumulated data (counting them as
items). It did not address the NULL-state path or the nullable declaration.
* SPARK-58095 concerns _empty_ input (zero rows) in combine's eval/serialize,
which is a different code path and a different trigger from a NULL state value.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
