This is an automated email from the ASF dual-hosted git repository.
dtenedor pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new ef0ef73c68e3 [SPARK-60485][SQL] Initial support for non-constant
expressions in VALUES list (excluding correlation references and SQL UDFs)
ef0ef73c68e3 is described below
commit ef0ef73c68e3480073e677abba85044f96e6e363
Author: Daniel Tenedorio <[email protected]>
AuthorDate: Thu Oct 22 14:20:26 2026 -0700
[SPARK-60485][SQL] Initial support for non-constant expressions in VALUES
list (excluding correlation references and SQL UDFs)
### What changes were proposed in this pull request?
This PR allows VALUES lists (inline tables) to contain expressions that are
not constant-foldable, for example:
- nondeterministic functions such as `rand()` and `uuid()`;
- Scala/Java UDFs;
- higher-order functions with lambdas;
- `raise_error()`;
- uncorrelated scalar subqueries.
Such a VALUES list is planned as an `Expand` over a `OneRowRelation`, with
one projection per row, so its rows are evaluated by the existing `ExpandExec`
when the query runs.
VALUES lists that are valid today keep exactly their current behavior: they
go through the same validation, produce the same `LocalRelation` or
`ResolvedInlineTable`, and are evaluated by the unchanged `EvalInlineTables`
rule. The new code path only applies to VALUES lists that fail analysis today.
**Analysis**
- `EvaluateUnresolvedInlineTable.validateInputEvaluable` accepts every
VALUES list it accepts today. When some expressions of a list can only be
evaluated when the query runs, all its expressions other than constants must be
evaluable by themselves when the query runs, including the ones with
CURRENT_LIKE expressions. Expressions that are not evaluable are rejected with
the existing `INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE`
error:
- aggregate, window, and generator functions;
- references to columns of an outer query, e.g. `LATERAL (VALUES (t.a))`;
- subqueries other than uncorrelated scalar subqueries, such as `EXISTS`
and `IN`;
- unevaluable expressions that would have to be rewritten into separate
operators, such as SQL UDFs and Python UDFs.
- If every expression of a VALUES list is foldable or contains a
CURRENT_LIKE expression, which is what the current validation requires, the
list takes the current code path unchanged. An expression with a subquery
doesn't qualify through CURRENT_LIKE expressions, inside the subquery or next
to it, because the subquery can only be evaluated when the query runs. Neither
does a nondeterministic expression, such as `concat(current_user(), uuid())`,
because `CheckAnalysis` rejects nondete [...]
- Foldable cells are evaluated into literals during analysis, so
evaluation errors in constant cells are still reported during analysis.
- CURRENT_LIKE cells are replaced with literals at the end of the
analysis phase, like today.
- `ResolveSubquery` now resolves subqueries in `UnresolvedInlineTable`.
`CheckAnalysis` only reports
`UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.SCALAR_SUBQUERY_IN_VALUES` with the
conf off, so VALUES lists with scalar subqueries that stay unresolved report
the actual error, such as an unresolved column or a missing table.
- The single-pass resolver throws `ExplicitlyUnsupportedResolverFeature`
for scalar subqueries in VALUES lists, and for VALUES lists that become an
`Expand`. These queries fall back to the fixed-point analyzer.
**Execution**
- No new operators. `ExpandExec` applies the projections to the single row
of its child, so all the rows of the VALUES list are evaluated in one task,
with whole-stage codegen.
- Each projection is evaluated once, for partition 0. As a result, `VALUES
(rand(5)), (rand(5))` returns the same values as `SELECT rand(5) UNION ALL
SELECT rand(5)`.
- Scalar subqueries are planned and executed by `PlanSubqueries` and
`PlanAdaptiveSubqueries` before `ExpandExec` runs.
- Like in a `SELECT` list, `ColumnPruning` removes the expressions of
columns that are not read.
- `ExpandExec` fix: the generated code evaluated a column once per input
row when the column had equal expressions in all projections, even if they were
nondeterministic. For example, a nondeterministic UDF called the same way in
every row of a VALUES list returned the same value in every row, unlike the
interpreted path. Now only deterministic columns are shared.
**Configuration**
- New internal conf `spark.sql.inlineTable.nonFoldableExpressions.enabled`,
default `true`. Setting it to `false` restores the previous behavior.
**References to other DBMSes**
- These engines accept general expressions in VALUES lists, including
scalar subqueries: PostgreSQL, SQL Server, Oracle 23ai, MySQL 8.0.19+ and
SQLite. BigQuery documents scalar subqueries in `INSERT ... VALUES`.
- Snowflake and Hive keep restrictions similar to Spark's current behavior.
### Why are the changes needed?
Today, every expression in a VALUES list must be constant-foldable, with
the exception of expressions like `current_timestamp()`, because the analyzer
evaluates the rows on the driver. So queries such as the following fail:
- `INSERT INTO t VALUES (uuid(), ...)`
- `SELECT * FROM VALUES (rand())`
- `INSERT INTO t VALUES ((SELECT max(id) FROM t) + 1, ...)`
Users have to rewrite them as `INSERT ... SELECT` or as a `UNION ALL` of
`SELECT`s. Most other SQL engines accept these queries.
SPARK-28291 tracks UDFs in VALUES lists. This PR supports Scala/Java UDFs
there, but not SQL or Python UDFs.
### Does this PR introduce _any_ user-facing change?
Yes. Queries that failed before now succeed:
```sql
SELECT * FROM VALUES ('one', rand(5)), ('two', 3.0D) AS data(a, b);
-- Before: [INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE]
-- Cannot evaluate the expression "rand(5)" in inline table
definition.
-- After:
-- one 0.02390696427502892
-- two 3.0
INSERT INTO t VALUES ((SELECT max(id) FROM t) + 1);
-- Before:
[UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.SCALAR_SUBQUERY_IN_VALUES]
-- Scalar subqueries in the VALUES clause.
-- After: inserts the next id.
```
Expressions that still can't be evaluated in a VALUES list fail with
`INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE`. An inline
table with non-foldable expressions shows up as an `Expand` over a
`OneRowRelation` in plans. VALUES lists that are valid today are not affected.
### How was this patch tested?
New tests:
- `InlineTableNonFoldableExpressionsSuite` (23 tests), covering:
- evaluation on executors, in a single task;
- constant lists, which remain a `LocalTableScan`;
- lists that are valid today: the same analyzed plans and results with
the conf on and off, CURRENT_LIKE expressions inside a Scala UDF still
evaluated on the driver, and evaluation errors next to CURRENT_LIKE expressions
still reported by the optimizer;
- nondeterministic semantics: a distinct `uuid()` per row, `rand(5)`
matching `SELECT rand(5)`, stable values when a plan is re-executed, and equal
nondeterministic UDF calls evaluated separately for every row, with whole-stage
codegen on and off;
- `current_timestamp()` evaluated once per query;
- nondeterministic expressions next to CURRENT_LIKE expressions in the
same expression, including a nondeterministic Scala UDF, and the previous error
with the conf off;
- type coercion and nullability;
- uncorrelated scalar subqueries, including `INSERT ... VALUES` that
reads its target table, and subqueries with CURRENT_LIKE expressions inside
them or next to them;
- unresolved columns and missing tables next to and inside scalar
subqueries, in `SELECT` and `INSERT`;
- rejected cases: `LATERAL` outer references, also inside a scalar
subquery, `EXISTS` and `IN` (also with CURRENT_LIKE expressions), aggregate,
window and generator functions (also over CURRENT_LIKE expressions), SQL UDFs,
and Python UDFs;
- runtime errors (`raise_error`), and unread columns that are not
evaluated;
- `INSERT` with `DEFAULT` and `uuid()`;
- DataFrame self-joins;
- whole-stage codegen and AQE on and off;
- SQL metrics;
- a 1000-row list;
- the conf set to `false`.
- `ResolveInlineTablesSuite`, covering:
- validation;
- the `Expand` that replaces lists with non-foldable expressions, with
constant cells folded;
- lists without non-foldable expressions resolving to the same plans with
the conf on and off;
- aliased (`DEFAULT`) cells;
- CURRENT_LIKE expressions;
- subqueries with CURRENT_LIKE expressions, which take the `Expand` path
or are rejected;
- nondeterministic expressions next to CURRENT_LIKE expressions, which
take the `Expand` path;
- analysis-time errors in constant cells;
- `CheckAnalysis` of nondeterministic expressions.
- New queries in `inline-table.sql`. The `rand(5)` cases in
`inline-table.sql` and `udf/udf-inline-table.sql` now succeed.
- `SQLQuerySuite`: the SPARK-49659 test runs with the conf set to `false`
and checks that the query now works with the conf on.
Regenerated `scripting/randomly_generated_scripts.sql`:
- 22 scripts used to fail with `SCALAR_SUBQUERY_IN_VALUES` and exit through
their `EXIT HANDLER`. They contain `INSERT ... VALUES` statements with scalar
subqueries, and they now run to completion.
- These scripts share tables, so the data they now insert changes the
control flow of 6 later scripts:
- 3 now reach `UPDATE` statements, which are unsupported for these
Parquet tables.
- 3 no longer reach such statements.
- I verified every changed outcome by running the file with all `EXIT
HANDLER`s removed, once with the conf on and once with it off.
Existing suites:
- `Expand` execution: `WholeStageCodegenSuite`, `PlannerSuite`,
`AdaptivePartialAggregationSuite`, `DatasetUnpivotSuite`,
`DataFrameAggregateSuite`, and the `GROUPING SETS` and UNPIVOT golden files.
- Catalyst analyzer suites, including `AnalysisSuite`,
`AnalysisErrorSuite`, `ResolveSubquerySuite` and `PlanParserSuite`.
- `InlineTableParsingImprovementsSuite`, `ResolveDefaultColumnsSuite`, the
`*InsertSuite`s and the `*SubquerySuite`s.
- The single-pass resolver suites: `ResolverGuardSuite`,
`HybridAnalyzerSuite`, the `*ResolverSuite`s, `ResolutionValidatorSuite`,
`ExplicitlyUnsupportedResolverFeatureSuite`, `DataFrameAnalyzerTestGapsSuite`.
- The `SQLQueryTestSuite` golden files for inline tables, CTEs,
`create_view`, `insert`, session variables and scripting.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 5.5)
Closes #59681 from dtenedor/values-list-with-non-constant-exprs.
Authored-by: Daniel Tenedorio <[email protected]>
Signed-off-by: Daniel Tenedorio <[email protected]>
(cherry picked from commit 8d3e0c52c3902a1f82d0536425ecae18ba9e5ade)
Signed-off-by: Daniel Tenedorio <[email protected]>
---
docs/sql-ref-syntax-qry-select-inline-table.md | 14 +
.../spark/sql/catalyst/analysis/Analyzer.scala | 4 +
.../sql/catalyst/analysis/CheckAnalysis.scala | 4 +-
.../sql/catalyst/analysis/resolver/Resolver.scala | 10 +
.../resolver/SubqueryExpressionResolver.scala | 4 +
.../spark/sql/catalyst/analysis/unresolved.scala | 5 +-
.../sql/catalyst/optimizer/finishAnalysis.scala | 25 +-
.../util/EvaluateUnresolvedInlineTable.scala | 99 +++++-
.../org/apache/spark/sql/internal/SQLConf.scala | 13 +
.../analysis/ResolveInlineTablesSuite.scala | 181 ++++++++--
.../apache/spark/sql/execution/ExpandExec.scala | 6 +-
.../analyzer-results/inline-table.sql.out | 110 ++++++-
.../scripting/randomly_generated_scripts.sql.out | 56 ++--
.../analyzer-results/udf/udf-inline-table.sql.out | 16 +-
.../resources/sql-tests/inputs/inline-table.sql | 23 +-
.../sql-tests/inputs/udf/udf-inline-table.sql | 2 +-
.../sql-tests/results/inline-table.sql.out | 119 ++++++-
.../scripting/randomly_generated_scripts.sql.out | 112 +++----
.../sql-tests/results/udf/udf-inline-table.sql.out | 19 +-
.../InlineTableNonFoldableExpressionsSuite.scala | 365 +++++++++++++++++++++
.../scala/org/apache/spark/sql/SQLQuerySuite.scala | 25 +-
21 files changed, 1004 insertions(+), 208 deletions(-)
diff --git a/docs/sql-ref-syntax-qry-select-inline-table.md
b/docs/sql-ref-syntax-qry-select-inline-table.md
index 7f0372ea787e..7c1731046b7f 100644
--- a/docs/sql-ref-syntax-qry-select-inline-table.md
+++ b/docs/sql-ref-syntax-qry-select-inline-table.md
@@ -34,6 +34,11 @@ VALUES ( expression [ , ... ] ) [ table_alias ]
* **expression**
Specifies a combination of one or more values, operators and SQL functions
that results in a value.
+ Since Spark 4.4, the expression does not need to be constant:
nondeterministic functions such as
+ `rand()`, Scala and Java user-defined functions, and uncorrelated scalar
subqueries are evaluated
+ when the query runs. Aggregate, window and generator functions, SQL and
Python user-defined
+ functions, other kinds of subqueries, and references to columns of an
outer query are not
+ supported.
* **table_alias**
@@ -70,6 +75,15 @@ SELECT * FROM VALUES ("one", array(0, 1)), ("two", array(2,
3)) AS data(a, b);
|one|[0, 1]|
|two|[2, 3]|
+---+------+
+
+-- expressions that are not constant, such as nondeterministic functions and
scalar subqueries
+SELECT a, length(b) FROM VALUES (1, uuid()), ((SELECT max(id) FROM range(5)),
uuid()) AS data(a, b);
++---+---------+
+| a|length(b)|
++---+---------+
+| 1| 36|
+| 4| 36|
++---+---------+
```
### Related Statements
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
index 29384c9020d7..8c5237adba9b 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala
@@ -2801,6 +2801,10 @@ class Analyzer(
resolveSubQueries(j, j)
case tvf: UnresolvedTableValuedFunction =>
resolveSubQueries(tvf, tvf)
+ // An inline table has no output, so its subqueries cannot reference its
columns.
+ case t: UnresolvedInlineTable
+ if
conf.getConf(SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED) =>
+ resolveSubQueries(t, t)
case s: SupportsSubquery if s.childrenResolved =>
resolveSubQueries(s, s)
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
index 57ed47bcd0bb..f567268d964e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala
@@ -492,7 +492,9 @@ trait CheckAnalysis extends LookupCatalog with
QueryErrorsBase with PlanToString
throw SparkException.internalError(
s"V2Reference should be resolved during analysis: ${r.name}")
- case u: UnresolvedInlineTable if
unresolvedInlineTableContainsScalarSubquery(u) =>
+ case u: UnresolvedInlineTable
+ if
!SQLConf.get.getConf(SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED) &&
+ unresolvedInlineTableContainsScalarSubquery(u) =>
throw QueryCompilationErrors.inlineTableContainsScalarSubquery(u)
case command: V2PartitionCommand =>
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/Resolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/Resolver.scala
index 458836388a24..61c7170af464 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/Resolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/Resolver.scala
@@ -639,6 +639,9 @@ class Resolver(
* Note: By default if all the inline table expressions can be evaluated
eagerly, the parser
* would produce a [[LocalRelation]] and the analysis would just skip this
step and go straight
* to `resolveLocalRelation` (SPARK-48967, SPARK-49269).
+ *
+ * Inline tables with expressions that are evaluated when the query runs,
such as `rand()`,
+ * become an [[Expand]], which only the fixed-point Analyzer supports.
*/
private def resolveInlineTable(unresolvedInlineTable:
UnresolvedInlineTable): LogicalPlan = {
val withResolvedExpressions = UnresolvedInlineTable(
@@ -654,6 +657,13 @@ class Resolver(
val resolvedRelation = EvaluateUnresolvedInlineTable
.evaluateUnresolvedInlineTable(withResolvedExpressions)
+ resolvedRelation match {
+ case _: Expand =>
+ throw new ExplicitlyUnsupportedResolverFeature(
+ "VALUES lists with expressions that are evaluated when the query
runs")
+ case _ =>
+ }
+
resolve(resolvedRelation)
}
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/SubqueryExpressionResolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/SubqueryExpressionResolver.scala
index 574fca29d818..9a1693a3891f 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/SubqueryExpressionResolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/SubqueryExpressionResolver.scala
@@ -28,6 +28,7 @@ import org.apache.spark.sql.catalyst.expressions.{
}
import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
import org.apache.spark.sql.errors.QueryCompilationErrors
+import org.apache.spark.sql.internal.SQLConf
/**
* [[SubqueryExpressionResolver]] resolves specific [[SubqueryExpression]]s,
such as
@@ -54,6 +55,9 @@ class SubqueryExpressionResolver(expressionResolver:
ExpressionResolver, resolve
*/
def resolveScalarSubquery(unresolvedScalarSubquery: ScalarSubquery):
Expression = {
traversals.current.parentOperator match {
+ case _: UnresolvedInlineTable
+ if
conf.getConf(SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED) =>
+ throw new ExplicitlyUnsupportedResolverFeature("scalar subqueries in
VALUES lists")
case unresolvedInlineTable: UnresolvedInlineTable =>
throw
QueryCompilationErrors.inlineTableContainsScalarSubquery(unresolvedInlineTable)
case _ =>
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
index 6765af018b2a..f3cb4653040c 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/unresolved.scala
@@ -197,7 +197,10 @@ object UnresolvedRelation {
/**
* An inline table that has not been resolved yet. Once resolved, it is turned
by the analyzer into
- * a [[org.apache.spark.sql.catalyst.plans.logical.LocalRelation]].
+ * a [[org.apache.spark.sql.catalyst.plans.logical.LocalRelation]], or into an
+ * [[org.apache.spark.sql.catalyst.plans.logical.Expand]] over a
+ * [[org.apache.spark.sql.catalyst.plans.logical.OneRowRelation]] if some of
its expressions are
+ * evaluated when the query runs.
*
* @param names list of column names
* @param rows expressions for the data
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala
index 7644c1023613..c32995d973a1 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/finishAnalysis.scala
@@ -92,20 +92,23 @@ object EvalInlineTables extends Rule[LogicalPlan] with
CastSupport {
def eval(table: ResolvedInlineTable): LocalRelation = {
val newRows: Seq[InternalRow] =
- table.rows.map { row => InternalRow.fromSeq(row.map { e =>
- try {
- prepareForEval(e).eval()
- } catch {
- case NonFatal(ex) =>
- table.failAnalysis(
- errorClass =
"INVALID_INLINE_TABLE.FAILED_SQL_EXPRESSION_EVALUATION",
- messageParameters = Map("sqlExpr" -> toSQLExpr(e)),
- cause = ex)
- }})
- }
+ table.rows.map(row =>
InternalRow.fromSeq(row.map(evalExpression(table, _))))
LocalRelation(table.output, newRows)
}
+
+ /** Evaluates one of the expressions of the given inline table. */
+ def evalExpression(table: ResolvedInlineTable, e: Expression): Any = {
+ try {
+ prepareForEval(e).eval()
+ } catch {
+ case NonFatal(ex) =>
+ table.failAnalysis(
+ errorClass =
"INVALID_INLINE_TABLE.FAILED_SQL_EXPRESSION_EVALUATION",
+ messageParameters = Map("sqlExpr" -> toSQLExpr(e)),
+ cause = ex)
+ }
+ }
}
/**
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/EvaluateUnresolvedInlineTable.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/EvaluateUnresolvedInlineTable.scala
index 15c2e74fb4a1..e8d9e9b569ae 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/EvaluateUnresolvedInlineTable.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/EvaluateUnresolvedInlineTable.scala
@@ -18,12 +18,14 @@ package org.apache.spark.sql.catalyst.util
import org.apache.spark.sql.catalyst.SQLConfHelper
import org.apache.spark.sql.catalyst.analysis._
-import org.apache.spark.sql.catalyst.expressions.{AliasHelper, EvalHelper,
Expression}
+import org.apache.spark.sql.catalyst.expressions.{AliasHelper,
CommonExpressionDef, EvalHelper, Expression, Generator, LeafExpression,
Literal, OuterReference, ScalarSubquery, SubqueryExpression, Unevaluable,
WindowExpression, WindowFunction}
+import
org.apache.spark.sql.catalyst.expressions.aggregate.{AggregateExpression,
AggregateFunction}
import org.apache.spark.sql.catalyst.optimizer.EvalInlineTables
-import org.apache.spark.sql.catalyst.plans.logical.LogicalPlan
+import org.apache.spark.sql.catalyst.plans.logical.{Expand, LogicalPlan,
OneRowRelation}
import org.apache.spark.sql.catalyst.trees.TreePattern.CURRENT_LIKE
import org.apache.spark.sql.catalyst.types.DataTypeUtils
import org.apache.spark.sql.catalyst.util.TypeUtils.{toSQLExpr, toSQLId}
+import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{StringHelper, StringType, StructField,
StructType}
/**
@@ -32,6 +34,10 @@ import org.apache.spark.sql.types.{StringHelper, StringType,
StructField, Struct
* a child of [[InsertIntoStatement]].
* Use the method: [[EvaluateUnresolvedInlineTable.evaluate]] as the entry
point for this
* transformation.
+ *
+ * If [[SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED]] is true, an
inline table with
+ * expressions that can only be evaluated when the query runs, such as
`rand()`, is replaced with
+ * an [[Expand]] over a [[OneRowRelation]] instead, with one projection per
row.
*/
object EvaluateUnresolvedInlineTable extends SQLConfHelper
with AliasHelper with EvalHelper with CastSupport {
@@ -47,7 +53,11 @@ object EvaluateUnresolvedInlineTable extends SQLConfHelper
validateInputDimension(table)
validateInputEvaluable(table)
val resolvedTable = findCommonTypesAndCast(table, ignoreCollation)
- earlyEvalIfPossible(resolvedTable)
+ if (canEvalAllDuringAnalysis(table)) {
+ earlyEvalIfPossible(resolvedTable)
+ } else {
+ toExpand(resolvedTable)
+ }
}
/**
@@ -61,6 +71,48 @@ object EvaluateUnresolvedInlineTable extends SQLConfHelper
if (earlyEvalPossible) EvalInlineTables.eval(table) else table
}
+ /**
+ * Turns an inline table with expressions that are evaluated when the query
runs into an
+ * [[Expand]] over a [[OneRowRelation]], which produces one output row per
projection. The
+ * constant expressions are evaluated now (see [[isConstant]]).
+ */
+ private def toExpand(table: ResolvedInlineTable): Expand = {
+ val projections = table.rows.map(_.map {
+ case e if isConstant(e) =>
Literal(EvalInlineTables.evalExpression(table, e), e.dataType)
+ case e => e
+ })
+ Expand(projections, table.output, OneRowRelation())
+ }
+
+ private def runtimeEvaluationEnabled: Boolean =
+ conf.getConf(SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED)
+
+ /** Returns whether all the expressions of the inline table can be evaluated
during analysis. */
+ private def canEvalAllDuringAnalysis(table: UnresolvedInlineTable): Boolean =
+ table.rows.forall(_.forall(canEvalDuringAnalysis))
+
+ /**
+ * Returns whether the expression can be evaluated during the analysis
phase: it must be
+ * foldable, or contain CURRENT_LIKE expressions, which are replaced by
literals at the end of
+ * the analysis phase, but no subqueries, which can only be evaluated when
the query runs.
+ * Inline tables with only such expressions are evaluated during analysis. If
+ * [[SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED]] is true, the
expressions with
+ * CURRENT_LIKE expressions must also be deterministic, because a
[[ResolvedInlineTable]] cannot
+ * contain nondeterministic expressions.
+ */
+ private def canEvalDuringAnalysis(e: Expression): Boolean =
+ (e.containsPattern(CURRENT_LIKE) && !SubqueryExpression.hasSubquery(e) &&
+ (e.deterministic || !runtimeEvaluationEnabled)) ||
+ (e.resolved && trimAliases(prepareForEval(e)).foldable)
+
+ /**
+ * Returns whether the expression is a constant, which is evaluated during
the analysis phase
+ * even if the inline table becomes an [[Expand]]: it must be foldable, and
not contain
+ * CURRENT_LIKE expressions, which are replaced with literals at the end of
the analysis phase.
+ */
+ private def isConstant(e: Expression): Boolean =
+ !e.containsPattern(CURRENT_LIKE) && e.resolved &&
trimAliases(prepareForEval(e)).foldable
+
/**
* Validates the input data dimension:
* 1. All rows have the same cardinality.
@@ -85,18 +137,23 @@ object EvaluateUnresolvedInlineTable extends SQLConfHelper
}
/**
- * Validates that all inline table data are valid expressions that can be
evaluated
- * (in this they must be foldable).
- * Note that nondeterministic expressions are not supported since they are
not foldable.
- * Exception are CURRENT_LIKE expressions, which are replaced by a literal
in later stages.
+ * Validates that all inline table data are valid expressions that can be
evaluated: during the
+ * analysis phase (see [[canEvalDuringAnalysis]]), or, if
+ * [[SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED]] is true, when
the query runs (see
+ * [[isRuntimeEvaluable]]). In the latter case, all the expressions that are
not constants are
+ * evaluated when the query runs, including the ones with CURRENT_LIKE
expressions.
* This is package visible for unit testing.
*/
def validateInputEvaluable(table: UnresolvedInlineTable): Unit = {
+ val evaluatedWhenQueryRuns = runtimeEvaluationEnabled &&
!canEvalAllDuringAnalysis(table)
table.rows.foreach { row =>
row.foreach { e =>
- if (e.containsPattern(CURRENT_LIKE)) {
- // Do nothing.
- } else if (!e.resolved || !trimAliases(prepareForEval(e)).foldable) {
+ val evaluable = if (evaluatedWhenQueryRuns) {
+ isConstant(e) || (e.resolved && isRuntimeEvaluable(e))
+ } else {
+ canEvalDuringAnalysis(e)
+ }
+ if (!evaluable) {
e.failAnalysis(
errorClass =
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
messageParameters = Map("expr" -> toSQLExpr(e)))
@@ -105,6 +162,28 @@ object EvaluateUnresolvedInlineTable extends SQLConfHelper
}
}
+ /**
+ * Returns whether the resolved expression can be evaluated by itself when
the query runs.
+ * Aggregate, window and generator functions are not meaningful in a VALUES
list. Outer
+ * references and subqueries other than uncorrelated scalar subqueries are
not supported, and
+ * neither are the remaining unevaluable expressions, such as SQL and Python
UDFs, which would
+ * have to be rewritten into separate operators.
+ */
+ private def isRuntimeEvaluable(e: Expression): Boolean = {
+ !trimAliases(prepareForEval(e)).exists {
+ case _: AggregateExpression | _: AggregateFunction | _: WindowExpression
|
+ _: WindowFunction | _: Generator | _: OuterReference => true
+ case s: ScalarSubquery => s.isCorrelated
+ case _: SubqueryExpression => true
+ // Replaced with literals at the end of the analysis phase.
+ case l: LeafExpression if l.containsPattern(CURRENT_LIKE) => false
+ // Evaluated by the enclosing `With`.
+ case _: CommonExpressionDef => false
+ case _: Unevaluable => true
+ case _ => false
+ }
+ }
+
/**
* This function attempts to coerce inputs into consistent types.
*
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
index 6d532e546e90..1e8236ef67df 100644
--- a/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
+++ b/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
@@ -1815,6 +1815,19 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED =
+ buildConf("spark.sql.inlineTable.nonFoldableExpressions.enabled")
+ .internal()
+ .doc("When true, VALUES lists (inline tables) may contain expressions
that are not " +
+ "constant-foldable, such as nondeterministic functions, Scala/Java
UDFs, and " +
+ "uncorrelated scalar subqueries. Such an inline table is evaluated on
executors when " +
+ "the query runs instead of being folded into a local relation during
analysis. When " +
+ "false, these expressions are rejected during analysis.")
+ .version("4.4.0")
+ .withBindingPolicy(ConfigBindingPolicy.SESSION)
+ .booleanConf
+ .createWithDefault(true)
+
val ESCAPED_STRING_LITERALS =
buildConf("spark.sql.parser.escapedStringLiterals")
.internal()
.doc("When true, string literals (including regex patterns) remain escaped
in our SQL " +
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTablesSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTablesSuite.scala
index 50d88e90987c..6cb0a5df968d 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTablesSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/analysis/ResolveInlineTablesSuite.scala
@@ -20,12 +20,13 @@ package org.apache.spark.sql.catalyst.analysis
import org.scalatest.BeforeAndAfter
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.expressions.{Alias, Cast, CurrentTime,
CurrentTimestamp, Literal, Rand}
+import org.apache.spark.sql.catalyst.expressions.{Add, Alias,
AttributeReference, Cast, Concat, CreateArray, CurrentTime, CurrentTimestamp,
CurrentUser, Exists, Explode, InSubquery, ListQuery, Literal, OuterReference,
Rand, ScalarSubquery, Uuid}
import org.apache.spark.sql.catalyst.expressions.aggregate.Count
import org.apache.spark.sql.catalyst.optimizer.{ComputeCurrentTime,
EvalInlineTables}
-import org.apache.spark.sql.catalyst.plans.logical.LocalRelation
+import org.apache.spark.sql.catalyst.plans.logical.{Expand, LocalRelation,
OneRowRelation, Project}
import org.apache.spark.sql.catalyst.util.EvaluateUnresolvedInlineTable
-import org.apache.spark.sql.types.{LongType, NullType, TimestampNTZType,
TimestampType, TimeType}
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.types.{DoubleType, IntegerType, LongType,
NullType, TimestampNTZType, TimestampType, TimeType}
/**
* Unit tests for [[ResolveInlineTables]]. Note that there are also test cases
defined in
@@ -37,33 +38,62 @@ class ResolveInlineTablesSuite extends AnalysisTest with
BeforeAndAfter {
private def lit(v: Any): Literal = Literal(v)
test("validate inputs are foldable") {
- EvaluateUnresolvedInlineTable.validateInputEvaluable(
- UnresolvedInlineTable(Seq("c1", "c2"), Seq(Seq(lit(1)))))
-
- // Alias is OK
- EvaluateUnresolvedInlineTable.validateInputEvaluable(
- UnresolvedInlineTable(Seq("c1", "c2"), Seq(Seq(Alias(lit(1), "a")()))))
-
- // nondeterministic (rand) should not work
- intercept[AnalysisException] {
+ withSQLConf(SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED.key ->
"false") {
EvaluateUnresolvedInlineTable.validateInputEvaluable(
- UnresolvedInlineTable(Seq("c1"), Seq(Seq(Rand(1)))))
- }
- intercept[AnalysisException] {
+ UnresolvedInlineTable(Seq("c1", "c2"), Seq(Seq(lit(1)))))
+
+ // Alias is OK
EvaluateUnresolvedInlineTable.validateInputEvaluable(
- UnresolvedInlineTable(Seq("c1"), Seq(Seq(Cast(Rand(1),
TimestampNTZType)))))
+ UnresolvedInlineTable(Seq("c1", "c2"), Seq(Seq(Alias(lit(1), "a")()))))
+
+ // nondeterministic (rand) should not work
+ intercept[AnalysisException] {
+ EvaluateUnresolvedInlineTable.validateInputEvaluable(
+ UnresolvedInlineTable(Seq("c1"), Seq(Seq(Rand(1)))))
+ }
+ intercept[AnalysisException] {
+ EvaluateUnresolvedInlineTable.validateInputEvaluable(
+ UnresolvedInlineTable(Seq("c1"), Seq(Seq(Cast(Rand(1),
TimestampNTZType)))))
+ }
+
+ // aggregate should not work
+ intercept[AnalysisException] {
+ EvaluateUnresolvedInlineTable.validateInputEvaluable(
+ UnresolvedInlineTable(Seq("c1"), Seq(Seq(Count(lit(1))))))
+ }
+
+ // unresolved attribute should not work
+ intercept[AnalysisException] {
+ EvaluateUnresolvedInlineTable.validateInputEvaluable(
+ UnresolvedInlineTable(Seq("c1"), Seq(Seq(UnresolvedAttribute("A")))))
+ }
}
+ }
- // aggregate should not work
- intercept[AnalysisException] {
- EvaluateUnresolvedInlineTable.validateInputEvaluable(
- UnresolvedInlineTable(Seq("c1"), Seq(Seq(Count(lit(1))))))
+ test("validate inputs are foldable or can be evaluated when the query runs")
{
+ val oneRow = Project(Seq(Alias(lit(1), "a")()), OneRowRelation())
+ Seq(lit(1), Alias(lit(1), "a")(), Rand(1), Uuid(Some(1L)),
ScalarSubquery(oneRow)).foreach {
+ e =>
+ EvaluateUnresolvedInlineTable.validateInputEvaluable(
+ UnresolvedInlineTable(Seq("c1"), Seq(Seq(e))))
}
- // unresolved attribute should not work
- intercept[AnalysisException] {
- EvaluateUnresolvedInlineTable.validateInputEvaluable(
- UnresolvedInlineTable(Seq("c1"), Seq(Seq(UnresolvedAttribute("A")))))
+ val outerReference = OuterReference(AttributeReference("a", IntegerType)())
+ val correlatedPlan = Project(Seq(Alias(outerReference, "b")()),
OneRowRelation())
+ Seq(
+ Count(lit(1)),
+ Count(lit(1)).toAggregateExpression(),
+ Explode(CreateArray(Seq(lit(1), lit(2)))),
+ outerReference,
+ ScalarSubquery(correlatedPlan, outerAttrs = Seq(outerReference)),
+ Exists(oneRow),
+ InSubquery(Seq(lit(1)), ListQuery(oneRow, numCols = 1)),
+ UnresolvedAttribute("A")
+ ).foreach { e =>
+ intercept[AnalysisException] {
+ EvaluateUnresolvedInlineTable.validateInputEvaluable(
+ UnresolvedInlineTable(Seq("c1"), Seq(Seq(e))))
+ }
}
}
@@ -156,6 +186,109 @@ class ResolveInlineTablesSuite extends AnalysisTest with
BeforeAndAfter {
assert(data.head.getLong(0) == correct)
}
+ test("inline tables with non-foldable expressions are evaluated when the
query runs") {
+ val table = UnresolvedInlineTable(Seq("c1", "c2"),
+ Seq(Seq(lit(1), Rand(1)), Seq(Add(lit(2), lit(3)), lit(1.5))))
+ val resolved = ResolveInlineTables(table)
+ resolved match {
+ case Expand(projections, output, _: OneRowRelation) =>
+ assert(output.map(_.dataType) == Seq(IntegerType, DoubleType))
+ // Foldable expressions are evaluated during analysis.
+ assert(projections == Seq(Seq(lit(1), Rand(1)), Seq(lit(5), lit(1.5))))
+ case other => fail(s"Expected an Expand, but got $other")
+ }
+ assert(resolved.maxRows.contains(2L))
+ // The optimizer leaves the table in the plan, to be evaluated when the
query runs.
+ assert(EvalInlineTables(resolved) == resolved)
+ }
+
+ test("inline tables without non-foldable expressions are resolved as
before") {
+ val flag = SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED.key
+ Seq(
+ Seq(Seq(lit(1), Add(lit(2), lit(3))), Seq(lit(4L), lit(5))),
+ // CURRENT_LIKE expressions are replaced at the end of the analysis
phase, so the other
+ // expressions of the table are evaluated at that point too.
+ Seq(Seq(CurrentTimestamp(), Add(lit(2), lit(3))))
+ ).foreach { rows =>
+ val table = UnresolvedInlineTable(rows.head.indices.map(i => s"c$i"),
rows)
+ val resolved = ResolveInlineTables(table)
+ assert(!resolved.isInstanceOf[Expand])
+ comparePlans(
+ resolved, withSQLConf(flag -> "false")(ResolveInlineTables(table)),
checkAnalysis = false)
+ }
+ }
+
+ test("aliased foldable expressions are evaluated during analysis") {
+ // Column DEFAULT values are substituted as aliased expressions.
+ val table = UnresolvedInlineTable(Seq("c1"), Seq(Seq(Alias(lit(1),
"a")())))
+ assert(ResolveInlineTables(table).isInstanceOf[LocalRelation])
+
+ val mixed = UnresolvedInlineTable(Seq("c1", "c2"), Seq(Seq(Alias(lit(1),
"a")(), Rand(1))))
+ ResolveInlineTables(mixed) match {
+ case Expand(Seq(Seq(l: Literal, _: Rand)), _, _) => assert(l == lit(1))
+ case other => fail(s"Expected an Expand, but got $other")
+ }
+ }
+
+ test("CURRENT_LIKE expressions next to non-foldable expressions") {
+ val table = UnresolvedInlineTable(Seq("c1", "c2"),
+ Seq(Seq(CurrentTimestamp(), Rand(1)), Seq(CurrentTimestamp(), Rand(2))))
+ EvalInlineTables(ComputeCurrentTime(ResolveInlineTables(table))) match {
+ case Expand(Seq(Seq(ts1: Literal, _: Rand), Seq(ts2: Literal, _: Rand)),
_, _) =>
+ // Both CURRENT_TIMESTAMP expressions are evaluated to the same value.
+ assert(ts1 == ts2)
+ case other => fail(s"Expected an Expand, but got $other")
+ }
+ }
+
+ test("nondeterministic expressions next to CURRENT_LIKE expressions in the
same expression") {
+ // A ResolvedInlineTable cannot contain nondeterministic expressions.
+ val e = Concat(Seq(CurrentUser(), Uuid(Some(1L))))
+ val table = UnresolvedInlineTable(Seq("c1"), Seq(Seq(e)))
+ ResolveInlineTables(table) match {
+ case Expand(Seq(Seq(projection)), _, _: OneRowRelation) =>
assert(projection == e)
+ case other => fail(s"Expected an Expand, but got $other")
+ }
+ withSQLConf(SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED.key ->
"false") {
+ assert(ResolveInlineTables(table).isInstanceOf[ResolvedInlineTable])
+ }
+ }
+
+ test("subqueries with CURRENT_LIKE expressions are not evaluated during
analysis") {
+ // The tree patterns of a subquery expression include those of its plan.
+ val currentTimestampRow = Project(Seq(Alias(CurrentTimestamp(), "ts")()),
OneRowRelation())
+ Seq(
+ ScalarSubquery(currentTimestampRow),
+ CreateArray(Seq(CurrentTimestamp(), ScalarSubquery(currentTimestampRow)))
+ ).foreach { e =>
+ ResolveInlineTables(UnresolvedInlineTable(Seq("c1"), Seq(Seq(e)))) match
{
+ case Expand(Seq(Seq(projection)), _, _: OneRowRelation) =>
assert(projection == e)
+ case other => fail(s"Expected an Expand, but got $other")
+ }
+ }
+ Seq(
+ Exists(currentTimestampRow),
+ InSubquery(Seq(CurrentTimestamp()), ListQuery(currentTimestampRow,
numCols = 1))
+ ).foreach { e =>
+ val table = UnresolvedInlineTable(Seq("c1"), Seq(Seq(e)))
+ val ex = intercept[AnalysisException](ResolveInlineTables(table))
+ assert(ex.getCondition ==
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE")
+ }
+ }
+
+ test("evaluation errors of foldable expressions next to non-foldable
expressions") {
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ val table = UnresolvedInlineTable(Seq("c1", "c2"),
+ Seq(Seq(Cast(lit("abc"), IntegerType), Rand(1))))
+ val e = intercept[AnalysisException](ResolveInlineTables(table))
+ assert(e.getCondition ==
"INVALID_INLINE_TABLE.FAILED_SQL_EXPRESSION_EVALUATION")
+ }
+ }
+
+ test("analysis of inline tables with nondeterministic expressions") {
+ assertAnalysisSuccess(UnresolvedInlineTable(Seq("c1"), Seq(Seq(Rand(1)),
Seq(Rand(2)))))
+ }
+
test("nullability inference in convert") {
val table1 = UnresolvedInlineTable(Seq("c1"), Seq(Seq(lit(1)),
Seq(lit(2L))))
val converted1 =
EvaluateUnresolvedInlineTable.findCommonTypesAndCast(table1)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
index d68dc0df2aa5..ee31909549e0 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/ExpandExec.scala
@@ -147,9 +147,11 @@ case class ExpandExec(
// Tracks whether a column has the same output for all rows.
// Size of sameOutput array should equal N.
// If sameOutput(i) is true, then the i-th column has the same value for
all output rows given
- // an input row.
+ // an input row. Equal nondeterministic expressions can still produce a
different value for
+ // every output row.
val sameOutput: Array[Boolean] = output.indices.map { colIndex =>
- projections.map(p => p(colIndex)).toSet.size == 1
+ val column = projections.map(p => p(colIndex))
+ column.toSet.size == 1 && column.head.deterministic
}.toArray
// Bind all the branch expressions once up front, so that identical
expressions appearing in
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/inline-table.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/inline-table.sql.out
index 4aa144b59257..f0af900e5499 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/inline-table.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/inline-table.sql.out
@@ -95,21 +95,7 @@ Project [a#x, b#x]
-- !query
select * from values ("one", rand(5)), ("two", 3.0D) as data(a, b)
-- !query analysis
-org.apache.spark.sql.AnalysisException
-{
- "errorClass" :
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
- "sqlState" : "42000",
- "messageParameters" : {
- "expr" : "\"rand(5)\""
- },
- "queryContext" : [ {
- "objectType" : "",
- "objectName" : "",
- "startIndex" : 30,
- "stopIndex" : 36,
- "fragment" : "rand(5)"
- } ]
-}
+[Analyzer test output redacted due to nondeterminism]
-- !query
@@ -277,3 +263,97 @@ select * from values (1 as a, 2 as b)
-- !query analysis
Project [col1#x, col2#x]
+- LocalRelation [col1#x, col2#x]
+
+
+-- !query
+select count(distinct u) from values (uuid()), (uuid()), (uuid()) as data(u)
+-- !query analysis
+[Analyzer test output redacted due to nondeterminism]
+
+
+-- !query
+select count(distinct ct), count(distinct r) from values (current_timestamp(),
rand()), (current_timestamp(), rand()) as data(ct, r)
+-- !query analysis
+[Analyzer test output redacted due to nondeterminism]
+
+
+-- !query
+select a, b is not null from values (1 + 1, rand()), (cast('3' as int),
rand()) as data(a, b)
+-- !query analysis
+[Analyzer test output redacted due to nondeterminism]
+
+
+-- !query
+select * from values ((select 1) + (select 2), 'a'), ((select max(c) from
values (4), (5) as t(c)), 'b') as data(a, b)
+-- !query analysis
+Project [a#x, b#x]
++- SubqueryAlias data
+ +- Expand [[(scalar-subquery#x [] + scalar-subquery#x []), a],
[scalar-subquery#x [], b]], [a#x, b#x]
+ : :- Project [1 AS 1#x]
+ : : +- OneRowRelation
+ : :- Project [2 AS 2#x]
+ : : +- OneRowRelation
+ : +- Aggregate [max(c#x) AS max(c)#x]
+ : +- SubqueryAlias t
+ : +- LocalRelation [c#x]
+ +- OneRowRelation
+
+
+-- !query
+select * from values (1) as t(a), lateral (values (t.a, rand())) as v(b, r)
+-- !query analysis
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" :
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
+ "sqlState" : "42000",
+ "messageParameters" : {
+ "expr" : "\"outer(t.a)\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 52,
+ "stopIndex" : 54,
+ "fragment" : "t.a"
+ } ]
+}
+
+
+-- !query
+select * from values (exists (select 1), rand()) as data(a, b)
+-- !query analysis
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" :
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
+ "sqlState" : "42000",
+ "messageParameters" : {
+ "expr" : "\"exists()\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 23,
+ "stopIndex" : 39,
+ "fragment" : "exists (select 1)"
+ } ]
+}
+
+
+-- !query
+select * from values (row_number() over (order by 1), rand()) as data(a, b)
+-- !query analysis
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" :
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
+ "sqlState" : "42000",
+ "messageParameters" : {
+ "expr" : "\"row_number() OVER (ORDER BY 1 ASC NULLS FIRST ROWS BETWEEN
UNBOUNDED PRECEDING AND CURRENT ROW)\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 23,
+ "stopIndex" : 52,
+ "fragment" : "row_number() over (order by 1)"
+ } ]
+}
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/scripting/randomly_generated_scripts.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/scripting/randomly_generated_scripts.sql.out
index 793b2b37415d..81aa94f12ec6 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/scripting/randomly_generated_scripts.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/scripting/randomly_generated_scripts.sql.out
@@ -479,7 +479,7 @@ BEGIN
END;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -888,7 +888,7 @@ BEGIN
END FOR supplier_loop;
END
-- !query analysis
-LocalRelation <empty>
+LocalRelation [col1#x]
-- !query
@@ -994,7 +994,7 @@ BEGIN
END L4;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -1285,7 +1285,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -1372,7 +1372,7 @@ BEGIN
END;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -1661,7 +1661,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -2275,7 +2275,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -2351,7 +2351,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -2594,7 +2594,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -2672,7 +2672,7 @@ BEGIN
END L2;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -2767,7 +2767,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -3210,7 +3210,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -3318,7 +3318,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -3552,7 +3552,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -3863,7 +3863,7 @@ BEGIN
END;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -4147,7 +4147,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -4642,7 +4642,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation <empty>
+LocalRelation [col1#x]
-- !query
@@ -4911,7 +4911,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -5015,7 +5015,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -5154,7 +5154,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -5439,7 +5439,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -5472,7 +5472,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation <empty>
+LocalRelation [col1#x]
-- !query
@@ -5529,7 +5529,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -5644,7 +5644,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -5749,7 +5749,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -5990,7 +5990,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -6476,7 +6476,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
@@ -6700,7 +6700,7 @@ BEGIN
END L0;
END
-- !query analysis
-LocalRelation [col1#x]
+LocalRelation <empty>
-- !query
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/udf/udf-inline-table.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/udf/udf-inline-table.sql.out
index 786b5ac49b12..9bf3fc541f30 100644
---
a/sql/core/src/test/resources/sql-tests/analyzer-results/udf/udf-inline-table.sql.out
+++
b/sql/core/src/test/resources/sql-tests/analyzer-results/udf/udf-inline-table.sql.out
@@ -81,21 +81,7 @@ Project [cast(udf(cast(a#x as string)) as string) AS
udf(a)#x, b#x]
-- !query
select udf(a), b from values ("one", rand(5)), ("two", 3.0D) as data(a, b)
-- !query analysis
-org.apache.spark.sql.AnalysisException
-{
- "errorClass" :
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
- "sqlState" : "42000",
- "messageParameters" : {
- "expr" : "\"rand(5)\""
- },
- "queryContext" : [ {
- "objectType" : "",
- "objectName" : "",
- "startIndex" : 38,
- "stopIndex" : 44,
- "fragment" : "rand(5)"
- } ]
-}
+[Analyzer test output redacted due to nondeterminism]
-- !query
diff --git a/sql/core/src/test/resources/sql-tests/inputs/inline-table.sql
b/sql/core/src/test/resources/sql-tests/inputs/inline-table.sql
index bb66e5dc6b9f..641df4a8e0a1 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/inline-table.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/inline-table.sql
@@ -35,7 +35,7 @@ select * from values ("one", array(0, 1)), ("two", array(2,
3)) as data(a, b);
-- decimal and double coercion
select * from values ("one", 2.0), ("two", 3.0D) as data(a, b);
--- error reporting: nondeterministic function rand
+-- nondeterministic function rand
select * from values ("one", rand(5)), ("two", 3.0D) as data(a, b);
-- error reporting: different number of columns
@@ -75,3 +75,24 @@ select * from values (1 as id, 'a' as name), (2 as id, 'b'
as name) as t(a, b);
-- aliased expressions without table alias
select * from values (1 as a, 2 as b);
+
+-- nondeterministic functions have a different seed in every row
+select count(distinct u) from values (uuid()), (uuid()), (uuid()) as data(u);
+
+-- current_timestamp() is evaluated once per query, also next to non-foldable
expressions
+select count(distinct ct), count(distinct r) from values (current_timestamp(),
rand()), (current_timestamp(), rand()) as data(ct, r);
+
+-- foldable expressions next to non-foldable expressions
+select a, b is not null from values (1 + 1, rand()), (cast('3' as int),
rand()) as data(a, b);
+
+-- uncorrelated scalar subqueries
+select * from values ((select 1) + (select 2), 'a'), ((select max(c) from
values (4), (5) as t(c)), 'b') as data(a, b);
+
+-- error reporting: correlated reference
+select * from values (1) as t(a), lateral (values (t.a, rand())) as v(b, r);
+
+-- error reporting: EXISTS subquery
+select * from values (exists (select 1), rand()) as data(a, b);
+
+-- error reporting: window function
+select * from values (row_number() over (order by 1), rand()) as data(a, b);
diff --git
a/sql/core/src/test/resources/sql-tests/inputs/udf/udf-inline-table.sql
b/sql/core/src/test/resources/sql-tests/inputs/udf/udf-inline-table.sql
index 179e528d74ca..daa06b81febf 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/udf/udf-inline-table.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/udf/udf-inline-table.sql
@@ -32,7 +32,7 @@ select udf(a), b from values ("one", array(0, 1)), ("two",
array(2, 3)) as data(
-- decimal and double coercion
select udf(a), b from values ("one", 2.0), ("two", 3.0D) as data(a, b);
--- error reporting: nondeterministic function rand
+-- nondeterministic function rand
select udf(a), b from values ("one", rand(5)), ("two", 3.0D) as data(a, b);
-- error reporting: different number of columns
diff --git a/sql/core/src/test/resources/sql-tests/results/inline-table.sql.out
b/sql/core/src/test/resources/sql-tests/results/inline-table.sql.out
index dee75e7338aa..cc6553d8b21b 100644
--- a/sql/core/src/test/resources/sql-tests/results/inline-table.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/inline-table.sql.out
@@ -107,23 +107,10 @@ two 3.0
-- !query
select * from values ("one", rand(5)), ("two", 3.0D) as data(a, b)
-- !query schema
-struct<>
+struct<a:string,b:double>
-- !query output
-org.apache.spark.sql.AnalysisException
-{
- "errorClass" :
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
- "sqlState" : "42000",
- "messageParameters" : {
- "expr" : "\"rand(5)\""
- },
- "queryContext" : [ {
- "objectType" : "",
- "objectName" : "",
- "startIndex" : 30,
- "stopIndex" : 36,
- "fragment" : "rand(5)"
- } ]
-}
+one 0.02390696427502892
+two 3.0
-- !query
@@ -313,3 +300,103 @@ select * from values (1 as a, 2 as b)
struct<col1:int,col2:int>
-- !query output
1 2
+
+
+-- !query
+select count(distinct u) from values (uuid()), (uuid()), (uuid()) as data(u)
+-- !query schema
+struct<count(DISTINCT u):bigint>
+-- !query output
+3
+
+
+-- !query
+select count(distinct ct), count(distinct r) from values (current_timestamp(),
rand()), (current_timestamp(), rand()) as data(ct, r)
+-- !query schema
+struct<count(DISTINCT ct):bigint,count(DISTINCT r):bigint>
+-- !query output
+1 2
+
+
+-- !query
+select a, b is not null from values (1 + 1, rand()), (cast('3' as int),
rand()) as data(a, b)
+-- !query schema
+struct<a:int,(b IS NOT NULL):boolean>
+-- !query output
+2 true
+3 true
+
+
+-- !query
+select * from values ((select 1) + (select 2), 'a'), ((select max(c) from
values (4), (5) as t(c)), 'b') as data(a, b)
+-- !query schema
+struct<a:int,b:string>
+-- !query output
+3 a
+5 b
+
+
+-- !query
+select * from values (1) as t(a), lateral (values (t.a, rand())) as v(b, r)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" :
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
+ "sqlState" : "42000",
+ "messageParameters" : {
+ "expr" : "\"outer(t.a)\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 52,
+ "stopIndex" : 54,
+ "fragment" : "t.a"
+ } ]
+}
+
+
+-- !query
+select * from values (exists (select 1), rand()) as data(a, b)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" :
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
+ "sqlState" : "42000",
+ "messageParameters" : {
+ "expr" : "\"exists()\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 23,
+ "stopIndex" : 39,
+ "fragment" : "exists (select 1)"
+ } ]
+}
+
+
+-- !query
+select * from values (row_number() over (order by 1), rand()) as data(a, b)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.AnalysisException
+{
+ "errorClass" :
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
+ "sqlState" : "42000",
+ "messageParameters" : {
+ "expr" : "\"row_number() OVER (ORDER BY 1 ASC NULLS FIRST ROWS BETWEEN
UNBOUNDED PRECEDING AND CURRENT ROW)\""
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 23,
+ "stopIndex" : 52,
+ "fragment" : "row_number() over (order by 1)"
+ } ]
+}
diff --git
a/sql/core/src/test/resources/sql-tests/results/scripting/randomly_generated_scripts.sql.out
b/sql/core/src/test/resources/sql-tests/results/scripting/randomly_generated_scripts.sql.out
index 1583266025ea..160a7f6c4a94 100644
---
a/sql/core/src/test/resources/sql-tests/results/scripting/randomly_generated_scripts.sql.out
+++
b/sql/core/src/test/resources/sql-tests/results/scripting/randomly_generated_scripts.sql.out
@@ -495,9 +495,9 @@ BEGIN
END;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Orders processing error
+
-- !query
@@ -914,9 +914,9 @@ BEGIN
END FOR supplier_loop;
END
-- !query schema
-struct<>
+struct<col1:string>
-- !query output
-
+Supplier contact management error
-- !query
@@ -1024,9 +1024,9 @@ BEGIN
END L4;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Customer order calculation error
+
-- !query
@@ -1323,9 +1323,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Tenure analysis error
+
-- !query
@@ -1412,9 +1412,9 @@ BEGIN
END;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Order source analysis error
+
-- !query
@@ -1707,9 +1707,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Employee demographics error
+
-- !query
@@ -2339,9 +2339,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-City shipping error
+
-- !query
@@ -2417,9 +2417,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Employee hierarchy error
+
-- !query
@@ -2668,9 +2668,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Product price error
+
-- !query
@@ -2748,9 +2748,9 @@ BEGIN
END L2;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Employee diversity error
+
-- !query
@@ -2845,9 +2845,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Channel analysis error
+
-- !query
@@ -3300,9 +3300,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Order status error
+
-- !query
@@ -3410,9 +3410,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Department diversity error
+
-- !query
@@ -3652,9 +3652,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Product margin error
+
-- !query
@@ -3971,9 +3971,9 @@ BEGIN
END;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Color diversity error
+
-- !query
@@ -4261,9 +4261,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Product size error
+
-- !query
@@ -4770,9 +4770,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<>
+struct<col1:string>
-- !query output
-
+Order item status error
-- !query
@@ -5047,9 +5047,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Order-item linkage error
+
-- !query
@@ -5155,9 +5155,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Inventory movement error
+
-- !query
@@ -5298,9 +5298,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Tenure salary error
+
-- !query
@@ -5593,9 +5593,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Type order diversity error
+
-- !query
@@ -5628,9 +5628,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<>
+struct<col1:string>
-- !query output
-
+Offer code error
-- !query
@@ -5687,9 +5687,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Dept stagnation error
+
-- !query
@@ -5806,9 +5806,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Discount balancing error
+
-- !query
@@ -5915,9 +5915,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Order value error
+
-- !query
@@ -6162,9 +6162,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Bulk item error
+
-- !query
@@ -6666,9 +6666,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Status workflow error
+
-- !query
@@ -6898,9 +6898,9 @@ BEGIN
END L0;
END
-- !query schema
-struct<col1:string>
+struct<>
-- !query output
-Bundle error
+
-- !query
diff --git
a/sql/core/src/test/resources/sql-tests/results/udf/udf-inline-table.sql.out
b/sql/core/src/test/resources/sql-tests/results/udf/udf-inline-table.sql.out
index 3e84ec09c215..47e048c7e20f 100644
--- a/sql/core/src/test/resources/sql-tests/results/udf/udf-inline-table.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/udf/udf-inline-table.sql.out
@@ -91,23 +91,10 @@ two 3.0
-- !query
select udf(a), b from values ("one", rand(5)), ("two", 3.0D) as data(a, b)
-- !query schema
-struct<>
+struct<udf(a):string,b:double>
-- !query output
-org.apache.spark.sql.AnalysisException
-{
- "errorClass" :
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
- "sqlState" : "42000",
- "messageParameters" : {
- "expr" : "\"rand(5)\""
- },
- "queryContext" : [ {
- "objectType" : "",
- "objectName" : "",
- "startIndex" : 38,
- "stopIndex" : 44,
- "fragment" : "rand(5)"
- } ]
-}
+one 0.02390696427502892
+two 3.0
-- !query
diff --git
a/sql/core/src/test/scala/org/apache/spark/sql/InlineTableNonFoldableExpressionsSuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/InlineTableNonFoldableExpressionsSuite.scala
new file mode 100644
index 000000000000..d083d699c15d
--- /dev/null
+++
b/sql/core/src/test/scala/org/apache/spark/sql/InlineTableNonFoldableExpressionsSuite.scala
@@ -0,0 +1,365 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.spark.sql
+
+import java.util.UUID
+
+import org.apache.spark.{SparkRuntimeException, TaskContext}
+import org.apache.spark.sql.IntegratedUDFTestUtils.{registerTestUDF,
shouldTestPythonUDFs, TestPythonUDF}
+import org.apache.spark.sql.execution.{ExpandExec, LocalTableScanExec}
+import org.apache.spark.sql.execution.adaptive.AdaptiveSparkPlanHelper
+import org.apache.spark.sql.functions.udf
+import org.apache.spark.sql.internal.SQLConf
+import org.apache.spark.sql.test.SharedSparkSession
+import org.apache.spark.sql.types.{DoubleType, LongType, StructField,
StructType}
+
+/**
+ * End-to-end tests for VALUES lists with expressions that are not foldable,
which are evaluated
+ * by an `Expand` over a `OneRowRelation` when the query runs.
+ */
+class InlineTableNonFoldableExpressionsSuite
+ extends QueryTest with SharedSparkSession with AdaptiveSparkPlanHelper {
+ import testImplicits._
+
+ private def expandExecs(df: DataFrame): Seq[ExpandExec] =
+ collect(df.queryExecution.executedPlan) { case e: ExpandExec => e }
+
+ private def contextOf(query: String, fragment: String): ExpectedContext = {
+ val start = query.indexOf(fragment)
+ assert(start >= 0, s"'$fragment' is not part of '$query'")
+ ExpectedContext(fragment, start, start + fragment.length - 1)
+ }
+
+ test("non-foldable expressions are evaluated on executors") {
+ withUserDefinedFunction("in_task" -> true) {
+ spark.udf.register("in_task", udf(() => TaskContext.get() !=
null).asNondeterministic())
+ val df = sql("SELECT * FROM VALUES (1, in_task()), (2, in_task()) AS
t(id, in_task)")
+ checkAnswer(df, Seq(Row(1, true), Row(2, true)))
+ val execs = expandExecs(df)
+ assert(execs.size == 1)
+ // All the rows are evaluated in a single task.
+ assert(execs.head.execute().getNumPartitions == 1)
+ }
+ }
+
+ test("VALUES lists with only foldable expressions still become local
relations") {
+ val df = sql("SELECT * FROM VALUES (1, 'a'), (2 + 1, upper('b')) AS t(id,
s)")
+ checkAnswer(df, Seq(Row(1, "a"), Row(3, "B")))
+ assert(expandExecs(df).isEmpty)
+ assert(collect(df.queryExecution.executedPlan) { case l:
LocalTableScanExec => l }.size == 1)
+ }
+
+ test("VALUES lists that were supported before are evaluated during analysis
as before") {
+ withUserDefinedFunction("on_driver" -> true) {
+ spark.udf.register("on_driver", udf((s: String) => s != null &&
TaskContext.get() == null))
+ val queries = Seq(
+ "SELECT * FROM VALUES (1, 'a'), (2 + 1, upper('b')) AS t(a, b)",
+ "SELECT * FROM VALUES (current_date(), 1 + 1) AS t(a, b)",
+ // The UDF is not foldable, but it was supported because its input is
a CURRENT_LIKE
+ // expression. It is evaluated on the driver at the end of the
analysis phase.
+ "SELECT * FROM VALUES (on_driver(current_user())) AS t(a)")
+ queries.foreach { query =>
+ val df = sql(query)
+ val legacy = withSQLConf(
+ SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED.key ->
"false") {
+ sql(query)
+ }
+ comparePlans(
+ df.queryExecution.analyzed, legacy.queryExecution.analyzed,
checkAnalysis = false)
+ checkAnswer(df, legacy)
+ assert(expandExecs(df).isEmpty)
+ }
+ checkAnswer(sql(queries.last), Row(true))
+ }
+
+ // The expressions of inline tables with CURRENT_LIKE expressions are
evaluated at the end of
+ // the analysis phase, so evaluation errors are reported when the query is
optimized.
+ withSQLConf(SQLConf.ANSI_ENABLED.key -> "true") {
+ val df = sql("SELECT * FROM VALUES (current_date(), 1 / 0) AS t(a, b)")
+ val e = intercept[AnalysisException](df.collect())
+ assert(e.getCondition ==
"INVALID_INLINE_TABLE.FAILED_SQL_EXPRESSION_EVALUATION")
+ }
+ }
+
+ test("nondeterministic expressions") {
+ // Every expression has its own seed.
+ val uuids = sql("SELECT * FROM VALUES (uuid()), (uuid()),
(uuid())").collect()
+ assert(uuids.map(_.getString(0)).distinct.length == 3)
+
+ // Each row is evaluated like the only row of a one-row relation.
+ checkAnswer(
+ sql("SELECT * FROM VALUES (rand(5)), (rand(5))"),
+ sql("SELECT rand(5) UNION ALL SELECT rand(5)"))
+
+ // The seeds are chosen during analysis, so executing the plan again
returns the same values.
+ val df = sql("SELECT * FROM VALUES (rand(), 1), (rand(), 2)")
+ checkAnswer(df, df.collect().toSeq)
+ }
+
+ test("equal nondeterministic expressions are evaluated separately for every
row") {
+ withUserDefinedFunction("random_uuid" -> true) {
+ spark.udf.register("random_uuid", udf(() =>
UUID.randomUUID().toString).asNondeterministic())
+ Seq(true, false).foreach { codegen =>
+ withSQLConf(SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key ->
codegen.toString) {
+ val rows = sql("SELECT * FROM VALUES (random_uuid()),
(random_uuid())").collect()
+ assert(rows.map(_.getString(0)).distinct.length == 2)
+ }
+ }
+ }
+ }
+
+ test("CURRENT_LIKE expressions are evaluated once per query") {
+ val df = sql(
+ """SELECT count(DISTINCT ts), count(DISTINCT r)
+ |FROM VALUES (current_timestamp(), rand()), (current_timestamp(),
rand()) AS t(ts, r)
+ |""".stripMargin)
+ checkAnswer(df, Row(1, 2))
+ }
+
+ test("nondeterministic expressions next to CURRENT_LIKE expressions in the
same expression") {
+ checkAnswer(
+ sql(
+ """SELECT count(DISTINCT s), min(startswith(s, current_user()))
+ |FROM VALUES (concat(current_user(), uuid())),
(concat(current_user(), uuid())) AS t(s)
+ |""".stripMargin),
+ Row(2, true))
+ checkAnswer(
+ sql(
+ """SELECT d BETWEEN current_date() AND date_add(current_date(), 9)
+ |FROM VALUES (date_add(current_date(), CAST(rand() * 10 AS INT))) AS
t(d)
+ |""".stripMargin),
+ Row(true))
+ withUserDefinedFunction("nd_identity" -> true) {
+ spark.udf.register("nd_identity", udf((s: String) =>
s).asNondeterministic())
+ checkAnswer(
+ sql("SELECT s = current_user() FROM VALUES
(nd_identity(current_user())) AS t(s)"),
+ Row(true))
+ }
+
+ withSQLConf(SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED.key ->
"false") {
+ // The inline table is not evaluated when the query runs, so it cannot
contain them.
+ val e = intercept[AnalysisException](
+ sql("SELECT * FROM VALUES (concat(current_user(), uuid()))"))
+ assert(e.getCondition == "INVALID_NON_DETERMINISTIC_EXPRESSIONS")
+ }
+ }
+
+ test("type coercion and nullability") {
+ val df = sql("SELECT * FROM VALUES (1, rand()), (2L, NULL) AS t(a, b)")
+ assert(df.schema == StructType(Seq(
+ StructField("a", LongType, nullable = false),
+ StructField("b", DoubleType, nullable = true))))
+ checkAnswer(df.select($"a", $"b".isNull), Seq(Row(1L, false), Row(2L,
true)))
+ }
+
+ test("uncorrelated scalar subqueries") {
+ checkAnswer(sql("SELECT * FROM VALUES ((SELECT 1) + (SELECT 2), 'a')"),
Row(3, "a"))
+ withTable("t") {
+ sql("CREATE TABLE t(id INT) USING parquet")
+ sql("INSERT INTO t VALUES (1), (2)")
+ sql("INSERT INTO t VALUES ((SELECT max(id) FROM t) + 1)")
+ checkAnswer(sql("SELECT * FROM t"), Seq(Row(1), Row(2), Row(3)))
+ }
+ }
+
+ test("uncorrelated scalar subqueries with CURRENT_LIKE expressions") {
+ // The CURRENT_LIKE expressions inside and next to the subqueries share
the value of the rest
+ // of the query.
+ checkAnswer(
+ sql("SELECT d = current_date() FROM VALUES ((SELECT current_date())) AS
t(d)"),
+ Row(true))
+ checkAnswer(
+ sql(
+ """SELECT d = date_add(current_date(), 1)
+ |FROM VALUES (date_add(current_date(), (SELECT 1))) AS t(d)
+ |""".stripMargin),
+ Row(true))
+ withTable("t") {
+ sql("CREATE TABLE t(id INT, ts TIMESTAMP) USING parquet")
+ sql("INSERT INTO t VALUES (1, TIMESTAMP'2000-01-01 00:00:00')")
+ sql(
+ """INSERT INTO t VALUES
+ | ((SELECT max(id) FROM t WHERE ts < current_timestamp()) + 1,
current_timestamp())
+ |""".stripMargin)
+ checkAnswer(sql("SELECT id FROM t"), Seq(Row(1), Row(2)))
+ }
+ }
+
+ test("unresolved references in VALUES lists with scalar subqueries") {
+ val unknownColumn = "SELECT * FROM VALUES ((SELECT 1), unknown_col)"
+ checkError(
+ exception = intercept[AnalysisException](sql(unknownColumn)),
+ condition = "UNRESOLVED_COLUMN.WITHOUT_SUGGESTION",
+ parameters = Map("objectName" -> "`unknown_col`"),
+ context = contextOf(unknownColumn, "unknown_col"))
+ val missingTable = "SELECT * FROM VALUES ((SELECT x FROM missing_table))"
+ checkError(
+ exception = intercept[AnalysisException](sql(missingTable)),
+ condition = "TABLE_OR_VIEW_NOT_FOUND",
+ parameters = Map("relationName" -> "`missing_table`"),
+ context = contextOf(missingTable, "missing_table"))
+ withTable("t") {
+ sql("CREATE TABLE t(id INT) USING parquet")
+ val insert = "INSERT INTO t VALUES ((SELECT max(id) FROM missing_table))"
+ checkError(
+ exception = intercept[AnalysisException](sql(insert)),
+ condition = "TABLE_OR_VIEW_NOT_FOUND",
+ parameters = Map("relationName" -> "`missing_table`"),
+ context = contextOf(insert, "missing_table"))
+ }
+ }
+
+ test("correlated references and subqueries other than scalar subqueries are
rejected") {
+ withTempView("t") {
+ sql("CREATE TEMPORARY VIEW t AS SELECT * FROM VALUES (1) AS t(a)")
+ val query = "SELECT * FROM t, LATERAL (VALUES (t.a, rand())) AS v(b, r)"
+ checkError(
+ exception = intercept[AnalysisException](sql(query)),
+ condition =
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
+ parameters = Map("expr" -> "\"outer(t.a)\""),
+ context = contextOf(query, "t.a"))
+
+ // The subquery can only reference the columns of the inline table,
which has none.
+ val nested = "SELECT * FROM t, LATERAL (VALUES ((SELECT t.a), rand()))
AS v(b, r)"
+ checkError(
+ exception = intercept[AnalysisException](sql(nested)),
+ condition = "UNRESOLVED_COLUMN.WITHOUT_SUGGESTION",
+ parameters = Map("objectName" -> "`t`.`a`"),
+ context = contextOf(nested, "t.a"))
+ }
+ Seq(
+ "SELECT * FROM VALUES (EXISTS (SELECT 1), rand())",
+ "SELECT * FROM VALUES (1 IN (SELECT 1), rand())",
+ // CURRENT_LIKE expressions don't make them evaluable during analysis.
+ "SELECT * FROM VALUES (EXISTS (SELECT current_date()))",
+ "SELECT * FROM VALUES (current_date() IN (SELECT
current_date()))").foreach { query =>
+ val e = intercept[AnalysisException](sql(query))
+ assert(e.getCondition ==
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE")
+ }
+ }
+
+ test("aggregate, window and generator functions are rejected") {
+ Seq(
+ "SELECT * FROM VALUES (count(1), rand())",
+ "SELECT * FROM VALUES (row_number() OVER (ORDER BY 1), rand())",
+ "SELECT * FROM VALUES (explode(array(1, 2)), 1)",
+ // Also over CURRENT_LIKE expressions, next to expressions evaluated
when the query runs.
+ "SELECT * FROM VALUES (max(current_date()), rand())",
+ "SELECT * FROM VALUES (row_number() OVER (ORDER BY current_date()),
rand())",
+ "SELECT * FROM VALUES (explode(array(current_date())), rand())").foreach
{ query =>
+ val e = intercept[AnalysisException](sql(query))
+ assert(e.getCondition ==
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE")
+ }
+ }
+
+ test("SQL UDFs are rejected") {
+ withUserDefinedFunction("plus_one" -> true) {
+ sql("CREATE TEMPORARY FUNCTION plus_one(x INT) RETURNS INT RETURN x + 1")
+ val e = intercept[AnalysisException](sql("SELECT * FROM VALUES
(plus_one(1))"))
+ assert(e.getCondition ==
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE")
+ }
+ }
+
+ test("Python UDFs are rejected") {
+ assume(shouldTestPythonUDFs)
+ withUserDefinedFunction("py_udf" -> true) {
+ registerTestUDF(TestPythonUDF(name = "py_udf"), spark)
+ val e = intercept[AnalysisException](sql("SELECT * FROM VALUES
(py_udf(1))"))
+ assert(e.getCondition ==
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE")
+ }
+ }
+
+ test("evaluation errors are raised when the query runs") {
+ val df = sql("SELECT * FROM VALUES (1, raise_error('boom'))")
+ checkError(
+ exception = intercept[SparkRuntimeException](df.collect()),
+ condition = "USER_RAISED_EXCEPTION",
+ parameters = Map("errorMessage" -> "boom"))
+ }
+
+ test("the expressions of columns that are not read are not evaluated") {
+ // Like the expressions of a SELECT list, the optimizer removes them.
+ checkAnswer(sql("SELECT a FROM VALUES (1, raise_error('boom')) AS t(a,
b)"), Row(1))
+ }
+
+ test("INSERT INTO ... VALUES with non-foldable expressions and DEFAULT") {
+ withTable("t") {
+ sql("CREATE TABLE t(id INT, s STRING DEFAULT 'x', u STRING) USING
parquet")
+ sql("INSERT INTO t VALUES (1, DEFAULT, uuid()), (2, 'y', uuid())")
+ checkAnswer(sql("SELECT id, s, length(u) FROM t"), Seq(Row(1, "x", 36),
Row(2, "y", 36)))
+ checkAnswer(sql("SELECT count(DISTINCT u) FROM t"), Row(2))
+ }
+ }
+
+ test("self-join of an inline table with non-foldable expressions") {
+ val df = sql("SELECT * FROM VALUES (1, rand()), (2, rand()) AS t(id, r)")
+ val joined = df.as("a").join(df.as("b"), $"a.id" === $"b.id")
+ // Both sides are the same plan, so they produce the same values.
+ checkAnswer(joined.select($"a.id", $"a.r" === $"b.r"), Seq(Row(1, true),
Row(2, true)))
+ }
+
+ test("whole-stage codegen and adaptive query execution") {
+ Seq(true, false).foreach { codegen =>
+ Seq(true, false).foreach { aqe =>
+ withSQLConf(
+ SQLConf.WHOLESTAGE_CODEGEN_ENABLED.key -> codegen.toString,
+ SQLConf.ADAPTIVE_EXECUTION_ENABLED.key -> aqe.toString) {
+ val df = sql(
+ """SELECT a + 1, b IS NOT NULL, c
+ |FROM VALUES (1, rand(), (SELECT 'x')), (2, rand(), 'y') AS t(a,
b, c)
+ |""".stripMargin)
+ checkAnswer(df, Seq(Row(2, true, "x"), Row(3, true, "y")))
+ assert(expandExecs(df).size == 1)
+ }
+ }
+ }
+ }
+
+ test("SQL metrics") {
+ val df = sql("SELECT * FROM VALUES (rand()), (rand()), (rand())")
+ df.collect()
+ assert(expandExecs(df).head.metrics("numOutputRows").value == 3)
+ }
+
+ test("large inline table with non-foldable expressions") {
+ val rows = (1 to 1000).map(i => s"($i, rand())").mkString(", ")
+ checkAnswer(
+ sql(s"SELECT count(*), count(DISTINCT a), count(DISTINCT b) FROM VALUES
$rows AS t(a, b)"),
+ Row(1000, 1000, 1000))
+ }
+
+ test("non-foldable expressions are rejected if the feature is disabled") {
+ withSQLConf(SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED.key ->
"false") {
+ val query = "SELECT * FROM VALUES (1, rand(5))"
+ checkError(
+ exception = intercept[AnalysisException](sql(query)),
+ condition =
"INVALID_INLINE_TABLE.CANNOT_EVALUATE_EXPRESSION_IN_INLINE_TABLE",
+ parameters = Map("expr" -> "\"rand(5)\""),
+ context = contextOf(query, "rand(5)"))
+
+ Seq("(SELECT 1)", "(SELECT current_date())").foreach { subquery =>
+ val subqueryQuery = s"SELECT * FROM VALUES ($subquery)"
+ checkError(
+ exception = intercept[AnalysisException](sql(subqueryQuery)),
+ condition =
"UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.SCALAR_SUBQUERY_IN_VALUES",
+ parameters = Map.empty,
+ context = contextOf(subqueryQuery, s"VALUES ($subquery)"))
+ }
+ }
+ }
+}
diff --git a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
index 2aa37ba2fa6e..a83059664162 100644
--- a/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
+++ b/sql/core/src/test/scala/org/apache/spark/sql/SQLQuerySuite.scala
@@ -4965,18 +4965,21 @@ class SQLQuerySuite extends SharedSparkSession with
AdaptiveSparkPlanHelper
}
test("SPARK-49659: Unsupported scalar subqueries in VALUES") {
- checkError(
- exception = intercept[AnalysisException](
- sql("SELECT * FROM VALUES ((SELECT 1) + (SELECT 2))")
- ),
- condition =
"UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.SCALAR_SUBQUERY_IN_VALUES",
- parameters = Map(),
- context = ExpectedContext(
- fragment = "VALUES ((SELECT 1) + (SELECT 2))",
- start = 14,
- stop = 45
+ withSQLConf(SQLConf.INLINE_TABLE_NON_FOLDABLE_EXPRESSIONS_ENABLED.key ->
"false") {
+ checkError(
+ exception = intercept[AnalysisException](
+ sql("SELECT * FROM VALUES ((SELECT 1) + (SELECT 2))")
+ ),
+ condition =
"UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.SCALAR_SUBQUERY_IN_VALUES",
+ parameters = Map(),
+ context = ExpectedContext(
+ fragment = "VALUES ((SELECT 1) + (SELECT 2))",
+ start = 14,
+ stop = 45
+ )
)
- )
+ }
+ checkAnswer(sql("SELECT * FROM VALUES ((SELECT 1) + (SELECT 2))"), Row(3))
}
test("SPARK-49743: OptimizeCsvJsonExpr does not change schema when pruning
struct") {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]