This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new fbc2c8c4ce26 [SPARK-60597][SQL] Keep the output of a top-level join on
a hidden USING join key in the single-pass resolver
fbc2c8c4ce26 is described below
commit fbc2c8c4ce26d528c583fca096465cd899a5e12c
Author: Dejan Krakovic <[email protected]>
AuthorDate: Wed Oct 28 18:02:54 2026 +0000
[SPARK-60597][SQL] Keep the output of a top-level join on a hidden USING
join key in the single-pass resolver
### What changes were proposed in this pull request?
The single-pass resolver fails dual-run for a top-level join whose
condition references the hidden key of a USING join below it, when nothing
above the join projects its output:
```sql
FROM nt1 JOIN nt2 USING (k) JOIN nt2 x ON nt2.k = x.k
```
```
HYBRID_ANALYZER_EXCEPTION.OUTPUT_SCHEMA_COMPARISON_MISMATCH
```
To resolve `nt2.k`, both analyzers add the hidden key to the output of the
USING join's Project. The fixed-point analyzer's `AddMetadataColumns` then
wraps the outer join in a Project with the join's original output. The
single-pass resolver restores the original output only from the resolver of the
operator above (`RetainsOriginalJoinOutput`), so a top-level join kept the
extra key in its output.
This PR adds `retainTopLevelJoinOutput` to `JoinResolver`. It wraps a
top-level regular join in a Project with the join's output when the join
condition references an attribute found only in the hidden output and that
attribute is in the join's own output, setting the join as the base operator as
the USING join path does. The condition's referenced attributes are used only
when the join has a condition. A LEFT SEMI or LEFT ANTI join outputs only its
left side, so a hidden key of its r [...]
It also makes `ThriftServerQueryTestBase.isNeedSort` sort the results of
queries that start with `FROM`, as `SQLQueryTestSuite` does for any query whose
plan is not sorted. Without it, `ThriftServerQueryTestSuite` compares the new
FROM-first query in server row order against the sorted golden output. The
expected and the actual output go through the same sort, so the other goldens
with FROM-first queries (`unpivot.sql`, `named-window.sql`) stay consistent.
Out of scope here: joins with such a condition below a `Filter`, `Limit`,
`Sort` or `SubqueryAlias` with no projecting operator above, and a condition
that references the hidden key only inside a subquery, for example `EXISTS
(SELECT 1 FROM nt2 y WHERE y.k = nt2.k)`. Both behave as before this PR.
### Why are the changes needed?
With dual-run enabled, which is the default in tests, the query above fails
analysis instead of returning its rows. The single-pass resolver must produce
the same plan as the fixed-point analyzer for the queries `ResolverGuard`
accepts.
### Does this PR introduce _any_ user-facing change?
No. The single-pass resolver is off by default, and dual-run is enabled
only in tests.
### How was this patch tested?
Added two queries to `using-join.sql`: the top-level join above, and the
same join with a correlated scalar subquery in its condition, which still fails
with `UNSUPPORTED_CORRELATED_SCALAR_SUBQUERY` as it does in the fixed-point
analyzer. Without the fix the first query fails with
`OUTPUT_SCHEMA_COMPARISON_MISMATCH`. Also added a top-level LEFT SEMI join and
a LEFT ANTI join whose condition references a hidden key of the right side,
which stay bare joins (wrapping them fails dual-run [...]
Also ran `SQLQueryTestSuite` for the join inputs,
`ThriftServerQueryTestSuite` for `using-join.sql`, `DataFrameJoinSuite`,
`JoinSuite`, `SQLQuerySuite` and the catalyst suites under
`org.apache.spark.sql.catalyst.analysis.resolver`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
This pull request and its description were written by Isaac.
Closes #59843 from dejankrak-db/SPARK-60597-single-pass-top-level-join.
Authored-by: Dejan Krakovic <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit d5df722feab8866b91c165947b47c436aa958c05)
Signed-off-by: Uros Bojanic <[email protected]>
---
.../catalyst/analysis/resolver/JoinResolver.scala | 56 +++++++-
.../resolver/RetainsOriginalJoinOutput.scala | 4 +-
.../sql-tests/analyzer-results/using-join.sql.out | 159 +++++++++++++++++++++
.../test/resources/sql-tests/inputs/using-join.sql | 18 +++
.../resources/sql-tests/results/using-join.sql.out | 76 ++++++++++
.../thriftserver/ThriftServerQueryTestSuite.scala | 3 +
6 files changed, 312 insertions(+), 4 deletions(-)
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/JoinResolver.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/JoinResolver.scala
index d035e8405fad..4287483cc810 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/JoinResolver.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/JoinResolver.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.catalyst.analysis.resolver
-import java.util.HashSet
+import java.util.{HashMap, HashSet}
import org.apache.spark.sql.catalyst.analysis.NaturalAndUsingJoinResolution
import org.apache.spark.sql.catalyst.expressions.{Attribute, Expression,
ExprId, NamedExpression}
@@ -28,7 +28,8 @@ import org.apache.spark.sql.catalyst.util._
/**
* Resolves [[Join]] operator by resolving its left and right children and its
join condition. If
* the unresolved join is [[NaturalJoin]] or [[UsingJoin]], the resulting
operator will be
- * [[Project]], otherwise it will be [[Join]].
+ * [[Project]], otherwise it will be [[Join]], or a [[Project]] over it that
keeps the join's output
+ * (see `retainTopLevelJoinOutput`).
*/
class JoinResolver(
override val resolver: Resolver,
@@ -245,6 +246,8 @@ class JoinResolver(
* - Resolve join condition.
* - Overwrite [[NameScope.output]] with join's output and overwrite
[[NameScope.hiddenOutput]]
* using the result of `computeHiddenOutputForRegularJoin` call.
+ * - Wrap a top-level [[Join]] in a [[Project]] with its output if
necessary (see
+ * `retainTopLevelJoinOutput`).
* - Wrap the [[Join]] in [[WithCTE]] if necessary.
*/
private def handleRegularJoin(
@@ -259,6 +262,12 @@ class JoinResolver(
rightNameScope = rightNameScope
)
+ val conditionReferencedAttributes = if
(partiallyResolvedJoin.condition.isDefined) {
+ expressionResolver.getLastReferencedAttributes
+ } else {
+ new HashMap[ExprId, Attribute]
+ }
+
val newOutput = Join.computeOutput(
joinType = partiallyResolvedJoin.joinType,
leftOutput = leftNameScope.output,
@@ -275,7 +284,11 @@ class JoinResolver(
hiddenOutput = Some(newHiddenOutput)
)
- val resolvedJoin = partiallyResolvedJoin.copy(condition =
resolvedCondition)
+ val resolvedJoin = retainTopLevelJoinOutput(
+ unresolvedJoin = unresolvedJoin,
+ resolvedJoin = partiallyResolvedJoin.copy(condition = resolvedCondition),
+ conditionReferencedAttributes = conditionReferencedAttributes
+ )
cteRegistry.currentScope.tryPutWithCTE(
unresolvedOperator = unresolvedJoin,
@@ -283,6 +296,43 @@ class JoinResolver(
)
}
+ /**
+ * Wraps a top-level [[Join]] in a [[Project]] with the join's output when
its condition
+ * references a column found only in the hidden output, such as the hidden
key of a USING join
+ * below, which is added to a child's output, and the column is in the
join's own output. A
+ * LEFT SEMI or LEFT ANTI join outputs only its left side, so a hidden key
of its right side does
+ * not change its output, and fixed-point does not wrap it either. A join
that is not top-level
+ * is wrapped by the resolver of the operator above it instead (see
+ * [[RetainsOriginalJoinOutput]]). For example:
+ *
+ * {{{
+ * FROM nt1 JOIN nt2 USING (k) JOIN nt2 x ON nt2.k = x.k
+ *
+ * Project [k#1, v1#2, v2#3, k#5, v2#6]
+ * +- Join Inner, (k#4 = k#5)
+ * :- Project [k#1, v1#2, v2#3, k#4]
+ * : +- Join Inner, (k#1 = k#4)
+ * +- ...
+ * }}}
+ */
+ private def retainTopLevelJoinOutput(
+ unresolvedJoin: Join,
+ resolvedJoin: Join,
+ conditionReferencedAttributes: HashMap[ExprId, Attribute]): LogicalPlan
= {
+ def referencesHiddenOutput: Boolean = {
+ val joinOutput = resolvedJoin.outputSet
+ scopes.current
+ .resolveMissingAttributesByHiddenOutput(conditionReferencedAttributes)
+ .exists(joinOutput.contains)
+ }
+ if (unresolvedJoin.containsTag(ResolverTag.TOP_LEVEL_OPERATOR) &&
referencesHiddenOutput) {
+ operatorResolutionContextStack.current.baseOperator = Some(resolvedJoin)
+ Project(scopes.current.output, resolvedJoin)
+ } else {
+ resolvedJoin
+ }
+ }
+
/**
* Compute new hidden output for the regular [[Join]]s. This new output
contains attributes from
* the main output (computed using the [[Join.computeOutput]]) the qualified
access only
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/RetainsOriginalJoinOutput.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/RetainsOriginalJoinOutput.scala
index 11ac01b0a08a..8244209429bd 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/RetainsOriginalJoinOutput.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/resolver/RetainsOriginalJoinOutput.scala
@@ -32,7 +32,9 @@ trait RetainsOriginalJoinOutput {
* changed when metadata columns are added to [[Project]] nodes below the
[[Join]]. This is
* necessary in order to stay compatible with fixed-point analyzer. Instead
of doing this in
* [[JoinResolver]] we must do this here, because while resolving [[Join]]
we still don't know if
- * we should add a [[Project]] or not. For example consider the following
query:
+ * we should add a [[Project]] or not. A top-level [[Join]] has no operator
above it, so
+ * [[JoinResolver]] wraps it itself (see
`JoinResolver.retainTopLevelJoinOutput`). For example
+ * consider the following query:
*
* {{{
* -- tables: nt1(k, v1), nt2(k, v2), nt3(k, v3)
diff --git
a/sql/core/src/test/resources/sql-tests/analyzer-results/using-join.sql.out
b/sql/core/src/test/resources/sql-tests/analyzer-results/using-join.sql.out
index 25340dabe275..6edde46f265e 100644
--- a/sql/core/src/test/resources/sql-tests/analyzer-results/using-join.sql.out
+++ b/sql/core/src/test/resources/sql-tests/analyzer-results/using-join.sql.out
@@ -911,3 +911,162 @@ Project [k#x, k#x]
+- Project [k#x, v2#x]
+- SubqueryAlias nt2
+- LocalRelation [k#x, v2#x]
+
+
+-- !query
+FROM nt1 JOIN nt2 USING (k) JOIN nt2 x ON nt2.k = x.k
+-- !query analysis
+Project [k#x, v1#x, v2#x, k#x, v2#x]
++- Join Inner, (k#x = k#x)
+ :- Project [k#x, v1#x, v2#x, k#x]
+ : +- Join Inner, (k#x = k#x)
+ : :- SubqueryAlias nt1
+ : : +- View (`nt1`, [k#x, v1#x])
+ : : +- Project [cast(k#x as string) AS k#x, cast(v1#x as int) AS
v1#x]
+ : : +- Project [k#x, v1#x]
+ : : +- SubqueryAlias nt1
+ : : +- LocalRelation [k#x, v1#x]
+ : +- SubqueryAlias nt2
+ : +- View (`nt2`, [k#x, v2#x])
+ : +- Project [cast(k#x as string) AS k#x, cast(v2#x as int) AS
v2#x]
+ : +- Project [k#x, v2#x]
+ : +- SubqueryAlias nt2
+ : +- LocalRelation [k#x, v2#x]
+ +- SubqueryAlias x
+ +- SubqueryAlias nt2
+ +- View (`nt2`, [k#x, v2#x])
+ +- Project [cast(k#x as string) AS k#x, cast(v2#x as int) AS v2#x]
+ +- Project [k#x, v2#x]
+ +- SubqueryAlias nt2
+ +- LocalRelation [k#x, v2#x]
+
+
+-- !query
+FROM nt1 JOIN nt2 USING (k) JOIN nt2 x
+ ON nt2.k = x.k AND x.v2 = (SELECT max(y.v2) FROM nt2 y WHERE y.k = x.k)
+-- !query analysis
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" :
"UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.UNSUPPORTED_CORRELATED_SCALAR_SUBQUERY",
+ "sqlState" : "0A000",
+ "messageParameters" : {
+ "treeNode" : "Join Inner, ((k#x = k#x) AND (v2#x = scalar-subquery#x
[k#x]))\n: +- Aggregate [max(v2#x) AS max(v2)#x]\n: +- Filter (k#x =
outer(k#x))\n: +- SubqueryAlias y\n: +- SubqueryAlias nt2\n:
+- View (`nt2`, [k#x, v2#x])\n: +- Project [cast(k#x
as string) AS k#x, cast(v2#x as int) AS v2#x]\n: +- Project
[k#x, v2#x]\n: +- SubqueryAlias nt2\n:
+- LocalRelation [k# [...]
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 29,
+ "stopIndex" : 112,
+ "fragment" : "JOIN nt2 x\n ON nt2.k = x.k AND x.v2 = (SELECT max(y.v2)
FROM nt2 y WHERE y.k = x.k)"
+ } ]
+}
+
+
+-- !query
+FROM nt2 x LEFT SEMI JOIN (nt1 JOIN nt2 USING (k)) ON x.k = nt2.k
+-- !query analysis
+Join LeftSemi, (k#x = k#x)
+:- SubqueryAlias x
+: +- SubqueryAlias nt2
+: +- View (`nt2`, [k#x, v2#x])
+: +- Project [cast(k#x as string) AS k#x, cast(v2#x as int) AS v2#x]
+: +- Project [k#x, v2#x]
+: +- SubqueryAlias nt2
+: +- LocalRelation [k#x, v2#x]
++- Project [k#x, v1#x, v2#x, k#x]
+ +- Join Inner, (k#x = k#x)
+ :- SubqueryAlias nt1
+ : +- View (`nt1`, [k#x, v1#x])
+ : +- Project [cast(k#x as string) AS k#x, cast(v1#x as int) AS v1#x]
+ : +- Project [k#x, v1#x]
+ : +- SubqueryAlias nt1
+ : +- LocalRelation [k#x, v1#x]
+ +- SubqueryAlias nt2
+ +- View (`nt2`, [k#x, v2#x])
+ +- Project [cast(k#x as string) AS k#x, cast(v2#x as int) AS v2#x]
+ +- Project [k#x, v2#x]
+ +- SubqueryAlias nt2
+ +- LocalRelation [k#x, v2#x]
+
+
+-- !query
+FROM nt2 x LEFT ANTI JOIN (nt1 JOIN nt2 USING (k)) ON x.k = nt2.k
+-- !query analysis
+Join LeftAnti, (k#x = k#x)
+:- SubqueryAlias x
+: +- SubqueryAlias nt2
+: +- View (`nt2`, [k#x, v2#x])
+: +- Project [cast(k#x as string) AS k#x, cast(v2#x as int) AS v2#x]
+: +- Project [k#x, v2#x]
+: +- SubqueryAlias nt2
+: +- LocalRelation [k#x, v2#x]
++- Project [k#x, v1#x, v2#x, k#x]
+ +- Join Inner, (k#x = k#x)
+ :- SubqueryAlias nt1
+ : +- View (`nt1`, [k#x, v1#x])
+ : +- Project [cast(k#x as string) AS k#x, cast(v1#x as int) AS v1#x]
+ : +- Project [k#x, v1#x]
+ : +- SubqueryAlias nt1
+ : +- LocalRelation [k#x, v1#x]
+ +- SubqueryAlias nt2
+ +- View (`nt2`, [k#x, v2#x])
+ +- Project [cast(k#x as string) AS k#x, cast(v2#x as int) AS v2#x]
+ +- Project [k#x, v2#x]
+ +- SubqueryAlias nt2
+ +- LocalRelation [k#x, v2#x]
+
+
+-- !query
+FROM nt1 JOIN nt2 USING (k) LEFT SEMI JOIN nt2 x ON nt2.k = x.k
+-- !query analysis
+Project [k#x, v1#x, v2#x]
++- Join LeftSemi, (k#x = k#x)
+ :- Project [k#x, v1#x, v2#x, k#x]
+ : +- Join Inner, (k#x = k#x)
+ : :- SubqueryAlias nt1
+ : : +- View (`nt1`, [k#x, v1#x])
+ : : +- Project [cast(k#x as string) AS k#x, cast(v1#x as int) AS
v1#x]
+ : : +- Project [k#x, v1#x]
+ : : +- SubqueryAlias nt1
+ : : +- LocalRelation [k#x, v1#x]
+ : +- SubqueryAlias nt2
+ : +- View (`nt2`, [k#x, v2#x])
+ : +- Project [cast(k#x as string) AS k#x, cast(v2#x as int) AS
v2#x]
+ : +- Project [k#x, v2#x]
+ : +- SubqueryAlias nt2
+ : +- LocalRelation [k#x, v2#x]
+ +- SubqueryAlias x
+ +- SubqueryAlias nt2
+ +- View (`nt2`, [k#x, v2#x])
+ +- Project [cast(k#x as string) AS k#x, cast(v2#x as int) AS v2#x]
+ +- Project [k#x, v2#x]
+ +- SubqueryAlias nt2
+ +- LocalRelation [k#x, v2#x]
+
+
+-- !query
+FROM nt2 x LEFT JOIN (nt1 JOIN nt2 USING (k)) ON x.k = nt2.k
+-- !query analysis
+Project [k#x, v2#x, k#x, v1#x, v2#x]
++- Join LeftOuter, (k#x = k#x)
+ :- SubqueryAlias x
+ : +- SubqueryAlias nt2
+ : +- View (`nt2`, [k#x, v2#x])
+ : +- Project [cast(k#x as string) AS k#x, cast(v2#x as int) AS v2#x]
+ : +- Project [k#x, v2#x]
+ : +- SubqueryAlias nt2
+ : +- LocalRelation [k#x, v2#x]
+ +- Project [k#x, v1#x, v2#x, k#x]
+ +- Join Inner, (k#x = k#x)
+ :- SubqueryAlias nt1
+ : +- View (`nt1`, [k#x, v1#x])
+ : +- Project [cast(k#x as string) AS k#x, cast(v1#x as int) AS
v1#x]
+ : +- Project [k#x, v1#x]
+ : +- SubqueryAlias nt1
+ : +- LocalRelation [k#x, v1#x]
+ +- SubqueryAlias nt2
+ +- View (`nt2`, [k#x, v2#x])
+ +- Project [cast(k#x as string) AS k#x, cast(v2#x as int) AS
v2#x]
+ +- Project [k#x, v2#x]
+ +- SubqueryAlias nt2
+ +- LocalRelation [k#x, v2#x]
diff --git a/sql/core/src/test/resources/sql-tests/inputs/using-join.sql
b/sql/core/src/test/resources/sql-tests/inputs/using-join.sql
index 8a4d24021da5..58a521c3d16b 100644
--- a/sql/core/src/test/resources/sql-tests/inputs/using-join.sql
+++ b/sql/core/src/test/resources/sql-tests/inputs/using-join.sql
@@ -105,3 +105,21 @@ SELECT nt2.* FROM (nt1 left outer join nt2 using (k)) join
nt2 x on x.v2 = 1;
-- Same, with the hidden nt2.k in the join condition
SELECT nt2.k, x.k FROM (nt1 left outer join nt2 using (k)) join nt2 x on nt2.k
= x.k;
+
+-- A top-level join without a SELECT, with the hidden nt2.k in the join
condition
+FROM nt1 JOIN nt2 USING (k) JOIN nt2 x ON nt2.k = x.k;
+
+-- The same join with a correlated scalar subquery in the condition, which a
Join cannot host
+FROM nt1 JOIN nt2 USING (k) JOIN nt2 x
+ ON nt2.k = x.k AND x.v2 = (SELECT max(y.v2) FROM nt2 y WHERE y.k = x.k);
+
+-- A top-level LEFT SEMI or LEFT ANTI join outputs only its left side, so a
hidden key of its right
+-- side in the condition does not change its output
+FROM nt2 x LEFT SEMI JOIN (nt1 JOIN nt2 USING (k)) ON x.k = nt2.k;
+
+FROM nt2 x LEFT ANTI JOIN (nt1 JOIN nt2 USING (k)) ON x.k = nt2.k;
+
+-- A hidden key of the left side of a LEFT SEMI join, and of the right side of
a LEFT OUTER join
+FROM nt1 JOIN nt2 USING (k) LEFT SEMI JOIN nt2 x ON nt2.k = x.k;
+
+FROM nt2 x LEFT JOIN (nt1 JOIN nt2 USING (k)) ON x.k = nt2.k;
diff --git a/sql/core/src/test/resources/sql-tests/results/using-join.sql.out
b/sql/core/src/test/resources/sql-tests/results/using-join.sql.out
index ff1a5815c0b1..3a4f479c9818 100644
--- a/sql/core/src/test/resources/sql-tests/results/using-join.sql.out
+++ b/sql/core/src/test/resources/sql-tests/results/using-join.sql.out
@@ -479,3 +479,79 @@ one one
one one
one one
two two
+
+
+-- !query
+FROM nt1 JOIN nt2 USING (k) JOIN nt2 x ON nt2.k = x.k
+-- !query schema
+struct<k:string,v1:int,v2:int,k:string,v2:int>
+-- !query output
+one 1 1 one 1
+one 1 1 one 5
+one 1 5 one 1
+one 1 5 one 5
+two 2 22 two 22
+
+
+-- !query
+FROM nt1 JOIN nt2 USING (k) JOIN nt2 x
+ ON nt2.k = x.k AND x.v2 = (SELECT max(y.v2) FROM nt2 y WHERE y.k = x.k)
+-- !query schema
+struct<>
+-- !query output
+org.apache.spark.sql.catalyst.ExtendedAnalysisException
+{
+ "errorClass" :
"UNSUPPORTED_SUBQUERY_EXPRESSION_CATEGORY.UNSUPPORTED_CORRELATED_SCALAR_SUBQUERY",
+ "sqlState" : "0A000",
+ "messageParameters" : {
+ "treeNode" : "Join Inner, ((k#x = k#x) AND (v2#x = scalar-subquery#x
[k#x]))\n: +- Aggregate [max(v2#x) AS max(v2)#x]\n: +- Filter (k#x =
outer(k#x))\n: +- SubqueryAlias y\n: +- SubqueryAlias nt2\n:
+- View (`nt2`, [k#x, v2#x])\n: +- Project [cast(k#x
as string) AS k#x, cast(v2#x as int) AS v2#x]\n: +- Project
[k#x, v2#x]\n: +- SubqueryAlias nt2\n:
+- LocalRelation [k# [...]
+ },
+ "queryContext" : [ {
+ "objectType" : "",
+ "objectName" : "",
+ "startIndex" : 29,
+ "stopIndex" : 112,
+ "fragment" : "JOIN nt2 x\n ON nt2.k = x.k AND x.v2 = (SELECT max(y.v2)
FROM nt2 y WHERE y.k = x.k)"
+ } ]
+}
+
+
+-- !query
+FROM nt2 x LEFT SEMI JOIN (nt1 JOIN nt2 USING (k)) ON x.k = nt2.k
+-- !query schema
+struct<k:string,v2:int>
+-- !query output
+one 1
+one 5
+two 22
+
+
+-- !query
+FROM nt2 x LEFT ANTI JOIN (nt1 JOIN nt2 USING (k)) ON x.k = nt2.k
+-- !query schema
+struct<k:string,v2:int>
+-- !query output
+four 4
+
+
+-- !query
+FROM nt1 JOIN nt2 USING (k) LEFT SEMI JOIN nt2 x ON nt2.k = x.k
+-- !query schema
+struct<k:string,v1:int,v2:int>
+-- !query output
+one 1 1
+one 1 5
+two 2 22
+
+
+-- !query
+FROM nt2 x LEFT JOIN (nt1 JOIN nt2 USING (k)) ON x.k = nt2.k
+-- !query schema
+struct<k:string,v2:int,k:string,v1:int,v2:int>
+-- !query output
+four 4 NULL NULL NULL
+one 1 one 1 1
+one 1 one 1 5
+one 5 one 1 1
+one 5 one 1 5
+two 22 two 2 22
diff --git
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
index 401b3c126d85..5fc9412ed62b 100644
---
a/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
+++
b/sql/hive-thriftserver/src/test/scala/org/apache/spark/sql/hive/thriftserver/ThriftServerQueryTestSuite.scala
@@ -345,6 +345,9 @@ class ThriftServerQueryTestSuite extends SQLQueryTestSuite
with SharedThriftServ
upperCase.startsWith("SELECT ") || upperCase.startsWith("SELECT\n") ||
upperCase.startsWith("WITH ") || upperCase.startsWith("WITH\n") ||
upperCase.startsWith("VALUES ") || upperCase.startsWith("VALUES\n") ||
+ // A query may start with its FROM clause (e.g. `FROM t1 JOIN t2 ...` in
using-join.sql);
+ // its row order is as unspecified as a SELECT's.
+ upperCase.startsWith("FROM ") || upperCase.startsWith("FROM\n") ||
// postgreSQL/union.sql
upperCase.startsWith("(")
}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
