This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/master by this push:
new f8c6dd8f0265 chore(spark): prune dead code in the six extended SQL AST
builders (#19455)
f8c6dd8f0265 is described below
commit f8c6dd8f02655856c7b0317615eb8a03f3ea8ade
Author: voonhous <[email protected]>
AuthorDate: Sun Aug 2 18:40:27 2026 +0800
chore(spark): prune dead code in the six extended SQL AST builders (#19455)
* chore(spark): prune dead code in the six extended SQL AST builders
#19132 pruned the forked parser to Hudi-routed statements; remove the code
that became unreachable in the six HoodieSpark*ExtendedSqlAstBuilders
(3.3/3.4/3.5/4.0/4.1/4.2):
- uncalled methods: createSchema, createStructType and both
visitPropertyKeys overloads (scaladoc references updated)
- visitBooleanLiteral: unreachable from SQL in both keyword modes.
TRUE/FALSE are ansiNonReserved, so a bare true/false in
transform-argument position always parses as a column reference, and
OPTIONS/TBLPROPERTIES boolean values go through getText without
visiting the constant. visitNullLiteral stays: under ANSI keyword mode
f(null, id) parses as a null literal.
- branches the grammar cannot produce: getSingleFieldReference's
empty-arguments arm (the transform rule requires at least one
argument), visitTransformArgument's getOrElse(throw ...) (a
transformArgument is always qualifiedName or constant),
visitCreateFileFormat's trailing case _ (STORED AS and STORED BY are
mutually exclusive grammar alternatives)
- unused imports, including ParserUtils.entry which is shadowed by an
identical local def in visitRowFormatDelimited
Deliberately kept, deviating from the issue: the case _ arm of
validateRowFormatFileFormat. It is reachable, e.g. via CREATE TABLE t
(b BLOB) ROW FORMAT SERDE 'x' STORED BY 'y': rowFormat and
createFileFormat coexist in createTableClauses, STORED BY leaves
createFileFormatCtx.fileFormat null, and (rowFormatCtx, null) falls past
the three typed arms. Removing it would turn a clean ParseException into
a scala.MatchError.
Fixes #19451
* fix(spark): restore visitBooleanLiteral and address review findings on
the pruning
Review of the pruning commit found #19451's premise wrong for one deletion
and surfaced several smaller cleanups:
- Restore visitBooleanLiteral in all six builders. The issue claimed
TRUE/FALSE are both ansiNonReserved; only TRUE is. Under ANSI keyword
mode (SQL_standard_keyword_behavior = conf.ansiEnabled, the Spark 4.x
default) a bare false in transform-argument position parses as a
BooleanLiteralContext, and without the visitor the generated base
visitor returns null and LiteralValue(lit.value, ...) throws a raw NPE
that escapes parsePlan (only ParseException/AnalysisException are
caught). The scaladoc now records the reachability constraint, the
wrong comment in TestBlobDataType is corrected, and an ANSI-mode
regression test asserts false yields a BooleanType literal while true
stays a column reference.
- Revert visitTransformArgument's .get to getOrElse(throw ParseException)
so a hypothetical future grammar change fails as a parse error instead
of a NoSuchElementException.
- Replace the stale 'should never happen' comment on
validateRowFormatFileFormat's case _ arm with the real reachability
explanation (ROW FORMAT ... STORED BY), and cover that arm plus the
(_, TableFileFormatContext) arm with tests.
- Align the visitCreateTable scaladoc drift #19132 left behind: the
CreateTableAsSelect and AS-select_statement lines in 3.3/3.5 and the
AS-select_statement lines in 4.1.
- Drop the BlobType import that the types._ wildcard already supplies
(3.3/3.4/3.5/4.0), matching 4.1/4.2.
- Fix a 6-vs-4-space indent drift in the 3.3 copy.
Verified: reactor compiles for spark3.3/3.4/3.5, CI-style scala-2.13
builds for spark4.0/4.1/4.2, and the TestBlobDataType suite (10/10).
---
.../sql/hudi/dml/schema/TestBlobDataType.scala | 39 ++++++++--
.../HoodieSpark3_3ExtendedSqlAstBuilder.scala | 78 +++++---------------
.../HoodieSpark3_4ExtendedSqlAstBuilder.scala | 75 +++++---------------
.../HoodieSpark3_5ExtendedSqlAstBuilder.scala | 82 +++++-----------------
.../HoodieSpark4_0ExtendedSqlAstBuilder.scala | 77 +++++---------------
.../HoodieSpark4_1ExtendedSqlAstBuilder.scala | 79 +++++----------------
.../HoodieSpark4_2ExtendedSqlAstBuilder.scala | 76 +++++---------------
7 files changed, 138 insertions(+), 368 deletions(-)
diff --git
a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestBlobDataType.scala
b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestBlobDataType.scala
index 4eae30286ba9..42d7bfcfba68 100644
---
a/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestBlobDataType.scala
+++
b/hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/spark/sql/hudi/dml/schema/TestBlobDataType.scala
@@ -454,11 +454,10 @@ class TestBlobDataType extends HoodieSparkSqlTestBase
with ExtendedParserTestHel
// to #19449: the extended parser enables ANSI reserved-keyword
enforcement whenever
// spark.sql.ansi.enabled is set (the Spark 4.x default) instead of
following Spark's
// spark.sql.ansi.enforceReservedKeywords, which makes a bare TIMESTAMP
token unparseable
- // there (a bug, not a Spark 4 constraint). A bare true/false in this
position parses as a
- // column reference in BOTH keyword modes (TRUE/FALSE are ansiNonReserved,
so qualifiedName
- // wins), leaving visitBooleanLiteral unreachable (#19451); null is a
column reference only
- // under the default non-ANSI keyword mode and a null literal under ANSI,
so visitNullLiteral
- // is mode-dependent and both are left unasserted here.
+ // there (a bug, not a Spark 4 constraint). Boolean and null literals are
keyword-mode
+ // dependent: only TRUE is ansiNonReserved, so a bare true always parses
as a column
+ // reference, while false and null are column references under the default
non-ANSI keyword
+ // mode but typed literals under ANSI mode (both cases are asserted below).
val plan = parseCreateTable(
s"""
|CREATE TABLE blob_lit_tbl (
@@ -485,6 +484,21 @@ class TestBlobDataType extends HoodieSparkSqlTestBase with
ExtendedParserTestHel
firstLiteralArg(transformByName(plan, "mu_ivl_t")).dataType)
assertResult(YearMonthIntervalType(YearMonthIntervalType.YEAR,
YearMonthIntervalType.MONTH))(
firstLiteralArg(transformByName(plan, "uu_ivl_t")).dataType)
+
+ // Under ANSI keyword mode a bare false and a bare null must survive
visitBooleanLiteral
+ // and visitNullLiteral as typed literals, while a bare true stays a
column reference
+ // (TRUE is ansiNonReserved; FALSE and NULL are not).
+ withSQLConf("spark.sql.ansi.enabled" -> "true") {
+ val ansiPlan = parseCreateTable(
+ "CREATE TABLE blob_bool_tbl (id BIGINT, data BLOB) USING hudi " +
+ "PARTITIONED BY (bool_t(false, id), true_t(true, id), null_t(null,
id))")
+ assertResult(BooleanType)(firstLiteralArg(transformByName(ansiPlan,
"bool_t")).dataType)
+ assertResult(false)(firstLiteralArg(transformByName(ansiPlan,
"bool_t")).value)
+ assertResult(NullType)(firstLiteralArg(transformByName(ansiPlan,
"null_t")).dataType)
+ assertResult(null)(firstLiteralArg(transformByName(ansiPlan,
"null_t")).value)
+ assertResult(Seq(Seq("true"), Seq("id")))(
+ transformFieldRefs(transformByName(ansiPlan, "true_t")))
+ }
}
test("Test parse CREATE TABLE with BLOB column and invalid partition
transforms") {
@@ -539,6 +553,14 @@ class TestBlobDataType extends HoodieSparkSqlTestBase with
ExtendedParserTestHel
"ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE")
assertResult(Some("TEXTFILE"))(ff6.tableSpec.serde.get.storedAs)
assertResult(",")(ff6.tableSpec.serde.get.serdeProperties("field.delim"))
+ // Any ROW FORMAT combined with STORED AS INPUTFORMAT/OUTPUTFORMAT is
accepted (the
+ // table-file-format arm of validateRowFormatFileFormat).
+ val ff7 = parseCreateTable("CREATE TABLE blob_ff7 (id BIGINT, data BLOB) "
+
+ "ROW FORMAT SERDE 'com.example.Serde' " +
+ "STORED AS INPUTFORMAT 'com.example.InFmt' OUTPUTFORMAT
'com.example.OutFmt'")
+ assertResult(Some("com.example.Serde"))(ff7.tableSpec.serde.get.serde)
+ assertResult(Some(FormatClasses("com.example.InFmt",
"com.example.OutFmt")))(
+ ff7.tableSpec.serde.get.formatClasses)
// ROW FORMAT DELIMITED with a non-text file format is rejected.
checkExceptionContain(
@@ -556,6 +578,13 @@ class TestBlobDataType extends HoodieSparkSqlTestBase with
ExtendedParserTestHel
checkExceptionContain(
"CREATE TABLE blob_ferr3 (id BIGINT, data BLOB) STORED BY
'com.example.Handler'")(
"Operation not allowed: STORED BY")
+ // ROW FORMAT combined with STORED BY leaves no file format for the row
format to pair with;
+ // validateRowFormatFileFormat's catch-all arm rejects the combination
before the STORED BY
+ // error can fire.
+ checkExceptionContain(
+ "CREATE TABLE blob_ferr5 (id BIGINT, data BLOB) " +
+ "ROW FORMAT SERDE 'com.example.Serde' STORED BY
'com.example.Handler'")(
+ "Unexpected combination of")
// A USING provider combined with a serde clause is not allowed.
checkExceptionContain(
"CREATE TABLE blob_ferr4 (id BIGINT, data BLOB) USING hudi STORED AS
PARQUET")(
diff --git
a/hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_3ExtendedSqlAstBuilder.scala
b/hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_3ExtendedSqlAstBuilder.scala
index 935bb438151e..06a51d02bf24 100644
---
a/hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_3ExtendedSqlAstBuilder.scala
+++
b/hudi-spark-datasource/hudi-spark3.3.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_3ExtendedSqlAstBuilder.scala
@@ -21,26 +21,24 @@ import
org.apache.hudi.spark.sql.parser.{HoodieSqlBaseBaseVisitor, HoodieSqlBase
import org.apache.hudi.spark.sql.parser.HoodieSqlBaseParser._
import org.antlr.v4.runtime.{ParserRuleContext, Token}
-import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
+import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis._
-import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat}
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.aggregate.{First, Last}
import org.apache.spark.sql.catalyst.parser.{ParseException, ParserInterface}
-import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, entry, escapedIdentifier, operationNotAllowed, source,
string, stringWithoutUnescape, validate, withOrigin, EnhancedLogicalPlan}
+import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, operationNotAllowed, source, string, stringWithoutUnescape,
validate, withOrigin}
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.util.{truncatedString, CharVarcharUtils,
DateTimeUtils, IntervalUtils}
+import org.apache.spark.sql.catalyst.util.{DateTimeUtils, IntervalUtils}
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.BucketSpecHelper
import org.apache.spark.sql.connector.catalog.TableCatalog
import org.apache.spark.sql.connector.expressions.{ApplyTransform,
BucketTransform, DaysTransform, Expression => V2Expression, FieldReference,
HoursTransform, IdentityTransform, LiteralValue, MonthsTransform, Transform,
YearsTransform}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.BlobType
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
import org.apache.spark.util.Utils.isTesting
@@ -208,14 +206,18 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
/**
- * Create a NULL literal expression.
+ * Create a NULL literal expression. Reachable under ANSI keyword mode,
where NULL (like
+ * FALSE) is a reserved word, so f(null, id) in transform-argument position
parses as a
+ * null literal instead of a column reference.
*/
override def visitNullLiteral(ctx: NullLiteralContext): Literal =
withOrigin(ctx) {
Literal(null)
}
/**
- * Create a Boolean literal expression.
+ * Create a Boolean literal expression. Reachable under ANSI keyword mode,
where FALSE is a
+ * reserved word (only TRUE is ansiNonReserved), so a bare false in
transform-argument
+ * position parses as a boolean literal instead of a column reference.
*/
override def visitBooleanLiteral(ctx: BooleanLiteralContext): Literal =
withOrigin(ctx) {
if (ctx.getText.toBoolean) {
@@ -642,13 +644,6 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create top level table schema.
- */
- protected def createSchema(ctx: ColTypeListContext): StructType = {
- StructType(Option(ctx).toSeq.flatMap(visitColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -691,13 +686,6 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create a [[StructType]] from a sequence of [[StructField]]s.
- */
- protected def createStructType(ctx: ComplexColTypeListContext): StructType =
{
- StructType(Option(ctx).toSeq.flatMap(visitComplexColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -776,7 +764,7 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a table property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitTablePropertyList(
ctx: TablePropertyListContext):
Map[String, String] = withOrigin(ctx) {
@@ -803,19 +791,6 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[TablePropertyListContext]], assuming no
values are specified.
- */
- def visitPropertyKeys(ctx: TablePropertyListContext): Seq[String] = {
- val props = visitTablePropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A table property key can either be String or a collection of dot
separated elements. This
* function extracts the property key based on whether its a string literal
or a table property
@@ -925,10 +900,6 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
lazy val name: String = ctx.identifier.getText
if (arguments.size > 1) {
throw new ParseException(s"Too many arguments for transform $name",
ctx)
- } else if (arguments.isEmpty) {
- throw
-
- new ParseException(s"Not enough arguments for transform $name", ctx)
} else {
getFieldReference(ctx, arguments.head)
}
@@ -1044,8 +1015,6 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
SerdeInfo(storedAs = Some(c.identifier.getText))
case (null, storageHandler) =>
operationNotAllowed("STORED BY", ctx)
- case _ =>
- throw new ParseException("Expected either STORED AS or STORED BY, not
both", ctx)
}
}
@@ -1144,7 +1113,8 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
s"ROW FORMAT DELIMITED is only compatible with 'textfile', not
'$fmt'", parentCtx)
}
case _ =>
- // should never happen
+ // Reachable: ROW FORMAT ... STORED BY 'handler' leaves
createFileFormatCtx.fileFormat
+ // null, so none of the typed arms above match.
def str(ctx: ParserRuleContext): String = {
(0 until ctx.getChildCount).map { i => ctx.getChild(i).getText
}.mkString(" ")
}
@@ -1230,14 +1200,13 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf:
SQLConf, delegate: ParserInterfa
}
/**
- * Create a table, returning a [[CreateTable]] or [[CreateTableAsSelect]]
logical plan.
+ * Create a table, returning a [[CreateTable]] logical plan.
*
* Expected format:
* {{{
* CREATE [TEMPORARY] TABLE [IF NOT EXISTS] [db_name.]table_name
* [USING table_provider]
- * create_table_clauses
- * [[AS] select_statement];
+ * create_table_clauses;
*
* create_table_clauses (order insensitive):
* [PARTITIONED BY (partition_fields)]
@@ -1360,7 +1329,7 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitPropertyList(ctx: PropertyListContext): Map[String,
String] = withOrigin(ctx) {
val properties = ctx.property.asScala.map { property =>
@@ -1386,19 +1355,6 @@ class HoodieSpark3_3ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[PropertyListContext]], assuming no values
are specified.
- */
- def visitPropertyKeys(ctx: PropertyListContext): Seq[String] = {
- val props = visitPropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A property key can either be String or a collection of dot separated
elements. This
* function extracts the property key based on whether its a string literal
or a property
diff --git
a/hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_4ExtendedSqlAstBuilder.scala
b/hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_4ExtendedSqlAstBuilder.scala
index cc49e8983d1f..695cf3fe1462 100644
---
a/hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_4ExtendedSqlAstBuilder.scala
+++
b/hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_4ExtendedSqlAstBuilder.scala
@@ -21,26 +21,24 @@ import
org.apache.hudi.spark.sql.parser.{HoodieSqlBaseBaseVisitor, HoodieSqlBase
import org.apache.hudi.spark.sql.parser.HoodieSqlBaseParser._
import org.antlr.v4.runtime.{ParserRuleContext, Token}
-import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
+import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis._
-import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat}
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.aggregate.{First, Last}
import org.apache.spark.sql.catalyst.parser.{ParseException, ParserInterface}
-import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, entry, escapedIdentifier, operationNotAllowed, source,
string, stringWithoutUnescape, validate, withOrigin, EnhancedLogicalPlan}
+import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, operationNotAllowed, source, string, stringWithoutUnescape,
validate, withOrigin}
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.util.{truncatedString, CharVarcharUtils,
DateTimeUtils, IntervalUtils}
+import org.apache.spark.sql.catalyst.util.{DateTimeUtils, IntervalUtils}
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.BucketSpecHelper
import org.apache.spark.sql.connector.catalog.TableCatalog
import org.apache.spark.sql.connector.expressions.{ApplyTransform,
BucketTransform, DaysTransform, Expression => V2Expression, FieldReference,
HoursTransform, IdentityTransform, LiteralValue, MonthsTransform, Transform,
YearsTransform}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.BlobType
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
import org.apache.spark.util.Utils.isTesting
@@ -208,14 +206,18 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
/**
- * Create a NULL literal expression.
+ * Create a NULL literal expression. Reachable under ANSI keyword mode,
where NULL (like
+ * FALSE) is a reserved word, so f(null, id) in transform-argument position
parses as a
+ * null literal instead of a column reference.
*/
override def visitNullLiteral(ctx: NullLiteralContext): Literal =
withOrigin(ctx) {
Literal(null)
}
/**
- * Create a Boolean literal expression.
+ * Create a Boolean literal expression. Reachable under ANSI keyword mode,
where FALSE is a
+ * reserved word (only TRUE is ansiNonReserved), so a bare false in
transform-argument
+ * position parses as a boolean literal instead of a column reference.
*/
override def visitBooleanLiteral(ctx: BooleanLiteralContext): Literal =
withOrigin(ctx) {
if (ctx.getText.toBoolean) {
@@ -642,13 +644,6 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create top level table schema.
- */
- protected def createSchema(ctx: ColTypeListContext): StructType = {
- StructType(Option(ctx).toSeq.flatMap(visitColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -691,13 +686,6 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create a [[StructType]] from a sequence of [[StructField]]s.
- */
- protected def createStructType(ctx: ComplexColTypeListContext): StructType =
{
- StructType(Option(ctx).toSeq.flatMap(visitComplexColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -776,7 +764,7 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a table property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitTablePropertyList(
ctx: TablePropertyListContext):
Map[String, String] = withOrigin(ctx) {
@@ -803,19 +791,6 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[TablePropertyListContext]], assuming no
values are specified.
- */
- def visitPropertyKeys(ctx: TablePropertyListContext): Seq[String] = {
- val props = visitTablePropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A table property key can either be String or a collection of dot
separated elements. This
* function extracts the property key based on whether its a string literal
or a table property
@@ -925,10 +900,6 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
lazy val name: String = ctx.identifier.getText
if (arguments.size > 1) {
throw new ParseException(s"Too many arguments for transform $name",
ctx)
- } else if (arguments.isEmpty) {
- throw
-
- new ParseException(s"Not enough arguments for transform $name", ctx)
} else {
getFieldReference(ctx, arguments.head)
}
@@ -1044,8 +1015,6 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
SerdeInfo(storedAs = Some(c.identifier.getText))
case (null, storageHandler) =>
operationNotAllowed("STORED BY", ctx)
- case _ =>
- throw new ParseException("Expected either STORED AS or STORED BY, not
both", ctx)
}
}
@@ -1144,7 +1113,8 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
s"ROW FORMAT DELIMITED is only compatible with 'textfile', not
'$fmt'", parentCtx)
}
case _ =>
- // should never happen
+ // Reachable: ROW FORMAT ... STORED BY 'handler' leaves
createFileFormatCtx.fileFormat
+ // null, so none of the typed arms above match.
def str(ctx: ParserRuleContext): String = {
(0 until ctx.getChildCount).map { i => ctx.getChild(i).getText
}.mkString(" ")
}
@@ -1276,7 +1246,7 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
// partition transforms for BucketSpec was moved inside parser
// https://issues.apache.org/jira/browse/SPARK-37923
val partitioning =
- partitionExpressions(partTransforms, partCols, ctx) ++
bucketSpec.map(_.asTransform)
+ partitionExpressions(partTransforms, partCols, ctx) ++
bucketSpec.map(_.asTransform)
val tableSpec = TableSpec(properties, provider, options, location, comment,
serdeInfo, external)
@@ -1359,7 +1329,7 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitPropertyList(ctx: PropertyListContext): Map[String,
String] = withOrigin(ctx) {
val properties = ctx.property.asScala.map { property =>
@@ -1385,19 +1355,6 @@ class HoodieSpark3_4ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[PropertyListContext]], assuming no values
are specified.
- */
- def visitPropertyKeys(ctx: PropertyListContext): Seq[String] = {
- val props = visitPropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A property key can either be String or a collection of dot separated
elements. This
* function extracts the property key based on whether its a string literal
or a property
diff --git
a/hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_5ExtendedSqlAstBuilder.scala
b/hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_5ExtendedSqlAstBuilder.scala
index b837d548240a..1cfa82c02136 100644
---
a/hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_5ExtendedSqlAstBuilder.scala
+++
b/hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark3_5ExtendedSqlAstBuilder.scala
@@ -21,26 +21,24 @@ import
org.apache.hudi.spark.sql.parser.{HoodieSqlBaseBaseVisitor, HoodieSqlBase
import org.apache.hudi.spark.sql.parser.HoodieSqlBaseParser._
import org.antlr.v4.runtime.{ParserRuleContext, Token}
-import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
+import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis._
-import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat}
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.aggregate.{First, Last}
-import org.apache.spark.sql.catalyst.parser.{EnhancedLogicalPlan,
ParseException, ParserInterface}
-import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, entry, escapedIdentifier, operationNotAllowed, source,
string, stringWithoutUnescape, validate, withOrigin}
+import org.apache.spark.sql.catalyst.parser.{ParseException, ParserInterface}
+import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, operationNotAllowed, source, string, stringWithoutUnescape,
validate, withOrigin}
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.util.{truncatedString, CharVarcharUtils,
DateTimeUtils, IntervalUtils}
+import org.apache.spark.sql.catalyst.util.{DateTimeUtils, IntervalUtils}
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.BucketSpecHelper
import org.apache.spark.sql.connector.catalog.TableCatalog
import org.apache.spark.sql.connector.expressions.{ApplyTransform,
BucketTransform, DaysTransform, Expression => V2Expression, FieldReference,
HoursTransform, IdentityTransform, LiteralValue, MonthsTransform, Transform,
YearsTransform}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.BlobType
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
import org.apache.spark.util.Utils.isTesting
@@ -208,14 +206,18 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
/**
- * Create a NULL literal expression.
+ * Create a NULL literal expression. Reachable under ANSI keyword mode,
where NULL (like
+ * FALSE) is a reserved word, so f(null, id) in transform-argument position
parses as a
+ * null literal instead of a column reference.
*/
override def visitNullLiteral(ctx: NullLiteralContext): Literal =
withOrigin(ctx) {
Literal(null)
}
/**
- * Create a Boolean literal expression.
+ * Create a Boolean literal expression. Reachable under ANSI keyword mode,
where FALSE is a
+ * reserved word (only TRUE is ansiNonReserved), so a bare false in
transform-argument
+ * position parses as a boolean literal instead of a column reference.
*/
override def visitBooleanLiteral(ctx: BooleanLiteralContext): Literal =
withOrigin(ctx) {
if (ctx.getText.toBoolean) {
@@ -642,13 +644,6 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create top level table schema.
- */
- protected def createSchema(ctx: ColTypeListContext): StructType = {
- StructType(Option(ctx).toSeq.flatMap(visitColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -691,13 +686,6 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create a [[StructType]] from a sequence of [[StructField]]s.
- */
- protected def createStructType(ctx: ComplexColTypeListContext): StructType =
{
- StructType(Option(ctx).toSeq.flatMap(visitComplexColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -776,7 +764,7 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a table property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitTablePropertyList(
ctx: TablePropertyListContext):
Map[String, String] = withOrigin(ctx) {
@@ -803,19 +791,6 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[TablePropertyListContext]], assuming no
values are specified.
- */
- def visitPropertyKeys(ctx: TablePropertyListContext): Seq[String] = {
- val props = visitTablePropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A table property key can either be String or a collection of dot
separated elements. This
* function extracts the property key based on whether its a string literal
or a table property
@@ -925,10 +900,6 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
lazy val name: String = ctx.identifier.getText
if (arguments.size > 1) {
throw new ParseException(s"Too many arguments for transform $name",
ctx)
- } else if (arguments.isEmpty) {
- throw
-
- new ParseException(s"Not enough arguments for transform $name", ctx)
} else {
getFieldReference(ctx, arguments.head)
}
@@ -1044,8 +1015,6 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
SerdeInfo(storedAs = Some(c.identifier.getText))
case (null, storageHandler) =>
operationNotAllowed("STORED BY", ctx)
- case _ =>
- throw new ParseException("Expected either STORED AS or STORED BY, not
both", ctx)
}
}
@@ -1144,7 +1113,8 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
s"ROW FORMAT DELIMITED is only compatible with 'textfile', not
'$fmt'", parentCtx)
}
case _ =>
- // should never happen
+ // Reachable: ROW FORMAT ... STORED BY 'handler' leaves
createFileFormatCtx.fileFormat
+ // null, so none of the typed arms above match.
def str(ctx: ParserRuleContext): String = {
(0 until ctx.getChildCount).map { i => ctx.getChild(i).getText
}.mkString(" ")
}
@@ -1230,14 +1200,13 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf:
SQLConf, delegate: ParserInterfa
}
/**
- * Create a table, returning a [[CreateTable]] or [[CreateTableAsSelect]]
logical plan.
+ * Create a table, returning a [[CreateTable]] logical plan.
*
* Expected format:
* {{{
* CREATE [TEMPORARY] TABLE [IF NOT EXISTS] [db_name.]table_name
* [USING table_provider]
- * create_table_clauses
- * [[AS] select_statement];
+ * create_table_clauses;
*
* create_table_clauses (order insensitive):
* [PARTITIONED BY (partition_fields)]
@@ -1277,7 +1246,7 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
// partition transforms for BucketSpec was moved inside parser
// https://issues.apache.org/jira/browse/SPARK-37923
val partitioning =
- partitionExpressions(partTransforms, partCols, ctx) ++
bucketSpec.map(_.asTransform)
+ partitionExpressions(partTransforms, partCols, ctx) ++
bucketSpec.map(_.asTransform)
val tableSpec = TableSpec(properties, provider, options, location, comment,
serdeInfo, external)
@@ -1360,7 +1329,7 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitPropertyList(ctx: PropertyListContext): Map[String,
String] = withOrigin(ctx) {
val properties = ctx.property.asScala.map { property =>
@@ -1386,19 +1355,6 @@ class HoodieSpark3_5ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[PropertyListContext]], assuming no values
are specified.
- */
- def visitPropertyKeys(ctx: PropertyListContext): Seq[String] = {
- val props = visitPropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A property key can either be String or a collection of dot separated
elements. This
* function extracts the property key based on whether its a string literal
or a property
diff --git
a/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_0ExtendedSqlAstBuilder.scala
b/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_0ExtendedSqlAstBuilder.scala
index 4fb0356b4f7b..08863dc81a33 100644
---
a/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_0ExtendedSqlAstBuilder.scala
+++
b/hudi-spark-datasource/hudi-spark4.0.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_0ExtendedSqlAstBuilder.scala
@@ -21,26 +21,24 @@ import
org.apache.hudi.spark.sql.parser.{HoodieSqlBaseBaseVisitor, HoodieSqlBase
import org.apache.hudi.spark.sql.parser.HoodieSqlBaseParser._
import org.antlr.v4.runtime.{ParserRuleContext, Token}
-import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
+import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis._
-import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat}
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.aggregate.{First, Last}
-import org.apache.spark.sql.catalyst.parser.{EnhancedLogicalPlan,
ParseException, ParserInterface}
-import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, entry, escapedIdentifier, operationNotAllowed, source,
string, stringWithoutUnescape, validate, withOrigin}
+import org.apache.spark.sql.catalyst.parser.{ParseException, ParserInterface}
+import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, operationNotAllowed, source, string, stringWithoutUnescape,
validate, withOrigin}
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.util.{truncatedString, CharVarcharUtils,
DateTimeUtils, IntervalUtils}
+import org.apache.spark.sql.catalyst.util.{DateTimeUtils, IntervalUtils}
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.BucketSpecHelper
import org.apache.spark.sql.connector.catalog.TableCatalog
import org.apache.spark.sql.connector.expressions.{ApplyTransform,
BucketTransform, DaysTransform, Expression => V2Expression, FieldReference,
HoursTransform, IdentityTransform, LiteralValue, MonthsTransform, Transform,
YearsTransform}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.BlobType
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
import org.apache.spark.util.Utils.isTesting
@@ -208,14 +206,18 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
/**
- * Create a NULL literal expression.
+ * Create a NULL literal expression. Reachable under ANSI keyword mode,
where NULL (like
+ * FALSE) is a reserved word, so f(null, id) in transform-argument position
parses as a
+ * null literal instead of a column reference.
*/
override def visitNullLiteral(ctx: NullLiteralContext): Literal =
withOrigin(ctx) {
Literal(null)
}
/**
- * Create a Boolean literal expression.
+ * Create a Boolean literal expression. Reachable under ANSI keyword mode,
where FALSE is a
+ * reserved word (only TRUE is ansiNonReserved), so a bare false in
transform-argument
+ * position parses as a boolean literal instead of a column reference.
*/
override def visitBooleanLiteral(ctx: BooleanLiteralContext): Literal =
withOrigin(ctx) {
if (ctx.getText.toBoolean) {
@@ -642,13 +644,6 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create top level table schema.
- */
- protected def createSchema(ctx: ColTypeListContext): StructType = {
- StructType(Option(ctx).toSeq.flatMap(visitColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -691,13 +686,6 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create a [[StructType]] from a sequence of [[StructField]]s.
- */
- protected def createStructType(ctx: ComplexColTypeListContext): StructType =
{
- StructType(Option(ctx).toSeq.flatMap(visitComplexColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -776,7 +764,7 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a table property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitTablePropertyList(
ctx: TablePropertyListContext):
Map[String, String] = withOrigin(ctx) {
@@ -803,19 +791,6 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[TablePropertyListContext]], assuming no
values are specified.
- */
- def visitPropertyKeys(ctx: TablePropertyListContext): Seq[String] = {
- val props = visitTablePropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A table property key can either be String or a collection of dot
separated elements. This
* function extracts the property key based on whether its a string literal
or a table property
@@ -925,10 +900,6 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
lazy val name: String = ctx.identifier.getText
if (arguments.size > 1) {
throw new ParseException(s"Too many arguments for transform $name",
ctx)
- } else if (arguments.isEmpty) {
- throw
-
- new ParseException(s"Not enough arguments for transform $name", ctx)
} else {
getFieldReference(ctx, arguments.head)
}
@@ -1044,8 +1015,6 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
SerdeInfo(storedAs = Some(c.identifier.getText))
case (null, storageHandler) =>
operationNotAllowed("STORED BY", ctx)
- case _ =>
- throw new ParseException("Expected either STORED AS or STORED BY, not
both", ctx)
}
}
@@ -1144,7 +1113,8 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
s"ROW FORMAT DELIMITED is only compatible with 'textfile', not
'$fmt'", parentCtx)
}
case _ =>
- // should never happen
+ // Reachable: ROW FORMAT ... STORED BY 'handler' leaves
createFileFormatCtx.fileFormat
+ // null, so none of the typed arms above match.
def str(ctx: ParserRuleContext): String = {
(0 until ctx.getChildCount).map { i => ctx.getChild(i).getText
}.mkString(" ")
}
@@ -1276,7 +1246,7 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
// partition transforms for BucketSpec was moved inside parser
// https://issues.apache.org/jira/browse/SPARK-37923
val partitioning =
- partitionExpressions(partTransforms, partCols, ctx) ++
bucketSpec.map(_.asTransform)
+ partitionExpressions(partTransforms, partCols, ctx) ++
bucketSpec.map(_.asTransform)
val tableSpec = TableSpec(properties, provider, options, location, comment,
Option.empty, serdeInfo, external)
@@ -1359,7 +1329,7 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitPropertyList(ctx: PropertyListContext): Map[String,
String] = withOrigin(ctx) {
val properties = ctx.property.asScala.map { property =>
@@ -1385,19 +1355,6 @@ class HoodieSpark4_0ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[PropertyListContext]], assuming no values
are specified.
- */
- def visitPropertyKeys(ctx: PropertyListContext): Seq[String] = {
- val props = visitPropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A property key can either be String or a collection of dot separated
elements. This
* function extracts the property key based on whether its a string literal
or a property
diff --git
a/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_1ExtendedSqlAstBuilder.scala
b/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_1ExtendedSqlAstBuilder.scala
index 421c6b4fe802..74b1192bef6c 100644
---
a/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_1ExtendedSqlAstBuilder.scala
+++
b/hudi-spark-datasource/hudi-spark4.1.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_1ExtendedSqlAstBuilder.scala
@@ -21,19 +21,18 @@ import
org.apache.hudi.spark.sql.parser.{HoodieSqlBaseBaseVisitor, HoodieSqlBase
import org.apache.hudi.spark.sql.parser.HoodieSqlBaseParser._
import org.antlr.v4.runtime.{ParserRuleContext, Token}
-import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
+import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis._
-import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat}
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.aggregate.{First, Last}
-import org.apache.spark.sql.catalyst.parser.{EnhancedLogicalPlan,
ParseException, ParserInterface}
-import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, entry, escapedIdentifier, operationNotAllowed, source,
string, stringWithoutUnescape, validate, withOrigin}
+import org.apache.spark.sql.catalyst.parser.{ParseException, ParserInterface}
+import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, operationNotAllowed, source, string, stringWithoutUnescape,
validate, withOrigin}
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.util.{truncatedString, CharVarcharUtils,
DateTimeUtils, IntervalUtils}
+import org.apache.spark.sql.catalyst.util.{DateTimeUtils, IntervalUtils}
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.BucketSpecHelper
import org.apache.spark.sql.connector.catalog.TableCatalog
@@ -207,14 +206,18 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
/**
- * Create a NULL literal expression.
+ * Create a NULL literal expression. Reachable under ANSI keyword mode,
where NULL (like
+ * FALSE) is a reserved word, so f(null, id) in transform-argument position
parses as a
+ * null literal instead of a column reference.
*/
override def visitNullLiteral(ctx: NullLiteralContext): Literal =
withOrigin(ctx) {
Literal(null)
}
/**
- * Create a Boolean literal expression.
+ * Create a Boolean literal expression. Reachable under ANSI keyword mode,
where FALSE is a
+ * reserved word (only TRUE is ansiNonReserved), so a bare false in
transform-argument
+ * position parses as a boolean literal instead of a column reference.
*/
override def visitBooleanLiteral(ctx: BooleanLiteralContext): Literal =
withOrigin(ctx) {
if (ctx.getText.toBoolean) {
@@ -641,13 +644,6 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create top level table schema.
- */
- protected def createSchema(ctx: ColTypeListContext): StructType = {
- StructType(Option(ctx).toSeq.flatMap(visitColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -690,13 +686,6 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create a [[StructType]] from a sequence of [[StructField]]s.
- */
- protected def createStructType(ctx: ComplexColTypeListContext): StructType =
{
- StructType(Option(ctx).toSeq.flatMap(visitComplexColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -775,7 +764,7 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a table property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitTablePropertyList(
ctx: TablePropertyListContext):
Map[String, String] = withOrigin(ctx) {
@@ -802,19 +791,6 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[TablePropertyListContext]], assuming no
values are specified.
- */
- def visitPropertyKeys(ctx: TablePropertyListContext): Seq[String] = {
- val props = visitTablePropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A table property key can either be String or a collection of dot
separated elements. This
* function extracts the property key based on whether its a string literal
or a table property
@@ -924,10 +900,6 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
lazy val name: String = ctx.identifier.getText
if (arguments.size > 1) {
throw new ParseException(s"Too many arguments for transform $name",
ctx)
- } else if (arguments.isEmpty) {
- throw
-
- new ParseException(s"Not enough arguments for transform $name", ctx)
} else {
getFieldReference(ctx, arguments.head)
}
@@ -1043,8 +1015,6 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
SerdeInfo(storedAs = Some(c.identifier.getText))
case (null, storageHandler) =>
operationNotAllowed("STORED BY", ctx)
- case _ =>
- throw new ParseException("Expected either STORED AS or STORED BY, not
both", ctx)
}
}
@@ -1143,7 +1113,8 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
s"ROW FORMAT DELIMITED is only compatible with 'textfile', not
'$fmt'", parentCtx)
}
case _ =>
- // should never happen
+ // Reachable: ROW FORMAT ... STORED BY 'handler' leaves
createFileFormatCtx.fileFormat
+ // null, so none of the typed arms above match.
def str(ctx: ParserRuleContext): String = {
(0 until ctx.getChildCount).map { i => ctx.getChild(i).getText
}.mkString(" ")
}
@@ -1235,8 +1206,7 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
* {{{
* CREATE [TEMPORARY] TABLE [IF NOT EXISTS] [db_name.]table_name
* [USING table_provider]
- * create_table_clauses
- * [[AS] select_statement];
+ * create_table_clauses;
*
* create_table_clauses (order insensitive):
* [PARTITIONED BY (partition_fields)]
@@ -1276,7 +1246,7 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
// partition transforms for BucketSpec was moved inside parser
// https://issues.apache.org/jira/browse/SPARK-37923
val partitioning =
- partitionExpressions(partTransforms, partCols, ctx) ++
bucketSpec.map(_.asTransform)
+ partitionExpressions(partTransforms, partCols, ctx) ++
bucketSpec.map(_.asTransform)
val tableSpec = TableSpec(properties, provider, options, location, comment,
Option.empty, serdeInfo, external)
@@ -1359,7 +1329,7 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitPropertyList(ctx: PropertyListContext): Map[String,
String] = withOrigin(ctx) {
val properties = ctx.property.asScala.map { property =>
@@ -1385,19 +1355,6 @@ class HoodieSpark4_1ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[PropertyListContext]], assuming no values
are specified.
- */
- def visitPropertyKeys(ctx: PropertyListContext): Seq[String] = {
- val props = visitPropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A property key can either be String or a collection of dot separated
elements. This
* function extracts the property key based on whether its a string literal
or a property
diff --git
a/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_2ExtendedSqlAstBuilder.scala
b/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_2ExtendedSqlAstBuilder.scala
index 198723f0e709..94066110b720 100644
---
a/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_2ExtendedSqlAstBuilder.scala
+++
b/hudi-spark-datasource/hudi-spark4.2.x/src/main/scala/org/apache/spark/sql/parser/HoodieSpark4_2ExtendedSqlAstBuilder.scala
@@ -21,19 +21,18 @@ import
org.apache.hudi.spark.sql.parser.{HoodieSqlBaseBaseVisitor, HoodieSqlBase
import org.apache.hudi.spark.sql.parser.HoodieSqlBaseParser._
import org.antlr.v4.runtime.{ParserRuleContext, Token}
-import org.antlr.v4.runtime.tree.{ParseTree, RuleNode, TerminalNode}
+import org.antlr.v4.runtime.tree.{ParseTree, RuleNode}
import org.apache.spark.internal.Logging
import org.apache.spark.sql.AnalysisException
-import org.apache.spark.sql.catalyst.{FunctionIdentifier, TableIdentifier}
+import org.apache.spark.sql.catalyst.TableIdentifier
import org.apache.spark.sql.catalyst.analysis._
-import org.apache.spark.sql.catalyst.catalog.{BucketSpec, CatalogStorageFormat}
+import org.apache.spark.sql.catalyst.catalog.BucketSpec
import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.expressions.aggregate.{First, Last}
-import org.apache.spark.sql.catalyst.parser.{EnhancedLogicalPlan,
ParseException, ParserInterface}
-import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, entry, escapedIdentifier, operationNotAllowed, source,
string, stringWithoutUnescape, validate, withOrigin}
+import org.apache.spark.sql.catalyst.parser.{ParseException, ParserInterface}
+import
org.apache.spark.sql.catalyst.parser.ParserUtils.{checkDuplicateClauses,
checkDuplicateKeys, operationNotAllowed, source, string, stringWithoutUnescape,
validate, withOrigin}
import org.apache.spark.sql.catalyst.plans._
import org.apache.spark.sql.catalyst.plans.logical._
-import org.apache.spark.sql.catalyst.util.{truncatedString, CharVarcharUtils,
DateTimeUtils, IntervalUtils}
+import org.apache.spark.sql.catalyst.util.{DateTimeUtils, IntervalUtils}
import org.apache.spark.sql.catalyst.util.DateTimeUtils._
import
org.apache.spark.sql.connector.catalog.CatalogV2Implicits.BucketSpecHelper
import org.apache.spark.sql.connector.catalog.TableCatalog
@@ -207,14 +206,18 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
/**
- * Create a NULL literal expression.
+ * Create a NULL literal expression. Reachable under ANSI keyword mode,
where NULL (like
+ * FALSE) is a reserved word, so f(null, id) in transform-argument position
parses as a
+ * null literal instead of a column reference.
*/
override def visitNullLiteral(ctx: NullLiteralContext): Literal =
withOrigin(ctx) {
Literal(null)
}
/**
- * Create a Boolean literal expression.
+ * Create a Boolean literal expression. Reachable under ANSI keyword mode,
where FALSE is a
+ * reserved word (only TRUE is ansiNonReserved), so a bare false in
transform-argument
+ * position parses as a boolean literal instead of a column reference.
*/
override def visitBooleanLiteral(ctx: BooleanLiteralContext): Literal =
withOrigin(ctx) {
if (ctx.getText.toBoolean) {
@@ -641,13 +644,6 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create top level table schema.
- */
- protected def createSchema(ctx: ColTypeListContext): StructType = {
- StructType(Option(ctx).toSeq.flatMap(visitColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -690,13 +686,6 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
}
}
- /**
- * Create a [[StructType]] from a sequence of [[StructField]]s.
- */
- protected def createStructType(ctx: ComplexColTypeListContext): StructType =
{
- StructType(Option(ctx).toSeq.flatMap(visitComplexColTypeList))
- }
-
/**
* Create a [[StructType]] from a number of column definitions.
*/
@@ -775,7 +764,7 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a table property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitTablePropertyList(
ctx: TablePropertyListContext):
Map[String, String] = withOrigin(ctx) {
@@ -802,19 +791,6 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[TablePropertyListContext]], assuming no
values are specified.
- */
- def visitPropertyKeys(ctx: TablePropertyListContext): Seq[String] = {
- val props = visitTablePropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A table property key can either be String or a collection of dot
separated elements. This
* function extracts the property key based on whether its a string literal
or a table property
@@ -924,10 +900,6 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
lazy val name: String = ctx.identifier.getText
if (arguments.size > 1) {
throw new ParseException(s"Too many arguments for transform $name",
ctx)
- } else if (arguments.isEmpty) {
- throw
-
- new ParseException(s"Not enough arguments for transform $name", ctx)
} else {
getFieldReference(ctx, arguments.head)
}
@@ -1043,8 +1015,6 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
SerdeInfo(storedAs = Some(c.identifier.getText))
case (null, storageHandler) =>
operationNotAllowed("STORED BY", ctx)
- case _ =>
- throw new ParseException("Expected either STORED AS or STORED BY, not
both", ctx)
}
}
@@ -1143,7 +1113,8 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
s"ROW FORMAT DELIMITED is only compatible with 'textfile', not
'$fmt'", parentCtx)
}
case _ =>
- // should never happen
+ // Reachable: ROW FORMAT ... STORED BY 'handler' leaves
createFileFormatCtx.fileFormat
+ // null, so none of the typed arms above match.
def str(ctx: ParserRuleContext): String = {
(0 until ctx.getChildCount).map { i => ctx.getChild(i).getText
}.mkString(" ")
}
@@ -1275,7 +1246,7 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
// partition transforms for BucketSpec was moved inside parser
// https://issues.apache.org/jira/browse/SPARK-37923
val partitioning =
- partitionExpressions(partTransforms, partCols, ctx) ++
bucketSpec.map(_.asTransform)
+ partitionExpressions(partTransforms, partCols, ctx) ++
bucketSpec.map(_.asTransform)
val tableSpec = TableSpec(properties, provider, options, location, comment,
Option.empty, serdeInfo, external)
@@ -1358,7 +1329,7 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
/**
* Convert a property list into a key-value map.
- * This should be called through [[visitPropertyKeyValues]] or
[[visitPropertyKeys]].
+ * This should be called through [[visitPropertyKeyValues]].
*/
override def visitPropertyList(ctx: PropertyListContext): Map[String,
String] = withOrigin(ctx) {
val properties = ctx.property.asScala.map { property =>
@@ -1384,19 +1355,6 @@ class HoodieSpark4_2ExtendedSqlAstBuilder(conf: SQLConf,
delegate: ParserInterfa
props
}
- /**
- * Parse a list of keys from a [[PropertyListContext]], assuming no values
are specified.
- */
- def visitPropertyKeys(ctx: PropertyListContext): Seq[String] = {
- val props = visitPropertyList(ctx)
- val badKeys = props.filter { case (_, v) => v != null }.keys
- if (badKeys.nonEmpty) {
- operationNotAllowed(
- s"Values should not be specified for key(s): ${badKeys.mkString("[",
",", "]")}", ctx)
- }
- props.keys.toSeq
- }
-
/**
* A property key can either be String or a collection of dot separated
elements. This
* function extracts the property key based on whether its a string literal
or a property
