rangareddy commented on issue #16806:
URL: https://github.com/apache/hudi/issues/16806#issuecomment-5408420651
**Findings: fixed. Closing - and answering the open question in this thread
about the `SELECT … FROM` variant.**
@deepakpanda93's assessment above is right, and the remaining doubt ("the
merged tests covered VALUES") can be resolved from the code rather than by
retesting.
**The fix is PR #12692**, whose title is `[HUDI-8898] Support INSERT SQL
statement with a subset of columns in **Spark 3.5**` (merged 2025-01-25). It
touched exactly the relevant files:
-
`hudi-spark-datasource/hudi-spark3.5.x/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieSpark35Analysis.scala`
-
`hudi-spark-datasource/hudi-spark/src/main/scala/org/apache/spark/sql/hudi/analysis/HoodieAnalysis.scala`
- `.../dml/insert/TestInsertTable.scala`
**Why `SELECT … FROM` is covered by the same path.** In
`HoodieSpark35Analysis.scala`, `HoodieSpark35ResolveColumnsForInsertInto`
matches on the user-specified column list, not on the shape of the query:
```scala
case i@InsertIntoStatement(table, _, _, query, _, _, _)
if ... && i.userSpecifiedCols.nonEmpty &&
i.table.isInstanceOf[LogicalRelation]
```
and then:
```scala
// Create a project if this INSERT has a user-specified column list.
val hasColumnList = insert.userSpecifiedCols.nonEmpty
val query = if (hasColumnList) {
createProjectForByNameQuery(tblName, insert)
} else {
insert.query
}
val newQuery = TableOutputResolver.resolveOutputColumns(
tblName, expectedColumns, query,
byName = hasColumnList || insert.byName, conf, supportColDefaultValue =
true)
```
`createProjectForByNameQuery` wraps `insert.query` whatever that query is -
it never inspects whether the source is a `VALUES` literal or a subquery. So
`INSERT INTO hudi_table (id, dt, ts) select id, dt, ts from (...)` resolves by
name and pads the unspecified columns through exactly the same code path as the
`VALUES` form. There is no separate branch that the `SELECT` variant could miss.
Test coverage backs that up: `TestInsertTable.scala:34` (`Test Insert Into
with subset of columns`) and `:45` (the Parquet comparison), with
`SELECT`-sourced subset inserts used throughout that suite rather than only
`VALUES`.
Also worth confirming there is **no throw** on the Spark 3.5 path - the
`AnalysisException("hudi not support specified cols when enable default
columns, …")` that caused this class of failure exists only in
`hudi-spark-datasource/hudi-spark3.4.x/src/main/scala/org/apache/spark/sql/HoodieSpark34CatalystPlanUtils.scala:74`.
**That last point is why the Spark 3.4 sibling, #16805 (HUDI-8911), stays
open** - the throw is still live on the 3.4 path, so the two tickets are
genuinely in different states despite describing the same feature.
Closing as fixed. Worth resolving JIRA HUDI-8912 to match, and noting it as
a duplicate of HUDI-8898.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
