szehon-ho commented on code in PR #57644:
URL: https://github.com/apache/spark/pull/57644#discussion_r3798212958
##########
sql/pipelines/src/main/scala/org/apache/spark/sql/pipelines/graph/DatasetManager.scala:
##########
@@ -317,9 +318,23 @@ object DatasetManager extends Logging {
val (catalog, identifier) =
PipelinesCatalogUtils.resolveTableCatalog(context.spark,
table.identifier)
- val outputSchema = table.specifiedSchema.getOrElse(
- inferredSchemas(table.identifier).asNullable
- )
+ val outputSchema = table.specifiedSchema match {
Review Comment:
Relaxing the validation breaks an invariant the planning path depends on.
`validateUserSpecifiedSchemas` used to require `inferredSchema == ss`, so a
declared schema was always exactly what the table materialized with, and
`VirtualTableInput.load` relies on that -- it uses the declaration as-is and
never consults the materialized table:
```scala
val deducedSchema = specifiedSchema match {
// If the user specified a schema, use it directly.
case Some(ss) => ss
```
With `outputSchema` now differing from `ss`, a same-graph downstream dataset
is planned against the declared columns but reads the real table at execution:
`reanalyzeFlow` builds a subgraph whose `tables` is only the destination,
`dfsInternal(..., stopAtMaterializationPoints = true)` skips the upstream
table, so it is not in `context.allInputs` and `GraphIdentifierManager`
classifies it as an `ExternalDatasetIdentifier` -- a plain read of the
four-column catalog table.
The worked example and the fix I'd suggest are in the review body. I traced
this statically rather than running it, so I can't say whether it surfaces as
an "extra fields" `AnalysisException` or as silent evolution downstream; either
way I think it needs a test with a downstream `SELECT *` over an AUTO CDC
target that has a data-only declaration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
