LuciferYang commented on PR #58409:
URL: https://github.com/apache/spark/pull/58409#issuecomment-5541737696
Thanks for staying on this, @cloud-fan, and you are right about the branch:
`getOrInferFileFormatSchema` does catch the duplicate-column check and continue
with a warning (`DataSource.scala:239-245`, SPARK-18108 and SPARK-21144). I had
not found it, and my earlier replies answered the symptom rather than the
mechanism, which is probably why the question kept coming back. Let me lay out
what I read this time so you can tell me where it is wrong.
As I read it there are two checks on the V1 path, looking at different
things. The warn-and-continue one checks `(dataSchema ++
partitionSchema).map(_.name)`, so what it tolerates is a name shared between
the data schema and the partition schema. The throwing one is
`checkSchemaColumnNameDuplication(hs.dataSchema, equality)` in
`resolveRelation`, and that is what a duplicate *inside* the data schema hits.
It sits after the relation-building match and applies to its result, so every
`HadoopFsRelation` branch goes through it, the streaming-sink metadata one and
the `CatalogFileIndex` one included, and `equality` is `conf.resolver`.
The tolerated case seems unable to leave two occurrences behind, because of
`DataSource.scala:222-223`: a user-specified schema becomes the data schema
minus every field whose name matches a partition column, so the field that made
the concatenation ambiguous is the one that gets removed, and if the duplicated
name is the partition column's own then both copies go. Printing the resolved
relation's two schemas:
```
user schema (a, p), partition column p: loaded, data=a
partition=p, rows=[2,2],[0,0],[1,1]
user schema (p, p), partition column p: loaded, data=<empty>
partition=p, rows=[2],[0],[1]
user schema (x, x), no partition column: AnalysisException
[COLUMN_ALREADY_EXISTS] The column `x` already exists
```
Two places did look like they could bite, and I think your instinct about
the weak check is right about one of them. Streaming is where a duplicate
really does travel on it: `readStream.schema("x int, x int")` survives stream
setup, since `sourceSchema` goes through `getOrInferFileFormatSchema`.
`FileStreamSource.getBatch` then builds each micro-batch with
`resolveRelation(checkFilesExist = false, readOnly = true)`, so the query fails
at first-batch planning rather than at analysis, which is worth its own JIRA
even though no relation reaches `buildReader`. The other is
`HiveMetastoreCatalog.convertToLogicalRelation`, the one construction site that
skips the check outright, but it is only ever called with `ParquetFileFormat`
or `OrcFileFormat`, and a Hive avro table stays a `HiveTableRelation` and reads
through the Hive SerDe.
So as far as I can tell `dataSchema.fieldIndex` is never asked to choose
between two occurrences. You know this code much better than I do, though, so
if there is a path I have walked past, please say which one and I will open a
follow-up JIRA for the ordinal plumbing, since this PR is already merged and
cannot carry the change anyway. A query I can run would help me see it fastest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
