szehon-ho commented on PR #58092:
URL: https://github.com/apache/spark/pull/58092#issuecomment-5349724171
Correction to one detail in my previous comment — why the fixture cannot
move up to `DescribeTableSuiteBase`. The conclusion holds, but the mechanism I
cited was for a provider the Hive suite does not use.
`HiveExternalCatalog.scala:342` (`schema = StructType(EMPTY_DATA_SCHEMA ++
table.partitionSchema)`) is inside `createDataSourceTable`, so it only fires
for a data source provider. The Hive suite's `defaultUsing` is `USING HIVE`, so
`getProvider()` returns `hive` and the fixture is a Hive SerDe table — which is
created without complaint.
What actually fails is reading it back. `restoreHiveSerdeTable` calls
`reorderSchema`, which resolves each declared partition column by name and
rejects the table when one is absent from the schema:
```
org.apache.spark.sql.AnalysisException: The metadata is corrupted. Unable to
find the partition
column names from the schema. schema: struct<id:int,actual_part:string>.
Partition columns: [declared_part]
at
org.apache.spark.sql.hive.HiveExternalCatalog.$anonfun$reorderSchema$3(HiveExternalCatalog.scala:895)
at
org.apache.spark.sql.hive.HiveExternalCatalog.reorderSchema(HiveExternalCatalog.scala:889)
at
org.apache.spark.sql.hive.HiveExternalCatalog.restoreHiveSerdeTable(HiveExternalCatalog.scala:913)
```
I confirmed this by actually moving the test into
`v1.DescribeTableSuiteBase` and running `hive/testOnly
*command.DescribeTableSuite`; both command versions fail on the fixture itself.
So the table cannot be described under the Hive catalog either way, and the
test stays in the V1 in-memory suite. The PR description has been updated to
describe this path rather than the creation-time one.
Also re-ran the suites after the last push, no failures: `sql/testOnly
*DescribeTableSuite` (106 tests, V1 and V2 suites) and `hive/testOnly
*command.DescribeTableSuite` (68 tests).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
