pan3793 commented on PR #7622:
URL: https://github.com/apache/kyuubi/pull/7622#issuecomment-5264779609
The external overwrite test does not cover the change it ships with:
reverting only `HiveBatchWrite.scala` to master leaves `HiveQuerySuite` at
24/24 on both a case-insensitive FS (APFS) and a case-sensitive one (ext4).
Reason in the inline thread. Suggest dropping the test and noting in the
description that the alignment is not observable with bundled Hive 2.3.10.
The other three hunks are load-bearing, each verified by reverting it alone:
- by-name split -> both new tests fail
- lower-cased `outputPartColumns` -> uppercase test fails with `Partition
spec {favoritecolor=, FavoriteColor=blue} contains non-partition columns`
- `createTable` column order -> both new tests fail with `CANNOT_SAFELY_CAST`
For the last one, consider scoping the order to the V2 table instead of
rebuilding the `CatalogTable`. A reordered `CatalogTable` breaks the invariant
that partition columns are trailing, which `partitionSchema` asserts and
`HiveTable.dataSchema`/`partitionSchema` inherit, and `newCatalogTable`
silently drops fields it does not forward, e.g. `collation` on Spark 4.2. Only
`Table.schema()` feeds by-position output resolution:
```scala
case class HiveTable(
sparkSession: SparkSession,
catalogTable: CatalogTable,
hiveTableCatalog: HiveTableCatalog,
requestedSchema: Option[StructType] = None)
override def schema(): StructType =
requestedSchema.getOrElse(catalogTable.schema)
// HiveTableCatalog.createTable
loadTable(ident).asInstanceOf[HiveTable].copy(requestedSchema = Some(schema))
```
Full proposal, 23 insertions / 80 deletions against your head, in
https://github.com/pan3793/kyuubi/tree/branch-kyuubi-6403, CI at
https://github.com/pan3793/kyuubi/actions/runs/31581643348. `HiveQuerySuite` +
`HiveCatalogSuite` 46/46 with `-Pspark-3.5`, `dev/reformat` clean.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
