deepakpanda93 opened a new pull request, #19551: URL: https://github.com/apache/hudi/pull/19551
### Describe the issue this Pull Request addresses Closes #15648. The Schema Evolution page told users to disable `hive.metastore.disallow.incompatible.col.type.changes` when an `ALTER TABLE` fails, but never showed **how**: > :::note > When using hive metastore, please disable `hive.metastore.disallow.incompatible.col.type.changes` if you encounter this error: > `The following columns have types incompatible with the existing columns in their respective positions`. > ::: That matters because the same page documents adding a column at an arbitrary position (`FIRST` / `AFTER`), which is exactly what trips the check. As HUDI-5459 notes, the way to disable it differs depending on which metastore the engine talks to, and the note covered neither variant. ### Summary and Changelog The metastore evaluates this check against **its own** configuration — `HiveAlterHandler#alterTable` reads `handler.getConf()` — so where the property has to be set depends on the metastore in play. Replaced the bare note with a `### Disabling the Hive metastore column type compatibility check` subsection covering all three mechanisms: | Deployment | How to disable | |---|---| | Spark with its own embedded metastore (no `hive.metastore.uris`) | `--conf 'spark.hadoop.hive.metastore.disallow.incompatible.col.type.changes=false'` — the metastore shares the Spark JVM, so the client conf *is* the handler conf | | Remote Hive metastore service | Set the property in the metastore's `hive-site.xml` and restart the service | | Remote HMS, single session | `set metaconf:hive.metastore.disallow.incompatible.col.type.changes=false;` — routes through `SetProcessor` to `IMetaStoreClient#setMetaConf`; the property is a `metaConfVars` entry, so the server accepts the override | Applied to `website/docs/schema_evolution.md` (next) and `website/versioned_docs/version-1.2.0/schema_evolution.md` (the current released docs), per the next-plus-current convention used in #19473. The nine older versioned copies carry the same bare note and were left alone — happy to widen if preferred. ### Verification Every claim was first checked against Hive `rel/release-3.1.3` and Spark `v3.5.1` sources, then reproduced on a Spark 3.5.7 + Hive 3.1.3 (standalone HMS) + MinIO stack using `hudi-spark3.5-bundle_2.12:1.2.0`. Reproducer: a Hudi table `(id int, name string, ts bigint)` with `hoodie.schema.on.read.enable=true` set at session level, then `ALTER TABLE t ADD COLUMNS (age int AFTER id)` — which shifts an `int` into a position a `string` occupied, incompatible per `ColumnType#areColTypesCompatible`. | Scenario | `hive.metastore.uris` | Property set | Result | |---|---|---|---| | Embedded metastore, default | *(unset)* | — | :x: `...types incompatible with the existing columns in their respective positions : age` | | Embedded + `spark.hadoop.…=false` | *(unset)* | client | :white_check_mark: succeeds; `describe` → `id, age, name, ts` | | Remote HMS, default | `thrift://…:9083` | — | :x: same error | | Remote HMS + `spark.hadoop.…=false` | `thrift://…:9083` | client | :x: **still fails** | | Remote HMS + property in the HMS's `hive-site.xml` | `thrift://…:9083` | server | :white_check_mark: succeeds | | Remote HMS + `setMetaConf` | `thrift://…:9083` | pushed to server | :white_check_mark: succeeds | Rows 2 and 4 are why the section is split by deployment: the same property, set the same way, works against an embedded metastore and is silently ignored against a remote one. Also confirmed that a standalone metastore does read `hive-site.xml` (`MetastoreConf` adds it ahead of `metastore-site.xml`), so the middle remedy is correct for that deployment. Site build: `npm run build` passes, with the warning set byte-identical to a baseline build of the same base commit (no new broken links or anchors). Both `/docs/schema_evolution` and `/docs/next/schema_evolution` were loaded from `npm run serve` and render the new section, with all pre-existing anchors on the page preserved. ### Impact Documentation only. No code, config, or behaviour change. ### Risk Level none ### Documentation Update This PR is the documentation update — the Schema Evolution page, `/docs/schema_evolution` and `/docs/next/schema_evolution`. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
