voonhous commented on PR #9743:
URL: https://github.com/apache/hudi/pull/9743#issuecomment-1867582951
@jonvex @xiarixiaoyao
Apologies for necro-ing this PR. I was revisiting this PR today and noticed
that using Hudi's comprehensive schema evolution using that is managed via
Hudi's `InternalSchema` is still supported via write paths using the Dataframe
API.
i.e. dataframe.write.format("hudi").options(...).mode(...).save(basePath)
Where the options require these 2 confguration:
```
hoodie.schema.on.read.enable=true
hoodie.datasource.write.reconcile.schema=true
```
This means that we can still make use of Hudi comprehensive schema evolution
to perform schema evolution. This behaviour is consistent with what we used to
have in the past, which is good!
However, this means of using schema evolution is not really documented and I
am wondering if the community has any plans to ensure that the end-to-end flow
for this use-case is error-free.
For now, there are 2 entrypoints for Spark in which Hudi comprehensive
schema evolution can be done. One, via Spark-SQL, and the other via Dataframe
API as i described above just now.
Considering the case where tables are sync-ed to a hive-catalogue, which is
one of the more common use cases. Spark-SQL current does this via
`sparkSession.sessionState.catalog.externalCatalog.alterTableDataSchema`.
Hence, hive-sync is done by Spark's code.
However, when Dataframe API, especially Deltrastreamer, hive-sync is done
using `hudi-hive-sync`. Something that Hudi manages internally.
Referring to the code below, if one does a Hudi comprehensive schema
evolution outside of the scope defined here, hive-sync will fail, although
UPSERT succeeds.
https://github.com/apache/hudi/blob/f0356de4e42006035bcce5feec5cae52fc147e46/hudi-sync/hudi-hive-sync/src/main/java/org/apache/hudi/hive/util/HiveSchemaUtil.java#L424-L439
# Questions
1. Is the community planning to support Hudi comprehensive schema evolution
via Dataframe API?
2. If so, a refactoring might be in store to move InternalSchema into the
different hudi-sync implementations such that there is a translation of Hudi's
`InternalSchema` type to XYZ-sync type.
3. If we are not doing this, should we document this behaviour and
explicitly let users know what our intended usage pattern for schema evolution
is? i.e. users should stop all their write jobs, perform hudi comprehensive
schema evolution via Spark-SQL, then resume their write jobs via
deltastreamer/other non-Spark-SQL writes.
@TengHuo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
