TengHuo commented on PR #9743: URL: https://github.com/apache/hudi/pull/9743#issuecomment-1869231265
> # Questions > 1. Is the community planning to support Hudi comprehensive schema evolution via Dataframe API? > 2. If so, a refactoring might be in store to move InternalSchema into the different hudi-sync implementations such that there is a translation of Hudi's `InternalSchema` type to XYZ-sync type. > 3. If we are not doing this, should we document this behaviour and explicitly let users know what our intended usage pattern for schema evolution is? i.e. users should stop all their write jobs, perform hudi comprehensive schema evolution via Spark-SQL, then resume their write jobs via deltastreamer/other non-Spark-SQL writes. Sorry for necro-ing this MR again. Complement about this question, > 1. Is the community planning to support Hudi comprehensive schema evolution via Dataframe API? 1. Is the community planning to support Hudi comprehensive schema evolution via `SparkRDDWriteClient` API and `HiveSyncTool`?** This issue mainly impact the pipelines of `HoodieDeltaStreamer`, e.g. `UPSERT` a batch of data with schema evolution succeeded, but hive sync could fail, then causing schema inconsistent issues when user run a query. And add a little bit more about the current `HoodieSyncTool`. As we understand, `HoodieSyncTool` could be used as an independent tool to sync Hudi table information to some outside meta data management service, e.g. HMS, with the method `HoodieSyncTool#syncHoodieTable`. So, there is no input parameters for this method, it needs to infer the schema and partition changes by itself. Thus, in the method `HiveSyncTool#syncHoodieTable`, it will load Hudi table schema information from HDFS and HMS, and do the schema comparison work in the method `HiveSchemaUtil#getSchemaDifference`. As Voon mentioned, this method has its own logic to decide the schema type update role, which is not totally compatible with the current Hudi Schema Evolution. For solving this issue, we are thinking about utilising the information from `InternalSchema` in `HoodieSyncTool`, and let each meta sync tool to decide how to translate these `TableChange` to its own meta service. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
