vinishjail97 commented on issue #871: URL: https://github.com/apache/incubator-xtable/issues/871#issuecomment-5108865728
Some notes from the Slack discussion, for context: - The generated Iceberg table itself is valid and readable. @kvutukuru confirmed successful reads from AWS Athena (Trino) and Spark on AWS EMR. The failure is specific to Databricks' Iceberg ingestion path. - To be clear about the setup: XTable is not being used to produce Delta here. The source is Hudi, XTable generates Iceberg metadata, and Databricks users want to read that Iceberg table. Databricks routes external Iceberg through its convert/clone machinery, which is where the partition-evolution restriction bites. See the Databricks docs on [converting Iceberg files to Delta Lake](https://docs.databricks.com/gcp/en/ingestion/data-migration/convert-to-delta#converting-a-directory-of-parquet-or-iceberg-files-in-an-external-location-to-delta-lake). - Possible workaround to try on the Databricks side, which may relax the restriction without any XTable change: `spark.databricks.delta.convert.iceberg.partitionEvolution.enabled`. Worth testing before we settle on the XTable-side fix, since it would unblock tables that already carry the extra spec. - Independent of the workaround, the extra unpartitioned spec is an artifact worth removing on our side, since it misrepresents the table's history to any strict consumer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
