vinishjail97 commented on issue #871: URL: https://github.com/apache/incubator-xtable/issues/871#issuecomment-5108975528
This looks like a limitation on the Databricks side rather than something wrong with the metadata XTable produces. Based on their docs, Databricks does not appear to read Iceberg natively in this path, it converts the Iceberg table to Delta: https://docs.databricks.com/gcp/en/ingestion/data-migration/convert-to-delta#converting-a-directory-of-parquet-or-iceberg-files-in-an-external-location-to-delta-lake [Partition evolution](https://iceberg.apache.org/docs/latest/evolution/#partition-evolution) is a core Iceberg feature, and the table here is queryable from native Iceberg engines, which @kvutukuru already confirmed with AWS Athena (Trino) and Spark on EMR. So the restriction comes from what the Delta conversion supports, not from the Iceberg spec. I asked one of the XTable users who runs this in production, and they pointed to a Databricks side config, `spark.databricks.delta.convert.iceberg.partitionEvolution.enabled`, which appears to relax this check. I could not find it in the Databricks docs for this path, so the pointer comes from their production experience rather than from published documentation. @kvutukuru would you be able to try setting it and report back whether it unblocks the read? If it works it would also help tables that have already been synced and are carrying the extra spec. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
