[
https://issues.apache.org/jira/browse/IMPALA-11618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17886863#comment-17886863
]
Zoltán Borók-Nagy commented on IMPALA-11618:
--------------------------------------------
By default, we don't allow data files be outside of the table directory. AFAIK
this is the problem with Iceberg tables created via
spark_catalog.system.snapshot().
One can try to make such tables readable by Impala by setting table property
write.object-storage.enabled to true.
> Impala fails to read Iceberg snapshot tables created by Spark using call
> spark_catalog.system.snapshot()
> ---------------------------------------------------------------------------------------------------------
>
> Key: IMPALA-11618
> URL: https://issues.apache.org/jira/browse/IMPALA-11618
> Project: IMPALA
> Issue Type: Bug
> Components: Catalog
> Affects Versions: Impala 4.1.0
> Reporter: David Rorke
> Assignee: Zoltán Borók-Nagy
> Priority: Major
> Labels: impala-iceberg
>
> I get an exception from catalog when Impala tries to read an Iceberg
> "snapshot" table created by a spark job using this API:
> [https://iceberg.apache.org/docs/latest/spark-procedures/#snapshot]
> {noformat}
> Query submitted at: 2022-09-27 09:43:15 (Coordinator:
> http://coordinator-0.coordinator-int.impala-1662757723-t7hk.svc.cluster.local:25000)
> ERROR: AnalysisException: Failed to load metadata for table:
> 'customer_iceberg_snapshot'
> CAUSED BY: TableLoadingException: Could not load table
> tpcds_3000_string_parquet_external.customer_iceberg_snapshot from catalog
> CAUSED BY: TException:
> TGetPartialCatalogObjectResponse(status:TStatus(status_code:GENERAL,
> error_msgs:[IcebergTableLoadingException: Error loading metadata for Iceberg
> table
> s3a://drorke-dwxperf2/warehouse/tablespace/external/hive/tpcds_3000_string_parquet_external.db/customer_iceberg_snapshot
> CAUSED BY: RuntimeException: FileSystem returned an unexpected path
> s3a://drorke-dwxperf2/warehouse/tablespace/external/hive/tpcds_3000_string_parquet_external.db/customer/0b443b4d91612be0-af2edb1b00000000_1194661597_data.0.parq
> for a file within
> s3a://drorke-dwxperf2/warehouse/tablespace/external/hive/tpcds_3000_string_parquet_external.db/customer_iceberg_snapshot]),
> lookup_status:OK)
> Looks like Impala gets confused in FileSystemUtils.relativizePath() because
> the final directory component in the new snapshot table name
> (customer_iceberg_snapshot) is different from the corresponding path
> component in the file.
> {noformat}
> It looks like Impala gets confused in FileSystemUtils.relativizePath()
> because the final directory component in the new snapshot table name
> (customer_iceberg_snapshot) is different from the corresponding path
> component in the file.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
