ashishkumar50 opened a new issue, #18921: URL: https://github.com/apache/iceberg/issues/18921
### Feature Request / Improvement **Background** Apache Iceberg supports different storage systems through its FileIO abstraction. These storage systems may expose the same underlying data through multiple access protocols or filesystem interfaces. For example, an object storage system may provide access through native filesystem APIs, S3-compatible APIs, or other storage-specific protocols. Different compute engines may prefer different access mechanisms depending on their runtime environment, performance requirements, and available integrations. Currently, Iceberg file locations are persisted in table metadata and are used by compute engines to access metadata files, manifest lists, manifests, data files, and delete files. When multiple engines access the same Iceberg table through different storage protocols, differences in URI schemes and filesystem implementations can create interoperability challenges. One catalog supplies one base location (one scheme). Relative paths resolve against the catalog-provided base; if that base is storage://…, an engine wanting a different physical protocol still needs runtime, engine-side translation. Existing v1–v3 tables (and absolute paths in v4) keep fully-qualified URIs. Relative paths only help new writes; this FileIO serves the installed base with no metadata rewrite. **Problem Statement** There is a need to decouple the storage location represented in Iceberg metadata from the protocol or filesystem implementation used by a compute engine to access the underlying data. A consistent, canonical URI representation would allow multiple compute engines to access the same Iceberg table through their preferred storage protocols without requiring protocol-specific locations to be persisted in table metadata. Without such a mechanism, tables created through one storage access protocol may not be directly accessible to engines configured to use another protocol, even when both protocols address the same underlying storage. Additionally, Iceberg Format V4 introduces support for relative file paths, where file locations can be stored relative to the table location. This requires URI resolution mechanisms to account for both relative and absolute paths while maintaining consistent storage access across different protocols. **Proposed Solution** Introduce a CanonicalUriFileIO implementation that separates the canonical URI representation persisted in Iceberg metadata from the physical URI used for storage operations. The implementation should provide a configurable mechanism to resolve canonical URIs into protocol-specific URIs at runtime, based on the storage access configuration of the compute engine. The proposed implementation should: 1. Maintain a consistent canonical URI representation in Iceberg table metadata, independent of the underlying storage access protocol. 2. Resolve canonical URIs to the appropriate physical storage URI at runtime. 3. Delegate filesystem operations to the configured underlying FileIO implementation. 4. Apply URI resolution consistently to all file operations performed through this FileIO — manifest lists, manifests, data files, delete files, and metadata files when written/read via this FileIO. 5. Ensure that newly written file locations preserve the appropriate logical representation (canonical absolute URI or relative path) according to the Iceberg table format version. 6. Support configurable URI mappings without introducing dependencies on any specific storage system or filesystem implementation. **Example** Consider a storage system that exposes the same underlying data through multiple protocols: ``` Compute engine Canonical URI in Iceberg metadata Physical access URI Engine A storage://warehouse/db/table/... protocol-a://warehouse/db/table/... Engine B storage://warehouse/db/table/... protocol-b://warehouse/db/table/... Engine C storage://warehouse/db/table/... protocol-c://warehouse/db/table/... ``` All engines operate on the same Iceberg table and share the same canonical metadata locations, while each engine resolves those locations to its configured storage access protocol. The URI mapping and underlying FileIO implementation should be configurable and extensible. **Expected Benefits** - Cross-engine interoperability: Allow different compute engines to access the same Iceberg tables using different storage protocols. - Protocol independence: Avoid coupling Iceberg metadata locations to a particular filesystem or storage access implementation. - Storage flexibility: Support storage systems exposing multiple access protocols or filesystem interfaces. - Consistent metadata: Maintain stable and consistent file locations across different compute engines. - Extensibility: Provide a reusable mechanism that can be adopted by different storage providers and integrations. - Reduced migration overhead: Avoid rewriting table metadata solely to accommodate a different storage access protocol, where the underlying locations can be resolved consistently. **Scope** - Introduce canonical URI resolution through the FileIO abstraction. - Support configurable URI scheme mappings. - Support delegation to different underlying FileIO implementations. - Ensure consistent URI resolution for read, write, delete, and other supported file operations. **Compatibility** The implementation should remain compatible with existing Iceberg table formats and catalog interfaces. No changes to the Iceberg table format specification should be required. Existing FileIO implementations and tables should continue to work without behavioral changes when canonical URI resolution is not configured. ### Query engine Spark ### Willingness to contribute - [x] I can contribute this improvement/feature independently - [ ] I would be willing to contribute this improvement/feature with guidance from the Iceberg community - [ ] I cannot contribute this improvement/feature at this time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
