elliotsteene-swap opened a new issue, #61388: URL: https://github.com/apache/doris/issues/61388
## Feature Request: Support `header.*` Properties for REST Iceberg Catalog ### Summary Apache Doris should support passing custom HTTP headers to the Iceberg REST catalog client via `header.<Header-Name>` catalog properties. This is part of the standard Iceberg REST catalog specification and is required to integrate with catalogs such as **GCP BigLake Metastore** that mandate custom headers for authentication and routing. ### Motivation and Use Case #### GCP BigLake Metastore [Google Cloud BigLake Metastore](https://cloud.google.com/bigquery/docs/iceberg-tables) exposes an Iceberg REST catalog endpoint but requires callers to include a `x-goog-user-project` header on every request to identify the billing project. Without this header, requests are rejected. The standard way to supply this in the Iceberg ecosystem is via the `header.x-goog-user-project` catalog property, which the Iceberg REST client translates into an HTTP header on all outgoing requests. Example of the expected Doris `CREATE CATALOG` syntax once this feature is supported: ```sql CREATE CATALOG biglake_catalog PROPERTIES ( 'type' = 'iceberg', 'iceberg.catalog.type' = 'rest', 'iceberg.rest.uri' = 'https://biglakeapi.googleapis.com/v1/projects/<PROJECT>/locations/<REGION>/catalogs/<CATALOG>', 'warehouse' = 'projects/<PROJECT>/locations/<REGION>/catalogs/<CATALOG>', 'header.x-goog-user-project' = '<BILLING_PROJECT>' ); ``` Without `header.*` support, it is impossible to use Doris with BigLake Metastore (and any other REST catalog that relies on custom request headers for auth/routing). ### Standard Iceberg Behaviour The `header.*` property prefix is a first-class convention in the Iceberg REST catalog client (`org.apache.iceberg.rest.RESTCatalog`). Any property whose key starts with `header.` is stripped of the prefix and injected as an HTTP header on every request the client makes. This behaviour is documented and used across the Iceberg ecosystem: - **Apache Iceberg Java / Spark** – `spark.sql.catalog.my_catalog.header.X-Catalog-Name=default` - **PyIceberg** – `header.x-goog-user-project`, `header.X-Iceberg-Access-Delegation`, etc. ([docs](https://py.iceberg.apache.org/configuration/)) - **Iceberg Rust client** – recently added the same feature ### Root Cause Doris appears to pass only an explicitly enumerated set of properties to the underlying `RESTCatalog` instance rather than forwarding all catalog properties. As a result, `header.*` keys are silently dropped before reaching the Iceberg client, even though the client itself is capable of processing them. The fix is to ensure that all catalog properties (or at minimum those with a `header.` prefix) are forwarded to the `restProperties` map that is handed to the Iceberg `RESTCatalog`. ### Expected Behaviour Any `header.<Name> = <value>` property specified in `CREATE CATALOG ... PROPERTIES (...)` should be forwarded to the Iceberg REST catalog client and sent as the HTTP header `<Name>: <value>` on all requests to the REST catalog endpoint. ### Alternatives Considered - **Network-layer proxy** – injecting headers via a sidecar/reverse-proxy is possible but fragile, adds operational complexity, and should not be required for a standard protocol feature. - **OAuth2 / SigV4** – these are separate authentication mechanisms and do not cover arbitrary header requirements such as billing-project routing. ### Environment - Apache Doris version: 3.x / 4.x (latest) - Feature affects: Iceberg REST catalog integration (`iceberg.catalog.type = 'rest'`) ### References - Apache Iceberg REST catalog spec – header properties - [PyIceberg configuration docs](https://py.iceberg.apache.org/configuration/) - [GCP BigLake Metastore Iceberg REST API](https://cloud.google.com/bigquery/docs/iceberg-tables) - Related Doris issue: #23671 (property pass-through for REST catalog) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
