This is an automated email from the ASF dual-hosted git repository. github-actions[bot] pushed a commit to branch cherry-pick-becbd502-to-branch-1.3 in repository https://gitbox.apache.org/repos/asf/gravitino.git
commit 0f7a86fa592827a132d56f252f1d16085bd20474 Author: roryqi <[email protected]> AuthorDate: Tue Jun 9 22:33:48 2026 +0800 [MINOR] docs: Document default ResolvingFileIO (#11511) ### What changes were proposed in this pull request? Document the default `ResolvingFileIO` behavior for the Iceberg catalog and Iceberg REST service, including supported URI schemes, explicit overrides, and storage bundle requirements. ### Why are the changes needed? PR #10904 changed the default FileIO behavior, but the existing documentation still states that `io-impl` has no default value. ### Does this PR introduce _any_ user-facing change? No. This PR only documents the behavior introduced by #10904. ### How was this patch tested? - `./gradlew spotlessApply` - `./gradlew :docs:build` --- docs/iceberg-rest-service.md | 51 +++++++++++++++++++------------- docs/lakehouse-iceberg-catalog.md | 61 +++++++++++++++++++++++---------------- 2 files changed, 67 insertions(+), 45 deletions(-) diff --git a/docs/iceberg-rest-service.md b/docs/iceberg-rest-service.md index ed951ba5d6..511b0ead3f 100644 --- a/docs/iceberg-rest-service.md +++ b/docs/iceberg-rest-service.md @@ -440,14 +440,25 @@ Refer to [Access Control](./security/access-control.md) for the complete list of ### Storage +If `gravitino.iceberg-rest.io-impl` is not configured, the Iceberg REST service uses +`org.apache.iceberg.io.ResolvingFileIO`, which selects a `FileIO` implementation based +on the URI scheme: + +- S3: `s3`, `s3a`, or `s3n` +- OSS: `oss` +- GCS: `gs` or `gcs` +- ADLS: `abfs`, `abfss`, `wasb`, or `wasbs` +- To override the default, explicitly configure `gravitino.iceberg-rest.io-impl`. +- Ensure that the corresponding storage bundle is available in the Iceberg REST server classpath. + #### S3 -| Configuration item | Description | Default value | Required | Since Version | -|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|------------------------------------------------|------------------| -| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg, use `org.apache.iceberg.aws.s3.S3FileIO` for S3. | (none) | No | 0.6.0-incubating | -| `gravitino.iceberg-rest.s3-endpoint` | An alternative endpoint of the S3 service, This could be used for S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. | (none) | No | 0.6.0-incubating | -| `gravitino.iceberg-rest.s3-region` | The region of the S3 service, like `us-west-2`. | (none) | No | 0.6.0-incubating | -| `gravitino.iceberg-rest.s3-path-style-access` | Whether to use path style access for S3. | false | No | 0.9.0-incubating | +| Configuration item | Description | Default value | Required | Since Version | +|-----------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|----------|------------------| +| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg. Set it to `org.apache.iceberg.aws.s3.S3FileIO` to explicitly use S3FileIO. | `org.apache.iceberg.io.ResolvingFileIO` | No | 0.6.0-incubating | +| `gravitino.iceberg-rest.s3-endpoint` | An alternative endpoint of the S3 service, This could be used for S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. | (none) | No | 0.6.0-incubating | +| `gravitino.iceberg-rest.s3-region` | The region of the S3 service, like `us-west-2`. | (none) | No | 0.6.0-incubating | +| `gravitino.iceberg-rest.s3-path-style-access` | Whether to use path style access for S3. | false | No | 0.9.0-incubating | For other Iceberg s3 properties not managed by Gravitino like `s3.sse.type`, you could config it directly by `gravitino.iceberg-rest.s3.sse.type`. @@ -461,11 +472,11 @@ Refer to [S3 credentials](./security/credential-vending.md#s3-credentials) for c #### OSS -| Configuration item | Description | Default value | Required | Since Version | -|---------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|------------------------------------------------------|------------------| -| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg, use `org.apache.iceberg.aliyun.oss.OSSFileIO` for OSS. | (none) | No | 0.6.0-incubating | -| `gravitino.iceberg-rest.oss-endpoint` | The endpoint of Aliyun OSS service. | (none) | No | 0.7.0-incubating | -| `gravitino.iceberg-rest.oss-region` | The region of the OSS service, like `oss-cn-hangzhou`, only used when `credential-providers` is `oss-token`. | (none) | No | 0.8.0-incubating | +| Configuration item | Description | Default value | Required | Since Version | +|---------------------------------------|---------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|----------|------------------| +| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg. Set it to `org.apache.iceberg.aliyun.oss.OSSFileIO` to explicitly use OSSFileIO. | `org.apache.iceberg.io.ResolvingFileIO` | No | 0.6.0-incubating | +| `gravitino.iceberg-rest.oss-endpoint` | The endpoint of Aliyun OSS service. | (none) | No | 0.7.0-incubating | +| `gravitino.iceberg-rest.oss-region` | The region of the OSS service, like `oss-cn-hangzhou`, only used when `credential-providers` is `oss-token`. | (none) | No | 0.8.0-incubating | For other Iceberg OSS properties not managed by Gravitino like `client.security-token`, you could config it directly by `gravitino.iceberg-rest.client.security-token`. @@ -481,9 +492,9 @@ Please set the `gravitino.iceberg-rest.warehouse` parameter to `oss://{bucket_na Supports using static GCS credential file or generating GCS token to access GCS data. -| Configuration item | Description | Default value | Required | Since Version | -|---------------------------------------------------|----------------------------------------------------------------------------------------------------|---------------|----------|------------------| -| `gravitino.iceberg-rest.io-impl` | The io implementation for `FileIO` in Iceberg, use `org.apache.iceberg.gcp.gcs.GCSFileIO` for GCS. | (none) | No | 0.6.0-incubating | +| Configuration item | Description | Default value | Required | Since Version | +|----------------------------------|------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|----------|------------------| +| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg. Set it to `org.apache.iceberg.gcp.gcs.GCSFileIO` to explicitly use GCSFileIO. | `org.apache.iceberg.io.ResolvingFileIO` | No | 0.6.0-incubating | For other Iceberg GCS properties not managed by Gravitino like `gcs.project-id`, you could config it directly by `gravitino.iceberg-rest.gcs.project-id`. @@ -499,9 +510,9 @@ Please set `gravitino.iceberg-rest.warehouse` to `gs://{bucket_name}/${prefix_na #### ADLS -| Configuration item | Description | Default value | Required | Since Version | -|-----------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|------------------| -| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg, use `org.apache.iceberg.azure.adlsv2.ADLSFileIO` for ADLS. | (none) | Yes | 0.8.0-incubating | +| Configuration item | Description | Default value | Required | Since Version | +|----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|----------|------------------| +| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg. Set it to `org.apache.iceberg.azure.adlsv2.ADLSFileIO` to explicitly use ADLSFileIO. | `org.apache.iceberg.io.ResolvingFileIO` | No | 0.8.0-incubating | For other Iceberg ADLS properties not managed by Gravitino like `adls.read.block-size-bytes`, you could config it directly by `gravitino.iceberg-rest.adls.read.block-size-bytes`. @@ -523,9 +534,9 @@ Builds with Hadoop 2.10.x. There may be compatibility issues when accessing Hado For storages not managed by Gravitino directly, configure them through custom catalog properties. -| Configuration item | Description | Default value | Required | Since Version | -|----------------------------------|-----------------------------------------------------------------------------------------|---------------|----------|------------------| -| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg; use the fully qualified classname. | (none) | No | 0.6.0-incubating | +| Configuration item | Description | Default value | Required | Since Version | +|----------------------------------|---------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|----------|------------------| +| `gravitino.iceberg-rest.io-impl` | The IO implementation for `FileIO` in Iceberg. Use the fully qualified class name to override the default implementation. | `org.apache.iceberg.io.ResolvingFileIO` | No | 0.6.0-incubating | To pass custom properties such as `security-token` to your custom `FileIO`, configure them via `gravitino.iceberg-rest.security-token`. The `security-token` is included in the properties when the `FileIO` initialize method is invoked. diff --git a/docs/lakehouse-iceberg-catalog.md b/docs/lakehouse-iceberg-catalog.md index 7b2bd9f6a9..1ca30280fc 100644 --- a/docs/lakehouse-iceberg-catalog.md +++ b/docs/lakehouse-iceberg-catalog.md @@ -105,16 +105,27 @@ To access a non-default catalog, set `warehouse` to the catalog name. This uses #### S3 +If `io-impl` is not configured, the Iceberg catalog uses +`org.apache.iceberg.io.ResolvingFileIO`, which selects a `FileIO` implementation based +on the URI scheme: + +- S3: `s3`, `s3a`, or `s3n` +- OSS: `oss` +- GCS: `gs` or `gcs` +- ADLS: `abfs`, `abfss`, `wasb`, or `wasbs` +- To override the default, explicitly configure `io-impl`. +- Ensure that the corresponding storage bundle is available in the Iceberg catalog classpath. + Supports using static access-key-id and secret-access-key to access S3 data. -| Configuration item | Description | Default value | Required | Since Version | -|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------|----------|------------------| -| `io-impl` | The io implementation for `FileIO` in Iceberg, use `org.apache.iceberg.aws.s3.S3FileIO` for s3. | (none) | No | 0.6.0-incubating | -| `s3-access-key-id` | The static access key ID used to access S3 data. | (none) | No | 0.6.0-incubating | -| `s3-secret-access-key` | The static secret access key used to access S3 data. | (none) | No | 0.6.0-incubating | -| `s3-endpoint` | An alternative endpoint of the S3 service, This could be used for S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. | (none) | No | 0.6.0-incubating | -| `s3-region` | The region of the S3 service, like `us-west-2`. | (none) | No | 0.6.0-incubating | -| `s3-path-style-access` | Whether to use path style access for S3. | false | No | 0.9.0-incubating | +| Configuration item | Description | Default value | Required | Since Version | +|------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|----------|------------------| +| `io-impl` | The IO implementation for `FileIO` in Iceberg. Set it to `org.apache.iceberg.aws.s3.S3FileIO` to explicitly use S3FileIO. | `org.apache.iceberg.io.ResolvingFileIO` | No | 0.6.0-incubating | +| `s3-access-key-id` | The static access key ID used to access S3 data. | (none) | No | 0.6.0-incubating | +| `s3-secret-access-key` | The static secret access key used to access S3 data. | (none) | No | 0.6.0-incubating | +| `s3-endpoint` | An alternative endpoint of the S3 service, This could be used for S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. | (none) | No | 0.6.0-incubating | +| `s3-region` | The region of the S3 service, like `us-west-2`. | (none) | No | 0.6.0-incubating | +| `s3-path-style-access` | Whether to use path style access for S3. | false | No | 0.9.0-incubating | For other Iceberg s3 properties not managed by Gravitino like `s3.sse.type`, you could config it directly by `gravitino.bypass.s3.sse.type`. @@ -133,12 +144,12 @@ Since Gravitino 1.1.0, the Gravitino Iceberg AWS bundle jar has already included Gravitino Iceberg REST service supports using static access-key-id and secret-access-key to access OSS data. -| Configuration item | Description | Default value | Required | Since Version | -|-------------------------|-------------------------------------------------------------------------------------------------------|---------------|----------|------------------| -| `io-impl` | The IO implementation for `FileIO` in Iceberg, use `org.apache.iceberg.aliyun.oss.OSSFileIO` for OSS. | (none) | No | 0.6.0-incubating | -| `oss-access-key-id` | The static access key ID used to access OSS data. | (none) | No | 0.7.0-incubating | -| `oss-secret-access-key` | The static secret access key used to access OSS data. | (none) | No | 0.7.0-incubating | -| `oss-endpoint` | The endpoint of Aliyun OSS service. | (none) | No | 0.7.0-incubating | +| Configuration item | Description | Default value | Required | Since Version | +|-------------------------|---------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|----------|------------------| +| `io-impl` | The IO implementation for `FileIO` in Iceberg. Set it to `org.apache.iceberg.aliyun.oss.OSSFileIO` to explicitly use OSSFileIO. | `org.apache.iceberg.io.ResolvingFileIO` | No | 0.6.0-incubating | +| `oss-access-key-id` | The static access key ID used to access OSS data. | (none) | No | 0.7.0-incubating | +| `oss-secret-access-key` | The static secret access key used to access OSS data. | (none) | No | 0.7.0-incubating | +| `oss-endpoint` | The endpoint of Aliyun OSS service. | (none) | No | 0.7.0-incubating | For other Iceberg OSS properties not managed by Gravitino like `client.security-token`, you could config it directly by `gravitino.bypass.client.security-token`. @@ -154,9 +165,9 @@ Since Gravitino 1.1.0, the Gravitino Iceberg aliyun bundle jar has already inclu Supports using google credential file to access GCS data. -| Configuration item | Description | Default value | Required | Since Version | -|--------------------|----------------------------------------------------------------------------------------------------|---------------|----------|------------------| -| `io-impl` | The io implementation for `FileIO` in Iceberg, use `org.apache.iceberg.gcp.gcs.GCSFileIO` for GCS. | (none) | No | 0.6.0-incubating | +| Configuration item | Description | Default value | Required | Since Version | +|--------------------|------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|----------|------------------| +| `io-impl` | The IO implementation for `FileIO` in Iceberg. Set it to `org.apache.iceberg.gcp.gcs.GCSFileIO` to explicitly use GCSFileIO. | `org.apache.iceberg.io.ResolvingFileIO` | No | 0.6.0-incubating | For other Iceberg GCS properties not managed by Gravitino like `gcs.project-id`, you could config it directly by `gravitino.bypass.gcs.project-id`. @@ -174,11 +185,11 @@ Since Gravitino 1.1.0, the Gravitino Iceberg GCP bundle jar has already included Supports using Azure account name and secret key to access ADLS data. -| Configuration item | Description | Default value | Required | Since Version | -|------------------------------|-----------------------------------------------------------------------------------------------------------|---------------|----------|------------------| -| `io-impl` | The io implementation for `FileIO` in Iceberg, use `org.apache.iceberg.azure.adlsv2.ADLSFileIO` for ADLS. | (none) | No | 0.6.0-incubating | -| `azure-storage-account-name` | The static storage account name used to access ADLS data. | (none) | No | 0.8.0-incubating | -| `azure-storage-account-key` | The static storage account key used to access ADLS data. | (none) | No | 0.8.0-incubating | +| Configuration item | Description | Default value | Required | Since Version | +|------------------------------|-------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|----------|------------------| +| `io-impl` | The IO implementation for `FileIO` in Iceberg. Set it to `org.apache.iceberg.azure.adlsv2.ADLSFileIO` to explicitly use ADLSFileIO. | `org.apache.iceberg.io.ResolvingFileIO` | No | 0.6.0-incubating | +| `azure-storage-account-name` | The static storage account name used to access ADLS data. | (none) | No | 0.8.0-incubating | +| `azure-storage-account-key` | The static storage account key used to access ADLS data. | (none) | No | 0.8.0-incubating | For other Iceberg ADLS properties not managed by Gravitino like `adls.read.block-size-bytes`, you could config it directly by `gravitino.iceberg-rest.adls.read.block-size-bytes`. @@ -194,9 +205,9 @@ Since Gravitino 1.1.0, the Gravitino Iceberg Azure bundle jar has already includ For other storages that are not managed by Gravitino directly, you can manage them through custom catalog properties. -| Configuration item | Description | Default value | Required | Since Version | -|--------------------|-----------------------------------------------------------------------------------------|---------------|----------|------------------| -| `io-impl` | The IO implementation for `FileIO` in Iceberg; use the fully qualified classname. | (none) | No | 0.6.0-incubating | +| Configuration item | Description | Default value | Required | Since Version | +|--------------------|---------------------------------------------------------------------------------------------------------------------------|-----------------------------------------|----------|------------------| +| `io-impl` | The IO implementation for `FileIO` in Iceberg. Use the fully qualified class name to override the default implementation. | `org.apache.iceberg.io.ResolvingFileIO` | No | 0.6.0-incubating | To pass custom properties such as `security-token` to your custom `FileIO`, you can directly configure it by `gravitino.bypass.security-token`. `security-token` will be included in the properties when the initialize method of `FileIO` is invoked.
