jackye1995 commented on a change in pull request #1891: URL: https://github.com/apache/iceberg/pull/1891#discussion_r553676871
########## File path: site/docs/aws.md ########## @@ -0,0 +1,187 @@ +<!-- + - Licensed to the Apache Software Foundation (ASF) under one or more + - contributor license agreements. See the NOTICE file distributed with + - this work for additional information regarding copyright ownership. + - The ASF licenses this file to You under the Apache License, Version 2.0 + - (the "License"); you may not use this file except in compliance with + - the License. You may obtain a copy of the License at + - + - http://www.apache.org/licenses/LICENSE-2.0 + - + - Unless required by applicable law or agreed to in writing, software + - distributed under the License is distributed on an "AS IS" BASIS, + - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + - See the License for the specific language governing permissions and + - limitations under the License. + --> + +# Iceberg AWS Integrations + +Iceberg provides integration with different AWS services through the `iceberg-aws` module. +This section describes how to use Iceberg with AWS. + +## Runtime Packages + +The `iceberg-aws` module is bundled with Spark and Flink engine runtimes. +However, the AWS clients are not bundled so that you can use the same client version as your application. +You will need to provide the AWS v2 SDK because that is what Iceberg depends on. +You can choose to use the [AWS SDK bundle](https://mvnrepository.com/artifact/software.amazon.awssdk/bundle), +or individual AWS client packages (Glue, S3, DynamoDB, KMS, STS) if you would like to have a minimal dependency footprint. + +For example, to use AWS features with Spark 3 and AWS clients version 2.15.40, you can start the SQL shell with: + +```sh +spark-sql --packages org.apache.iceberg:iceberg-spark3-runtime:0.11.0,software.amazon.awssdk:bundle:2.15.40 \ + --conf spark.sql.catalog.my_catalog=org.apache.iceberg.spark.SparkCatalog \ + --conf spark.sql.catalog.my_catalog.warehouse=s3://my-bucket/my/key/prefix \ + --conf spark.sql.catalog.my_catalog.catalog-impl=org.apache.iceberg.aws.glue.GlueCatalog \ + --conf spark.sql.catalog.my_catalog.lock.impl=org.apache.iceberg.aws.glue.DynamoLockManager \ + --conf spark.sql.catalog.my_catalog.lock.table=myGlueLockTable +``` + +As you can see, In the shell command, we use `--packages` to specify the additional AWS bundle dependency with its version as `2.15.40`. + +## Glue Catalog + +Iceberg enables the use of [AWS Glue](https://aws.amazon.com/glue) as the `Catalog` implementation. +When used, an Iceberg namespace is stored as a [Glue Database](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-databases.html), +an Iceberg table is stored as a [Glue Table](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-tables.html), +and every Iceberg table version is stored as a [Glue TableVersion](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-catalog-tables.html#aws-glue-api-catalog-tables-TableVersion). +You can start using Glue catalog by specifying the `catalog-impl` as `org.apache.iceberg.aws.glue.GlueCatalog`, +just like what is shown in the [runtime packages](#runtime-packages) section above. +More details about loading the catalog can be found in individual engine pages, such as [Spark](../spark/#loading-a-custom-catalog) and [Flink](../flink/#creating-catalogs-and-using-catalogs). + +### Glue Catalog ID + +You can specify the Glue catalog ID through `gluecatalog.id` catalog property to point to a Glue catalog in a different AWS account. +The Glue catalog ID is your numeric AWS account ID. +If the Glue catalog is in a different region, you should configure you AWS client to point to the correct region, +see more details in [AWS client customization](#aws-client-customization). + +### Skip Archive + +By default, Glue stores all the table versions created and user can rollback a table to any historical version if needed. +However, if you are streaming data to Iceberg, this will easily create a lot of Glue table versions. +Therefore, it is recommended to turn off the archive feature in Glue by setting `gluecatalog.skip-archive` to true. +For more details, please read [Glue Quotas](https://docs.aws.amazon.com/general/latest/gr/glue.html) and the [UpdateTable API](https://docs.aws.amazon.com/glue/latest/webapi/API_UpdateTable.html). + +### DynamoDB for commit locking + +Glue does not have a strong guarantee over concurrent updates to a table. +Although it throws `ConcurrentModificationException` when detecting two processes updating a table at the same time, +there is no guarantee that one update would not clobber the other update. +Therefore, DynamoDB can be used for Glue, so that for every commit, +`GlueCatalog` first obtains a lock using a helper DynamoDB table and then try to safely modify the Glue table. + +To enable this feature, use the lock related catalog properties: + +1. Set `lock.impl` as `org.apache.iceberg.aws.glue.DynamoLockManager`. +2. Set `lock.table` as the DynamoDB table name you would like to use. If the lock table with the given name does not exist in DynamoDB, a new table is created with billing mode set as [pay-per-request](https://aws.amazon.com/blogs/aws/amazon-dynamodb-on-demand-no-capacity-planning-and-pay-per-request-pricing). + +Other lock related catalog properties can also be used to adjust locking behaviors such as heartbeat interval. +For more details, please refer to [Lock catalog properties](../configuration/#lock-catalog-properties). + +### Warehouse Location + +By default, Glue only allows a warehouse location in S3 because of the use of `S3FileIO`. +To store data in a different local or cloud store, Glue catalog can switch to use `HadoopFileIO` +or any custom FileIO through the mechanism described in the [custom FileIO](../custom-catalog/#custom-file-io-implementation) section. + +### Table Location + +By default, the root location for a table `my_table` of namespace `my_ns` is at `my-warehouse-location/my-ns.db/my-table`. +This root location can be changed at both namespace and table level. + +To use a different path prefix for all tables under a namespace, use any AWS Glue client SDK you like to update the `locationUri` attribute of the corresponding Glue database. +For example, you can update the `locationUri` of `my_ns` to `s3://my-ns-bucket`, +then any newly created table will have a default root location under the new prefix. +For example, a new table `my_table_2` will have its root location at `s3://my-ns-bucket/my_table_2`. + +To use a completely different root path for a specific table, set the `location` table property to be the desired root path value. + +## S3 FileIO + +Iceberg allows users to write data to S3 through `S3FileIO`. +`GlueCatalog` by default uses this FileIO, and other catalogs can load this FileIO using the `io-impl` catalog property. + +### Progressive Multipart Upload + +`S3FileIO` implements a customized progressive multipart upload algorithm to upload data. +Data files are uploaded by parts in parallel as soon as each part is ready, +and each file part is deleted as soon as its upload process completes. +This provides maximized upload speed and minimized local disk usage during uploads. +Here are the configurations that users can tune related to this feature: + +| Property | Default | Description | +| --------------------------------- | -------------------------------------------------- | ------------------------------------------------------ | +| s3fileio.multipart.num-threads | the available number of processors in the system | number of threads to use for uploading parts to S3 (shared across all output streams) | +| s3fileio.multipart.part.size | 32MB | the size of a single part for multipart upload requests | +| s3fileio.multipart.threshold | 1.5 | the threshold expressed as a factor times the multipart size at which to switch from uploading using a single put object request to uploading using multipart upload | +| s3fileio.staging.dir | `java.io.tmpdir` property value | the directory to hold temporary files | Review comment: Looks like there are multiple discussions around the config key names, and these names for `s3fileio` are not designed by me but Daniel. Let me put up another thread for this discussion before release. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
