momo-jun commented on code in PR #15835: URL: https://github.com/apache/pulsar/pull/15835#discussion_r885147196
########## site2/docs/tiered-storage-s3.md: ########## @@ -0,0 +1,247 @@ +--- +id: tiered-storage-s3 +title: Use S3 offloader with Pulsar +sidebar_label: "S3 offloader" +--- + +S3 offloader is introduced to serve S3-compatible storage, which means that the storage employs the S3 API as its “language" and applications that speak the S3 API are able to plug and play with S3-compatible storage. + +This chapter guides you through every step of installing and configuring the S3 offloader and using it with Pulsar. + +## Installation + +Follow the steps below to install the S3 offloader. + +### Prerequisite + +- Pulsar: 2.9.2 or later versions + +### Steps + +This example uses Pulsar 2.9.2. + +1. [Download the Pulsar tarball](getting-started-standalone.md#install-pulsar-using-binary-release). + +2. Download and untar the Pulsar offloaders package, then copy the Pulsar offloaders as `offloaders` in the Pulsar directory. See [Install tiered storage offloaders](getting-started-standalone.md#install-tiered-storage-offloaders-optional). + + **Output** + + As shown from the output, Pulsar uses [Apache jclouds](https://jclouds.apache.org) to support [AWS S3](https://aws.amazon.com/s3/), [GCS](https://cloud.google.com/storage/), [Azure](https://portal.azure.com/#home), and [Aliyun OSS](https://www.aliyun.com/product/oss) for long-term storage. + + ``` + + tiered-storage-file-system-2.9.2.nar + tiered-storage-jcloud-2.9.2.nar + + ``` + + :::note + + * If you run Pulsar in a bare-metal cluster, ensure that `offloaders` tarball is unzipped in every broker's Pulsar directory. + * If you run Pulsar in Docker or deploy Pulsar using a Docker image (such as K8s and DCOS), you can use the `apachepulsar/pulsar-all` image. The `apachepulsar/pulsar-all` image has already bundled tiered storage offloaders. + + ::: + +## Configuration + +:::note + +Before offloading data from BookKeeper to S3-compatible storage, you need to configure some properties of the S3 offload driver. Besides, you can also configure the S3 offloader to run it automatically or trigger it manually. + +::: + + +### Configure S3 offloader driver + +You can configure the S3 offloader driver in the configuration file `broker.conf` or `standalone.conf`. + +- **Required** configurations are as below. + + | Required configuration | Description | Example value | Review Comment: Example value -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
