showuon commented on code in PR #14528: URL: https://github.com/apache/kafka/pull/14528#discussion_r1360448888
########## docs/ops.html: ########## @@ -3984,27 +3984,73 @@ <h5 class="anchor-heading"><a id="tiered_storage_config_topic" class="anchor-lin If unset, The value in <code>retention.ms</code> and <code>retention.bytes</code> will be used. </p> -<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">Configurations Example</a></h4> +<h4 class="anchor-heading"><a id="tiered_storage_config_ex" class="anchor-link"></a><a href="#tiered_storage_config_ex">Quick Start Example</a></h4> + +<p>Apache Kafka doesn't provide an out-of-the-box RemoteStorageManager implementation. To have a preview of the tiered storage + feature, the `LocalTieredStorage` implemented for integration test can be used, which will create a temporary directory in local storage + to simulate the remote storage. +</p> + +<p>To adopt the `LocalTieredStorage`, the test library needs to be built locally</p> +<pre>./gradlew :storage:testJar</pre> +<p>After build successfully, there should be a `kafka-storage-x.x.x-SNAPSHOT-test.jar` file under `storage/build/libs`. +Next, setting configurations in the broker side to enable tiered storage feature.</p> -<p>Here is a sample configuration to enable tiered storage feature in broker side: <pre> # Sample Zookeeper/Kraft broker server.properties listening on PLAINTEXT://:9092 remote.log.storage.system.enable=true -# Please provide the implementation for remoteStorageManager. This is the mandatory configuration for tiered storage. -# remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.NoOpRemoteStorageManager + # Using the "PLAINTEXT" listener for the clients in RemoteLogMetadataManager to talk to the brokers. remote.log.metadata.manager.listener.name=PLAINTEXT + +# Please provide the implementation info for remoteStorageManager. This is the mandatory configuration for tiered storage. +# Here, we use the `LocalTieredStorage` built above. +remote.log.storage.manager.class.name=org.apache.kafka.server.log.remote.storage.LocalTieredStorage +remote.log.storage.manager.class.path=/Users/luke/dev/kafka/storage/build/libs/kafka-storage-x.x.x-SNAPSHOT-test.jar + +# Configure the directory used for `LocalTieredStorage` +rsm.config.dir=/tmp/kafka-remote-storage + +# This needs to be changed if broker numbers are more than 1 +rlmm.config.remote.log.metadata.topic.replication.factor=1 </pre> </p> -<p>After broker is started, creating a topic with tiered storage enabled, and a small log time retention value to try this feature: -<pre>bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000 +<p>After broker is started, creating a topic with tiered storage enabled with configs: + +<pre> +# remote.storage.enable=true -> enabling tiered storage +# local.retention.ms=1000 -> the time before local segments are moving to remote storage (and then deleted) +# retention.ms=300000 -> when segments exceed this time, the segments in remote storage will be deleted +# segment.bytes=1000000 -> for test only, to speed up the log segment rolling interval +# file.delete.delay.ms=10000 -> for test only, to speed up the file delete delay + +bin/kafka-topics.sh --create --topic tieredTopic --bootstrap-server localhost:9092 --config remote.storage.enable=true --config local.retention.ms=1000 --config retention.ms=300000 --config segment.bytes=1000000 --config file.delete.delay.ms=1000 </pre> </p> +<p>Try to send messages to the `tieredTopic` topic to roll the log segment:</p> + +<pre> +bin/kafka-producer-perf-test.sh --topic tieredTopic --num-records 1000 --record-size 1024 --throughput -1 --producer-props bootstrap.servers=localhost:9092 Review Comment: Updated the num-records to 1200 and tested locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
