[
https://issues.apache.org/jira/browse/HDFS-17977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18113009#comment-18113009
]
ASF GitHub Bot commented on HDFS-17977:
---------------------------------------
rdhabalia commented on code in PR #8719:
URL: https://github.com/apache/hadoop/pull/8719#discussion_r3962284880
##########
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/HdfsStressTest.java:
##########
@@ -0,0 +1,735 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hdfs;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.net.InetSocketAddress;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Properties;
+import java.util.Random;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ThreadLocalRandom;
+import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.AtomicReference;
+
+import org.apache.hadoop.classification.VisibleForTesting;
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.conf.Configured;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.fs.permission.FsPermission;
+import org.apache.hadoop.util.Tool;
+import org.apache.hadoop.util.ToolRunner;
+
+/**
+ * A compact, self-contained HDFS DataNode read/write stress tester.
+ *
+ * <p>{@code TestDFSIO} is the standard HDFS I/O benchmark, but it launches a
+ * MapReduce job scheduled by YARN over the whole cluster, which makes it hard
+ * to (a) generate a <em>controlled</em> QPS / throughput, (b) target a
specific
+ * subset of DataNodes (e.g. a replica set), and (c) obtain client-side latency
+ * <em>distributions</em> (p50/p95/p99). It also loads the entire cluster and
+ * takes a long time to saturate a targeted set of nodes. This tool fills that
+ * gap with a single-process, no-MapReduce load generator that:
+ *
+ * <ul>
+ * <li>drives a configurable, steady read and/or write throughput (MB/s) with
+ * a global rate limiter (optionally ramping up for acceleration
testing);</li>
+ * <li>targets specific DataNodes via HDFS favored-nodes hints so load lands
+ * on a chosen replica set;</li>
+ * <li>uses a configurable block/file size;</li>
+ * <li>runs a pre-test phase that pre-creates a large set of files so that
the
+ * measured reads are cold reads (larger than DataNode page cache);</li>
+ * <li>records client-side read/write latency distributions
+ * (p50, p75, p95, p99, min, max, mean, stddev) and effective QPS.</li>
+ * </ul>
+ *
+ * <h3>How to run</h3>
+ * <pre>
+ * # From a client host with the HDFS configuration on the classpath. The
+ * # tool ships in the hadoop-hdfs test jar:
+ * hadoop jar hadoop-hdfs-<version>-tests.jar \
+ * org.apache.hadoop.hdfs.HdfsStressTest /path/to/stress.properties
+ * </pre>
+ *
+ * <p>The single argument is a Java properties file describing the workload.
+ * Any property may also be supplied on the command line with
+ * {@code -D<key>=<value>} (ToolRunner options), which takes precedence.
+ *
+ * <h3>Example {@code stress.properties}</h3>
+ * <pre>
+ * # Target a specific replica set (host:port of the DataNode xfer port).
+ *
favoredDataNodes=dn1.example.com:9866,dn2.example.com:9866,dn3.example.com:9866
+ * replication=3
+ * blockSizeMB=128
+ *
+ * # Write workload.
+ * testWriteDirectory=/tmp/hdfs-stress/write
+ * writeThroughputMB=200
+ *
+ * # Read workload (cold reads from files created in the pre-test phase).
+ * testReadDirectories=/tmp/hdfs-stress/read
+ * readThroughputMB=200
+ *
+ * # Pre-test: create enough read files to exceed the DataNode page cache
+ * # (typically ~2x the DataNode memory) so reads are served from disk.
+ * testReadFileSizeGB=64
+ * preTestWriteThroughputMB=400
+ * preTestWriteDurationSeconds=600
+ *
+ * # Main measurement window.
+ * testDurationSeconds=300
+ *
+ * # Optional acceleration stress test: linearly ramp throughput from the
+ * # start value above to these end values over the test duration.
+ * endWriteThroughputMB=600
+ * endReadThroughputMB=600
+ *
+ * # -1 => auto (a fixed worker pool; the rate limiter enforces throughput).
+ * writeThreads=-1
+ * readThreads=-1
+ * </pre>
+ *
+ * <h3>Configuration reference</h3>
+ * <p>All keys are read from the properties file (or overridden with
+ * {@code -Dkey=value}). Sizes are plain numbers in the unit named by the key.
+ * <table border="1">
+ * <caption>HdfsStressTest properties</caption>
+ * <tr><th>Property</th><th>Default</th><th>Description</th></tr>
+ * <tr><td>{@code favoredDataNodes}</td><td>(none)</td>
+ * <td>Comma-separated {@code host:port} list of DataNode transfer ports.
+ * Written blocks are pinned to these nodes via HDFS favored-nodes hints,
+ * so load lands on a chosen replica set instead of the whole
cluster.</td></tr>
+ * <tr><td>{@code replication}</td><td>3</td>
+ * <td>Replication factor for files created by the tool.</td></tr>
+ * <tr><td>{@code blockSizeMB}</td><td>128</td>
+ * <td>Block/file size in MB. Each write and each read operation moves
one
+ * block-sized file, so this also sets the I/O unit for latency
stats.</td></tr>
+ * <tr><td>{@code testWriteDirectory}</td><td>(none)</td>
+ * <td>HDFS directory for the write workload. Omit to disable
writes.</td></tr>
+ * <tr><td>{@code writeThroughputMB}</td><td>0</td>
+ * <td>Target sustained write throughput in MB/s (0 disables
writes).</td></tr>
+ * <tr><td>{@code endWriteThroughputMB}</td><td>0 (no ramp)</td>
+ * <td>If greater than {@code writeThroughputMB}, throughput ramps
linearly
+ * from {@code writeThroughputMB} to this value over the run
(acceleration
+ * / find-the-knee test); otherwise throughput stays constant.</td></tr>
+ * <tr><td>{@code writeThreads}</td><td>-1 (auto)</td>
+ * <td>Writer worker threads; {@code -1} uses a fixed pool and lets the
+ * rate limiter govern throughput.</td></tr>
+ * <tr><td>{@code testReadDirectories}</td><td>(none)</td>
+ * <td>Comma-separated HDFS directories for the read workload and for the
+ * pre-test cold-read corpus. Omit to disable reads.</td></tr>
+ * <tr><td>{@code readThroughputMB}</td><td>0</td>
+ * <td>Target sustained read throughput in MB/s (0 disables
reads).</td></tr>
+ * <tr><td>{@code endReadThroughputMB}</td><td>0 (no ramp)</td>
+ * <td>Optional linear read-throughput ramp end value; ramps only when
+ * greater than {@code readThroughputMB} (see the write ramp).</td></tr>
+ * <tr><td>{@code readThreads}</td><td>-1 (auto)</td>
+ * <td>Reader worker threads; {@code -1} uses a fixed pool.</td></tr>
+ * <tr><td>{@code testReadFileSizeGB}</td><td>0</td>
+ * <td>Total size of the cold-read corpus to pre-create, in GB. Set this
+ * larger than the aggregate OS page cache of the target DataNodes (a
good
+ * rule of thumb is ~2x their RAM) so reads cannot be served from
cache.</td></tr>
+ * <tr><td>{@code preTestWriteThroughputMB}</td><td>0 (unlimited)</td>
+ * <td>Throughput (MB/s) used while generating the cold-read corpus. The
+ * default of {@code 0} (or any non-positive value) means "build the
+ * corpus as fast as the client can" (no pacing), which is usually what
+ * you want; set a positive value only to keep corpus creation from
itself
+ * saturating the cluster.</td></tr>
+ * <tr><td>{@code preTestWriteDurationSeconds}</td><td>0 (unbounded)</td>
+ * <td>Safety cap on the pre-test phase; it stops when either the corpus
+ * reaches {@code testReadFileSizeGB} or this many seconds
elapse.</td></tr>
+ * <tr><td>{@code testDurationSeconds}</td><td>60</td>
+ * <td>Length of the measured read/write window.</td></tr>
+ * </table>
+ *
+ * <h3>Cold reads: avoiding OS page-cache hits with a pre-test corpus</h3>
+ * <p>A read benchmark is only meaningful if it exercises the DataNode disks
+ * rather than the operating-system page cache. If reads keep hitting the same
+ * small set of recently written blocks, the DataNodes simply serve them from
+ * RAM and the numbers reflect memory bandwidth, not HDFS/disk performance.
+ *
+ * <p>To force cold reads, the pre-test phase writes a corpus of size
+ * {@code testReadFileSizeGB} <em>once</em>, sized deliberately larger than the
+ * combined page cache (main memory) of the target DataNodes. Because the
corpus
+ * does not fit in memory, the kernel continuously evicts older pages as newer
+ * blocks are written, so by the time the measured phase reads a given file
+ * again its pages have already been evicted and are no longer in the page
+ * cache. Every measured read therefore falls through to disk, giving a true
+ * cold-read result. The read workload also spreads its picks across the whole
+ * corpus (rather than replaying a hot subset) to keep the page-cache hit rate
+ * near zero. Pick {@code testReadFileSizeGB} at roughly twice the target
+ * DataNode RAM for a comfortable margin.
+ *
+ * <h3>Distributing load across multiple clients</h3>
+ * <p>A single client process is limited by its own CPU, NIC and JVM. To drive
+ * higher aggregate load, or to model many real writers/readers, run the tool
+ * on several client hosts at once against the same cluster; the total offered
+ * load is the sum of the per-client {@code writeThroughputMB} /
+ * {@code readThroughputMB}. Give each client a distinct
+ * {@code testWriteDirectory} (and, if pre-generating separate corpora,
distinct
+ * {@code testReadDirectories}) so the clients do not collide on paths, point
+ * them at the same {@code favoredDataNodes} to concentrate load on one replica
+ * set, and start them together so their measurement windows overlap. Aggregate
+ * the per-client latency distributions and throughput to get the cluster-wide
+ * result.
+ */
+public class HdfsStressTest extends Configured implements Tool {
+
+ private static final long MB = 1024L * 1024L;
+ private static final long GB = 1024L * MB;
+ private static final int IO_BUFFER_BYTES = (int) MB;
+ private static final int DEFAULT_AUTO_THREADS = 16;
+
+ // --
> HDFS-Test Add a compact standalone HDFS DataNode read/write stress tester
> -------------------------------------------------------------------------
>
> Key: HDFS-17977
> URL: https://issues.apache.org/jira/browse/HDFS-17977
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: benchmarks, hdfs, test
> Reporter: Rajan Dhabalia
> Priority: Major
> Labels: pull-request-available
>
> h2. Summary
> Add a compact, single-process HDFS read/write stress tester that generates
> controlled throughput against targeted DataNodes and reports client-side
> latency distributions.
> h2. Description
> h3. Motivation
> `TestDFSIO` is the standard HDFS I/O benchmark but is not well suited for
> targeted DataNode stress testing.
> Key limitations:
> * Requires MapReduce/YARN.
> * Load is distributed through MapReduce scheduling rather than directly
> targeting DataNodes.
> * Limited control over steady throughput/QPS against selected DataNodes.
> * Primarily reports aggregate throughput rather than client-side p50/p95/p99
> latency.
> * Does not provide a reliable mechanism for cold-read workloads targeting
> disk I/O.
> `HdfsStressTest` provides a lightweight standalone tool for controlled
> DataNode performance and stress testing.
> h3. Approach
> Introduce `HdfsStressTest`, a single-process load generator that depends only
> on the HDFS client.
> Key capabilities:
> # *Controlled throughput*
> ## Global token-bucket rate limiter.
> ## Configurable read/write throughput.
> ## Optional linear throughput ramp.
> # *Targeted DataNodes*
> ## Use HDFS favored-nodes hints to direct writes to selected DataNodes.
> # *Configurable I/O size*
> ## Configurable block/file size.
> ## Block-sized operations for consistent latency measurements.
> # *Cold-read workload*
> ## Pre-create a read corpus before testing.
> ## Corpus can exceed DataNode page cache to reduce cache effects.
> # *Latency and throughput metrics*
> ## p50, p75, p95, p99, min, max, mean, and standard deviation.
> ## Effective QPS and throughput for reads/writes.
> # *Client-side scale-out*
> ## Multiple clients can run concurrently.
> ## Aggregate load is the sum of configured load across clients.
> Configuration is provided through a Java properties file, with individual
> properties optionally overridden using `-D` through `ToolRunner`.
> h3. Configuration
> ||Property||Default||Description||
> |`favoredDataNodes`|None|Comma-separated DataNode host:port list used as
> favored nodes for writes|
> |`replication`|3|Replication factor for generated files|
> |`blockSizeMB`|128|Block/file size for read/write operations|
> |`testWriteDirectory`|None|HDFS directory for write workload; omit to disable
> writes|
> |`writeThroughputMB`|0|Target write throughput; `0` disables writes|
> |`endWriteThroughputMB`|0|Optional end value for linear write-throughput ramp|
> |`writeThreads`|-1|Number of writer threads; `-1` uses automatic
> configuration|
> |`testReadDirectories`|None|HDFS directories for read workload/cold-read
> corpus|
> |`readThroughputMB`|0|Target read throughput; `0` disables reads|
> |`endReadThroughputMB`|0|Optional end value for linear read-throughput ramp|
> |`readThreads`|-1|Number of reader threads; `-1` uses automatic configuration|
> |`testReadFileSizeGB`|0|Size of pre-created cold-read corpus; `0` disables
> corpus generation|
> |`preTestWriteThroughputMB`|0|Optional throughput limit for corpus generation|
> |`preTestWriteDurationSeconds`|0|Optional time limit for corpus generation|
> |`testDurationSeconds`|60|Duration of measured workload|
> h3. Example
> {code:java}
> hadoop jar hadoop-hdfs-<version>-tests.jar \
> org.apache.hadoop.hdfs.HdfsStressTest \
> /path/to/stress.properties
> {code}
> h3. Results
> The stress tester provides:
> * Controlled read/write load against targeted DataNodes.
> * Reproducible workloads against specific replica sets.
> * Cold-read workloads for storage-path testing.
> * Client-side latency distributions and tail latency.
> * Effective throughput and QPS measurements.
> * Execution without MapReduce/YARN.
> * Scale-out through multiple client processes.
> The tool complements rather than replaces `TestDFSIO`.
> h3. Comparison with TestDFSIO
> ||Dimension||TestDFSIO||HdfsStressTest||
> |Target specific DataNodes|Limited|Supported through favored-nodes hints|
> |Controlled throughput|Limited by MapReduce|Explicit control|
> |Throughput ramp|No|Supported|
> |Cold-read workload|Not specifically supported|Supported|
> |Client-side latency|Aggregate metrics|p50/p75/p95/p99 and other statistics|
> |External dependencies|MapReduce/YARN|HDFS client only|
> |Targeted DataNode stress|Difficult|Primary use case|
> |Scale-out|MapReduce-based|Multiple standalone clients|
> |Single-process execution|No|Yes|
> h3. Expected Impact
> Improves reproducibility and control for DataNode performance testing through:
> * Precise offered-load control.
> * Targeted DataNode/replica-set testing.
> * Repeatable cold-read workloads.
> * Client-side tail-latency measurements.
> * Lightweight execution without MapReduce/YARN.
> * Easy scale-out using multiple client processes.
> The primary benefit is {*}better observability and control during DataNode
> performance and stress testing{*}, rather than production throughput
> improvement.
> h2. Robustness and Input Validation
> The tool validates configuration at startup to prevent misleading benchmark
> results:
> * `blockSizeMB` must be positive.
> * Read workloads (`readThroughputMB > 0`) require a positive
> `testReadFileSizeGB`.
> * If `preTestWriteDurationSeconds` limits corpus generation before the
> requested `testReadFileSizeGB` is reached, the tool prints a *WARNING* that
> reads may be served from OS page cache and recommends increasing or removing
> the time limit.
> h2. Testing
> `TestHdfsStressTest` (MiniDFSCluster) covers:
> * Block-sized file creation with configured replication and payload.
> * Full-file reads to EOF.
> * Pre-test cold-read corpus generation across multiple directories.
> * Corpus generation duration limits and cache warnings.
> * Write-only execution through `ToolRunner`.
> `TestHdfsStressTestHelpers` provides cluster-free unit tests for:
> * Token-bucket rate limiting and pacing.
> * Runtime rate changes.
> * Latency conversion and sorting.
> * Bounded-memory reservoir sampling.
> * Configuration validation.
> * Valid read-only and write-only configurations.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
