ermahesh opened a new pull request, #11219:
URL: https://github.com/apache/ozone/pull/11219
## What changes were proposed in this pull request?
Follow-up from HDDS-14524.
The `dfsrw` (`dfs-read-write-validator`) freon workload issues exactly one
read
after every write, so its read/write mix is pinned at 1:1. Real workloads are
rarely balanced, and the benchmark is more useful when the mix can be tuned
toward read-heavy or write-heavy.
This PR adds a `--read-write-ratio` option that sets how many reads each
write is
followed by:
| `--read-write-ratio` | effect |
| --- | --- |
| `1.0` (default) | one read per write, the workload as it is today |
| `4` | read-heavy: each write is read back four times |
| `0.25` | write-heavy: one read back every fourth write |
The existing per-path CRC32 validation and stale-read detection are
unaffected. A
read still picks a random file from the history of the thread that wrote it
and
compares against the checksum of the most recent write of that path, so both
corruption and an overwritten path returning older bytes are still detected.
### Why a ratio rather than a `--read-threads` / `--write-threads` split
A read-only thread would have no write history of its own, so it would have
to
read paths owned by other threads. Those paths are overwritten concurrently,
which is precisely the situation the stale-read check treats as a failure.
Splitting threads would therefore have meant weakening the validation, so
every
read is kept inside the thread that wrote the file.
### Fractional ratios
A ratio that is not a whole number is accumulated per thread rather than
rounded
on each write:
```java
private int readsDue(double readsPerWrite) {
readCredit += readsPerWrite;
int reads = (int) readCredit;
readCredit -= reads;
return reads;
}
```
Rounding per write would collapse every ratio below 0.5 to zero reads,
leaving a
"validator" that never validates. Carrying the remainder makes `0.25` read
back
every fourth write and `1.5` alternate between one and two reads. At the
default
of `1.0` the credit lands on exactly 1.0 every time, so the default path is
unchanged.
### Notes
* `-n` keeps its meaning: one task is one write, and the number of files
written
for a given `-n` does not change with the ratio.
* The `file-write` and `file-read-validate` timers are unchanged, so the two
sides of the mix stay separately measurable.
* A ratio that is not a positive finite number is rejected at startup,
alongside
the existing `--size`, `--buffer` and `--max-files-per-thread` checks.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16354
## How was this patch tested?
New unit test `TestHadoopFsReadWriteRatio` in `hadoop-ozone/freon`.
The existing integration test `TestHadoopFsReadWriteValidator` is unchanged
and
continues to cover the default behaviour against a real cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
