FrankChen021 opened a new pull request #10203:
URL: https://github.com/apache/druid/pull/10203
### Description
This PR solves #9998
In Druid, there're lots of byte-related properties. In the world of big
data, the value of these properties are usually very large.
For example,
````
druid.server.maxSize=300000000000
````
At first glance of this configuration, it's hard to tell how many bytes
`druid.server.maxSize` hold. User have to count the number digit by digit to
know it's 300_000_000_000. Changing its value is also error-prone.
So this PR adds support of unit suffix to the value of these properties,
like how we specify the value of `Xms` or `Xmx` properties of JVM, to make
configurations more convenient and intuitive.
With this PR, we could set `druid.server.maxSize` as below:
````
druid.server.maxSize=300m
# or
druid.server.maxSize=300g
# or
druid.server.maxSize=3t
````
Because K(or M/G/T) is ambiguous in computer, whether it's 1000 or 1024 is
context related. This PR defines the supported units and their bases as follow
| Unit | Description | Base |
|---|---|---|
| K | Kilo Decimal Byte | 1000 |
| M | Mega Decimal Byte | 1000_000 |
| G | Giga Decimal Byte | 1000_000_000 |
| T | Tera Decimal Byte | 1000_000_000_1000 |
| P | Peta Decimal Byte | 1000_000_000_1000_1000 |
| KiB | Kilo Binary Byte | 1024 |
| MiB | Mega Binary Byte | 1024 * 1024 |
| GiB | Giga Binary Byte | 1024 * 1024 * 1024 |
| TiB | Tera Binary Byte | 1024 * 1024 * 1024 * 1024 |
| PiB | Peta Binary Byte | 1024 * 1024 * 1024 * 1024 |
So,
````
druid.server.maxSize=300m # 300_000_000 bytes
````
````
druid.server.maxSize=300MiB # 300*1024*1024 bytes
````
Of course, the old ways are still supported
````
druid.server.maxSize=300000000 # 300M bytes
````
### Implementation
This PR
1. adds a `Bytes` class to parse the new format of value and turns it into
long-format value. With this class, it's easy to turn number formatted property
into this new format
2. adds a `BytesRange` annotation to constraint the range of a `Bytes`
property
3. changes 8 properties to support this feature
- druid.cache.sizeInBytes
- druid.indexer.runner.maxZnodeBytes
- druid.processing.buffer.sizeBytes
- druid.server.maxSize
- druid.server.http.maxScatterGatherBytes
- druid.broker.http.maxQueuedBytes
- druid.processing.buffer.sizeBytes
- maxSize within druid.segmentCache.locations
4. changes the default value of properties above in example configuration
files to use the simplified format
<hr>
This PR has:
- [X] been self-reviewed.
- [X] added documentation for new or modified features or behaviors.
- [X] added Javadocs for most classes and all non-trivial methods. Linked
related entities via Javadoc links.
- [X] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold for [code
coverage](https://github.com/apache/druid/blob/master/dev/code-review/code-coverage.md)
is met.
- [X] been tested in a test Druid cluster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
