[
https://issues.apache.org/jira/browse/HBASE-27264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17573733#comment-17573733
]
Bryan Beaudreault commented on HBASE-27264:
-------------------------------------------
My only thought here is that the existing unified.encoded.blocksize.ratio
config is a bit hard to configure, and now we're adding 2 more configs in a
similar area. I wonder if there's some sort of simplification we can do here to
make it easier on users. Often block encoding and compression go hand-in-hand.
Can we have a single unified config for them? Or is there some other easier way
to auto-tune these for users, or at least add logs/metrics to make it easier to
know what to set it to?
> Add options to consider compressed size when delimiting blocks during hfile
> writes
> ----------------------------------------------------------------------------------
>
> Key: HBASE-27264
> URL: https://issues.apache.org/jira/browse/HBASE-27264
> Project: HBase
> Issue Type: New Feature
> Reporter: Wellington Chevreuil
> Assignee: Wellington Chevreuil
> Priority: Major
>
> In HBASE-27232 we had modified "hbase.writer.unified.encoded.blocksize.ratio"
> property soo that it can allow for the encoded size to be considered when
> delimiting hfiles blocks during writes.
> Here we propose two additional properties,"hbase.block.size.limit.compressed"
> and "hbase.block.size.max.compressed" that would allow for consider the
> compressed size (if compression is in use) for delimiting blocks during hfile
> writing. When compression is enabled, certain datasets can have very high
> compression efficiency, so that the default 64KB block size and 10GB max file
> size can lead to hfiles with very large number of blocks.
> In this proposal, "hbase.block.size.limit.compressed" is a boolean flag that
> switches to compressed size for delimiting blocks, and
> "hbase.block.size.max.compressed" is an int with the limit, in bytes for the
> compressed block size, in order to avoid very large uncompressed blocks
> (defaulting to 320KB).
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
