This is an automated email from the ASF dual-hosted git repository.
dataroaring pushed a commit to branch branch-2.0
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-2.0 by this push:
new 8a5496f2f3e [cherry-pick](branch-2.0) Pick "[Fix](LZ4 compression) Fix
wrong LZ4 compression max input size limit (#41239)" (#41504)
8a5496f2f3e is described below
commit 8a5496f2f3e74a5783001f8153056727ff159410
Author: abmdocrt <[email protected]>
AuthorDate: Tue Oct 8 11:52:41 2024 +0800
[cherry-pick](branch-2.0) Pick "[Fix](LZ4 compression) Fix wrong LZ4
compression max input size limit (#41239)" (#41504)
## Proposed changes
LZ4 compression max supported value is LZ4_MAX_INPUT_SIZE, which is
0x7E000000(2,113,929,216 bytes). Doris use wrong max size INT_MAX, which
is 2,147,483,647, to check. If input data size is between this two size,
then it can pass the check but LZ4 compression will fail.
This PR fix it.
---
be/src/util/block_compression.cpp | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git a/be/src/util/block_compression.cpp
b/be/src/util/block_compression.cpp
index 7afbafa9790..1c2a65b5337 100644
--- a/be/src/util/block_compression.cpp
+++ b/be/src/util/block_compression.cpp
@@ -108,11 +108,13 @@ public:
}
Status compress(const Slice& input, faststring* output) override {
- if (input.size > INT_MAX) {
+ if (input.size > LZ4_MAX_INPUT_SIZE) {
return Status::InvalidArgument(
- "LZ4 not support those case(input.size>INT_MAX), maybe you
should change "
- "fragment_transmission_compression_codec to snappy,
size={}",
- input.size);
+ "LZ4 not support those
case(input.size>LZ4_MAX_INPUT_SIZE), maybe you should "
+ "change "
+ "fragment_transmission_compression_codec to snappy,
input.size={}, "
+ "LZ4_MAX_INPUT_SIZE={}",
+ input.size, LZ4_MAX_INPUT_SIZE);
}
Context* context;
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
