0AyanamiRei opened a new pull request, #64711:
URL: https://github.com/apache/doris/pull/64711
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Stream load and HTTP stream load parse compressed CSV data
through a legacy format enum list when estimating async group commit content
length, while newer compress_type-based inputs such as ZSTD remain
FORMAT_CSV_PLAIN or FORMAT_JSON. This makes ZSTD stream loads unsupported by
the shared load-format parser and misses compressed-load handling in group
commit paths. This change teaches LoadUtil to parse ZSTD, centralizes
compressed load detection by compress_type plus legacy format types, applies it
to stream load and HTTP stream group commit planning, and adds zstd regression
coverage for CSV, JSON, group commit stream load, and group commit HTTP stream
paths.
### Release note
Support ZSTD compressed stream load and HTTP stream load inputs.
### Check List (For Author)
- Test:
- Unit Test: ./run-be-ut.sh --run --filter=LoadUtilTest.*
- Regression test: ./run-regression-test.sh --run -d load_p0/stream_load
-s test_stream_load_compress_type -c
'jdbc:mysql://127.0.0.1:33520/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round'
-ha 127.0.0.1:33500
- Regression test: ./run-regression-test.sh --run -d load_p0/stream_load
-s test_group_commit_stream_load -c
'jdbc:mysql://127.0.0.1:33520/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round'
-ha 127.0.0.1:33500
- Regression test: ./run-regression-test.sh --run -d load_p0/http_stream
-s test_group_commit_http_stream -c
'jdbc:mysql://127.0.0.1:33520/?useLocalSessionState=true&allowLoadLocalInfile=true&zeroDateTimeBehavior=round'
-ha 127.0.0.1:33500
- Static check: ./build-support/clang-format.sh &&
./build-support/check-format.sh
- Static check attempted:
CLANG_TIDY_BINARY=/data/data3/huangruixin/include/src-master/ldb_toolchain/bin/clang-tidy
./build-support/run-clang-tidy.sh (blocked by existing/environment
diagnostics: existing HTTP action and LoadUtilTest lint findings plus stddef.h
not found; new zstd helper warning fixed and load_util.cpp/load_util.h reported
clean)
- Behavior changed: Yes. Stream load and HTTP stream load now accept ZSTD as
compress_type and treat it as compressed input for async group commit content
length estimation.
- Does this need documentation: No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
