zjw1111 commented on code in PR #3457:
URL: https://github.com/apache/avro/pull/3457#discussion_r2268463573
##########
lang/c++/impl/DataFile.cc:
##########
@@ -491,22 +494,39 @@ void DataFileReaderBase::readDataBlock() {
if (decompressed_size == ZSTD_CONTENTSIZE_ERROR) {
throw Exception("ZSTD: Not a valid compressed frame");
} else if (decompressed_size == ZSTD_CONTENTSIZE_UNKNOWN) {
- throw Exception("ZSTD: Unable to determine decompressed size");
- }
-
- // Decompress the data
- uncompressed.clear();
- uncompressed.resize(decompressed_size);
- size_t result = ZSTD_decompress(
- uncompressed.data(), decompressed_size,
- reinterpret_cast<const char *>(compressed_.data()),
compressed_.size());
-
- if (ZSTD_isError(result)) {
- throw Exception("ZSTD decompression error: {}",
ZSTD_getErrorName(result));
- }
- if (result != decompressed_size) {
- throw Exception("ZSTD: Decompressed size mismatch: expected {},
got {}",
- decompressed_size, result);
+ // Stream decompress the data
+ ZSTD_DCtx *dctx = ZSTD_createDCtx();
+ if (!dctx) {
+ throw Exception("ZSTD decompression error: ZSTD_createDCtx()
failed");
+ }
+ ZSTD_inBuffer in{compressed_.data(), compressed_.size(), 0};
+ std::vector<char> tmp(ZSTD_DStreamOutSize());
+ ZSTD_outBuffer out{tmp.data(), tmp.size(), 0};
+ size_t ret;
+ do {
+ out.pos = 0;
+ ret = ZSTD_decompressStream(dctx, &out, &in);
Review Comment:
It tested by `testCompatibilityZstdCodec()`.
The current C++ impl uses batch writing (`ZSTD_compress`) to write
ZSTD-encoded files, rather than streaming writing (`ZSTD_compressStream2`).
Therefore, it is not possible to test `ZSTD_decompressStream` by first writing
and then reading within a C++ unittest. Instead, I test it using a ZSTD-encoded
file written in Java and then read it from C++.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@avro.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
