Hello Zoltan Borok-Nagy, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/24071
to look at the new patch set (#13).
Change subject: IMPALA-14755:(part 1) Implement Puffin Blob reader and File
writer
......................................................................
IMPALA-14755:(part 1) Implement Puffin Blob reader and File writer
This is the first part of a multi-part implementation adding support for
Iceberg deletion vectors stored in Puffin files. This commit introduces
the core infrastructure for reading and writing Puffin format files
containing deletion vector blobs.
This commit adds:
- Generic BlobReader template base class for reading blob data from HDFS
with specialized DeletionVectorBlobReader for Puffin deletion vectors
- PuffinWriter that writes Puffin files with deletion vector blobs,
supporting merging of existing and new deletion vectors
- Puffin data structures (BlobMetadata, BlobData, File) and serialization
- Integration with table sink pipeline via new PUFFIN THdfsFileFormat
- Extended OutputPartition with PuffinWriteResult for tracking DV metadata
- CRC32 checksums for blob integrity and RoaringBitmap64::Or() for DV merging
- Updated Thrift/FlatBuffer schemas for deletion vector metadata
Testing:
- as this patch is a the first part of a multi-part implementation,
the functionality is validated only in part 2. For reader/writer
sanity checks, manual validation performed with Spark 3.5.5 with
Iceberg 1.10.1 installed.
Change-Id: I068a071f9db907064ccec8568db5234863eb4587
---
M be/CMakeLists.txt
M be/src/exec/CMakeLists.txt
A be/src/exec/blob-reader.h
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/iceberg-delete-builder.cc
M be/src/exec/output-partition.h
A be/src/exec/puffin/CMakeLists.txt
A be/src/exec/puffin/blob.h
A be/src/exec/puffin/puffin-writer.cc
A be/src/exec/puffin/puffin-writer.h
M be/src/exec/table-sink-base.cc
M be/src/service/query-options-test.cc
M be/src/util/hash-util.h
M be/src/util/roaring-bitmap-test.cc
M be/src/util/roaring-bitmap.h
M be/src/util/thash128-util.h
M common/fbs/IcebergObjects.fbs
M common/protobuf/planner.proto
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/DataSinks.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/Types.thrift
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/util/IcebergUtil.java
M testdata/workloads/functional-query/queries/QueryTest/set.test
26 files changed, 1,185 insertions(+), 15 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/24071/13
--
To view, visit http://gerrit.cloudera.org:8080/24071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I068a071f9db907064ccec8568db5234863eb4587
Gerrit-Change-Number: 24071
Gerrit-PatchSet: 13
Gerrit-Owner: Peter Rozsa <[email protected]>
Gerrit-Reviewer: Impala Public Jenkins <[email protected]>
Gerrit-Reviewer: Peter Rozsa <[email protected]>
Gerrit-Reviewer: Zoltan Borok-Nagy <[email protected]>
