gripleaf commented on code in PR #242:
URL: https://github.com/apache/paimon-cpp/pull/242#discussion_r3850629030
##########
src/paimon/core/manifest/manifest_file.cpp:
##########
@@ -93,8 +93,9 @@ Status ManifestFile::ReadBucketEntries(const std::string&
file_name, int32_t buc
file_name,
[this, bucket, entries](const std::shared_ptr<arrow::StructArray>&
batch) -> Status {
const arrow::ArrayVector& fields = batch->fields();
+ ColumnarRow row(fields, pool_, /*row_id=*/0);
Review Comment:
You are right that Validate() is O(k) rather than row-wise. However, our
workload scans thousands of small manifest files. Since batching does not cross
file boundaries and most files produce only one underfilled batch, Validate()
effectively runs once per file.
Each file also creates new Arrow DataType objects, so type fingerprints are
recomputed instead of reused. This is why Validate → TypeEquals →
LoadFingerprint accounts for about 25.9% of total CPU in the profile, despite
Validate() not scanning every row.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
