gripleaf commented on code in PR #242:
URL: https://github.com/apache/paimon-cpp/pull/242#discussion_r3843180401
##########
src/paimon/format/avro/avro_direct_decoder.cpp:
##########
@@ -57,6 +57,20 @@ Status DecodeFieldToBuilder(const ::avro::NodePtr& avro_node,
::avro::Decoder* decoder, arrow::ArrayBuilder*
array_builder,
AvroDirectDecoder::DecodeContext* ctx);
+Status ReserveBuilderCapacityImpl(arrow::ArrayBuilder* array_builder, int64_t
capacity) {
+ PAIMON_RETURN_NOT_OK_FROM_ARROW(array_builder->Reserve(capacity));
+ if (array_builder->type()->id() != arrow::Type::STRUCT) {
+ return Status::OK();
+ }
+
+ auto* struct_builder = checked_cast<arrow::StructBuilder*>(array_builder);
+ for (int32_t i = 0; i < struct_builder->num_fields(); ++i) {
+ PAIMON_RETURN_NOT_OK(
+ ReserveBuilderCapacityImpl(struct_builder->field_builder(i),
capacity));
+ }
+ return Status::OK();
+}
Review Comment:
before:
<img width="3988" height="1058" alt="image"
src="https://github.com/user-attachments/assets/f93657e9-e246-41e5-8405-ccddd8c2ae29";
/>
after:
<img width="3964" height="996" alt="image"
src="https://github.com/user-attachments/assets/a2f2b4e3-f8ed-47fd-8b4c-759e2643a840";
/>
<img width="1280" height="510" alt="image"
src="https://github.com/user-attachments/assets/0fc87b1a-bad4-4ea8-9a57-553833a563e5";
/>
> ReserveRowAlignedBuilderCapacity -> ReserveBuilderCapacityImpl
CPU Profile Comparison:
<img width="976" height="324" alt="image"
src="https://github.com/user-attachments/assets/dbeb7034-0411-4edb-8343-ca135e77db69";
/>
The optimized implementation reserves builder capacity before decoding.
This converts repeated reallocations into a small one-time allocation and
reduces the major resizing and reallocation hotspots by approximately 70%.
The profiles clearly demonstrate reduced memory-management CPU overhead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
