sakshichitnis27 commented on code in PR #7511:
URL: https://github.com/apache/datafusion-comet/pull/7511#discussion_r4566109237
##########
native/shuffle/src/spark_unsafe/row.rs:
##########
@@ -831,237 +739,34 @@ fn append_struct_fields_field_major(
fields: &arrow::datatypes::Fields,
) -> Result<(), CometError> {
let num_rows = row_end - row_start;
- let num_fields = fields.len();
-
- // First pass: Build struct validity and collect which structs are null
- // We use a Vec<bool> for simplicity; could use a bitset for better memory
+ let mut struct_addresses = Vec::with_capacity(num_rows);
+ let mut struct_sizes = Vec::with_capacity(num_rows);
let mut struct_is_null = Vec::with_capacity(num_rows);
for i in row_start..row_end {
read_row_at!(parent_row, row_addresses_ptr, row_sizes_ptr, i);
-
let is_null = parent_row.is_null_at(column_idx);
struct_is_null.push(is_null);
if is_null {
struct_builder.append_null();
+ struct_addresses.push(0);
Review Comment:
Added in af87dcdb, using your suggested wording.
I also updated the PR description per your review: it now says this is a
no-behavior-change refactor with neutral performance, and the `--profile dev`
figures are replaced with release-profile numbers. I built `main` and this
branch separately and ran `struct_conversion` interleaved. Nested and deeply
nested cases came out at 0.98 to 1.07 PR รท main per run. Flat came out neutral
to slightly faster (0.83 to 1.01, and 0.91 to 0.98 with the order reversed),
which I am not claiming as a speedup. @andygrove could you take another look
when you have a moment?
##########
native/shuffle/src/spark_unsafe/row.rs:
##########
@@ -369,7 +369,6 @@ pub(super) fn append_field(
///
/// Unlike `append_struct_fields_field_major`, this function takes slices of
row addresses,
Review Comment:
Thanks, updated in af87dcdb. The doc comment now says
`append_nested_struct_fields_field_major` is the single field-major
implementation for both top-level and nested structs, that the top-level caller
`append_struct_fields_field_major` builds the address, size and validity slices
from the parent rows first, and that nested structs build them from their own
rows and recurse. The stale "helper for nested structs" wording is gone.
@comphead could you take another look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
