siddubakka commented on PR #12217:
URL: https://github.com/apache/arrow-rs/pull/12217#issuecomment-6984861725

   @etseidl Thanks for running these, the numbers are useful.
   
   You are right that this removes a real optimization. The regressions sit 
where I would expect them: `primitive_all_null` and `struct_all_null` default 
cases are up about 68 percent, `fsb` about 19, and the `map`, `struct_non_null` 
and `string_non_null` `ndv` cases in the 8 to 16 range. The suite total is 
roughly flat across the four shards though, and a few of the `ndv` high 
cardinality cases got faster, so the cost looks concentrated rather than broad. 
Either way it is a real cost.
   
   One clarification, not a counterargument: the compact level representation 
from #9831 stays, only the batch granularity in `write_batch_internal` changes.
   
   On the tradeoff, I think your read is the right one. The page row count 
limit is already best effort on the other two paths since pages fill to the 
next `write_batch_size` boundary, so matching that overflow behavior for 
compact data costs the optimization without buying a real guarantee. I am happy 
to write the docs section you described instead: a `WriterProperties` section 
covering how `write_batch_size`, `data_page_row_count_limit` and dictionary 
page size interact, with best effort wording, linked from the setters.
   
   @adriangb said earlier that fixing the implementation was better than 
documenting it. If you two still prefer the code fix I will keep this PR as is. 
Otherwise I will close it and open the docs PR against the issue. Which way 
would you like it?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to