alamb opened a new issue, #11281:
URL: https://github.com/apache/datafusion/issues/11281
Further to the performance discussion @alamb - the
StringBuilder pattern you suggested in
https://github.com/apache/datafusion/pull/11136#discussion_r1657725214 does
seem to materially improve performance:
```
Extract data page statistics for String/extract_statistics/String
time: [15.368 µs 15.405 µs 15.446 µs]
change: [-68.672% -68.540% -68.409%] (p = 0.00 <
0.05)
Performance has improved.
Found 4 outliers among 100 measurements (4.00%)
4 (4.00%) high mild
```
So seems like a worthwhile thing to go ahead with? I think there are several
places where we can do something similar.
One question - I notice in that ticket that you appended nulls for missing
values. However, I think in most cases, missing values are simply omitted
because all the None values are removed by flattening. So, in general, users of
the data page statistics will need to check whether or not the length of the
array matches the number of actual data pages? This is different from how the
row group statistics are handled - they will instead have a null value for any
missing statistics.
Is this difference in behaviour expected or just a side effect of the
implementation.
_Originally posted by @efredine in
https://github.com/apache/datafusion/issues/10922#issuecomment-2207434579_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
