RatulDawar commented on PR #22780:
URL: https://github.com/apache/datafusion/pull/22780#issuecomment-4638421894
@neilconway I reviewd it the fix looks good. One thing worth nothing here
is that this also improves the byte estimation.
Which looked like this earlier for 10M rows, 800MB input, 4 partitions,
Partial + COUNT(*) :
```
output_rows = input_rows = 10,000,000
total_byte_size = (input_bytes / input_rows) × output_rows
= (800,000,000 / 10,000,000) × 10,000,000
= 800,000,000
```
The newer better estimation looks like this:
```
output_rows = partition_count = 4
total_byte_size = (input_bytes / input_rows) × output_rows
= (800,000,000 / 10,000,000) × 4
= 80 × 4
= 320
```
Althougth the width of the estimation is still off as here the number of
output columns also converge, and would be off for very wide tables.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
