avantgardnerio commented on PR #2106:
URL:
https://github.com/apache/datafusion-ballista/pull/2106#issuecomment-5024544895
I'm considering case B here, now that we have multi-partition-per-task,
there's some instances where it could be better to hash parition:
```
Case A: 1 input partition per task (4 tasks total)
- Sort-shuffle: 4 tasks × 1 input each = 4 files total. Downstream reader
for output k opens 4 files, seeks to index[k] in each.
- Hash-writer (M×K layout): 4 tasks × 4 outputs each = 16 files total.
Downstream reader for output k opens 4 files (one per producer task), reads
each whole.
Case B: 4 input partitions per task (1 task total, the
multi-partition-tasks payoff)
- Sort-shuffle: 1 task × 4 inputs = 4 files total. Downstream reader for
output k still opens 4 files with 4 index seeks — same file count as Case A.
- Hash-writer: 1 task × 4 outputs = 4 files total. Downstream reader for
output k opens 1 file, reads whole.
```
But what I was working towards anyway was removing that hashing from
`ShuffleWriter` entirely:
```
RepartitionExec(Hash) → ShuffleWriterExec(None)
```
Which I think this PR still allows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
