On 26.08.23 05:39, j...@bloow.edu wrote:
On Friday, 25 August 2023 at 21:31:37 UTC, Ali Çehreli wrote:
On 8/25/23 14:27, j...@bloow.edu wrote:
> "A work unit is a set of consecutive elements of range to be
processed
> by a worker thread between communication with any other
thread. The
> number of elements processed per work unit is controlled by
the
> workUnitSize parameter. "
>
> So the question is how to rebalance these work units?
Ok, your question brings me back from summer hibernation. :)
This is what I do:
- Sort the tasks in decreasing time order; the ones that will take the
most time should go first.
- Use a work unit size of 1.
The longest running task will start first. You can't get better than
that. When I print some progress reporting, I see that most of the
time N-1 tasks have finished and we are waiting for that one longest
running task.
Ali
"back to sleep"
I do not know the amount of time they will run. They are files that are
being downloaded and I neither know the file size nor the download
rate(in fact, the actual download happens externally).
While I could use work unit of size 1 then problem then is I would be
downloading N files at once and that will cause other problems if N is
large(and sometimes it is).
There should be a "work unit size" and a "max simultaneous workers".
Then I could set the work unit size to 1 and say the max simultaneous
workers to 8 to get 8 simultaneous downloads without stalling.
I think thats what is implemented atm ...
`taskPool` creates a `TaskPool` of size `defaultPoolThreads` (defaulting
to totalCPUs - 1). The work unit size is only there to optimize for
small workloads where task / thread switching would be a big performance
problem (I guess). So in your case a work unit size of 1 should be good.
Did you try this already?
Kind regards,
Christian
