George: apologies for mis-identifying yourself as OP.
Israel:
On 22/02/19 6:04 AM, Israel Brewster wrote:
Actually not a ’toy example’ at all. It is simply the first step in
gridding some data I am working with - a problem that is solved by tools
like SatPy, but unfortunately I can’t use SatPy because it doesn’t
recognize my file format, and you can’t load data directly. Writing a
custom file importer for SatPy is probably my next step.
Not to focus on the word "toy", the governing issue is of setup cost cf
the acceleration afforded by the parallel processing. In this case, the
former is/was more-or-less as high as the latter, and your efforts were
insufficiently rewarded.
That said, if the computer was concurrently performing this task and a
number of others, the number of cores available to you would decrease.
At which point, speeds start heading backwards!
This is largely speculation because only you know the task, objectives,
and circumstances - however, for those 'playing along at home' and
learning from your experiment...
That said, the entire process took around 60 seconds to run. As this
step was taking 10, I figured it would be low-hanging fruit for speeding
up the process. Obviously I was wrong. For what it’s worth, I did manage
to re-factor the code, so instead of generating the entire grid
up-front, I generate the boxes as needed to calculate the overlap with
the data grid. This brought the processing time down to around 40
seconds, so a definite improvement there.
Doing it on-demand. Now you're talking! Plus, if you're able to 'fit'
the data into each box as it is created, that will help justify the
setup/tear-down overhead cost for each async process.
Well done!
---
Israel Brewster
Software Engineer
Alaska Volcano Observatory
Geophysical Institute - UAF
2156 Koyukuk Drive
Fairbanks AK 99775-7320
Work: 907-474-5172
cell: 907-328-9145
On Feb 20, 2019, at 4:30 PM, DL Neil <pythonl...@danceswithmice.info
<mailto:pythonl...@danceswithmice.info>> wrote:
George
On 21/02/19 1:15 PM, george trojan wrote:
def create_box(x_y):
return geometry.box(x_y[0] - 1, x_y[1], x_y[0], x_y[1] - 1)
x_range = range(1, 1001)
y_range = range(1, 801)
x_y_range = list(itertools.product(x_range, y_range))
grid = list(map(create_box, x_y_range))
Which creates and populates an 800x1000 “grid” (represented as a flat
list
at this point) of “boxes”, where a box is a shapely.geometry.box(). This
takes about 10 seconds to run.
Looking at this, I am thinking it would lend itself well to
parallelization. Since the box at each “coordinate" is independent of all
others, it seems I should be able to simply split the list up into chunks
and process each chunk in parallel on a separate core. To that end, I
created a multiprocessing pool:
I recall a similar discussion when folk were being encouraged to move
away from monolithic and straight-line processing to modular functions
- it is more (CPU-time) efficient to run in a straight line; than it
is to repeatedly call, set-up, execute, and return-from a function or
sub-routine! ie there is an over-head to many/all constructs!
Isn't the 'problem' that it is a 'toy example'? That the amount of
computing within each parallel process is small in relation to the
inherent 'overhead'.
Thus, if the code performed a reasonable analytical task within each
box after it had been defined (increased CPU load), would you then
notice the expected difference between the single- and multi-process
implementations?
From AKL to AK
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list
--
Regards =dn
--
https://mail.python.org/mailman/listinfo/python-list
