Your original reporting of 6.9s vs 59s (8x or more), compared to experiment #2
below where you report the gap closing to 23s vs 19s (20% gap), makes me think
that the biggest bang for buck is going to be either parallelizing across frame
range iterations (either entirely, or at least making the -o task asynchronous
so that it can get started on the next iteration even while it's waiting on the
I/O for the prior frame iterations). I'm sure there are other things we can do
to close the gap, but they're never going to make a dent in of the frame
parallelization issue without that being addressed explicitly.
I really appreciate your bringing this to our attention. Even though we use
frame range iteration extensively in production, I've always done detailed
benchmarks on individual frames, thinking that it's the individual commands
that should be optimized, and admit now that I've never really considered how
the frame iteration itself could introduce serialization that is hiding a
potentially huge performance gap. It's a very exciting prospect to think that a
large (potentially factor-of-several) speedup might be possible in many
real-world oiiotool use cases, just by changing the primary axis of concurrency
(parallelize frame range iterations rather than parallelizing pieces of the
image within each frame).
I'm neck deep in some other things at the moment, but as soon as I have that
off my plate, this is something I intend to return to and see if any fast
progress can be made that will have a big perf benefit to anybody using frame
ranges.
-- lg
> On May 9, 2023, at 1:06 PM, Simon Björk <bjork.si...@gmail.com> wrote:
>
> Thanks for the super detailed reply Larry, very interesting! It must have
> taken forever to write :).
>
> First of all I'm not complaining at all about oiiotool and I'm perfectly
> happy if the python bindings can be used to speed up the results (via
> multiprocessing). I just assumed it would be the other way around.
>
> I did to the test you asked for.
>
> 1. Crop a single (8K) exr file:
> oiiotool: 1.53s
> oiiotool (with modifiers): 1.27s
> python (no multiprocessing): 1.08s
>
> 2. Crop a sequence of (8K) exr files:
> oiiotool: 26.23s
> oiiotool (with modifiers): 23.43
> python (loop over files without multiprocessing): 19.36s
>
> So python still seems to be faster. One thing to note here is that I'm using
> a very old version of oiio (2.1.9) due to the hassle of getting stuff to work
> on Windows. Maybe the results come out different in newer versions.
>
> Regarding your thoughts about improving oiiotool I'm not sure. I mean it
> would be nice if oiiotool could operate in parall for sequences, but for me
> personally it's totally fine to use python for such cases (as I would imagine
> it would be a lot of work to get oiio to work that way).
>
> @Daniel: Thanks for the suggestion, I'll try to do some more tests!
>
> /Simon
>
>
>
>
> -------------------------------
> Simon Björk
> Compositor/TD
>
> +46 (0)70-2859503
> www.bjorkvisuals.com <http://www.bjorkvisuals.com/>
>
> Den tis 9 maj 2023 kl 21:19 skrev Daniel Flehner Heen
> <flehnerhee...@gmail.com <mailto:flehnerhee...@gmail.com>>:
> HI Simon!
>
> Not certain, but you might be running into something I've experienced before
> regarding image cache size(?). However, I experienced the opposite where
> Python was slower than oiiotool due to different defaults.
> For the long explanation from Larry please search for the email with this
> subject: "Python - ImageBufAlgo.channels taking it's time (OIIO - 2.3.9.1)"
>
> Have you tried upping your cache size (in MB) with the "--cache" flag?
> Docs:
> https://openimageio.readthedocs.io/en/v2.4.11.0/oiiotool.html#cmdoption-cache
> <https://openimageio.readthedocs.io/en/v2.4.11.0/oiiotool.html#cmdoption-cache>
> On Mon, May 8, 2023 at 11:41 AM Simon Björk <bjork.si...@gmail.com
> <mailto:bjork.si...@gmail.com>> wrote:
> I'm trying to crop a sequence of (8k) exr files and it seems like oiiotool is
> quite a bit slower than using the python bindings and mulitprocessing.
>
> Is this expected? I was under the assumption that it's always better/faster
> to use oiiotool if possible. I've tried changing the --threads argument but
> the results are the same. I'm on Windows.
>
> oiiotool path/to/src.3935-3954%06d.exr --crop 6640x5760+1000+0 --runstats -o
> path/to/dst.3935-3954%06d.exr
> --------------------------------
> Time: 59 seconds
> import sys
> import os
> import time
> from multiprocessing.dummy import Pool as ThreadPool
>
> import OpenImageIO as oiio
>
> oiio.attribute("threads", 1)
>
> def crop(path):
> im = oiio.ImageBuf(path)
> new_im = oiio.ImageBufAlgo.crop(im, oiio.ROI(1000, 7640, 0, 5760))
> new_filepath = "{0}/{1}".format("D:/tmp/exr_crop", os.path.basename(path))
> new_im.write(new_filepath)
>
> dir_path = "D:/tmp/exr_src"
> files = ["{0}/{1}".format(dir_path, x) for x in os.listdir(dir_path)]
>
> st = time.time()
>
> pool = ThreadPool(48)
> results = pool.map(crop, files)
>
> et = time.time()
>
> print("Total time: {0}".format(et-st))
>
> --------------------------------
> Time: 6.9 seconds
>
> /Simon
>
>
>
> -------------------------------
> Simon Björk
> Compositor/TD
>
> +46 (0)70-2859503
> www.bjorkvisuals.com
> <http://www.bjorkvisuals.com/>_______________________________________________
> Oiio-dev mailing list
> Oiio-dev@lists.openimageio.org <mailto:Oiio-dev@lists.openimageio.org>
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
>
>
> --
> -Daniel
> _______________________________________________
> Oiio-dev mailing list
> Oiio-dev@lists.openimageio.org <mailto:Oiio-dev@lists.openimageio.org>
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
> <http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org>
> _______________________________________________
> Oiio-dev mailing list
> Oiio-dev@lists.openimageio.org
> http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
--
Larry Gritz
l...@larrygritz.com
_______________________________________________
Oiio-dev mailing list
Oiio-dev@lists.openimageio.org
http://lists.openimageio.org/listinfo.cgi/oiio-dev-openimageio.org
