Perhaps I should clarify that (1) the goal would be to have blind-tee (and blind-cat if that is implemented) to use already existing functions that sends data between two files, and have these functions use splice when possible), so would only use tee explicitly, not splice, and (2) a 1 hour long blind video with the resolution 1920x1080@30 is 6.5TB large, so we are talking about a massive amount of data that is beeing sent between processes. In total, a 1 hour long video could require houndreds of terabytes being sent between process. Double that if it is a 60 fps video (which you will actually on the Internet nowadays) and multiple that by 4 for a 4K video, and double that again for a stereoscopic video (you can find all of these things on Blu-ray videos). So we could be talking about a couple of petabytes of data for a profession video, and 100TB for an amateur video, not just a couple of gigabytes. Try to copy that about of data with dd (in parallell of course), and you might find that even halving that time[1] would be nice, even if most if the time in the rendering process is usually spent on rendering effects, stacking videos, and transcoding.
[1] 42 hours per petabyte. ________________________________________ From: Mattias Andrée [[email protected]] Sent: 12 December 2017 11:32 To: hackers mail list Subject: RE: [hackers] [blind] update todo: tee is too slow || Mattias Andrée When I rendered a video, tee used 100% while the other process was basically at 0, for more than 50% of the rendering time. I ran it multiply times to verify that is was correct. An alternative solution would be to use sockets, but that would require changes to the shell. Optimising tee seems like the sensible alternative. Besides, normally when you use tools like tee and cat, you expect it to finish within milliseconds, even for larger files, here we are talking about time intervals between seconds and a few hours, with most if not all CPU:s at 90% to 100%, so optimisations do not hurt, especially not when as significant as using tee and splice. It may be unfortunate to have to use both tee–splice and read–write (which is required both for platforms not supporting tee and splice, and for file types not supporting them), but considering how little complexity this adds, — it's not much more than a duplication of a function, — I think it is a trade-off worth considering. ________________________________________ From: isabella parakiss [[email protected]] Sent: 12 December 2017 10:58 To: hackers mail list Subject: Re: [hackers] [blind] update todo: tee is too slow || Mattias Andrée bullshit On 12/4/17, [email protected] <[email protected]> wrote: > commit d8aa45da86d1128149fd7ab6ac3725bf8e88a1b1 > Author: Mattias Andrée <[email protected]> > AuthorDate: Mon Dec 4 22:35:59 2017 +0100 > Commit: Mattias Andrée <[email protected]> > CommitDate: Mon Dec 4 22:35:59 2017 +0100 > > update todo: tee is too slow > > Signed-off-by: Mattias Andrée <[email protected]> > > diff --git a/TODO b/TODO > index b0bbde7..408e942 100644 > --- a/TODO > +++ b/TODO > @@ -1,3 +1,7 @@ > +blind-tee (and tee(1)) is too slow (bottleneck) and must be reimplemented > +using tee(2) and splice(2). cat(1) may also be too slow, if this is the > +case, add blind-splice that just copies stdin to stdout using splice(2). > + > blind-transform affine transformation by matrix multiplication, > -[xy] for > tiling, -s for > improve quality on downscaling (pixels' > neighbours must not change) > blind-apply-map remap pixels (distortion) using the X and Y > values, -[xy] > for tiling, -s for > >
