It may look insane on the surface level. I am not commited to changes listed in the TODO, most of them are just ideas that I will have to evalute later whether they are worth implementing. Although I'm pretty convinced most of them are good. ________________________________________ From: isabella parakiss [[email protected]] Sent: 12 December 2017 13:31 To: hackers mail list Subject: Re: [hackers] [blind] update todo: tee is too slow || Mattias Andrée
don't you realize how insane the whole thing is? On 12/12/17, Mattias Andrée <[email protected]> wrote: > Perhaps I should clarify that (1) the goal would be to have blind-tee > (and blind-cat if that is implemented) to use already existing functions > that sends data between two files, and have these functions use splice > when possible), so would only use tee explicitly, not splice, and (2) > a 1 hour long blind video with the resolution 1920x1080@30 is 6.5TB > large, so we are talking about a massive amount of data that is beeing > sent between processes. In total, a 1 hour long video could require > houndreds of terabytes being sent between process. Double that if it is a > 60 fps video (which you will actually on the Internet nowadays) and > multiple that by 4 for a 4K video, and double that again for a stereoscopic > video (you can find all of these things on Blu-ray videos). So we could > be talking about a couple of petabytes of data for a profession video, > and 100TB for an amateur video, not just a couple of gigabytes. Try to > copy that about of data with dd (in parallell of course), and you might > find that even halving that time[1] would be nice, even if most if the time > in the rendering process is usually spent on rendering effects, stacking > videos, and transcoding. > > [1] 42 hours per petabyte. > > ________________________________________ > From: Mattias Andrée [[email protected]] > Sent: 12 December 2017 11:32 > To: hackers mail list > Subject: RE: [hackers] [blind] update todo: tee is too slow || Mattias > Andrée > > When I rendered a video, tee used 100% while the other process was > basically at 0, for more than 50% of the rendering time. I ran it multiply > times to verify that is was correct. An alternative solution would be > to use sockets, but that would require changes to the shell. Optimising > tee seems like the sensible alternative. Besides, normally when you use > tools like tee and cat, you expect it to finish within milliseconds, even > for larger files, here we are talking about time intervals between seconds > and a few hours, with most if not all CPU:s at 90% to 100%, so > optimisations > do not hurt, especially not when as significant as using tee and splice. It > may be unfortunate to have to use both tee–splice and read–write (which > is required both for platforms not supporting tee and splice, and for file > types not supporting them), but considering how little complexity this > adds, > — it's not much more than a duplication of a function, — I think it is a > trade-off worth considering. > > ________________________________________ > From: isabella parakiss [[email protected]] > Sent: 12 December 2017 10:58 > To: hackers mail list > Subject: Re: [hackers] [blind] update todo: tee is too slow || Mattias > Andrée > > bullshit > > On 12/4/17, [email protected] <[email protected]> wrote: >> commit d8aa45da86d1128149fd7ab6ac3725bf8e88a1b1 >> Author: Mattias Andrée <[email protected]> >> AuthorDate: Mon Dec 4 22:35:59 2017 +0100 >> Commit: Mattias Andrée <[email protected]> >> CommitDate: Mon Dec 4 22:35:59 2017 +0100 >> >> update todo: tee is too slow >> >> Signed-off-by: Mattias Andrée <[email protected]> >> >> diff --git a/TODO b/TODO >> index b0bbde7..408e942 100644 >> --- a/TODO >> +++ b/TODO >> @@ -1,3 +1,7 @@ >> +blind-tee (and tee(1)) is too slow (bottleneck) and must be >> reimplemented >> +using tee(2) and splice(2). cat(1) may also be too slow, if this is the >> +case, add blind-splice that just copies stdin to stdout using splice(2). >> + >> blind-transform affine transformation by matrix >> multiplication, -[xy] for >> tiling, -s for >> improve quality on downscaling (pixels' >> neighbours must not change) >> blind-apply-map remap pixels (distortion) using the X and Y >> values, -[xy] >> for tiling, -s for >> >> > > > >
