Eric Wong <[email protected]> writes: > > --------8<------- > Subject: [PATCH] git: fix asynchronous batching for deep pipelines > > ...By using non-blocking pipe writes. This avoids problems for > musl (and other libc) where getdelim(3) used by `git cat-file --batch*' > uses a smaller input buffer than glibc or FreeBSD libc. > > My key mistake was our check against MAX_INFLIGHT is only useful > for the initial batch of requests. It is not useful for > subsequent requests since git will drain the pipe at > unpredictable rates due to libc differences. > > To fix this problem, I initially tried to drain the read pipe > as long as readable data was pending. However, reading git > output without giving git more work would also limit parallelism > opportunities since we don't want git to sit idle, either. This > change ensures we keep both pipes reasonably full to reduce > stalls and maximize parallelism between git and public-inbox. > > While the limit set a few weeks ago in commit > 56e6e587745c (git: cap MAX_INFLIGHT value to POSIX minimum, 2022-12-21) > remains in place, any higher or lower limit will work. It may > be worth it to use an even lower limit to improve interactivity > w.r.t. Ctrl-C interrupts. > > I've tested the pre-56e6e587745c and even higher values on an > Alpine VM in the GCC Farm <https://cfarm.tetaneutral.net> > > Reported-by: Chris Brannon <[email protected]> > Tested-by: Chris Brannon <[email protected]> > Link: https://public-inbox.org/meta/[email protected]/T/ *SNIP*
Thanks! The conversion seems to have been successful, but it logged very many of the following: E: Resource temporarily unavailable at /usr/share/perl5/vendor_perl/PublicInbox/Git.pm line 320. I'm sorry, I didn't count how many. It seemed like multiple screenfuls at least. -- Chris
