Tim Rühsen <tim.rueh...@gmx.de> writes: > Thanks for the pointer to coproc, never heard of it ;-) (That means I > never had a problem that needed coproc). > > Anyways, copy&pasting the script results in a file '[1]' with bash 4.4.23.
Yeah, I'm not surprised there are bugs in it. > Also, wget -i - waits with downloading until stdin has been closed. How > can you circumvent that ? The more I think about the original problem, the more puzzled I am. The OP said that starting wget for each URL took a long time. But my experience is that starting processes is quite quick. (I once modified tar to compress each file individually with gzip before writing it to an Exabyte type. On a much slower processor than modern processors, the writing was not delayed by starting a process for each file written.) I suspect the delay is not starting wget but establishing the initial HTTP connection to the server. Probably a better approach to the problem is to download the files in batches on N consecutive URLs, where N is large enough that the HTTP startup time is well less than the total download time. Process each batch with a seperate invocation of wget, and exit the loop when an attempted batch doesn't create any new downloaded files (or, the last file in the batch doesn't exist), indicating there are no more files to download. Dale