Re: parallelizing the archiver

[Andrey Borodin]

> 10 сент. 2021 г., в 22:18, Bossart, Nathan <bossa...@amazon.com> написал(а):
> 
> I was thinking that archive_batch_size would be the maximum batch
> size.  If the archiver only finds a single file to archive, that's all
> it'd send to the archive command.  If it finds more, it'd send up to
> archive_batch_size to the command.

I think that a concept of a "batch" is misleading.
If you pass filenames via stdin you don't need to know all names upfront.
Just send more names to the pipe if achiver_command is still running one more 
segments just became available.
This way level of parallelism will adapt to the workload.

Best regards, Andrey Borodin.