If I may ask while we're at it: can we make some option to have GNU Parallel
NOT buffer things and/or not output things? Some operations are heavy on I/O
and not computationally intensive, so there's no need for either buffering or
output from them :)
> Dnia 9 luty 2018 o 09:45 hubert depesz lubaczewski <dep...@depesz.com>
> On Thu, Feb 08, 2018 at 07:50:22PM +0100, Ole Tange wrote:
> > In practice this means you need 2.3 GB of free RAM and 3.8 GB of free
> > virtual memory (RAM+swap) to run this.
> > Given your output from 'free' I think that if you add 2 GB of swap,
> > then that will solve your issue. Just because Perl loves swap, it does
> > not mean that it will start swapping - it just wants to feel safe that
> > it could if it wanted to. I have seen the same behaviour of other
> > programs, so my guess is that you might get more performance from your
> > system simply by adding a little swap space.
> > The other option is to use --cat as described on
> > http://lists.gnu.org/archive/html/parallel/2018-01/msg00027.html
> Yes, but --cat creates temp files, which I can't have as it slows down
> things considerably.
> Anyway - looks like I simply can't use parallel. It's memory
> requirements are (from my perspective) impossible to handle, and I still
> don't understand what exactly is the benefit of buffering stuff in