Script 'mail_helper' called by obssrc Hello community, here is the log from the commit of package gnu_parallel for openSUSE:Factory checked in at 2021-12-01 20:47:33 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/gnu_parallel (Old) and /work/SRC/openSUSE:Factory/.gnu_parallel.new.31177 (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "gnu_parallel" Wed Dec 1 20:47:33 2021 rev:70 rq:935004 version:20211122 Changes: -------- --- /work/SRC/openSUSE:Factory/gnu_parallel/gnu_parallel.changes 2021-10-11 15:32:01.402884034 +0200 +++ /work/SRC/openSUSE:Factory/.gnu_parallel.new.31177/gnu_parallel.changes 2021-12-02 02:13:37.610960479 +0100 @@ -1,0 +2,8 @@ +Sat Nov 27 12:51:44 UTC 2021 - Dirk M??ller <[email protected]> + +- update to 20211122: + * Bug fixes and man page updates. + * Don't postpone output until $jobslots jobs have been started. + * Bug fixes and man page updates. + +------------------------------------------------------------------- Old: ---- parallel-20210922.tar.bz2 parallel-20210922.tar.bz2.sig New: ---- parallel-20211122.tar.bz2 parallel-20211122.tar.bz2.sig ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ gnu_parallel.spec ++++++ --- /var/tmp/diff_new_pack.Tn8Yk7/_old 2021-12-02 02:13:38.210958649 +0100 +++ /var/tmp/diff_new_pack.Tn8Yk7/_new 2021-12-02 02:13:38.210958649 +0100 @@ -17,13 +17,12 @@ Name: gnu_parallel -Version: 20210922 +Version: 20211122 Release: 0 Summary: Shell tool for executing jobs in parallel License: GPL-3.0-or-later Group: Productivity/File utilities URL: https://www.gnu.org/software/parallel/ -#DL-URL: http://ftp.gnu.org/gnu/parallel/ Source: https://ftp.gnu.org/gnu/parallel/parallel-%version.tar.bz2 Source2: https://ftp.gnu.org/gnu/parallel/parallel-%version.tar.bz2.sig Source3: %name.keyring ++++++ parallel-20210922.tar.bz2 -> parallel-20211122.tar.bz2 ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/NEWS new/parallel-20211122/NEWS --- old/parallel-20210922/NEWS 2021-09-22 23:04:25.000000000 +0200 +++ new/parallel-20211122/NEWS 2021-11-23 00:26:33.000000000 +0100 @@ -1,13 +1,45 @@ -20210922 +20211122 New in this release: - * Bug fixes and man page updates. +News about GNU Parallel: + +* Otimizando o seu Hacking com o GNU Parallel - Mateus Buogo + https://www.youtube.com/watch?v=xIpPPZXDKGU + +* SMACK 12 - Do it faster! Simple ways to use all those cores (and + GPUs) efficiently. https://www.youtube.com/watch?v=lXDGY7NvOYk + + +20211022 + +New in this release: + +* Don't postpone output until $jobslots jobs have been started. + +* Bug fixes and man page updates. News about GNU Parallel: +* How to Install GNU Parallel using Anaconda | Linux + https://www.youtube.com/watch?v=UwDNVP-L0qA + +* Install GNU Parallel from source code | Linux + https://www.youtube.com/watch?v=PIyJH4ben5o + +* Compiling GNU Parallel on CentOS-7 + https://thelinuxcluster.com/2021/10/19/compiling-gnu-parallel-on-centos-7/ + + +20210922 + +New in this release: + +* Bug fixes and man page updates. + +News about GNU Parallel: * 3 Programs that will Level Up your Bash Game https://medium.com/codex/3-programs-that-will-level-up-your-bash-game-bf14e17e2291 diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/README new/parallel-20211122/README --- old/parallel-20210922/README 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/README 2021-11-23 00:35:26.000000000 +0100 @@ -57,11 +57,11 @@ Full installation of GNU Parallel is as simple as: - wget https://ftpmirror.gnu.org/parallel/parallel-20210922.tar.bz2 - wget https://ftpmirror.gnu.org/parallel/parallel-20210922.tar.bz2.sig - gpg parallel-20210922.tar.bz2.sig - bzip2 -dc parallel-20210922.tar.bz2 | tar xvf - - cd parallel-20210922 + wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2 + wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2.sig + gpg parallel-20211122.tar.bz2.sig + bzip2 -dc parallel-20211122.tar.bz2 | tar xvf - + cd parallel-20211122 ./configure && make && sudo make install @@ -70,11 +70,11 @@ If you are not root you can add ~/bin to your path and install in ~/bin and ~/share: - wget https://ftpmirror.gnu.org/parallel/parallel-20210922.tar.bz2 - wget https://ftpmirror.gnu.org/parallel/parallel-20210922.tar.bz2.sig - gpg parallel-20210922.tar.bz2.sig - bzip2 -dc parallel-20210922.tar.bz2 | tar xvf - - cd parallel-20210922 + wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2 + wget https://ftpmirror.gnu.org/parallel/parallel-20211122.tar.bz2.sig + gpg parallel-20211122.tar.bz2.sig + bzip2 -dc parallel-20211122.tar.bz2 | tar xvf - + cd parallel-20211122 ./configure --prefix=$HOME && make && make install Or if your system lacks 'make' you can simply copy src/parallel @@ -122,8 +122,8 @@ When using programs that use GNU Parallel to process data for publication please cite: - Tange, O. (2021, September 22). GNU Parallel 20210922 ('Vindelev'). - Zenodo. https://doi.org/10.5281/zenodo.5523272 + Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai'). + Zenodo. https://doi.org/10.5281/zenodo.5719513 Copyright (C) 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019, 2020, 2021 Ole Tange, http://ole.tange.dk and diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/configure new/parallel-20211122/configure --- old/parallel-20210922/configure 2021-09-22 23:18:57.000000000 +0200 +++ new/parallel-20211122/configure 2021-11-23 00:37:59.000000000 +0100 @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.69 for parallel 20210922. +# Generated by GNU Autoconf 2.69 for parallel 20211122. # # Report bugs to <[email protected]>. # @@ -579,8 +579,8 @@ # Identity of this package. PACKAGE_NAME='parallel' PACKAGE_TARNAME='parallel' -PACKAGE_VERSION='20210922' -PACKAGE_STRING='parallel 20210922' +PACKAGE_VERSION='20211122' +PACKAGE_STRING='parallel 20211122' PACKAGE_BUGREPORT='[email protected]' PACKAGE_URL='' @@ -1214,7 +1214,7 @@ # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures parallel 20210922 to adapt to many kinds of systems. +\`configure' configures parallel 20211122 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1281,7 +1281,7 @@ if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of parallel 20210922:";; + short | recursive ) echo "Configuration of parallel 20211122:";; esac cat <<\_ACEOF @@ -1357,7 +1357,7 @@ test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -parallel configure 20210922 +parallel configure 20211122 generated by GNU Autoconf 2.69 Copyright (C) 2012 Free Software Foundation, Inc. @@ -1374,7 +1374,7 @@ This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by parallel $as_me 20210922, which was +It was created by parallel $as_me 20211122, which was generated by GNU Autoconf 2.69. Invocation command line was $ $0 $@ @@ -2237,7 +2237,7 @@ # Define the identity of the package. PACKAGE='parallel' - VERSION='20210922' + VERSION='20211122' cat >>confdefs.h <<_ACEOF @@ -2880,7 +2880,7 @@ # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by parallel $as_me 20210922, which was +This file was extended by parallel $as_me 20211122, which was generated by GNU Autoconf 2.69. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -2942,7 +2942,7 @@ cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -parallel config.status 20210922 +parallel config.status 20211122 configured by $0, generated by GNU Autoconf 2.69, with options \\"\$ac_cs_config\\" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/configure.ac new/parallel-20211122/configure.ac --- old/parallel-20210922/configure.ac 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/configure.ac 2021-11-23 00:35:26.000000000 +0100 @@ -1,4 +1,4 @@ -AC_INIT([parallel], [20210922], [[email protected]]) +AC_INIT([parallel], [20211122], [[email protected]]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/env_parallel.1 new/parallel-20211122/src/env_parallel.1 --- old/parallel-20210922/src/env_parallel.1 2021-09-22 23:19:00.000000000 +0200 +++ new/parallel-20211122/src/env_parallel.1 2021-11-23 00:38:02.000000000 +0100 @@ -133,7 +133,7 @@ .\" ======================================================================== .\" .IX Title "ENV_PARALLEL 1" -.TH ENV_PARALLEL 1 "2021-09-22" "20210922" "parallel" +.TH ENV_PARALLEL 1 "2021-11-22" "20211122" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/env_parallel.ash new/parallel-20211122/src/env_parallel.ash --- old/parallel-20210922/src/env_parallel.ash 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/env_parallel.ash 2021-11-23 00:35:26.000000000 +0100 @@ -385,7 +385,7 @@ return 255 fi if [ "$_parset_NAME" = "--version" ] ; then - echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" + echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Foundation, Inc." echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/env_parallel.bash new/parallel-20211122/src/env_parallel.bash --- old/parallel-20210922/src/env_parallel.bash 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/env_parallel.bash 2021-11-23 00:35:26.000000000 +0100 @@ -384,7 +384,7 @@ return 255 fi if [ "$_parset_NAME" = "--version" ] ; then - echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" + echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Foundation, Inc." echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/env_parallel.dash new/parallel-20211122/src/env_parallel.dash --- old/parallel-20210922/src/env_parallel.dash 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/env_parallel.dash 2021-11-23 00:35:26.000000000 +0100 @@ -385,7 +385,7 @@ return 255 fi if [ "$_parset_NAME" = "--version" ] ; then - echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" + echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Foundation, Inc." echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/env_parallel.ksh new/parallel-20211122/src/env_parallel.ksh --- old/parallel-20210922/src/env_parallel.ksh 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/env_parallel.ksh 2021-11-23 00:35:26.000000000 +0100 @@ -363,7 +363,7 @@ return 255 fi if [ "$_parset_NAME" = "--version" ] ; then - echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" + echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Foundation, Inc." echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/env_parallel.mksh new/parallel-20211122/src/env_parallel.mksh --- old/parallel-20210922/src/env_parallel.mksh 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/env_parallel.mksh 2021-11-23 00:35:26.000000000 +0100 @@ -365,7 +365,7 @@ return 255 fi if [ "$_parset_NAME" = "--version" ] ; then - echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" + echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Foundation, Inc." echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/env_parallel.sh new/parallel-20211122/src/env_parallel.sh --- old/parallel-20210922/src/env_parallel.sh 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/env_parallel.sh 2021-11-23 00:35:26.000000000 +0100 @@ -390,7 +390,7 @@ return 255 fi if [ "$_parset_NAME" = "--version" ] ; then - echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" + echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Foundation, Inc." echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/env_parallel.zsh new/parallel-20211122/src/env_parallel.zsh --- old/parallel-20210922/src/env_parallel.zsh 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/env_parallel.zsh 2021-11-23 00:35:26.000000000 +0100 @@ -355,7 +355,7 @@ return 255 fi if [ "$_parset_NAME" = "--version" ] ; then - echo "parset 20210922 (GNU parallel `parallel --minversion 1`)" + echo "parset 20211122 (GNU parallel `parallel --minversion 1`)" echo "Copyright (C) 2007-2021 Ole Tange, http://ole.tange.dk and Free Software" echo "Foundation, Inc." echo "License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/niceload new/parallel-20211122/src/niceload --- old/parallel-20210922/src/niceload 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/niceload 2021-11-23 00:35:26.000000000 +0100 @@ -26,7 +26,7 @@ use strict; use Getopt::Long; $Global::progname="niceload"; -$Global::version = 20210922; +$Global::version = 20211122; Getopt::Long::Configure("bundling","require_order"); get_options_from_array(\@ARGV) || die_usage(); if($opt::version) { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/parallel new/parallel-20211122/src/parallel --- old/parallel-20210922/src/parallel 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/parallel 2021-11-23 00:35:26.000000000 +0100 @@ -566,14 +566,26 @@ "$file is not a seekable file."); ::wait_and_exit(255); } - my $header = find_header(\$buf,open_or_exit($file)); + + my $fh = open_or_exit($file); + my $firstlinelen = 0; + if($opt::skip_first_line) { + my $newline; + # Read a full line one byte at a time + while($firstlinelen += sysread($fh,$newline,1,0)) { + $newline eq "\n" and last; + } + } + my $header = find_header(\$buf,$fh); # find positions - my @pos = find_split_positions($file,int($Global::blocksize),$header); + my @pos = find_split_positions($file,int($Global::blocksize), + $header,$firstlinelen); # Make @cat_prepends my @cat_prepends = (); for(my $i=0; $i<$#pos; $i++) { push(@cat_prepends, - cat_partial($file, 0, length($header), $pos[$i], $pos[$i+1])); + cat_partial($file, $firstlinelen, $firstlinelen+length($header), + $pos[$i], $pos[$i+1])); } return @cat_prepends; } @@ -618,8 +630,8 @@ # $opt::recend # Returns: # @positions of block start/end - my($file, $block, $header) = @_; - my $headerlen = length $header; + my($file, $block, $header, $firstlinelen) = @_; + my $skiplen = $firstlinelen + length $header; my $size = -s $file; if(-b $file) { # $file is a blockdevice @@ -627,7 +639,8 @@ } $block = int $block; if($opt::groupby) { - return split_positions_for_group_by($file,$size,$block,$header); + return split_positions_for_group_by($file,$size,$block, + $header,$firstlinelen); } # The optimal dd blocksize for mint, redhat, solaris, openbsd = 2^17..2^20 # The optimal dd blocksize for freebsd = 2^15..2^17 @@ -637,8 +650,8 @@ my ($recstart,$recend) = recstartrecend(); my $recendrecstart = $recend.$recstart; my $fh = ::open_or_exit($file); - push(@pos,$headerlen); - for(my $pos = $block+$headerlen; $pos < $size; $pos += $block) { + push(@pos,$skiplen); + for(my $pos = $block+$skiplen; $pos < $size; $pos += $block) { my $buf; if($recendrecstart eq "") { # records ends anywhere @@ -745,14 +758,14 @@ return($v,$vpos); } - my ($file,$size,$block,$header) = @_; + my ($file,$size,$block,$header,$firstlinelen) = @_; my ($a,$b,$c,$apos,$bpos,$cpos); my @pos; $fh = open_or_exit($file); # Set $Global::group_by_column $Global::group_by_perlexpr group_by_loop($fh,$opt::recsep); # $xpos = linestart, $x = value at $xpos, $apos < $bpos < $cpos - $apos = length $header; + $apos = $firstlinelen + length $header; for(($a,$apos) = value_at($apos); $apos < $size;) { push @pos, $apos; $bpos = $apos + $block; @@ -967,6 +980,13 @@ my $in = *STDIN; my $timeout = $Global::blocktimeout; + if($opt::skip_first_line) { + my $newline; + # Read a full line one byte at a time + while(sysread($in,$newline,1,0)) { + $newline eq "\n" and last; + } + } my $header = find_header(\$buf,$in); my $anything_written; my $eof; @@ -2244,7 +2264,7 @@ sub init_globals() { # Defaults: - $Global::version = 20210922; + $Global::version = 20211122; $Global::progname = 'parallel'; $::name = "GNU Parallel"; $Global::infinity = 2**31; @@ -2275,14 +2295,16 @@ # = {.}.{+.} = {+/}/{/.}.{+.} # = {..}.{+..} = {+/}/{/..}.{+..} # = {...}.{+...} = {+/}/{/...}.{+...} - '{+/}' => 's:/[^/]*$::', - '{+.}' => 's:.*\.::', - '{+..}' => 's:.*\.([^.]*\.):$1:', - '{+...}' => 's:.*\.([^.]*\.[^.]*\.):$1:', - '{..}' => 's:\.[^/.]+$::; s:\.[^/.]+$::', - '{...}' => 's:\.[^/.]+$::; s:\.[^/.]+$::; s:\.[^/.]+$::', - '{/..}' => 's:.*/::; s:\.[^/.]+$::; s:\.[^/.]+$::', - '{/...}' => 's:.*/::; s:\.[^/.]+$::; s:\.[^/.]+$::; s:\.[^/.]+$::', + '{+/}' => 's:/[^/]*$:: || s:.*$::', + # a.b => b; a => '' + '{+.}' => 's:.*\.:: || s:.*$::', + # a.b.c => b.c; a.b => ''; a => '' + '{+..}' => 's:.*\.([^/.]+\.[^/.]+)$:$1: || s:.*$::', + '{+...}' => 's:.*\.([^/.]+\.[^/.]+\.[^/.]+)$:$1: || s:.*$::', + '{..}' => 's:\.[^/.]+\.[^/.]+$::', + '{...}' => 's:\.[^/.]+\.[^/.]+\.[^/.]+$::', + '{/..}' => 's:.*/::; s:\.[^/.]+\.[^/.]+$::', + '{/...}' => 's:.*/::; s:\.[^/.]+\.[^/.]+\.[^/.]+$::', # n choose k = Binomial coefficient '{choose_k}' => 'for $t (2..$#arg){ if($arg[$t-1] ge $arg[$t]) { skip() } }', # {##} = number of jobs @@ -5055,8 +5077,8 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - " Tange, O. (2021, September 22). GNU Parallel 20210922 ('Vindelev').", - " Zenodo. https://doi.org/10.5281/zenodo.5523272";, + " Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai').", + " Zenodo. https://doi.org/10.5281/zenodo.5719513";, "", # Before changing these lines, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice @@ -5088,8 +5110,8 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - " Tange, O. (2021, September 22). GNU Parallel 20210922 ('Vindelev').", - " Zenodo. https://doi.org/10.5281/zenodo.5523272";, + " Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai').", + " Zenodo. https://doi.org/10.5281/zenodo.5719513";, "", # Before changing these line, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and @@ -5214,20 +5236,20 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - "\@software{tange_2021_5523272,", + "\@software{tange_2021_5719513,", " author = {Tange, Ole},", - " title = {GNU Parallel 20210922 ('Vindelev')},", - " month = Sep,", + " title = {GNU Parallel 20211122 ('Peng Shuai')},", + " month = Nov,", " year = 2021,", " note = {{GNU Parallel is a general parallelizer to run", " multiple serial command line programs in parallel", " without changing them.}},", " publisher = {Zenodo},", - " doi = {10.5281/zenodo.5523272},", - " url = {https://doi.org/10.5281/zenodo.5523272}";, + " doi = {10.5281/zenodo.5719513},", + " url = {https://doi.org/10.5281/zenodo.5719513}";, "}", "", - "(Feel free to use \\nocite{tange_2021_5523272})", + "(Feel free to use \\nocite{tange_2021_5719513})", "", # Before changing these lines, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and @@ -7263,6 +7285,9 @@ $count_jobs_already_read = $Global::JobQueue->next_seq(); my $wait_time_for_getting_args = 0; my $start_time = time; + if($wanted_processes < $Global::infinity) { + $Global::dummy_jobs = 1; + } while(1) { $system_limit >= $wanted_processes and last; not $more_filehandles and last; @@ -11263,7 +11288,8 @@ } if($opt::sqlmaster) { - # Insert the V1..Vn for this $seq in SQL table instead of generating one + # Insert the V1..Vn for this $seq in SQL table + # instead of generating one $Global::sql->insert_records($self->seq(), $self->{'command'}, $self->{'arg_list_flat_orig'}); } @@ -11282,12 +11308,16 @@ *Arg::arg = $self->{'arg_list_flat_orig'}; my $quote_arg = ($Global::quote_replace and not $Global::quoting); + my $col; for my $perlexpr (keys %{$self->{'replacecount'}}) { - if($perlexpr =~ /^(\d+) /) { - # Positional - defined($record->[$1-1]) or next; - $self->{'len'}{$perlexpr} += - length $record->[$1-1]->replace($perlexpr,$quote_arg,$self); + if($perlexpr =~ /^(-?\d+)(?:\D.*|)$/) { + # Positional replacement string + # Deal with negative positional replacement string + $col = ($1 < 0) ? $1 : $1-1; + if(defined($record->[$col])) { + $self->{'len'}{$perlexpr} += + length $record->[$col]->replace($perlexpr,$quote_arg,$self); + } } else { for my $arg (@$record) { if(defined $arg) { @@ -11577,7 +11607,6 @@ # Worst case is BASE64 encoding 3 bytes -> 4 bytes $len = int($len*4/3); } - return $len; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/parallel.1 new/parallel-20211122/src/parallel.1 --- old/parallel-20210922/src/parallel.1 2021-09-21 22:36:45.000000000 +0200 +++ new/parallel-20211122/src/parallel.1 2021-11-07 22:53:23.000000000 +0100 @@ -133,7 +133,7 @@ .\" ======================================================================== .\" .IX Title "PARALLEL 1" -.TH PARALLEL 1 "2021-08-29" "20210822" "parallel" +.TH PARALLEL 1 "2021-10-24" "20211022" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l @@ -821,11 +821,11 @@ .Ve .Sp When used with \fB\-\-pipe\fR only pass full CSV-records. -.IP "\fB\-\-ctag\fR \fIstr\fR (beta testing)" 4 -.IX Item "--ctag str (beta testing)" +.IP "\fB\-\-ctag\fR \fIstr\fR" 4 +.IX Item "--ctag str" Color tag. See \fB\-\-tag\fR. -.IP "\fB\-\-ctagstring\fR \fIstr\fR (beta testing)" 4 -.IX Item "--ctagstring str (beta testing)" +.IP "\fB\-\-ctagstring\fR \fIstr\fR" 4 +.IX Item "--ctagstring str" Color tagstring. See \fB\-\-tagstring\fR. .IP "\fB\-\-delay\fR \fImytime\fR" 4 .IX Item "--delay mytime" @@ -1237,8 +1237,19 @@ .IX Item "-P N" .PD Number of jobslots on each machine. Run up to N jobs in parallel. 0 -means as many as possible. Default is 100% which will run one job per -\&\s-1CPU\s0 on each machine. +means as many as possible (this can take a while to +determine). Default is 100% which will run one job per \s-1CPU\s0 on each +machine. +.Sp +Due to a bug \fB\-j 0\fR will also evaluate replacement strings twice up +to the number of joblots: +.Sp +.Vb 4 +\& # This will not count from 1 but from number\-of\-jobslots +\& seq 10000 | parallel \-j0 echo \*(Aq{= $_ = $foo++; =}\*(Aq | head +\& # This will count from 1 +\& seq 10000 | parallel \-j100 echo \*(Aq{= $_ = $foo++; =}\*(Aq | head +.Ve .Sp If \fB\-\-semaphore\fR is set, the default is 1 thus making a mutex. .IP "\fB\-\-jobs\fR \fI+N\fR" 4 @@ -2728,8 +2739,8 @@ the jobs (with \fB\-\-timeout\fR, \fB\-\-memfree\fR, or \fB\-\-halt\fR). This is due to \s-1GNU\s0 \fBparallel\fR giving each child its own process group, which is then killed. Process groups are dependant on the tty. -.IP "\fB\-\-tag\fR (beta testing)" 4 -.IX Item "--tag (beta testing)" +.IP "\fB\-\-tag\fR" 4 +.IX Item "--tag" Tag lines with arguments. Each output line will be prepended with the arguments and \s-1TAB\s0 (\et). When combined with \fB\-\-onall\fR or \fB\-\-nonall\fR the lines will be prepended with the sshlogin instead. @@ -2737,8 +2748,8 @@ \&\fB\-\-tag\fR is ignored when using \fB\-u\fR. .Sp \&\fB\-\-ctag\fR gives the tag a color. -.IP "\fB\-\-tagstring\fR \fIstr\fR (beta testing)" 4 -.IX Item "--tagstring str (beta testing)" +.IP "\fB\-\-tagstring\fR \fIstr\fR" 4 +.IX Item "--tagstring str" Tag lines with a string. Each output line will be prepended with \&\fIstr\fR and \s-1TAB\s0 (\et). \fIstr\fR can contain replacement strings such as \&\fB{}\fR. @@ -3323,7 +3334,7 @@ .Ve .PP If your environment (aliases, variables, and functions) is small you -can copy the full environment without having to \fBexport \-f\fR +can copy the full environment without having to \fBexport \-f \fR anything. See \fBenv_parallel\fR. .SS "\s-1EXAMPLE:\s0 Function tester" .IX Subsection "EXAMPLE: Function tester" @@ -5028,6 +5039,10 @@ \& true >jobqueue; tail \-n+0 \-f jobqueue | parallel \-S .. .Ve .PP +Output only will be printed when reading the next input after a job +has finished: So you need to submit a job after the first has finished +to see the output from the first job. +.PP If you keep this running for a long time, jobqueue will grow. A way of removing the jobs already run is by making \s-1GNU\s0 \fBparallel\fR stop when it hits a special value and then restart. To use \fB\-\-eof\fR to make \s-1GNU\s0 @@ -5060,20 +5075,6 @@ .PP \&\s-1GNU\s0 \fBparallel\fR discovers if \fBjobfile\fR or \fB~/.parallel/sshloginfile\fR changes. -.PP -There is a a small issue when using \s-1GNU\s0 \fBparallel\fR as queue -system/batch manager: You have to submit JobSlot number of jobs before -they will start, and after that you can submit one at a time, and job -will start immediately if free slots are available. -.PP -Output from the running or completed jobs are held back and will only -be printed when the next job is started (unless you use \-\-ungroup or -\&\-\-line\-buffer, in which case the output from the jobs are printed -immediately). -.PP -E.g. if you have 10 jobslots then the output from the first completed -job will only be printed when job 11 has started, and the output of -second completed job will only be printed when job 12 has started. .SS "\s-1EXAMPLE: GNU\s0 Parallel as dir processor" .IX Subsection "EXAMPLE: GNU Parallel as dir processor" If you have a dir in which users drop files that needs to be processed diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/parallel.html new/parallel-20211122/src/parallel.html --- old/parallel-20210922/src/parallel.html 2021-09-21 22:36:45.000000000 +0200 +++ new/parallel-20211122/src/parallel.html 2021-11-07 22:53:22.000000000 +0100 @@ -792,13 +792,13 @@ <p>When used with <b>--pipe</b> only pass full CSV-records.</p> </dd> -<dt id="ctag-str-beta-testing"><b>--ctag</b> <i>str</i> (beta testing)</dt> +<dt id="ctag-str"><b>--ctag</b> <i>str</i></dt> <dd> <p>Color tag. See <b>--tag</b>.</p> </dd> -<dt id="ctagstring-str-beta-testing"><b>--ctagstring</b> <i>str</i> (beta testing)</dt> +<dt id="ctagstring-str"><b>--ctagstring</b> <i>str</i></dt> <dd> <p>Color tagstring. See <b>--tagstring</b>.</p> @@ -1247,7 +1247,14 @@ <dt id="P-N"><b>-P</b> <i>N</i></dt> <dd> -<p>Number of jobslots on each machine. Run up to N jobs in parallel. 0 means as many as possible. Default is 100% which will run one job per CPU on each machine.</p> +<p>Number of jobslots on each machine. Run up to N jobs in parallel. 0 means as many as possible (this can take a while to determine). Default is 100% which will run one job per CPU on each machine.</p> + +<p>Due to a bug <b>-j 0</b> will also evaluate replacement strings twice up to the number of joblots:</p> + +<pre><code> # This will not count from 1 but from number-of-jobslots + seq 10000 | parallel -j0 echo '{= $_ = $foo++; =}' | head + # This will count from 1 + seq 10000 | parallel -j100 echo '{= $_ = $foo++; =}' | head</code></pre> <p>If <b>--semaphore</b> is set, the default is 1 thus making a mutex.</p> @@ -2604,7 +2611,7 @@ <p>Using <b>--tty</b> unfortunately means that GNU <b>parallel</b> cannot kill the jobs (with <b>--timeout</b>, <b>--memfree</b>, or <b>--halt</b>). This is due to GNU <b>parallel</b> giving each child its own process group, which is then killed. Process groups are dependant on the tty.</p> </dd> -<dt id="tag-beta-testing"><b>--tag</b> (beta testing)</dt> +<dt id="tag"><b>--tag</b></dt> <dd> <p>Tag lines with arguments. Each output line will be prepended with the arguments and TAB (\t). When combined with <b>--onall</b> or <b>--nonall</b> the lines will be prepended with the sshlogin instead.</p> @@ -2614,7 +2621,7 @@ <p><b>--ctag</b> gives the tag a color.</p> </dd> -<dt id="tagstring-str-beta-testing"><b>--tagstring</b> <i>str</i> (beta testing)</dt> +<dt id="tagstring-str"><b>--tagstring</b> <i>str</i></dt> <dd> <p>Tag lines with a string. Each output line will be prepended with <i>str</i> and TAB (\t). <i>str</i> can contain replacement strings such as <b>{}</b>.</p> @@ -3127,7 +3134,7 @@ <pre><code> parallel --env doit -S server doit ::: 1 2 3 parallel --env doubleit -S server doubleit ::: 1 2 3 ::: a b</code></pre> -<p>If your environment (aliases, variables, and functions) is small you can copy the full environment without having to <b>export -f</b> anything. See <b>env_parallel</b>.</p> +<p>If your environment (aliases, variables, and functions) is small you can copy the full environment without having to <b>export -f </b> anything. See <b>env_parallel</b>.</p> <h2 id="EXAMPLE:-Function-tester">EXAMPLE: Function tester</h2> @@ -4352,6 +4359,8 @@ <pre><code> true >jobqueue; tail -n+0 -f jobqueue | parallel -S ..</code></pre> +<p>Output only will be printed when reading the next input after a job has finished: So you need to submit a job after the first has finished to see the output from the first job.</p> + <p>If you keep this running for a long time, jobqueue will grow. A way of removing the jobs already run is by making GNU <b>parallel</b> stop when it hits a special value and then restart. To use <b>--eof</b> to make GNU <b>parallel</b> exit, <b>tail</b> also needs to be forced to exit:</p> <pre><code> true >jobqueue; @@ -4377,12 +4386,6 @@ <p>GNU <b>parallel</b> discovers if <b>jobfile</b> or <b>~/.parallel/sshloginfile</b> changes.</p> -<p>There is a a small issue when using GNU <b>parallel</b> as queue system/batch manager: You have to submit JobSlot number of jobs before they will start, and after that you can submit one at a time, and job will start immediately if free slots are available.</p> - -<p>Output from the running or completed jobs are held back and will only be printed when the next job is started (unless you use --ungroup or --line-buffer, in which case the output from the jobs are printed immediately).</p> - -<p>E.g. if you have 10 jobslots then the output from the first completed job will only be printed when job 11 has started, and the output of second completed job will only be printed when job 12 has started.</p> - <h2 id="EXAMPLE:-GNU-Parallel-as-dir-processor">EXAMPLE: GNU Parallel as dir processor</h2> <p>If you have a dir in which users drop files that needs to be processed you can do this on GNU/Linux (If you know what <b>inotifywait</b> is called on other platforms file a bug report):</p> Binary files old/parallel-20210922/src/parallel.pdf and new/parallel-20211122/src/parallel.pdf differ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/parallel.pod new/parallel-20211122/src/parallel.pod --- old/parallel-20210922/src/parallel.pod 2021-08-29 22:30:00.000000000 +0200 +++ new/parallel-20211122/src/parallel.pod 2021-10-24 20:39:11.000000000 +0200 @@ -743,12 +743,12 @@ When used with B<--pipe> only pass full CSV-records. -=item B<--ctag> I<str> (beta testing) +=item B<--ctag> I<str> Color tag. See B<--tag>. -=item B<--ctagstring> I<str> (beta testing) +=item B<--ctagstring> I<str> Color tagstring. See B<--tagstring>. @@ -1197,8 +1197,17 @@ =item B<-P> I<N> Number of jobslots on each machine. Run up to N jobs in parallel. 0 -means as many as possible. Default is 100% which will run one job per -CPU on each machine. +means as many as possible (this can take a while to +determine). Default is 100% which will run one job per CPU on each +machine. + +Due to a bug B<-j 0> will also evaluate replacement strings twice up +to the number of joblots: + + # This will not count from 1 but from number-of-jobslots + seq 10000 | parallel -j0 echo '{= $_ = $foo++; =}' | head + # This will count from 1 + seq 10000 | parallel -j100 echo '{= $_ = $foo++; =}' | head If B<--semaphore> is set, the default is 1 thus making a mutex. @@ -2710,7 +2719,7 @@ then killed. Process groups are dependant on the tty. -=item B<--tag> (beta testing) +=item B<--tag> Tag lines with arguments. Each output line will be prepended with the arguments and TAB (\t). When combined with B<--onall> or B<--nonall> @@ -2721,7 +2730,7 @@ B<--ctag> gives the tag a color. -=item B<--tagstring> I<str> (beta testing) +=item B<--tagstring> I<str> Tag lines with a string. Each output line will be prepended with I<str> and TAB (\t). I<str> can contain replacement strings such as @@ -3284,7 +3293,7 @@ parallel --env doubleit -S server doubleit ::: 1 2 3 ::: a b If your environment (aliases, variables, and functions) is small you -can copy the full environment without having to B<export -f> +can copy the full environment without having to B<export -f > anything. See B<env_parallel>. @@ -4773,6 +4782,10 @@ true >jobqueue; tail -n+0 -f jobqueue | parallel -S .. +Output only will be printed when reading the next input after a job +has finished: So you need to submit a job after the first has finished +to see the output from the first job. + If you keep this running for a long time, jobqueue will grow. A way of removing the jobs already run is by making GNU B<parallel> stop when it hits a special value and then restart. To use B<--eof> to make GNU @@ -4802,20 +4815,6 @@ GNU B<parallel> discovers if B<jobfile> or B<~/.parallel/sshloginfile> changes. -There is a a small issue when using GNU B<parallel> as queue -system/batch manager: You have to submit JobSlot number of jobs before -they will start, and after that you can submit one at a time, and job -will start immediately if free slots are available. - -Output from the running or completed jobs are held back and will only -be printed when the next job is started (unless you use --ungroup or ---line-buffer, in which case the output from the jobs are printed -immediately). - -E.g. if you have 10 jobslots then the output from the first completed -job will only be printed when job 11 has started, and the output of -second completed job will only be printed when job 12 has started. - =head2 EXAMPLE: GNU Parallel as dir processor diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/parallel.rst new/parallel-20211122/src/parallel.rst --- old/parallel-20210922/src/parallel.rst 2021-08-30 00:00:54.000000000 +0200 +++ new/parallel-20211122/src/parallel.rst 2021-11-07 22:53:22.000000000 +0100 @@ -883,13 +883,13 @@ -- \ **--ctag**\ \ *str*\ (beta testing) +- \ **--ctag**\ \ *str*\ Color tag. See \ **--tag**\ . -- \ **--ctagstring**\ \ *str*\ (beta testing) +- \ **--ctagstring**\ \ *str*\ Color tagstring. See \ **--tagstring**\ . @@ -1452,8 +1452,21 @@ - \ **-P**\ \ *N*\ Number of jobslots on each machine. Run up to N jobs in parallel. 0 - means as many as possible. Default is 100% which will run one job per - CPU on each machine. + means as many as possible (this can take a while to + determine). Default is 100% which will run one job per CPU on each + machine. + + Due to a bug \ **-j 0**\ will also evaluate replacement strings twice up + to the number of joblots: + + + .. code-block:: perl + + # This will not count from 1 but from number-of-jobslots + seq 10000 | parallel -j0 echo '{= $_ = $foo++; =}' | head + # This will count from 1 + seq 10000 | parallel -j100 echo '{= $_ = $foo++; =}' | head + If \ **--semaphore**\ is set, the default is 1 thus making a mutex. @@ -3309,7 +3322,7 @@ -- \ **--tag**\ (beta testing) +- \ **--tag**\ Tag lines with arguments. Each output line will be prepended with the arguments and TAB (\t). When combined with \ **--onall**\ or \ **--nonall**\ @@ -3321,7 +3334,7 @@ -- \ **--tagstring**\ \ *str*\ (beta testing) +- \ **--tagstring**\ \ *str*\ Tag lines with a string. Each output line will be prepended with \ *str*\ and TAB (\t). \ *str*\ can contain replacement strings such as @@ -4114,7 +4127,7 @@ If your environment (aliases, variables, and functions) is small you -can copy the full environment without having to \ **export -f**\ +can copy the full environment without having to \ **export -f **\ anything. See \ **env_parallel**\ . @@ -6378,6 +6391,10 @@ true >jobqueue; tail -n+0 -f jobqueue | parallel -S .. +Output only will be printed when reading the next input after a job +has finished: So you need to submit a job after the first has finished +to see the output from the first job. + If you keep this running for a long time, jobqueue will grow. A way of removing the jobs already run is by making GNU \ **parallel**\ stop when it hits a special value and then restart. To use \ **--eof**\ to make GNU @@ -6415,20 +6432,6 @@ GNU \ **parallel**\ discovers if \ **jobfile**\ or \ **~/.parallel/sshloginfile**\ changes. -There is a a small issue when using GNU \ **parallel**\ as queue -system/batch manager: You have to submit JobSlot number of jobs before -they will start, and after that you can submit one at a time, and job -will start immediately if free slots are available. - -Output from the running or completed jobs are held back and will only -be printed when the next job is started (unless you use --ungroup or ---line-buffer, in which case the output from the jobs are printed -immediately). - -E.g. if you have 10 jobslots then the output from the first completed -job will only be printed when job 11 has started, and the output of -second completed job will only be printed when job 12 has started. - EXAMPLE: GNU Parallel as dir processor ====================================== diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/parallel.texi new/parallel-20211122/src/parallel.texi --- old/parallel-20210922/src/parallel.texi 2021-09-21 22:36:47.000000000 +0200 +++ new/parallel-20211122/src/parallel.texi 2021-11-07 22:53:27.000000000 +0100 @@ -825,13 +825,13 @@ When used with @strong{--pipe} only pass full CSV-records. -@item @strong{--ctag} @emph{str} (beta testing) -@anchor{@strong{--ctag} @emph{str} (beta testing)} +@item @strong{--ctag} @emph{str} +@anchor{@strong{--ctag} @emph{str}} Color tag. See @strong{--tag}. -@item @strong{--ctagstring} @emph{str} (beta testing) -@anchor{@strong{--ctagstring} @emph{str} (beta testing)} +@item @strong{--ctagstring} @emph{str} +@anchor{@strong{--ctagstring} @emph{str}} Color tagstring. See @strong{--tagstring}. @@ -1326,8 +1326,19 @@ @anchor{@strong{-P} @emph{N}} Number of jobslots on each machine. Run up to N jobs in parallel. 0 -means as many as possible. Default is 100% which will run one job per -CPU on each machine. +means as many as possible (this can take a while to +determine). Default is 100% which will run one job per CPU on each +machine. + +Due to a bug @strong{-j 0} will also evaluate replacement strings twice up +to the number of joblots: + +@verbatim + # This will not count from 1 but from number-of-jobslots + seq 10000 | parallel -j0 echo '{= $_ = $foo++; =}' | head + # This will count from 1 + seq 10000 | parallel -j100 echo '{= $_ = $foo++; =}' | head +@end verbatim If @strong{--semaphore} is set, the default is 1 thus making a mutex. @@ -2964,8 +2975,8 @@ to GNU @strong{parallel} giving each child its own process group, which is then killed. Process groups are dependant on the tty. -@item @strong{--tag} (beta testing) -@anchor{@strong{--tag} (beta testing)} +@item @strong{--tag} +@anchor{@strong{--tag}} Tag lines with arguments. Each output line will be prepended with the arguments and TAB (\t). When combined with @strong{--onall} or @strong{--nonall} @@ -2975,8 +2986,8 @@ @strong{--ctag} gives the tag a color. -@item @strong{--tagstring} @emph{str} (beta testing) -@anchor{@strong{--tagstring} @emph{str} (beta testing)} +@item @strong{--tagstring} @emph{str} +@anchor{@strong{--tagstring} @emph{str}} Tag lines with a string. Each output line will be prepended with @emph{str} and TAB (\t). @emph{str} can contain replacement strings such as @@ -3706,7 +3717,7 @@ @end verbatim If your environment (aliases, variables, and functions) is small you -can copy the full environment without having to @strong{export -f} +can copy the full environment without having to @strong{export -f } anything. See @strong{env_parallel}. @node EXAMPLE: Function tester @@ -5535,6 +5546,10 @@ true >jobqueue; tail -n+0 -f jobqueue | parallel -S .. @end verbatim +Output only will be printed when reading the next input after a job +has finished: So you need to submit a job after the first has finished +to see the output from the first job. + If you keep this running for a long time, jobqueue will grow. A way of removing the jobs already run is by making GNU @strong{parallel} stop when it hits a special value and then restart. To use @strong{--eof} to make GNU @@ -5568,20 +5583,6 @@ GNU @strong{parallel} discovers if @strong{jobfile} or @strong{~/.parallel/sshloginfile} changes. -There is a a small issue when using GNU @strong{parallel} as queue -system/batch manager: You have to submit JobSlot number of jobs before -they will start, and after that you can submit one at a time, and job -will start immediately if free slots are available. - -Output from the running or completed jobs are held back and will only -be printed when the next job is started (unless you use --ungroup or ---line-buffer, in which case the output from the jobs are printed -immediately). - -E.g. if you have 10 jobslots then the output from the first completed -job will only be printed when job 11 has started, and the output of -second completed job will only be printed when job 12 has started. - @node EXAMPLE: GNU Parallel as dir processor @section EXAMPLE: GNU Parallel as dir processor diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/parallel_alternatives.7 new/parallel-20211122/src/parallel_alternatives.7 --- old/parallel-20210922/src/parallel_alternatives.7 2021-08-23 08:42:55.000000000 +0200 +++ new/parallel-20211122/src/parallel_alternatives.7 2021-10-21 21:20:32.000000000 +0200 @@ -133,7 +133,7 @@ .\" ======================================================================== .\" .IX Title "PARALLEL_ALTERNATIVES 7" -.TH PARALLEL_ALTERNATIVES 7 "2021-08-23" "20210822" "parallel" +.TH PARALLEL_ALTERNATIVES 7 "2021-10-09" "20210922" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/parsort new/parallel-20211122/src/parsort --- old/parallel-20210922/src/parsort 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/parsort 2021-11-23 00:35:26.000000000 +0100 @@ -122,9 +122,9 @@ "help" => \$opt::dummy, ) || exit(255); $Global::progname = ($0 =~ m:(^|/)([^/]+)$:)[1]; -$Global::version = 20210922; +$Global::version = 20211122; if($opt::version) { version(); exit 0; } -@Global::sortoptions = +@Global::sortoptions = grep { ! /^-D$/ } shell_quote(@ARGV_before[0..($#ARGV_before-$#ARGV-1)]); #if($opt::zero_terminated) { $/ = "\0"; } @@ -160,6 +160,7 @@ @Global::sortoptions, '::::', @files) || die; my @cmd = merge(<$par>); close $par; + debug(@cmd); # The command uses <(...) so it is incompatible with /bin/sh open(my $bash,"|-","bash") || die; print $bash @cmd; @@ -220,6 +221,13 @@ return $tmpfifo; } +sub debug { + # Returns: N/A + $opt::D or return; + @_ = grep { defined $_ ? $_ : "" } @_; + print STDERR @_[1..$#_]; +} + sub version() { # Returns: N/A print join diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/parsort.1 new/parallel-20211122/src/parsort.1 --- old/parallel-20210922/src/parsort.1 2021-09-22 23:19:00.000000000 +0200 +++ new/parallel-20211122/src/parsort.1 2021-11-23 00:38:02.000000000 +0100 @@ -133,7 +133,7 @@ .\" ======================================================================== .\" .IX Title "PARSORT 1" -.TH PARSORT 1 "2021-09-22" "20210922" "parallel" +.TH PARSORT 1 "2021-11-22" "20211122" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/sem new/parallel-20211122/src/sem --- old/parallel-20210922/src/sem 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/sem 2021-11-23 00:35:26.000000000 +0100 @@ -566,14 +566,26 @@ "$file is not a seekable file."); ::wait_and_exit(255); } - my $header = find_header(\$buf,open_or_exit($file)); + + my $fh = open_or_exit($file); + my $firstlinelen = 0; + if($opt::skip_first_line) { + my $newline; + # Read a full line one byte at a time + while($firstlinelen += sysread($fh,$newline,1,0)) { + $newline eq "\n" and last; + } + } + my $header = find_header(\$buf,$fh); # find positions - my @pos = find_split_positions($file,int($Global::blocksize),$header); + my @pos = find_split_positions($file,int($Global::blocksize), + $header,$firstlinelen); # Make @cat_prepends my @cat_prepends = (); for(my $i=0; $i<$#pos; $i++) { push(@cat_prepends, - cat_partial($file, 0, length($header), $pos[$i], $pos[$i+1])); + cat_partial($file, $firstlinelen, $firstlinelen+length($header), + $pos[$i], $pos[$i+1])); } return @cat_prepends; } @@ -618,8 +630,8 @@ # $opt::recend # Returns: # @positions of block start/end - my($file, $block, $header) = @_; - my $headerlen = length $header; + my($file, $block, $header, $firstlinelen) = @_; + my $skiplen = $firstlinelen + length $header; my $size = -s $file; if(-b $file) { # $file is a blockdevice @@ -627,7 +639,8 @@ } $block = int $block; if($opt::groupby) { - return split_positions_for_group_by($file,$size,$block,$header); + return split_positions_for_group_by($file,$size,$block, + $header,$firstlinelen); } # The optimal dd blocksize for mint, redhat, solaris, openbsd = 2^17..2^20 # The optimal dd blocksize for freebsd = 2^15..2^17 @@ -637,8 +650,8 @@ my ($recstart,$recend) = recstartrecend(); my $recendrecstart = $recend.$recstart; my $fh = ::open_or_exit($file); - push(@pos,$headerlen); - for(my $pos = $block+$headerlen; $pos < $size; $pos += $block) { + push(@pos,$skiplen); + for(my $pos = $block+$skiplen; $pos < $size; $pos += $block) { my $buf; if($recendrecstart eq "") { # records ends anywhere @@ -745,14 +758,14 @@ return($v,$vpos); } - my ($file,$size,$block,$header) = @_; + my ($file,$size,$block,$header,$firstlinelen) = @_; my ($a,$b,$c,$apos,$bpos,$cpos); my @pos; $fh = open_or_exit($file); # Set $Global::group_by_column $Global::group_by_perlexpr group_by_loop($fh,$opt::recsep); # $xpos = linestart, $x = value at $xpos, $apos < $bpos < $cpos - $apos = length $header; + $apos = $firstlinelen + length $header; for(($a,$apos) = value_at($apos); $apos < $size;) { push @pos, $apos; $bpos = $apos + $block; @@ -967,6 +980,13 @@ my $in = *STDIN; my $timeout = $Global::blocktimeout; + if($opt::skip_first_line) { + my $newline; + # Read a full line one byte at a time + while(sysread($in,$newline,1,0)) { + $newline eq "\n" and last; + } + } my $header = find_header(\$buf,$in); my $anything_written; my $eof; @@ -2244,7 +2264,7 @@ sub init_globals() { # Defaults: - $Global::version = 20210922; + $Global::version = 20211122; $Global::progname = 'parallel'; $::name = "GNU Parallel"; $Global::infinity = 2**31; @@ -2275,14 +2295,16 @@ # = {.}.{+.} = {+/}/{/.}.{+.} # = {..}.{+..} = {+/}/{/..}.{+..} # = {...}.{+...} = {+/}/{/...}.{+...} - '{+/}' => 's:/[^/]*$::', - '{+.}' => 's:.*\.::', - '{+..}' => 's:.*\.([^.]*\.):$1:', - '{+...}' => 's:.*\.([^.]*\.[^.]*\.):$1:', - '{..}' => 's:\.[^/.]+$::; s:\.[^/.]+$::', - '{...}' => 's:\.[^/.]+$::; s:\.[^/.]+$::; s:\.[^/.]+$::', - '{/..}' => 's:.*/::; s:\.[^/.]+$::; s:\.[^/.]+$::', - '{/...}' => 's:.*/::; s:\.[^/.]+$::; s:\.[^/.]+$::; s:\.[^/.]+$::', + '{+/}' => 's:/[^/]*$:: || s:.*$::', + # a.b => b; a => '' + '{+.}' => 's:.*\.:: || s:.*$::', + # a.b.c => b.c; a.b => ''; a => '' + '{+..}' => 's:.*\.([^/.]+\.[^/.]+)$:$1: || s:.*$::', + '{+...}' => 's:.*\.([^/.]+\.[^/.]+\.[^/.]+)$:$1: || s:.*$::', + '{..}' => 's:\.[^/.]+\.[^/.]+$::', + '{...}' => 's:\.[^/.]+\.[^/.]+\.[^/.]+$::', + '{/..}' => 's:.*/::; s:\.[^/.]+\.[^/.]+$::', + '{/...}' => 's:.*/::; s:\.[^/.]+\.[^/.]+\.[^/.]+$::', # n choose k = Binomial coefficient '{choose_k}' => 'for $t (2..$#arg){ if($arg[$t-1] ge $arg[$t]) { skip() } }', # {##} = number of jobs @@ -5055,8 +5077,8 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - " Tange, O. (2021, September 22). GNU Parallel 20210922 ('Vindelev').", - " Zenodo. https://doi.org/10.5281/zenodo.5523272";, + " Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai').", + " Zenodo. https://doi.org/10.5281/zenodo.5719513";, "", # Before changing these lines, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice @@ -5088,8 +5110,8 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - " Tange, O. (2021, September 22). GNU Parallel 20210922 ('Vindelev').", - " Zenodo. https://doi.org/10.5281/zenodo.5523272";, + " Tange, O. (2021, November 22). GNU Parallel 20211122 ('Peng Shuai').", + " Zenodo. https://doi.org/10.5281/zenodo.5719513";, "", # Before changing these line, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and @@ -5214,20 +5236,20 @@ "If you use programs that use GNU Parallel to process data for an article in a", "scientific publication, please cite:", "", - "\@software{tange_2021_5523272,", + "\@software{tange_2021_5719513,", " author = {Tange, Ole},", - " title = {GNU Parallel 20210922 ('Vindelev')},", - " month = Sep,", + " title = {GNU Parallel 20211122 ('Peng Shuai')},", + " month = Nov,", " year = 2021,", " note = {{GNU Parallel is a general parallelizer to run", " multiple serial command line programs in parallel", " without changing them.}},", " publisher = {Zenodo},", - " doi = {10.5281/zenodo.5523272},", - " url = {https://doi.org/10.5281/zenodo.5523272}";, + " doi = {10.5281/zenodo.5719513},", + " url = {https://doi.org/10.5281/zenodo.5719513}";, "}", "", - "(Feel free to use \\nocite{tange_2021_5523272})", + "(Feel free to use \\nocite{tange_2021_5719513})", "", # Before changing these lines, please read # https://www.gnu.org/software/parallel/parallel_design.html#Citation-notice and @@ -7263,6 +7285,9 @@ $count_jobs_already_read = $Global::JobQueue->next_seq(); my $wait_time_for_getting_args = 0; my $start_time = time; + if($wanted_processes < $Global::infinity) { + $Global::dummy_jobs = 1; + } while(1) { $system_limit >= $wanted_processes and last; not $more_filehandles and last; @@ -11263,7 +11288,8 @@ } if($opt::sqlmaster) { - # Insert the V1..Vn for this $seq in SQL table instead of generating one + # Insert the V1..Vn for this $seq in SQL table + # instead of generating one $Global::sql->insert_records($self->seq(), $self->{'command'}, $self->{'arg_list_flat_orig'}); } @@ -11282,12 +11308,16 @@ *Arg::arg = $self->{'arg_list_flat_orig'}; my $quote_arg = ($Global::quote_replace and not $Global::quoting); + my $col; for my $perlexpr (keys %{$self->{'replacecount'}}) { - if($perlexpr =~ /^(\d+) /) { - # Positional - defined($record->[$1-1]) or next; - $self->{'len'}{$perlexpr} += - length $record->[$1-1]->replace($perlexpr,$quote_arg,$self); + if($perlexpr =~ /^(-?\d+)(?:\D.*|)$/) { + # Positional replacement string + # Deal with negative positional replacement string + $col = ($1 < 0) ? $1 : $1-1; + if(defined($record->[$col])) { + $self->{'len'}{$perlexpr} += + length $record->[$col]->replace($perlexpr,$quote_arg,$self); + } } else { for my $arg (@$record) { if(defined $arg) { @@ -11577,7 +11607,6 @@ # Worst case is BASE64 encoding 3 bytes -> 4 bytes $len = int($len*4/3); } - return $len; } diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/sql new/parallel-20211122/src/sql --- old/parallel-20210922/src/sql 2021-09-22 23:17:48.000000000 +0200 +++ new/parallel-20211122/src/sql 2021-11-23 00:35:26.000000000 +0100 @@ -600,7 +600,7 @@ exit ($err); sub parse_options { - $Global::version = 20210922; + $Global::version = 20211122; $Global::progname = 'sql'; # This must be done first as this may exec myself diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20210922/src/sql.1 new/parallel-20211122/src/sql.1 --- old/parallel-20210922/src/sql.1 2021-09-22 23:19:00.000000000 +0200 +++ new/parallel-20211122/src/sql.1 2021-11-23 00:38:02.000000000 +0100 @@ -133,7 +133,7 @@ .\" ======================================================================== .\" .IX Title "SQL 1" -.TH SQL 1 "2021-09-22" "20210922" "parallel" +.TH SQL 1 "2021-11-22" "20211122" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l ++++++ parallel-20210922.tar.bz2.sig -> parallel-20211122.tar.bz2.sig ++++++ --- /work/SRC/openSUSE:Factory/gnu_parallel/parallel-20210922.tar.bz2.sig 2021-10-11 15:32:01.494884182 +0200 +++ /work/SRC/openSUSE:Factory/.gnu_parallel.new.31177/parallel-20211122.tar.bz2.sig 2021-12-02 02:13:37.734960101 +0100 @@ -2,7 +2,7 @@ # To check the signature run: # echo | gpg -# gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve parallel-20210922.tar.bz2.sig +# gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve parallel-20211122.tar.bz2.sig echo | gpg 2>/dev/null gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve $0 @@ -10,32 +10,32 @@ -----BEGIN PGP SIGNATURE----- -iQUHBAABCgAdFiEEzaAaQgjE90UGEH570atFFoiIiIgFAmFLnfAACgkQ0atFFoiI -iIiilCagkPmhpnXB9EzxNfdHD6nk1eMIiaS/KcJ4Gme2aoAFJI52+spOd2svwK4G -YmLQJEDCsWEVNns7G9OBq12JWQA+QbJ+QEsJe5qnFf52qlPL7kppVDNHhcsgyboV -9rJTdpmNczV3Ch//sYLye5cfovcccoZVzHJiouYXvrBHibPaMeGu5YWHRxEBRA5z -lB9VZ+1KScB8DnazuZmkrgBaNUg34+xccnL0bcB7f0IdljZrrof0W+YjXkdHaNK6 -FYGy4APEgNTf9hjxPiLZB8+S7DHTi9lHaCvlBGdQWzjrcqRY50KP2WaM5SyLtlrG -YHbnTgbAEb9OLGEwuqWDAsVn7cMARPQYqfortcDejzwcLbsOxhKlTlnb5RPXD09J -Vb6++npFiWUGMp1Z+1Mhvm84pS19y3YCIaFMVhMroIni+5VCzEy0KblHv7AO9R2Y -b3pI43mNIvd0gA0zBaOHD6XDHbfrpPJoGtMkFC2gHjHfE1M2OjDPEujkO3Fbt2kM -OP3AH+vjbmMSyAeC9sVgYNXqkfURK0OShjAx+Cx74CBM6ATTnvQ1mdko8fn/Es5z -8n6ZA1P6MfJgnFs0kRB3TZiPUFE80tFyxLSMxwtiWqrQ7hr5owJNXbvfV59b6tRt -e1wSE9CSPPJnzvraI1aQWGh/9cJwFhKcaE1GfRbp6HUAgw22xuKEc9IEB0c3NN2j -hC2iD+HgkYWaN//m2DrL+Ykr4cW+Qk+CMCbqUeHNqnEiE68lmfU1tSOdMxCo0DNU -GC/6UjTGJHojIYMr/QdFFHls13aUkAKcAWnantgb/8a6S1jAYzy4XwaIPwlZHaJS -7hhuNmP2AtSdS97i7OGOL+OoJSsUKfDS9NZp13qV9xDDD/X77dihsIWzi92N5giE -qjzurfSmzpQAfEKSuYQcFQbrjuqYXPN6gH+WX2zFySaPgdbJ5fMeAe2kvDQfdIM1 -ulUgD7YWb7wpWfynI9PgehKMWXxMCbT0znrilknjPnml3T3PvFE1hknQaugRr7Tf -8qyniZwv3ScCsNRYJg+xLen2pAkNKS1w0vRWxb1SoAar+Sn7q5BtLO7YvjiWMIWi -qFSqPUTtsMdkM+dvrT0yH5v2yKc+oxARyOTEg5ddoZtnhWnxpHlk6IgAooGJAPbw -AKtOMo4EqwSV74vvSLeI755DgJhZk2efcNnxNuDO4Rf562KyNRkbF8UntV87Gvy0 -82aeX2EC7xYB8kY1zC1TP5qYbU2tkTKc021Jab7QDmL006pvXp3mgdVIxK+XN+Lg -o+4WjJ75Er5cQKIvn2V67C2o0SvGi37Uxj2TM53OudTIsju4uWs17fpiZC9prDsz -t4GDu/IcjIUGZpj+9LAiZ/oqwDfcXPxeB14S1ESakFbZ2IL/bPXnuDp0rhF6ta7Y -fZO2Sj3runfuBNV7upENHcu9GlugaCC9k4kmYtzjrH5FQYlcXYLcf3EYrI7Ukeck -2cwwMDJM5QfKJ0I1+DcMcZXA8IZrmgPmMxr1rxK7oil/AF+humyfcX9iCrxHgMxu -7WFDaW+a6owxDRjsXdkOJ3CMXGCHIAcSc4dMOT4T4vY0G94MEf0nq3m9GISxrSta -UJmZqGaHFx3rRcWgb5lMzw3LCw7fpiEi+a+mEjX0UuvHmqIXUQHuyxoh -=qmYO +iQUHBAABCgAdFiEEzaAaQgjE90UGEH570atFFoiIiIgFAmGcKpQACgkQ0atFFoiI +iIh85Caffd3Eds+Z5GZtLwo/nlNOCteLSCVe6rDJ8yCknUT36KCUTqd9Uxkx8lZa +rIIXO6ABiX/AzY24SIPDjTc3VOzve0rACwj56hEpCRw7GMuWwa90iqmTFj5GyK2Z +uf83zkVJ2HhPCk0V9rRb6HWvPIA1doKtaqPGt+al58T4TkVVUoYKPv+swvqxWPtL +gfha1G2ldWy7SUozJZv6+EC5ciJ4D2z+7vFJ29VVniKTLQ564YiiClO5G5mvTtyu +DAR6ZAMPhJZwUDhyykKeZCIzP110MK+69l0rJf1mKm8lwbFspdgH3Hi+QJ3cdFNF +fKGr5AAf9pc02bTsCAmLwdZEv5A0u3z3KT5LWiKoJEAzh3vIOtUcX831BjrkfX+y +PiMmiPs87TnvV51FaHh0ZcvzYHME9tM2ag21X+h7w2Y/0MwAuWzhnljDvMPPF3aN +5fH3snjtqMXmn5XgHRL8zyGF9COI1VLKOSKFSK3YcLz4hH1+tY7n3I5NUyQqH14S +7Pa5sdpxlzawUpm90PorszlKpmfDUH/DGTnaMA3zyy0znvir3/gZrnonrub5uPWe +ytNeMPncXe8QTUe6vB3VTDBBZkdDc7dOWCybj4SP8ZlexAbG3ym3aiDTcIyKR9lI +Gwhpvn4zy4hx7hwQrhi+UEKjTXYQDBYfRwI6+DO08RLDyyjfQ2x5fP7Qie+xXncX +3Vt7CHzUyH9rweCa0c6ScKDozXtbYcv3m7bE64bPdFLXJAgr/HRKyW9veUyKDtI2 +WuS3uc/IdjV8kkgvE990PgDb84yTSS1nJ6vkvzyf4omkk265Uw+OXCqD5G05lcVA ++YTRYyU25PhufuTmkNVX10uO9ht6ENY1gBj4DJyLvEGcUAXekdDOuX5ziVo99EXu +bF5/nV+hKcQBhcuf7FRl4z4qswMypzPcWWwiLN3Aqf6WXzUdz2vOTw79Ehm0Anc0 +qzIreQ3Uv1BIAGVzBvFXiKEOQECJofL9wegWJncTDttRzC7FYAhK7jsfrQFM8VCH +W4wUdQqeXyQy0ou3xJZr0Ryau64c6joaciHJGEImK0N/HEzjSA/43xQcbonvfpRO +a8YiIKvuM648PGlXofi3iP2qcpNKbQC+6kUirBCHn0s+2zq2d9cpXJK3O7qANSfU +e/ZnD55SltMepOI3lRgKtJJjANnBQZLdMcNWFI/oR2IwYR8+kRvezMzQPjBZCfHv +vRd15kUUDuVoc5U97r8fK3ynBsuiO3cmBNbgwq6+BVo622tSgfQwLbSbHSabYTbF +hwPjrh/5pcpnHXzHaHQ2TUivmuK1VIGiCUllpyripF////AFf+YY6Ns+DcSPe/5O +oLy8JKyNPzclRiKkerExSRQqeLg5uKyWIf/BwWuxvbZA/LgaQb6QtI0VryhxdGiz +OvVBCvAkPknctApkj24qKJEDxHz6jbIEAz3TGnbgH0tW8QVHa1D9SG39/jR0HhRc +Vg2coG8iZSrmSDZHHB+SCM+p7x51ha8ezaXME9SlDbZRLam4N4B1ZolnSlbng1un +RF7Viu/R/92R91GBWE2gavDqzlIcwagJtm0TUCyUling+Quez8EbWwCs19b4w2D7 +YQEt+6PFCOZkLtj/iZAABCczmfWKw6PtxJdF+JTAG3PK2qZJHi0zBHyf +=O68d -----END PGP SIGNATURE-----
