Hello community, here is the log from the commit of package gnu_parallel for openSUSE:Factory checked in at 2015-09-24 06:15:39 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/gnu_parallel (Old) and /work/SRC/openSUSE:Factory/.gnu_parallel.new (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "gnu_parallel" Changes: -------- --- /work/SRC/openSUSE:Factory/gnu_parallel/gnu_parallel.changes 2015-09-09 20:21:18.000000000 +0200 +++ /work/SRC/openSUSE:Factory/.gnu_parallel.new/gnu_parallel.changes 2015-09-24 06:16:07.000000000 +0200 @@ -1,0 +2,6 @@ +Tue Sep 22 07:11:31 UTC 2015 - [email protected] + +- Update to 20150922 + * Bug fixes and man page updates. + +------------------------------------------------------------------- Old: ---- parallel-20150822.tar.bz2 parallel-20150822.tar.bz2.sig New: ---- parallel-20150922.tar.bz2 parallel-20150922.tar.bz2.sig ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ gnu_parallel.spec ++++++ --- /var/tmp/diff_new_pack.SIpy15/_old 2015-09-24 06:16:07.000000000 +0200 +++ /var/tmp/diff_new_pack.SIpy15/_new 2015-09-24 06:16:07.000000000 +0200 @@ -17,7 +17,7 @@ Name: gnu_parallel -Version: 20150822 +Version: 20150922 Release: 0 Summary: Shell tool for executing jobs in parallel License: GPL-3.0+ ++++++ parallel-20150822.tar.bz2 -> parallel-20150922.tar.bz2 ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/NEWS new/parallel-20150922/NEWS --- old/parallel-20150822/NEWS 2015-08-22 11:41:38.000000000 +0200 +++ new/parallel-20150922/NEWS 2015-09-21 19:29:16.000000000 +0200 @@ -1,3 +1,35 @@ +20150922 + +* GNU Parallel was cited in: Flexible Modeling of Epidemics with an Empirical Bayes Framework http://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1004382 + +* GNU Parallel was cited in: BL1: 2D Potts Model with a Twist https://sucs.swan.ac.uk/~rjames93/Dissertation.pdf + +* GNU Parallel was cited in: DockBench: An Integrated Informatic Platform Bridging the Gap between the Robust Validation of Docking Protocols and Virtual Screening Simulations http://www.mdpi.com/1420-3049/20/6/9977/pdf + +* GNU Parallel was cited in: A Scalable Parallel Implementation of Evolutionary Algorithms for +Multi-Objective Optimization on GPUs http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7257074 + +* GNU Parallel was cited in: Tools and techniques for computational reproducibility http://biorxiv.org/content/biorxiv/early/2015/07/17/022707.full.pdf + +* GNU Parallel was cited in: How Can We Measure the Similarity Between Résumés of Selected Candidates for a Job? http://crawl.prod.proquest.com.s3.amazonaws.com/fpcache/c79b355e6e441a51de3c7fcd52866184.pdf?AWSAccessKeyId=AKIAJF7V7KNV2KKY2NUQ&Expires=1442658364&Signature=soRDi6Xqw1zHhSmBrqJ5KiD%2B8Sw%3D + +* GNU Parallel was cited in: Interplay of cell dynamics and epithelial tension during morphogenesis of the Drosophila pupal wing http://www.researchgate.net/profile/Raphael_Etournay/publication/279061859_Interplay_of_cell_dynamics_and_epithelial_tension_during_morphogenesis_of_the_Drosophila_pupal_wing/links/558a95ad08aeae8413bcceea.pdf + +* Third-party selling GNU Parallel T-shirts http://www.aliexpress.com/item/2015F-BSO-GNU-LOGO-GNU-PARALLEL-men-s-shirt-sleeve-visual-illusion-error/32464827966.html + +* Joys of gnu parallel http://scottolesen.com/index.php/2015/08/26/joys-of-gnu-parallel/ + +* Crop and resize images with bash and ImageMagick https://www.simonholywell.com/post/2015/08/image-resize-crop-bash-imagemagick/ + +* Three Ways to Script Processes in Parallel http://www.codeword.xyz/2015/09/02/three-ways-to-script-processes-in-parallel/ + +* What It Looks Like to Process 3.5 Million Books http://blog.gdeltproject.org/what-it-looks-like-to-process-3-5-million-books/ + +* L’Exploration De Données Twitter http://blog.inovia-conseil.fr/?p=233 + +* Bug fixes and man page updates. + + 20150822 * If $job->skip() is called in {= =} the job will not be run. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/README new/parallel-20150922/README --- old/parallel-20150822/README 2015-08-22 11:40:23.000000000 +0200 +++ new/parallel-20150922/README 2015-09-21 19:25:53.000000000 +0200 @@ -40,9 +40,9 @@ Full installation of GNU Parallel is as simple as: - wget http://ftpmirror.gnu.org/parallel/parallel-20150822.tar.bz2 - bzip2 -dc parallel-20150822.tar.bz2 | tar xvf - - cd parallel-20150822 + wget http://ftpmirror.gnu.org/parallel/parallel-20150922.tar.bz2 + bzip2 -dc parallel-20150922.tar.bz2 | tar xvf - + cd parallel-20150922 ./configure && make && sudo make install @@ -51,9 +51,9 @@ If you are not root you can add ~/bin to your path and install in ~/bin and ~/share: - wget http://ftpmirror.gnu.org/parallel/parallel-20150822.tar.bz2 - bzip2 -dc parallel-20150822.tar.bz2 | tar xvf - - cd parallel-20150822 + wget http://ftpmirror.gnu.org/parallel/parallel-20150922.tar.bz2 + bzip2 -dc parallel-20150922.tar.bz2 | tar xvf - + cd parallel-20150922 ./configure --prefix=$HOME && make && make install Or if your system lacks 'make' you can simply copy src/parallel diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/configure new/parallel-20150922/configure --- old/parallel-20150822/configure 2015-08-22 11:40:31.000000000 +0200 +++ new/parallel-20150922/configure 2015-09-21 19:26:08.000000000 +0200 @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.69 for parallel 20150822. +# Generated by GNU Autoconf 2.69 for parallel 20150922. # # Report bugs to <[email protected]>. # @@ -579,8 +579,8 @@ # Identity of this package. PACKAGE_NAME='parallel' PACKAGE_TARNAME='parallel' -PACKAGE_VERSION='20150822' -PACKAGE_STRING='parallel 20150822' +PACKAGE_VERSION='20150922' +PACKAGE_STRING='parallel 20150922' PACKAGE_BUGREPORT='[email protected]' PACKAGE_URL='' @@ -1203,7 +1203,7 @@ # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures parallel 20150822 to adapt to many kinds of systems. +\`configure' configures parallel 20150922 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1269,7 +1269,7 @@ if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of parallel 20150822:";; + short | recursive ) echo "Configuration of parallel 20150922:";; esac cat <<\_ACEOF @@ -1345,7 +1345,7 @@ test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -parallel configure 20150822 +parallel configure 20150922 generated by GNU Autoconf 2.69 Copyright (C) 2012 Free Software Foundation, Inc. @@ -1362,7 +1362,7 @@ This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by parallel $as_me 20150822, which was +It was created by parallel $as_me 20150922, which was generated by GNU Autoconf 2.69. Invocation command line was $ $0 $@ @@ -2225,7 +2225,7 @@ # Define the identity of the package. PACKAGE='parallel' - VERSION='20150822' + VERSION='20150922' cat >>confdefs.h <<_ACEOF @@ -2867,7 +2867,7 @@ # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by parallel $as_me 20150822, which was +This file was extended by parallel $as_me 20150922, which was generated by GNU Autoconf 2.69. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -2929,7 +2929,7 @@ cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -parallel config.status 20150822 +parallel config.status 20150922 configured by $0, generated by GNU Autoconf 2.69, with options \\"\$ac_cs_config\\" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/configure.ac new/parallel-20150922/configure.ac --- old/parallel-20150822/configure.ac 2015-08-22 11:40:23.000000000 +0200 +++ new/parallel-20150922/configure.ac 2015-09-21 19:25:53.000000000 +0200 @@ -1,4 +1,4 @@ -AC_INIT([parallel], [20150822], [[email protected]]) +AC_INIT([parallel], [20150922], [[email protected]]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/src/niceload new/parallel-20150922/src/niceload --- old/parallel-20150822/src/niceload 2015-08-22 11:40:23.000000000 +0200 +++ new/parallel-20150922/src/niceload 2015-09-21 19:25:53.000000000 +0200 @@ -24,7 +24,7 @@ use strict; use Getopt::Long; $Global::progname="niceload"; -$Global::version = 20150822; +$Global::version = 20150922; Getopt::Long::Configure("bundling","require_order"); get_options_from_array(\@ARGV) || die_usage(); if($opt::version) { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/src/parallel new/parallel-20150922/src/parallel --- old/parallel-20150822/src/parallel 2015-08-22 11:40:23.000000000 +0200 +++ new/parallel-20150922/src/parallel 2015-09-21 19:25:53.000000000 +0200 @@ -51,6 +51,7 @@ my @input_source_fh; if($opt::pipepart) { + # -a is used for data - not for command line args @input_source_fh = map { open_or_exit($_) } "/dev/null"; } else { @input_source_fh = map { open_or_exit($_) } @opt::a; @@ -77,9 +78,7 @@ for my $s (split /$delimiter/o, $line) { ::debug("init", "Colname: '$s'"); # Replace {colname} with {2} - # TODO accept configurable short hands - # TODO how to deal with headers in {=...=} - for(@command) { + for(@command,@Global::ret_files) { s:\{$s(|/|//|\.|/\.)\}:\{$id$1\}:g; } $Global::input_source_header{$id} = $s; @@ -115,16 +114,25 @@ # before starting any $Global::JobQueue->total_jobs(); } +# Compute $Global::max_jobs_running +for my $sshlogin (values %Global::host) { + $sshlogin->max_jobs_running(); +} if($opt::pipepart) { + if($opt::roundrobin) { + # Compute size of -a + my $size = 0; + $size += -s $_ for @opt::a; + $Global::max_jobs_running or ::die_bug("Global::max_jobs_running not set"); + # Set --blocksize = size / no of proc + $opt::blocksize = 1 + $size / $Global::max_jobs_running; + } @Global::cat_partials = map { pipe_part_files($_) } @opt::a; # Unget the command as many times as there are parts $Global::JobQueue->{'commandlinequeue'}->unget( map { $Global::JobQueue->{'commandlinequeue'}->get() } @Global::cat_partials ); } -for my $sshlogin (values %Global::host) { - $sshlogin->max_jobs_running(); -} init_run_jobs(); my $sem; @@ -916,7 +924,6 @@ if(defined $opt::colsep) { $Global::trim = 'lr'; } if(defined $opt::header) { $opt::colsep = defined $opt::colsep ? $opt::colsep : "\t"; } if(defined $opt::trim) { $Global::trim = $opt::trim; } - if(defined $opt::roundrobin) { $opt::pipe = 1; } if(defined $opt::arg_sep) { $Global::arg_sep = $opt::arg_sep; } if(defined $opt::arg_file_sep) { $Global::arg_file_sep = $opt::arg_file_sep; } if(defined $opt::number_of_cpus) { print SSHLogin::no_of_cpus(),"\n"; wait_and_exit(0); } @@ -1090,7 +1097,7 @@ sub init_globals { # Defaults: - $Global::version = 20150822; + $Global::version = 20150922; $Global::progname = 'parallel'; $Global::infinity = 2**31; $Global::debug = 0; @@ -3252,7 +3259,7 @@ my $sleepsum = 0; my $sleep = 0.001; my @dead; - + while(@pids and $sleepsum < $sleep_max) { if($Global::killall) { # Killall => don't run reaper @@ -4834,7 +4841,9 @@ $max_system_proc_reached and last; my $before_getting_arg = time; - get_args_or_jobs() or last; + if(!$opt::roundrobin) { + get_args_or_jobs() or last; + } $wait_time_for_getting_args += time - $before_getting_arg; $system_limit++; @@ -5997,10 +6006,10 @@ # Returns: # $wrapped_command = the wrapped command my $command = shift; - my $script = '$c="'.::perl_quote_scalar($command).'";'. + my $script = ::spacefree(0,q{ if(sysread(STDIN, $buf, 1)) { - open($fh, "|-", $c) || die; + open($fh, "|-", "@ARGV") || die; syswrite($fh, $buf); while($read = sysread(STDIN, $buf, 32768)) { syswrite($fh, $buf); @@ -6010,7 +6019,8 @@ } }); ::debug("run",'Empty wrap: perl -e '.::shell_quote_scalar($script)."\n"); - return 'perl -e '.::shell_quote_scalar($script); + return 'perl -e '.::shell_quote_scalar($script)." ". + ::shell_quote_scalar($Global::shell." -c ".::shell_quote_scalar($command)); } sub filter_through_compress { @@ -7773,6 +7783,8 @@ 'seq' => $seq, 'len' => \%len, 'arg_list' => [], + 'arg_list_flat' => [], + 'arg_list_flat_orig' => [undef], 'arg_queue' => $arg_queue, 'max_number_of_args' => $max_number_of_args, 'replacecount' => \%replacecount, @@ -7900,19 +7912,23 @@ # Returns: N/A my $self = shift; my $record = shift; + push @{$self->{'arg_list_flat_orig'}}, map { $_->orig() } @$record; + push @{$self->{'arg_list_flat'}}, @$record; push @{$self->{'arg_list'}}, $record; + # Make @arg available for {= =} + *Arg::arg = $self->{'arg_list_flat_orig'}; my $quote_arg = $Global::noquote ? 0 : not $Global::quoting; - my $rep; - for my $arg (@$record) { - if(defined $arg) { - for my $perlexpr (keys %{$self->{'replacecount'}}) { - # 50% faster than below - $self->{'len'}{$perlexpr} += length $arg->replace($perlexpr,$quote_arg,$self); - # $rep = $arg->replace($perlexpr,$quote_arg,$self); - # $self->{'len'}{$perlexpr} += length $rep; - # ::debug("length", "Length: ", length $rep, - # "(", $perlexpr, "=>", $rep, ")\n"); + for my $perlexpr (keys %{$self->{'replacecount'}}) { + if($perlexpr =~ /^(\d+) /) { + # Positional + defined($record->[$1-1]) or next; + $self->{'len'}{$perlexpr} += length $record->[$1-1]->replace($perlexpr,$quote_arg,$self); + } else { + for my $arg (@$record) { + if(defined $arg) { + $self->{'len'}{$perlexpr} += length $arg->replace($perlexpr,$quote_arg,$self); + } } } } @@ -7924,12 +7940,20 @@ # the last record my $self = shift; my $record = pop @{$self->{'arg_list'}}; + # pop off arguments from @$record + splice @{$self->{'arg_list_flat_orig'}}, -($#$record+1), $#$record+1; + splice @{$self->{'arg_list_flat'}}, -($#$record+1), $#$record+1; my $quote_arg = $Global::noquote ? 0 : not $Global::quoting; - for my $arg (@$record) { - if(defined $arg) { - for my $perlexpr (keys %{$self->{'replacecount'}}) { - $self->{'len'}{$perlexpr} -= - length $arg->replace($perlexpr,$quote_arg,$self); + for my $perlexpr (keys %{$self->{'replacecount'}}) { + if($perlexpr =~ /^(\d+) /) { + # Positional + defined($record->[$1-1]) or next; + $self->{'len'}{$perlexpr} -= length $record->[$1-1]->replace($perlexpr,$quote_arg,$self); + } else { + for my $arg (@$record) { + if(defined $arg) { + $self->{'len'}{$perlexpr} -= length $arg->replace($perlexpr,$quote_arg,$self); + } } } } @@ -7937,15 +7961,17 @@ } sub pop_all { - # Remove all arguments and zeros the length of replacement strings + # Remove all arguments and zeros the length of replacement perlexpr # Returns: # all records my $self = shift; my @popped = @{$self->{'arg_list'}}; - for my $replacement_string (keys %{$self->{'replacecount'}}) { - $self->{'len'}{$replacement_string} = 0; + for my $perlexpr (keys %{$self->{'replacecount'}}) { + $self->{'len'}{$perlexpr} = 0; } $self->{'arg_list'} = []; + $self->{'arg_list_flat_orig'} = [undef]; + $self->{'arg_list_flat'} = []; return @popped; } @@ -8127,53 +8153,45 @@ my $context_replace; my $perl_expressions_as_re; my @arg; + my %words_containing_replacement_strings; sub fish_out_words_containing_replacement_strings { - my %word; - for (@target) { - my $tt = $_; - ::debug("replace", "Target: $tt"); - # Command line template: - # a{1}b{}c{}d - # becomes: - # a{=1 $_=$_ =}b{= $_=$_ =}c{= $_=$_ =}d - # becomes: - # a\257<1 $_=$_ \257>b\257< $_=$_ \257>c\257< $_=$_ \257>d - # Input A B C (no context) becomes: - # A B C => aAbA B CcA B Cd - # Input A B C (context -X) becomes: - # A B C => aAbAcAd aAbBcBd aAbCcCd - if($context_replace) { - while($tt =~ s/([^\s\257]* # before {= + if(not $words_containing_replacement_strings{$context_replace,@target}) { + my %word; + for (@target) { + my $tt = $_; + ::debug("replace", "Target: $tt"); + # Command line template: + # a{1}b{}c{}d + # becomes: + # a{=1 $_=$_ =}b{= $_=$_ =}c{= $_=$_ =}d + # becomes: + # a\257<1 $_=$_ \257>b\257< $_=$_ \257>c\257< $_=$_ \257>d + # Input A B C (no context) becomes: + # A B C => aAbA B CcA B Cd + # Input A B C (context -X) becomes: + # A B C => aAbAcAd aAbBcBd aAbCcCd + if($context_replace) { + while($tt =~ s/([^\s\257]* # before {= (?: \257< # {= [^\257]*? # The perl expression \257> # =} [^\s\257]* # after =} )+)/ /x) { - # $1 = pre \257 perlexpr \257 post - $word{"$1"} ||= 1; - } - } else { - while($tt =~ s/( (?: \257<([^\257]*?)\257>) )//x) { - # $f = \257 perlexpr \257 - $word{$1} ||= 1; + # $1 = pre \257 perlexpr \257 post + $word{"$1"} ||= 1; + } + } else { + while($tt =~ s/( (?: \257<([^\257]*?)\257>) )//x) { + # $f = \257 perlexpr \257 + $word{$1} ||= 1; + } } } + @{$words_containing_replacement_strings{$context_replace,@target}} = keys %word } - return keys %word; - } - - sub flatten_arg_list { - my $arglist_ref = shift; - @arg = (); - for my $record (@$arglist_ref) { - # $self->{'arg_list'} = [ [Arg11, Arg12], [Arg21, Arg22], [Arg31, Arg32] ] - # Merge arg-objects from records into @arg for easy access - CORE::push @arg, @$record; - } - # Add one arg if empty to allow {#} and {%} to be computed only once - if(not @arg) { @arg = (Arg->new("")); } + return @{$words_containing_replacement_strings{$context_replace,@target}}; } sub replace_placeholders { @@ -8191,28 +8209,32 @@ my $quote = shift; my $quote_arg = shift; my %replace; + # -X = context replace (fish_out_words_containing_replacement_strings) $context_replace = $self->{'context_replace'}; @target = @$targetref; ::debug("replace", "Replace @target\n"); - # -X = context replace - # maybe multiple input sources - # maybe --xapply if(not @target) { # @target is empty: Return empty array return @target; } - # Fish out the words that have replacement strings in them - my @word = fish_out_words_containing_replacement_strings(); - flatten_arg_list($self->{'arg_list'}); # Make it possible to use $arg[2] in {= =} - @Arg::arg = (undef, map { $_->orig() } @arg); + *Arg::arg = $self->{'arg_list_flat_orig'}; + # Flat list: + # $self->{'arg_list'} = [ [Arg11, Arg12], [Arg21, Arg22], [Arg31, Arg32] ] + # $self->{'arg_list_flat'} = [ Arg11, Arg12, Arg21, Arg22, Arg31, Arg32 ] + if(not @{$self->{'arg_list_flat'}}) { + @{$self->{'arg_list_flat'}} = Arg->new(""); + } + my $argref = $self->{'arg_list_flat'}; # Number of arguments - used for positional arguments - my $n = $#arg+1; + my $n = $#$argref+1; - # This is actually a CommandLine-object, + # $self is actually a CommandLine-object, # but it looks nice to be able to say {= $job->slot() =} my $job = $self; - for my $word (@word) { + # Fish out the words that have replacement strings in them + for my $word ( + fish_out_words_containing_replacement_strings()) { # word = AB \257< perlexpr \257> CD \257< perlexpr \257> EF ::debug("replace", "Replacing in $word\n"); my $normal_replace; @@ -8222,7 +8244,7 @@ # push to replace word value $perl_expressions_as_re ||= join("|", map {s/^-?\d+//; "\Q$_\E"} keys %{$self->{'replacecount'}}); - for my $arg (@arg) { + for my $arg (@$argref) { my $val = $word; # Replace {= perl expr =} with value for each arg $val =~ s{\257<(-?\d+)?($perl_expressions_as_re)\257>} @@ -8230,8 +8252,8 @@ if($1) { # Positional replace # Find the relevant arg and replace it - ($arg[$1 > 0 ? $1-1 : $n+$1] ? # If defined: replace - $arg[$1 > 0 ? $1-1 : $n+$1]-> + ($argref->[$1 > 0 ? $1-1 : $n+$1] ? # If defined: replace + $argref->[$1 > 0 ? $1-1 : $n+$1]-> replace($2,$quote_arg,$self) : ""); } else { @@ -8254,7 +8276,6 @@ if($quote) { @target = ::shell_quote(@target); } - # ::debug("replace", "%replace=",::my_dump(%replace),"\n"); if(%replace) { # Substitute the replace strings with the replacement values # Must be sorted by length if a short word is a substring of a long word @@ -8282,7 +8303,7 @@ my $class = shift; my $commandref = shift; my $read_from = shift; - my $context_replace = shift; + my $context_replace = shift || 0; my $max_number_of_args = shift; my $return_files = shift; my @unget = (); @@ -8300,6 +8321,7 @@ } } # Replace replacement strings with {= perl expr =} + # '{=' 'perlexpr' '=}' => '{= perlexpr =}' @command = merge_rpl_parts(@command); # Protect matching inside {= perl expr =} @@ -8324,7 +8346,7 @@ # with the {= perl expr =} # Avoid replacing inside existing {= perl expr =} while(s{((^|\257>)[^\257]*?) # Don't replace after \257 unless \257> - \Q$rpl\E} + \Q$rpl\E} {$1\257<$Global::rpl{$rpl}\257>}xg) { } # Do the same for the positional replacement strings @@ -8332,11 +8354,14 @@ $posrpl = $rpl; if($posrpl =~ s/^\{//) { # Only do this if the shorthand start with { - s{\{(-?\d+)\Q$posrpl\E} - {\257<$1 $Global::rpl{$rpl}\257>}g; + while(s{((^|\257>)[^\257]*?) # Don't replace after \257 unless \257> + \{(-?\d+)\Q$posrpl\E} + {$1\257<$3 $Global::rpl{$rpl}\257>}xg) { + } } } } + # Add {} if no replacement strings in @command ($replacecount_ref, $len_ref, @command) = replacement_counts_and_lengths($return_files,@command); @@ -8817,7 +8842,13 @@ if($opt::xapply) { return $self->xapply_get(); } else { - return $self->nest_get(); + my $arglist = $self->nest_get(); + # Flush cached computed values + # To fix: parallel echo {%} ::: 1 2 ::: 4 3 + if($arglist) { + map { $_->flush_cache() } @$arglist; + } + return $arglist; } } @@ -8962,7 +8993,6 @@ return undef; } } -# ::debug("run", "read $arg\n"); # Remove delimiter $arg =~ s:$/$::; if($Global::end_of_file_string and @@ -9051,9 +9081,9 @@ sub Q { # Q alias for ::shell_quote_scalar # Run shell_quote_scalar once to set the reference to the sub - my @a = ::shell_quote_scalar(@_); + my $a = ::shell_quote_scalar(@_); *Q = \&::shell_quote_scalar; - return @a; + return $a; } { @@ -9069,33 +9099,47 @@ # This is actually a CommandLine-object, # but it looks nice to be able to say {= $job->slot() =} my $job = shift; - $perlexpr =~ s/^-?\d+ +//; # Positional replace treated as normal replace - local $_; - if($Global::trim eq "n") { - $_ = $self->{'orig'}; - } else { - $_ = trim_of($self->{'orig'}); - } - ::debug("replace", "eval ", $perlexpr, " ", $_, "\n"); - if(not $perleval{$perlexpr}) { - # Make an anonymous function of the $perlexpr - # And more importantly: Compile it only once - if($perleval{$perlexpr} = - eval('sub { no strict; no warnings; my $job = shift; '. - $perlexpr.' }')) { - # All is good + $perlexpr =~ s/^(-?\d+)? *//; # Positional replace treated as normal replace + if(not $self->{'cache'}{$perlexpr}) { + # Only compute the value once + # Use $_ as the variable to change + local $_; + if($Global::trim eq "n") { + $_ = $self->{'orig'}; } else { - # The eval failed. Maybe $perlexpr is invalid perl? - ::error("Cannot use $perlexpr: $@"); - ::wait_and_exit(255); + # Trim the input + $_ = trim_of($self->{'orig'}); } + ::debug("replace", "eval ", $perlexpr, " ", $_, "\n"); + if(not $perleval{$perlexpr}) { + # Make an anonymous function of the $perlexpr + # And more importantly: Compile it only once + if($perleval{$perlexpr} = + eval('sub { no strict; no warnings; my $job = shift; '. + $perlexpr.' }')) { + # All is good + } else { + # The eval failed. Maybe $perlexpr is invalid perl? + ::error("Cannot use $perlexpr: $@"); + ::wait_and_exit(255); + } + } + # Execute the function + $perleval{$perlexpr}->($job); + $self->{'cache'}{$perlexpr} = $_; } - # Execute the function - $perleval{$perlexpr}->($job); - return $quote ? ::shell_quote_scalar($_) : $_; + # Return the value quoted if needed + return($quote ? ::shell_quote_scalar($self->{'cache'}{$perlexpr}) + : $self->{'cache'}{$perlexpr}); } } +sub flush_cache { + # Flush cache of computed values + my $self = shift; + $self->{'cache'} = undef; +} + sub orig { my $self = shift; return $self->{'orig'}; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/src/parallel.1 new/parallel-20150922/src/parallel.1 --- old/parallel-20150822/src/parallel.1 2015-08-22 11:40:35.000000000 +0200 +++ new/parallel-20150922/src/parallel.1 2015-09-21 15:37:32.000000000 +0200 @@ -133,7 +133,7 @@ .\" ======================================================================== .\" .IX Title "PARALLEL 1" -.TH PARALLEL 1 "2015-08-22" "20150822" "parallel" +.TH PARALLEL 1 "2015-09-20" "20150822" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l @@ -596,11 +596,11 @@ citations, so if users do not know they should cite then that makes it harder to finance development. However, if you pay 10000 \s-1EUR,\s0 you should feel free to use \fB\-\-will\-cite\fR. -.IP "\fB\-\-block\fR \fIsize\fR (beta testing)" 9 -.IX Item "--block size (beta testing)" +.IP "\fB\-\-block\fR \fIsize\fR" 9 +.IX Item "--block size" .PD 0 -.IP "\fB\-\-block\-size\fR \fIsize\fR (beta testing)" 9 -.IX Item "--block-size size (beta testing)" +.IP "\fB\-\-block\-size\fR \fIsize\fR" 9 +.IX Item "--block-size size" .PD Size of block in bytes to read at a time. The \fIsize\fR can be postfixed with K, M, G, T, P, k, m, g, t, or p which would multiply the size @@ -1202,11 +1202,11 @@ of each job is saved in a file and the filename is then printed. .Sp See also: \fB\-\-results\fR -.IP "\fB\-\-pipe\fR (beta testing)" 9 -.IX Item "--pipe (beta testing)" +.IP "\fB\-\-pipe\fR" 9 +.IX Item "--pipe" .PD 0 -.IP "\fB\-\-spreadstdin\fR (beta testing)" 9 -.IX Item "--spreadstdin (beta testing)" +.IP "\fB\-\-spreadstdin\fR" 9 +.IX Item "--spreadstdin" .PD Spread input to jobs on stdin (standard input). Read a block of data from stdin (standard input) and give one block of data as input to one @@ -1329,8 +1329,8 @@ .IX Item "--no-keep-order" Overrides an earlier \fB\-\-keep\-order\fR (e.g. if set in \&\fB~/.parallel/config\fR). -.IP "\fB\-\-nice\fR \fIniceness\fR (beta testing)" 9 -.IX Item "--nice niceness (beta testing)" +.IP "\fB\-\-nice\fR \fIniceness\fR" 9 +.IX Item "--nice niceness" Run the command at this niceness. For simple commands you can just add \&\fBnice\fR in front of the command. But if the command consists of more sub commands (Like: ls|wc) then prepending \fBnice\fR will not always @@ -2556,7 +2556,7 @@ \& cat x1y*z1 > x1z1 .Ve .PP -So you end up with x1z1 .. x1z5 each containing the content of all +So you end up with x1z1 .. x5z5 each containing the content of all values of y. .SH "EXAMPLE: Breadth first parallel web crawler/mirrorer" .IX Header "EXAMPLE: Breadth first parallel web crawler/mirrorer" @@ -4123,11 +4123,8 @@ to do this. That results in extra quoting to get filename containing newline to work correctly. .PP -\&\fBmake \-j\fR has no support for grouping the output, therefore output -may run together, e.g. the first half of a line is from one process -and the last half of the line is from another process. The example -\&\fBParallel grep\fR cannot be done reliably with \fBmake \-j\fR because of -this. +\&\fBmake \-j\fR computes a dependency graph before running jobs. Jobs run +by \s-1GNU \s0\fBparallel\fR does not depend on eachother. .PP (Very early versions of \s-1GNU \s0\fBparallel\fR were coincidently implemented using \fBmake \-j\fR). @@ -4730,7 +4727,10 @@ should specify why you believe the problem is not fixed in that version. .IP "\(bu" 2 -A complete example that others can run that shows the problem +A minimal, complete, and verifiable example (See description on +http://stackoverflow.com/help/mcve). +.Sp +It should be a complete example that others can run that shows the problem including all files needed to run the example. This should preferably be small and simple, so try to remove as many options as possible. A combination of \fByes\fR, \fBseq\fR, \fBcat\fR, \fBecho\fR, and \fBsleep\fR can diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/src/parallel.html new/parallel-20150922/src/parallel.html --- old/parallel-20150822/src/parallel.html 2015-08-22 11:40:34.000000000 +0200 +++ new/parallel-20150922/src/parallel.html 2015-09-21 15:37:31.000000000 +0200 @@ -546,11 +546,11 @@ <p>If you use <b>--will-cite</b> in scripts to be run by others you are making it harder for others to see the citation notice. The development of GNU <b>parallel</b> is indirectly financed through citations, so if users do not know they should cite then that makes it harder to finance development. However, if you pay 10000 EUR, you should feel free to use <b>--will-cite</b>.</p> </dd> -<dt id="block-size-beta-testing"><b>--block</b> <i>size</i> (beta testing)</dt> +<dt id="block-size"><b>--block</b> <i>size</i></dt> <dd> </dd> -<dt id="block-size-size-beta-testing"><b>--block-size</b> <i>size</i> (beta testing)</dt> +<dt id="block-size-size"><b>--block-size</b> <i>size</i></dt> <dd> <p>Size of block in bytes to read at a time. The <i>size</i> can be postfixed with K, M, G, T, P, k, m, g, t, or p which would multiply the size with 1024, 1048576, 1073741824, 1099511627776, 1125899906842624, 1000, 1000000, 1000000000, 1000000000000, or 1000000000000000 respectively.</p> @@ -1182,11 +1182,11 @@ <p>See also: <b>--results</b></p> </dd> -<dt id="pipe-beta-testing"><b>--pipe</b> (beta testing)</dt> +<dt id="pipe"><b>--pipe</b></dt> <dd> </dd> -<dt id="spreadstdin-beta-testing"><b>--spreadstdin</b> (beta testing)</dt> +<dt id="spreadstdin"><b>--spreadstdin</b></dt> <dd> <p>Spread input to jobs on stdin (standard input). Read a block of data from stdin (standard input) and give one block of data as input to one job.</p> @@ -1310,7 +1310,7 @@ <p>Overrides an earlier <b>--keep-order</b> (e.g. if set in <b>~/.parallel/config</b>).</p> </dd> -<dt id="nice-niceness-beta-testing"><b>--nice</b> <i>niceness</i> (beta testing)</dt> +<dt id="nice-niceness"><b>--nice</b> <i>niceness</i></dt> <dd> <p>Run the command at this niceness. For simple commands you can just add <b>nice</b> in front of the command. But if the command consists of more sub commands (Like: ls|wc) then prepending <b>nice</b> will not always work. <b>--nice</b> will make sure all sub commands are niced - even on remote servers.</p> @@ -2382,7 +2382,7 @@ <pre><code> cat x1y*z1 > x1z1</code></pre> -<p>So you end up with x1z1 .. x1z5 each containing the content of all values of y.</p> +<p>So you end up with x1z1 .. x5z5 each containing the content of all values of y.</p> <h1 id="EXAMPLE:-Breadth-first-parallel-web-crawler-mirrorer">EXAMPLE: Breadth first parallel web crawler/mirrorer</h1> @@ -3453,7 +3453,7 @@ <p><b>make -j</b> can run jobs in parallel, but requires a crafted Makefile to do this. That results in extra quoting to get filename containing newline to work correctly.</p> -<p><b>make -j</b> has no support for grouping the output, therefore output may run together, e.g. the first half of a line is from one process and the last half of the line is from another process. The example <b>Parallel grep</b> cannot be done reliably with <b>make -j</b> because of this.</p> +<p><b>make -j</b> computes a dependency graph before running jobs. Jobs run by GNU <b>parallel</b> does not depend on eachother.</p> <p>(Very early versions of GNU <b>parallel</b> were coincidently implemented using <b>make -j</b>).</p> @@ -3921,7 +3921,9 @@ <li><p>The complete output of <b>parallel --version</b>. If you are not running the latest released version (see http://ftp.gnu.org/gnu/parallel/) you should specify why you believe the problem is not fixed in that version.</p> </li> -<li><p>A complete example that others can run that shows the problem including all files needed to run the example. This should preferably be small and simple, so try to remove as many options as possible. A combination of <b>yes</b>, <b>seq</b>, <b>cat</b>, <b>echo</b>, and <b>sleep</b> can reproduce most errors. If your example requires large files, see if you can make them by something like <b>seq 1000000</b> > <b>file</b> or <b>yes | head -n 10000000</b> > <b>file</b>. If your example requires remote execution, see if you can use <b>localhost</b> - maybe using another login.</p> +<li><p>A minimal, complete, and verifiable example (See description on http://stackoverflow.com/help/mcve).</p> + +<p>It should be a complete example that others can run that shows the problem including all files needed to run the example. This should preferably be small and simple, so try to remove as many options as possible. A combination of <b>yes</b>, <b>seq</b>, <b>cat</b>, <b>echo</b>, and <b>sleep</b> can reproduce most errors. If your example requires large files, see if you can make them by something like <b>seq 1000000</b> > <b>file</b> or <b>yes | head -n 10000000</b> > <b>file</b>. If your example requires remote execution, see if you can use <b>localhost</b> - maybe using another login.</p> </li> <li><p>The output of your example. If your problem is not easily reproduced by others, the output might help them figure out the problem.</p> Files old/parallel-20150822/src/parallel.pdf and new/parallel-20150922/src/parallel.pdf differ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/src/parallel.pod new/parallel-20150922/src/parallel.pod --- old/parallel-20150822/src/parallel.pod 2015-08-22 11:32:45.000000000 +0200 +++ new/parallel-20150922/src/parallel.pod 2015-09-20 09:04:58.000000000 +0200 @@ -491,9 +491,9 @@ should feel free to use B<--will-cite>. -=item B<--block> I<size> (beta testing) +=item B<--block> I<size> -=item B<--block-size> I<size> (beta testing) +=item B<--block-size> I<size> Size of block in bytes to read at a time. The I<size> can be postfixed with K, M, G, T, P, k, m, g, t, or p which would multiply the size @@ -1146,9 +1146,9 @@ See also: B<--results> -=item B<--pipe> (beta testing) +=item B<--pipe> -=item B<--spreadstdin> (beta testing) +=item B<--spreadstdin> Spread input to jobs on stdin (standard input). Read a block of data from stdin (standard input) and give one block of data as input to one @@ -1291,7 +1291,7 @@ B<~/.parallel/config>). -=item B<--nice> I<niceness> (beta testing) +=item B<--nice> I<niceness> Run the command at this niceness. For simple commands you can just add B<nice> in front of the command. But if the command consists of more @@ -2484,7 +2484,7 @@ cat x1y*z1 > x1z1 -So you end up with x1z1 .. x1z5 each containing the content of all +So you end up with x1z1 .. x5z5 each containing the content of all values of y. @@ -3900,11 +3900,8 @@ to do this. That results in extra quoting to get filename containing newline to work correctly. -B<make -j> has no support for grouping the output, therefore output -may run together, e.g. the first half of a line is from one process -and the last half of the line is from another process. The example -B<Parallel grep> cannot be done reliably with B<make -j> because of -this. +B<make -j> computes a dependency graph before running jobs. Jobs run +by GNU B<parallel> does not depend on eachother. (Very early versions of GNU B<parallel> were coincidently implemented using B<make -j>). @@ -4500,7 +4497,10 @@ =item * -A complete example that others can run that shows the problem +A minimal, complete, and verifiable example (See description on +http://stackoverflow.com/help/mcve). + +It should be a complete example that others can run that shows the problem including all files needed to run the example. This should preferably be small and simple, so try to remove as many options as possible. A combination of B<yes>, B<seq>, B<cat>, B<echo>, and B<sleep> can diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/src/parallel.texi new/parallel-20150922/src/parallel.texi --- old/parallel-20150822/src/parallel.texi 2015-08-22 11:40:45.000000000 +0200 +++ new/parallel-20150922/src/parallel.texi 2015-09-21 15:37:40.000000000 +0200 @@ -597,11 +597,11 @@ harder to finance development. However, if you pay 10000 EUR, you should feel free to use @strong{--will-cite}. -@item @strong{--block} @emph{size} (beta testing) -@anchor{@strong{--block} @emph{size} (beta testing)} +@item @strong{--block} @emph{size} +@anchor{@strong{--block} @emph{size}} -@item @strong{--block-size} @emph{size} (beta testing) -@anchor{@strong{--block-size} @emph{size} (beta testing)} +@item @strong{--block-size} @emph{size} +@anchor{@strong{--block-size} @emph{size}} Size of block in bytes to read at a time. The @emph{size} can be postfixed with K, M, G, T, P, k, m, g, t, or p which would multiply the size @@ -1307,11 +1307,11 @@ See also: @strong{--results} -@item @strong{--pipe} (beta testing) -@anchor{@strong{--pipe} (beta testing)} +@item @strong{--pipe} +@anchor{@strong{--pipe}} -@item @strong{--spreadstdin} (beta testing) -@anchor{@strong{--spreadstdin} (beta testing)} +@item @strong{--spreadstdin} +@anchor{@strong{--spreadstdin}} Spread input to jobs on stdin (standard input). Read a block of data from stdin (standard input) and give one block of data as input to one @@ -1458,8 +1458,8 @@ Overrides an earlier @strong{--keep-order} (e.g. if set in @strong{~/.parallel/config}). -@item @strong{--nice} @emph{niceness} (beta testing) -@anchor{@strong{--nice} @emph{niceness} (beta testing)} +@item @strong{--nice} @emph{niceness} +@anchor{@strong{--nice} @emph{niceness}} Run the command at this niceness. For simple commands you can just add @strong{nice} in front of the command. But if the command consists of more @@ -2818,7 +2818,7 @@ cat x1y*z1 > x1z1 @end verbatim -So you end up with x1z1 .. x1z5 each containing the content of all +So you end up with x1z1 .. x5z5 each containing the content of all values of y. @node EXAMPLE: Breadth first parallel web crawler/mirrorer @@ -4508,11 +4508,8 @@ to do this. That results in extra quoting to get filename containing newline to work correctly. -@strong{make -j} has no support for grouping the output, therefore output -may run together, e.g. the first half of a line is from one process -and the last half of the line is from another process. The example -@strong{Parallel grep} cannot be done reliably with @strong{make -j} because of -this. +@strong{make -j} computes a dependency graph before running jobs. Jobs run +by GNU @strong{parallel} does not depend on eachother. (Very early versions of GNU @strong{parallel} were coincidently implemented using @strong{make -j}). @@ -5186,7 +5183,10 @@ should specify why you believe the problem is not fixed in that version. -@item A complete example that others can run that shows the problem +@item A minimal, complete, and verifiable example (See description on +http://stackoverflow.com/help/mcve). + +It should be a complete example that others can run that shows the problem including all files needed to run the example. This should preferably be small and simple, so try to remove as many options as possible. A combination of @strong{yes}, @strong{seq}, @strong{cat}, @strong{echo}, and @strong{sleep} can diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/src/sem new/parallel-20150922/src/sem --- old/parallel-20150822/src/sem 2015-08-22 11:40:23.000000000 +0200 +++ new/parallel-20150922/src/sem 2015-09-21 19:25:53.000000000 +0200 @@ -51,6 +51,7 @@ my @input_source_fh; if($opt::pipepart) { + # -a is used for data - not for command line args @input_source_fh = map { open_or_exit($_) } "/dev/null"; } else { @input_source_fh = map { open_or_exit($_) } @opt::a; @@ -77,9 +78,7 @@ for my $s (split /$delimiter/o, $line) { ::debug("init", "Colname: '$s'"); # Replace {colname} with {2} - # TODO accept configurable short hands - # TODO how to deal with headers in {=...=} - for(@command) { + for(@command,@Global::ret_files) { s:\{$s(|/|//|\.|/\.)\}:\{$id$1\}:g; } $Global::input_source_header{$id} = $s; @@ -115,16 +114,25 @@ # before starting any $Global::JobQueue->total_jobs(); } +# Compute $Global::max_jobs_running +for my $sshlogin (values %Global::host) { + $sshlogin->max_jobs_running(); +} if($opt::pipepart) { + if($opt::roundrobin) { + # Compute size of -a + my $size = 0; + $size += -s $_ for @opt::a; + $Global::max_jobs_running or ::die_bug("Global::max_jobs_running not set"); + # Set --blocksize = size / no of proc + $opt::blocksize = 1 + $size / $Global::max_jobs_running; + } @Global::cat_partials = map { pipe_part_files($_) } @opt::a; # Unget the command as many times as there are parts $Global::JobQueue->{'commandlinequeue'}->unget( map { $Global::JobQueue->{'commandlinequeue'}->get() } @Global::cat_partials ); } -for my $sshlogin (values %Global::host) { - $sshlogin->max_jobs_running(); -} init_run_jobs(); my $sem; @@ -916,7 +924,6 @@ if(defined $opt::colsep) { $Global::trim = 'lr'; } if(defined $opt::header) { $opt::colsep = defined $opt::colsep ? $opt::colsep : "\t"; } if(defined $opt::trim) { $Global::trim = $opt::trim; } - if(defined $opt::roundrobin) { $opt::pipe = 1; } if(defined $opt::arg_sep) { $Global::arg_sep = $opt::arg_sep; } if(defined $opt::arg_file_sep) { $Global::arg_file_sep = $opt::arg_file_sep; } if(defined $opt::number_of_cpus) { print SSHLogin::no_of_cpus(),"\n"; wait_and_exit(0); } @@ -1090,7 +1097,7 @@ sub init_globals { # Defaults: - $Global::version = 20150822; + $Global::version = 20150922; $Global::progname = 'parallel'; $Global::infinity = 2**31; $Global::debug = 0; @@ -3252,7 +3259,7 @@ my $sleepsum = 0; my $sleep = 0.001; my @dead; - + while(@pids and $sleepsum < $sleep_max) { if($Global::killall) { # Killall => don't run reaper @@ -4834,7 +4841,9 @@ $max_system_proc_reached and last; my $before_getting_arg = time; - get_args_or_jobs() or last; + if(!$opt::roundrobin) { + get_args_or_jobs() or last; + } $wait_time_for_getting_args += time - $before_getting_arg; $system_limit++; @@ -5997,10 +6006,10 @@ # Returns: # $wrapped_command = the wrapped command my $command = shift; - my $script = '$c="'.::perl_quote_scalar($command).'";'. + my $script = ::spacefree(0,q{ if(sysread(STDIN, $buf, 1)) { - open($fh, "|-", $c) || die; + open($fh, "|-", "@ARGV") || die; syswrite($fh, $buf); while($read = sysread(STDIN, $buf, 32768)) { syswrite($fh, $buf); @@ -6010,7 +6019,8 @@ } }); ::debug("run",'Empty wrap: perl -e '.::shell_quote_scalar($script)."\n"); - return 'perl -e '.::shell_quote_scalar($script); + return 'perl -e '.::shell_quote_scalar($script)." ". + ::shell_quote_scalar($Global::shell." -c ".::shell_quote_scalar($command)); } sub filter_through_compress { @@ -7773,6 +7783,8 @@ 'seq' => $seq, 'len' => \%len, 'arg_list' => [], + 'arg_list_flat' => [], + 'arg_list_flat_orig' => [undef], 'arg_queue' => $arg_queue, 'max_number_of_args' => $max_number_of_args, 'replacecount' => \%replacecount, @@ -7900,19 +7912,23 @@ # Returns: N/A my $self = shift; my $record = shift; + push @{$self->{'arg_list_flat_orig'}}, map { $_->orig() } @$record; + push @{$self->{'arg_list_flat'}}, @$record; push @{$self->{'arg_list'}}, $record; + # Make @arg available for {= =} + *Arg::arg = $self->{'arg_list_flat_orig'}; my $quote_arg = $Global::noquote ? 0 : not $Global::quoting; - my $rep; - for my $arg (@$record) { - if(defined $arg) { - for my $perlexpr (keys %{$self->{'replacecount'}}) { - # 50% faster than below - $self->{'len'}{$perlexpr} += length $arg->replace($perlexpr,$quote_arg,$self); - # $rep = $arg->replace($perlexpr,$quote_arg,$self); - # $self->{'len'}{$perlexpr} += length $rep; - # ::debug("length", "Length: ", length $rep, - # "(", $perlexpr, "=>", $rep, ")\n"); + for my $perlexpr (keys %{$self->{'replacecount'}}) { + if($perlexpr =~ /^(\d+) /) { + # Positional + defined($record->[$1-1]) or next; + $self->{'len'}{$perlexpr} += length $record->[$1-1]->replace($perlexpr,$quote_arg,$self); + } else { + for my $arg (@$record) { + if(defined $arg) { + $self->{'len'}{$perlexpr} += length $arg->replace($perlexpr,$quote_arg,$self); + } } } } @@ -7924,12 +7940,20 @@ # the last record my $self = shift; my $record = pop @{$self->{'arg_list'}}; + # pop off arguments from @$record + splice @{$self->{'arg_list_flat_orig'}}, -($#$record+1), $#$record+1; + splice @{$self->{'arg_list_flat'}}, -($#$record+1), $#$record+1; my $quote_arg = $Global::noquote ? 0 : not $Global::quoting; - for my $arg (@$record) { - if(defined $arg) { - for my $perlexpr (keys %{$self->{'replacecount'}}) { - $self->{'len'}{$perlexpr} -= - length $arg->replace($perlexpr,$quote_arg,$self); + for my $perlexpr (keys %{$self->{'replacecount'}}) { + if($perlexpr =~ /^(\d+) /) { + # Positional + defined($record->[$1-1]) or next; + $self->{'len'}{$perlexpr} -= length $record->[$1-1]->replace($perlexpr,$quote_arg,$self); + } else { + for my $arg (@$record) { + if(defined $arg) { + $self->{'len'}{$perlexpr} -= length $arg->replace($perlexpr,$quote_arg,$self); + } } } } @@ -7937,15 +7961,17 @@ } sub pop_all { - # Remove all arguments and zeros the length of replacement strings + # Remove all arguments and zeros the length of replacement perlexpr # Returns: # all records my $self = shift; my @popped = @{$self->{'arg_list'}}; - for my $replacement_string (keys %{$self->{'replacecount'}}) { - $self->{'len'}{$replacement_string} = 0; + for my $perlexpr (keys %{$self->{'replacecount'}}) { + $self->{'len'}{$perlexpr} = 0; } $self->{'arg_list'} = []; + $self->{'arg_list_flat_orig'} = [undef]; + $self->{'arg_list_flat'} = []; return @popped; } @@ -8127,53 +8153,45 @@ my $context_replace; my $perl_expressions_as_re; my @arg; + my %words_containing_replacement_strings; sub fish_out_words_containing_replacement_strings { - my %word; - for (@target) { - my $tt = $_; - ::debug("replace", "Target: $tt"); - # Command line template: - # a{1}b{}c{}d - # becomes: - # a{=1 $_=$_ =}b{= $_=$_ =}c{= $_=$_ =}d - # becomes: - # a\257<1 $_=$_ \257>b\257< $_=$_ \257>c\257< $_=$_ \257>d - # Input A B C (no context) becomes: - # A B C => aAbA B CcA B Cd - # Input A B C (context -X) becomes: - # A B C => aAbAcAd aAbBcBd aAbCcCd - if($context_replace) { - while($tt =~ s/([^\s\257]* # before {= + if(not $words_containing_replacement_strings{$context_replace,@target}) { + my %word; + for (@target) { + my $tt = $_; + ::debug("replace", "Target: $tt"); + # Command line template: + # a{1}b{}c{}d + # becomes: + # a{=1 $_=$_ =}b{= $_=$_ =}c{= $_=$_ =}d + # becomes: + # a\257<1 $_=$_ \257>b\257< $_=$_ \257>c\257< $_=$_ \257>d + # Input A B C (no context) becomes: + # A B C => aAbA B CcA B Cd + # Input A B C (context -X) becomes: + # A B C => aAbAcAd aAbBcBd aAbCcCd + if($context_replace) { + while($tt =~ s/([^\s\257]* # before {= (?: \257< # {= [^\257]*? # The perl expression \257> # =} [^\s\257]* # after =} )+)/ /x) { - # $1 = pre \257 perlexpr \257 post - $word{"$1"} ||= 1; - } - } else { - while($tt =~ s/( (?: \257<([^\257]*?)\257>) )//x) { - # $f = \257 perlexpr \257 - $word{$1} ||= 1; + # $1 = pre \257 perlexpr \257 post + $word{"$1"} ||= 1; + } + } else { + while($tt =~ s/( (?: \257<([^\257]*?)\257>) )//x) { + # $f = \257 perlexpr \257 + $word{$1} ||= 1; + } } } + @{$words_containing_replacement_strings{$context_replace,@target}} = keys %word } - return keys %word; - } - - sub flatten_arg_list { - my $arglist_ref = shift; - @arg = (); - for my $record (@$arglist_ref) { - # $self->{'arg_list'} = [ [Arg11, Arg12], [Arg21, Arg22], [Arg31, Arg32] ] - # Merge arg-objects from records into @arg for easy access - CORE::push @arg, @$record; - } - # Add one arg if empty to allow {#} and {%} to be computed only once - if(not @arg) { @arg = (Arg->new("")); } + return @{$words_containing_replacement_strings{$context_replace,@target}}; } sub replace_placeholders { @@ -8191,28 +8209,32 @@ my $quote = shift; my $quote_arg = shift; my %replace; + # -X = context replace (fish_out_words_containing_replacement_strings) $context_replace = $self->{'context_replace'}; @target = @$targetref; ::debug("replace", "Replace @target\n"); - # -X = context replace - # maybe multiple input sources - # maybe --xapply if(not @target) { # @target is empty: Return empty array return @target; } - # Fish out the words that have replacement strings in them - my @word = fish_out_words_containing_replacement_strings(); - flatten_arg_list($self->{'arg_list'}); # Make it possible to use $arg[2] in {= =} - @Arg::arg = (undef, map { $_->orig() } @arg); + *Arg::arg = $self->{'arg_list_flat_orig'}; + # Flat list: + # $self->{'arg_list'} = [ [Arg11, Arg12], [Arg21, Arg22], [Arg31, Arg32] ] + # $self->{'arg_list_flat'} = [ Arg11, Arg12, Arg21, Arg22, Arg31, Arg32 ] + if(not @{$self->{'arg_list_flat'}}) { + @{$self->{'arg_list_flat'}} = Arg->new(""); + } + my $argref = $self->{'arg_list_flat'}; # Number of arguments - used for positional arguments - my $n = $#arg+1; + my $n = $#$argref+1; - # This is actually a CommandLine-object, + # $self is actually a CommandLine-object, # but it looks nice to be able to say {= $job->slot() =} my $job = $self; - for my $word (@word) { + # Fish out the words that have replacement strings in them + for my $word ( + fish_out_words_containing_replacement_strings()) { # word = AB \257< perlexpr \257> CD \257< perlexpr \257> EF ::debug("replace", "Replacing in $word\n"); my $normal_replace; @@ -8222,7 +8244,7 @@ # push to replace word value $perl_expressions_as_re ||= join("|", map {s/^-?\d+//; "\Q$_\E"} keys %{$self->{'replacecount'}}); - for my $arg (@arg) { + for my $arg (@$argref) { my $val = $word; # Replace {= perl expr =} with value for each arg $val =~ s{\257<(-?\d+)?($perl_expressions_as_re)\257>} @@ -8230,8 +8252,8 @@ if($1) { # Positional replace # Find the relevant arg and replace it - ($arg[$1 > 0 ? $1-1 : $n+$1] ? # If defined: replace - $arg[$1 > 0 ? $1-1 : $n+$1]-> + ($argref->[$1 > 0 ? $1-1 : $n+$1] ? # If defined: replace + $argref->[$1 > 0 ? $1-1 : $n+$1]-> replace($2,$quote_arg,$self) : ""); } else { @@ -8254,7 +8276,6 @@ if($quote) { @target = ::shell_quote(@target); } - # ::debug("replace", "%replace=",::my_dump(%replace),"\n"); if(%replace) { # Substitute the replace strings with the replacement values # Must be sorted by length if a short word is a substring of a long word @@ -8282,7 +8303,7 @@ my $class = shift; my $commandref = shift; my $read_from = shift; - my $context_replace = shift; + my $context_replace = shift || 0; my $max_number_of_args = shift; my $return_files = shift; my @unget = (); @@ -8300,6 +8321,7 @@ } } # Replace replacement strings with {= perl expr =} + # '{=' 'perlexpr' '=}' => '{= perlexpr =}' @command = merge_rpl_parts(@command); # Protect matching inside {= perl expr =} @@ -8324,7 +8346,7 @@ # with the {= perl expr =} # Avoid replacing inside existing {= perl expr =} while(s{((^|\257>)[^\257]*?) # Don't replace after \257 unless \257> - \Q$rpl\E} + \Q$rpl\E} {$1\257<$Global::rpl{$rpl}\257>}xg) { } # Do the same for the positional replacement strings @@ -8332,11 +8354,14 @@ $posrpl = $rpl; if($posrpl =~ s/^\{//) { # Only do this if the shorthand start with { - s{\{(-?\d+)\Q$posrpl\E} - {\257<$1 $Global::rpl{$rpl}\257>}g; + while(s{((^|\257>)[^\257]*?) # Don't replace after \257 unless \257> + \{(-?\d+)\Q$posrpl\E} + {$1\257<$3 $Global::rpl{$rpl}\257>}xg) { + } } } } + # Add {} if no replacement strings in @command ($replacecount_ref, $len_ref, @command) = replacement_counts_and_lengths($return_files,@command); @@ -8817,7 +8842,13 @@ if($opt::xapply) { return $self->xapply_get(); } else { - return $self->nest_get(); + my $arglist = $self->nest_get(); + # Flush cached computed values + # To fix: parallel echo {%} ::: 1 2 ::: 4 3 + if($arglist) { + map { $_->flush_cache() } @$arglist; + } + return $arglist; } } @@ -8962,7 +8993,6 @@ return undef; } } -# ::debug("run", "read $arg\n"); # Remove delimiter $arg =~ s:$/$::; if($Global::end_of_file_string and @@ -9051,9 +9081,9 @@ sub Q { # Q alias for ::shell_quote_scalar # Run shell_quote_scalar once to set the reference to the sub - my @a = ::shell_quote_scalar(@_); + my $a = ::shell_quote_scalar(@_); *Q = \&::shell_quote_scalar; - return @a; + return $a; } { @@ -9069,33 +9099,47 @@ # This is actually a CommandLine-object, # but it looks nice to be able to say {= $job->slot() =} my $job = shift; - $perlexpr =~ s/^-?\d+ +//; # Positional replace treated as normal replace - local $_; - if($Global::trim eq "n") { - $_ = $self->{'orig'}; - } else { - $_ = trim_of($self->{'orig'}); - } - ::debug("replace", "eval ", $perlexpr, " ", $_, "\n"); - if(not $perleval{$perlexpr}) { - # Make an anonymous function of the $perlexpr - # And more importantly: Compile it only once - if($perleval{$perlexpr} = - eval('sub { no strict; no warnings; my $job = shift; '. - $perlexpr.' }')) { - # All is good + $perlexpr =~ s/^(-?\d+)? *//; # Positional replace treated as normal replace + if(not $self->{'cache'}{$perlexpr}) { + # Only compute the value once + # Use $_ as the variable to change + local $_; + if($Global::trim eq "n") { + $_ = $self->{'orig'}; } else { - # The eval failed. Maybe $perlexpr is invalid perl? - ::error("Cannot use $perlexpr: $@"); - ::wait_and_exit(255); + # Trim the input + $_ = trim_of($self->{'orig'}); } + ::debug("replace", "eval ", $perlexpr, " ", $_, "\n"); + if(not $perleval{$perlexpr}) { + # Make an anonymous function of the $perlexpr + # And more importantly: Compile it only once + if($perleval{$perlexpr} = + eval('sub { no strict; no warnings; my $job = shift; '. + $perlexpr.' }')) { + # All is good + } else { + # The eval failed. Maybe $perlexpr is invalid perl? + ::error("Cannot use $perlexpr: $@"); + ::wait_and_exit(255); + } + } + # Execute the function + $perleval{$perlexpr}->($job); + $self->{'cache'}{$perlexpr} = $_; } - # Execute the function - $perleval{$perlexpr}->($job); - return $quote ? ::shell_quote_scalar($_) : $_; + # Return the value quoted if needed + return($quote ? ::shell_quote_scalar($self->{'cache'}{$perlexpr}) + : $self->{'cache'}{$perlexpr}); } } +sub flush_cache { + # Flush cache of computed values + my $self = shift; + $self->{'cache'} = undef; +} + sub orig { my $self = shift; return $self->{'orig'}; diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/src/sql new/parallel-20150922/src/sql --- old/parallel-20150822/src/sql 2015-08-22 11:40:23.000000000 +0200 +++ new/parallel-20150922/src/sql 2015-09-21 19:25:53.000000000 +0200 @@ -566,7 +566,7 @@ exit ($err); sub parse_options { - $Global::version = 20150822; + $Global::version = 20150922; $Global::progname = 'sql'; # This must be done first as this may exec myself diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20150822/src/sql.1 new/parallel-20150922/src/sql.1 --- old/parallel-20150822/src/sql.1 2015-08-22 11:40:33.000000000 +0200 +++ new/parallel-20150922/src/sql.1 2015-09-21 19:26:12.000000000 +0200 @@ -133,7 +133,7 @@ .\" ======================================================================== .\" .IX Title "SQL 1" -.TH SQL 1 "2015-08-22" "20150822" "parallel" +.TH SQL 1 "2015-09-21" "20150922" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l Files old/parallel-20150822/src/sql.pdf and new/parallel-20150922/src/sql.pdf differ ++++++ parallel-20150822.tar.bz2.sig -> parallel-20150922.tar.bz2.sig ++++++ --- /work/SRC/openSUSE:Factory/gnu_parallel/parallel-20150822.tar.bz2.sig 2015-09-09 20:21:18.000000000 +0200 +++ /work/SRC/openSUSE:Factory/.gnu_parallel.new/parallel-20150922.tar.bz2.sig 2015-09-24 06:16:07.000000000 +0200 @@ -2,41 +2,41 @@ # To check the signature run: # echo | gpg -# gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve parallel-20150822.tar.bz2.sig +# gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve parallel-20150922.tar.bz2.sig echo | gpg 2>/dev/null gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve $0 exit $? -----BEGIN PGP SIGNATURE----- -Version: GnuPG v2.0.22 (GNU/Linux) +Version: GnuPG v1 -iQTwBAABCgAGBQJV2EQJAAoJENGrRRaIiIiIBDwmoJ/yd2AbLHGy1mumdxWdcbr8 -xS8gSPgkTDCwcCaxHxIf88bONqvOBOHsdEOuGTAA2wFSGBKbALF6Qd0ucGxQm21Y -ka1UwHYkkjbnbMbS9myrWar2zFr5Bz8LyKBsunY4DjzZ/uiYqe1M2ohnEvxUvEUK -6jF2NYtTjI1QqN5xiaZCJxYmTSrdmNjuEbZ4X4ANCCGZn/4V2yQeByhm3+0R+Gn1 -S1WRJDGHJB3RnQClcXVoyDBOxqDTm/+/QtXeh5NMoNHKH6gmS6psQ3RvPPOfkwct -vNhwAZq6oTrHVn6hoKlK8BLWxNewRqCc6MFX5/Bg6cJyaf5ubjQUn/sayQJnv3zA -NbfGDt5mZdVZz371DHdM7G1lmyq8F4YfKyzuND5dFCua57+yQU72wnKp3pvEWMT3 -pdnkCkQ8MBZyx+RKFKHjYFPJEKDtMZ3ZZNag6/1+Fzt67AmAb18F4NOdH6vwTppn -sdPPXEaJlPIf/ij1xVnF11vrjDp3V6+deqraa1HD2XVj6g3Zrx9KewyLqeGDtYjU -rM240NVYABq99xcmhhx+P8Pc/4x0/XRTEuSmv26YDMJq2AyAJwwEFgAQCtpLbUsD -s1tM7eyniWmduhxQixMAS+yuuNAFp/ATWPafYO8kaKu94fQpmpYcyuWn9TNi+6aT -jZfo8U7xJ0fbl9qCCvUODqs63JdGT52x8MNLi0RsLY9dDxRgLUrqSqVT6CfczZAp -5sflqj7D/UZawK7BqnbtM+UyMCtFOV7vKZ7zd6cMuAoRRAjRNOG+83BPAxriWXtb -wun+gBf9xk1HlvDxi4j6FT0ro4KNfqZB1JrMiIYqMbpgqwN2p0WWb3I/Shi2RnKk -IaX3Esv6Ji3G2MPW/hhYcoHDWHb0vzbSvV5KRU4w7ALAwKFsy106/JgUfbGHsUK7 -3lNpfWjkXuVzqQJBb5Lw5L8rArLpL9w4XJyfxIKE4r0NcHJVcPFgSGh2aO9IsUQm -zGw6tIeKobRXkGWTsd8CV0yiKcCQjGF6wKxQbxCZpmCZUv3b8CIn/vgm56cka3Dc -JmAVhE4ZjNfNR0BF74Da/rkggj30/fu+quMDyVNerfRKfR34G+4TQ5pAEq5UICZ9 -+BGUkRGJxVKN+WCPn1+xYAPqu+3F6kaNQt9lJbct9lhDaIlfZ9/c42eK5X/4aQo/ -XE+Z+bg8sBxaK84Sht1g4dbV6H+7EpzL539cmCT4Hw1WYyGN8uoSB+KPA9geyPDN -KVcOcl6XNV/dvzjoK+FWJyKIld+pClQ8lJH+EZqHHvCnRIMXbPWxAotewwgiHSwu -wn+iyVdTMzZIxalsmQfb03MPeGDB8QznYYgYNer11tqaN8sNfs7nxvrDy2J+Qqdz -GY4S1rFYSLuAsG8H6JU7CbnVHEONDoB+vEJ6xsNYFRAcoJua8z8/1jB5ASKc10IK -idrtWEt/jjSi5q68xRaBVvDjh8UjHzOag+ndT3om01UofXi5ZeLXpGjO6BJOt8iK -7ek61IjEVuen6oud57/s61kruLIq/RJ4kbgK0drg6QElcRRK5riXfvr2M8TKc2dh -3yto8jcogMRMfsbdBQYOxYZCbyyDz56TbzT9lto6p2DwcIa5rxAG7NgEJnvua20S -eH4hBm8ynd/3lqb1mkarWJh8yA== -=51yS +iQTwBAABCgAGBQJWAEQgAAoJENGrRRaIiIiIJ0wmn0nOgAn1eLwnqfa9WLFeydND ++pM8Xt5hJVdo0ReNVec4zjk4DdBuGxqnYkPWDctHw0eelxcjxI74TMXx+K81ta1V +bEnVAgLOyeaXFXsn3FrtHajJxM7rO11vM8sCiBzCFq7v4qZY0CZOPO/UYC4Tu4Bf +nPNJXeAr3ZJVcZddI1Bnl1WwVEgDLZi/xAOalSsIk5/cp+X9GUwKRYt6cR8zQLuo +bbBDqPltCc0W6UNC9IGtJlU0nKG2ta4sftjya4xqsnfWtJO5T3zZ9qTNhppWjz9N +JmKfeWdenwhKR40aKEAj/TLM313IQWKBQ8GoU+XlvaqxvwPyF9zBFf8JYYafsEtT +hZs0GAkNPhIBo47IBT8+8kIK/ufNmTh9k+kB1spOZr5gkMCWreHDZzv0AjNbZ243 +BWcWnkFPz4VCLJ5FAOqgEiXgzJsxwBer6Oo0mmAH7vfHh9bZtkV+Sy1713JUUuCr +PsKJ1ky9IXw7MvOo9xmgS8YB+eZejSMLX5aQLhu8IHhTHzQ0gw9FuD0RJasSkRNK +M5P5xPAYIIQ2GMX5gkKpIJN1lzvUg59lj41LpZVlcHm6J4y7cup4IMsQJtkWPo1o +5YjLu/a1U5xixWOjB82PY4l44IJSKpLLGldMBCMl5V+FIyrTUC0Ui6dQO5fN8Jhj +gXa/qB97FZkZiDp8UuOtxxLKcd1bqkqZUDmPgw0pmvX2qg8Rxdui4mvXhr+zxE+K +sMTflXvQuNzBYFla1/v5lHNBbYeuaoEyHSW/tGqDPCSSbi9pEKNncZos3HBFroTg +NfwfOmChDqY44/7dx0R4j5MFA0Yl50w+O0WtiiXb8fSlkiTumwRh91XSeIoWg/xz +Y/8SEpZH5W13uHpgHvXPuLrvIL8F+VLlIulHkiEGye+g/Xzi2UoBdV80MtCSiI/K +hh7xiaKcn9ZGZPAPfvYo80K4sDaECRvCMnNZKH23Tyt0nf/1fqSdsMDK4QOwVa06 +H/MJ+MMRXpxwwCeRcVRzYIGDmh8dTBLhh+G697+ifhewXJubmiX8aMQ+y9nTJ0SI +yATFCuuy5hEbPAbjCoXYmoVBijk3tUeZWTxKg0664KyWbBFLpj+/pUZzr/B+LEq/ +T76bxNZTr+7rfwTZeC6xVMhfgBkEAz48kmH81iKqFXCTZdibl1MXQL/auC+a4tOI +HNcf41fe+Kr9TPURhzPmDWUQhp7eubuLVObyYtwsw4uUxz4noKdv6NRvNMy8pusD +Xz3mzcl1c3CVbhebHo03/FgkjpNt+W/RmsI6oF0MalHAvJOtABVJOA6GiJQy31cm +pCqfrB7cjlkoiI5Ek0Z79Y7bLnV6FDyZbC1f76+u/+QaRi3NwOyiX92fTntyIEgl +AGVwx0jBaXmggwWIVdJIGqNcCRLCkIE0x7QwNkRFqsAEivUKgoJSz4Yn7IEoY2VY +Jf+CnD2DmImKq7GT/gVA1PGC48SwkWVAGlXhTJSkkdMxN2a9RZR3GWyfTLN8zPQg +jVcYaWIhFqZALMIplPAzc1urx2tuxjaAzbGO2OJdPfh9iQJL1LLl67Ugokr5nN/j +BMWtRpMEKrTBIeWl7moJ6jck0TdDfaINgu222TKZkG/miI0/uc7XNd0YeBflY89y +ZfBgthXlGDRTLJ7RmG13Kc2y3A== +=dk+y -----END PGP SIGNATURE-----
