Hello community, here is the log from the commit of package gnu_parallel for openSUSE:Factory checked in at 2015-11-26 17:03:43 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Comparing /work/SRC/openSUSE:Factory/gnu_parallel (Old) and /work/SRC/openSUSE:Factory/.gnu_parallel.new (New) ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Package is "gnu_parallel" Changes: -------- --- /work/SRC/openSUSE:Factory/gnu_parallel/gnu_parallel.changes 2015-10-25 14:18:14.000000000 +0100 +++ /work/SRC/openSUSE:Factory/.gnu_parallel.new/gnu_parallel.changes 2015-11-26 17:03:45.000000000 +0100 @@ -1,0 +2,8 @@ +Mon Nov 23 12:11:14 UTC 2015 - [email protected] + +- Update to 20151122 + * Automating large numbers of tasks https://rcc.uchicago.edu/docs/tutorials/kicp-tutorials/running-jobs.html + * Max out your IOPs with GNU Parallel http://blog.bitratchet.com/2015/11/11/max-out-your-iops-with-gnu-parallel/ + * Bug fixes and man page updates. + +------------------------------------------------------------------- Old: ---- parallel-20151022.tar.bz2 parallel-20151022.tar.bz2.sig New: ---- parallel-20151122.tar.bz2 parallel-20151122.tar.bz2.sig ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ Other differences: ------------------ ++++++ gnu_parallel.spec ++++++ --- /var/tmp/diff_new_pack.Jj6Ims/_old 2015-11-26 17:03:46.000000000 +0100 +++ /var/tmp/diff_new_pack.Jj6Ims/_new 2015-11-26 17:03:46.000000000 +0100 @@ -17,16 +17,16 @@ Name: gnu_parallel -Version: 20151022 +Version: 20151122 Release: 0 Summary: Shell tool for executing jobs in parallel License: GPL-3.0+ Group: Productivity/File utilities Url: http://www.gnu.org/software/parallel/ #DL-URL: http://ftp.gnu.org/gnu/parallel/ -Source0: http://ftp.gnu.org/gnu/parallel/parallel-%{version}.tar.bz2 -Source1: http://ftp.gnu.org/gnu/parallel/parallel-%{version}.tar.bz2.sig -Source2: %{name}.keyring +Source: http://ftp.gnu.org/gnu/parallel/parallel-%version.tar.bz2 +Source2: http://ftp.gnu.org/gnu/parallel/parallel-%version.tar.bz2.sig +Source3: %name.keyring BuildRoot: %{_tmppath}/%{name}-%{version}-build BuildArch: noarch @@ -49,27 +49,27 @@ This subpackage contains the documentation for Parallel. %prep -%setup -q -n parallel-%{version} +%setup -q -n parallel-%version %build -%configure --docdir="%{_docdir}/%{name}" +%configure --docdir="%_docdir/%name" make %{?_smp_mflags} %install %make_install -cp -a CITATION COPYING NEWS README "%{buildroot}/%{_docdir}/%{name}/" +cp -a CITATION COPYING NEWS README "%buildroot/%_docdir/%name/" %files %defattr(-,root,root) -%{_bindir}/niceload -%{_bindir}/parallel -%{_bindir}/sem -%{_bindir}/sql -%{_mandir}/man1/*.1* -%{_mandir}/man7/*.7* +%_bindir/niceload +%_bindir/parallel +%_bindir/sem +%_bindir/sql +%_mandir/man1/*.1* +%_mandir/man7/*.7* %files doc %defattr(-,root,root) -%{_docdir}/%{name}/ +%_docdir/%name/ %changelog ++++++ parallel-20151022.tar.bz2 -> parallel-20151122.tar.bz2 ++++++ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/NEWS new/parallel-20151122/NEWS --- old/parallel-20151022/NEWS 2015-10-21 23:34:26.000000000 +0200 +++ new/parallel-20151122/NEWS 2015-11-22 19:49:29.000000000 +0100 @@ -1,8 +1,41 @@ +20151122 + +* GNU Parallel packaged for CERN CentOS: http://linuxsoft.cern.ch/cern/centos/7/cern/x86_64/repoview/parallel.html + +* GNU Parallel was cited in: The Outer Solar System Origins Survey: I. Design and First-Quarter Discoveries http://arxiv.org/pdf/1511.02895.pdf + +* GNU Parallel was cited in: Contrasting genetic architectures of schizophrenia and other complex diseases using fast variance-components analysis http://www.nature.com/ng/journal/vaop/ncurrent/full/ng.3431.html + +* GNU Parallel was cited in: Named-Entity Chunking for Norwegian Text using Support Vector Machines http://ojs.bibsys.no/index.php/NIK/article/viewFile/248/211 + +* GNU Parallel was cited in: Multiscale Estimation of Binding Kinetics Using Brownian Dynamics, Molecular Dynamics and Milestoning http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1004381#pcbi.1004381.ref072 + +* GNU Parallel was cited in: A Detailed Characterization of Errors in Logic Circuits due to Single-Event Transients http://ieeexplore.ieee.org/xpl/articleDetails.jsp?tp=&arnumber=7302348&url=http%3A%2F%2Fieeexplore.ieee.org%2Fxpls%2Fabs_all.jsp%3Farnumber%3D7302348 + +* GNU Parallel was cited in: Exploiting Multiple Levels of Parallelism in Sparse Matrix-Matrix Multiplication http://arxiv.org/pdf/1510.00844.pdf + +* GNU Parallel was cited in: Arabic Speaker Emotion Classification Using Rhythm Metrics and Neural Networks http://www.eurasip.org/Proceedings/Eusipco/Eusipco2015/papers/1570104855.pdf + +* GNU Parallel was cited in: Stride Search: a general algorithm for storm detection in high resolution climate data http://www.geosci-model-dev-discuss.net/8/7727/2015/gmdd-8-7727-2015.pdf + +* GNU Parallel was cited in: Decomposing Digital Paintings into Layers via RGB-space Geometry http://arxiv.org/pdf/1509.03335.pdf + +* GNU Parallel was cited in: Structure and evolutionary history of a large family of NLR proteins in the zebrafish http://www.biorxiv.org/content/biorxiv/early/2015/09/18/027151.full.pdf + +* GNU Parallel was cited in: Evolution of movement strategies under competitive interactions http://digital.csic.es/bitstream/10261/115973/1/evolution_movement_strategies_Kiziridis.pdf + +* Automating large numbers of tasks https://rcc.uchicago.edu/docs/tutorials/kicp-tutorials/running-jobs.html + +* Max out your IOPs with GNU Parallel http://blog.bitratchet.com/2015/11/11/max-out-your-iops-with-gnu-parallel/ + +* Bug fixes and man page updates. + + 20151022 -* --plus makes it possible to use {##} as a short had for {= - $_=$Global::JobQueue->total_jobs() =} which gives the the number of - jobs to run. +* --plus makes it possible to use {##} as a shorthand for + {= $_=$Global::JobQueue->total_jobs() =} which gives the the number + of jobs to run in total. * {= $_=$Global::JobQueue->total_jobs() =} is incompatible with -X, -m, and --xargs. @@ -37,30 +70,48 @@ 20150922 -* GNU Parallel was cited in: Flexible Modeling of Epidemics with an Empirical Bayes Framework http://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1004382 +* GNU Parallel was cited in: Flexible Modeling of Epidemics with an + Empirical Bayes Framework + http://journals.plos.org/ploscompbiol/article?id=10.1371%2Fjournal.pcbi.1004382 -* GNU Parallel was cited in: BL1: 2D Potts Model with a Twist https://sucs.swan.ac.uk/~rjames93/Dissertation.pdf +* GNU Parallel was cited in: BL1: 2D Potts Model with a Twist + https://sucs.swan.ac.uk/~rjames93/Dissertation.pdf -* GNU Parallel was cited in: DockBench: An Integrated Informatic Platform Bridging the Gap between the Robust Validation of Docking Protocols and Virtual Screening Simulations http://www.mdpi.com/1420-3049/20/6/9977/pdf +* GNU Parallel was cited in: DockBench: An Integrated Informatic + Platform Bridging the Gap between the Robust Validation of Docking + Protocols and Virtual Screening Simulations + http://www.mdpi.com/1420-3049/20/6/9977/pdf -* GNU Parallel was cited in: A Scalable Parallel Implementation of Evolutionary Algorithms for -Multi-Objective Optimization on GPUs http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7257074 +* GNU Parallel was cited in: A Scalable Parallel Implementation of + Evolutionary Algorithms for Multi-Objective Optimization on GPUs + http://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=7257074 -* GNU Parallel was cited in: Tools and techniques for computational reproducibility http://biorxiv.org/content/biorxiv/early/2015/07/17/022707.full.pdf +* GNU Parallel was cited in: Tools and techniques for computational + reproducibility + http://biorxiv.org/content/biorxiv/early/2015/07/17/022707.full.pdf -* GNU Parallel was cited in: How Can We Measure the Similarity Between Résumés of Selected Candidates for a Job? http://www.researchgate.net/publication/275954089_How_can_we_measure_the_similarity_between_rsums_of_selected_candidates_for_a_job +* GNU Parallel was cited in: How Can We Measure the Similarity Between + Résumés of Selected Candidates for a Job? + http://www.researchgate.net/publication/275954089_How_can_we_measure_the_similarity_between_rsums_of_selected_candidates_for_a_job -* GNU Parallel was cited in: Interplay of cell dynamics and epithelial tension during morphogenesis of the Drosophila pupal wing http://www.researchgate.net/profile/Raphael_Etournay/publication/279061859_Interplay_of_cell_dynamics_and_epithelial_tension_during_morphogenesis_of_the_Drosophila_pupal_wing/links/558a95ad08aeae8413bcceea.pdf +* GNU Parallel was cited in: Interplay of cell dynamics and epithelial + tension during morphogenesis of the Drosophila pupal wing + http://www.researchgate.net/profile/Raphael_Etournay/publication/279061859_Interplay_of_cell_dynamics_and_epithelial_tension_during_morphogenesis_of_the_Drosophila_pupal_wing/links/558a95ad08aeae8413bcceea.pdf -* Third-party selling GNU Parallel T-shirts http://www.aliexpress.com/item/2015F-BSO-GNU-LOGO-GNU-PARALLEL-men-s-shirt-sleeve-visual-illusion-error/32464827966.html +* Third-party selling GNU Parallel T-shirts + http://www.aliexpress.com/item/2015F-BSO-GNU-LOGO-GNU-PARALLEL-men-s-shirt-sleeve-visual-illusion-error/32464827966.html -* Joys of gnu parallel http://scottolesen.com/index.php/2015/08/26/joys-of-gnu-parallel/ +* Joys of gnu parallel + http://scottolesen.com/index.php/2015/08/26/joys-of-gnu-parallel/ -* Crop and resize images with bash and ImageMagick https://www.simonholywell.com/post/2015/08/image-resize-crop-bash-imagemagick/ +* Crop and resize images with bash and ImageMagick + https://www.simonholywell.com/post/2015/08/image-resize-crop-bash-imagemagick/ -* Three Ways to Script Processes in Parallel http://www.codeword.xyz/2015/09/02/three-ways-to-script-processes-in-parallel/ +* Three Ways to Script Processes in Parallel + http://www.codeword.xyz/2015/09/02/three-ways-to-script-processes-in-parallel/ -* What It Looks Like to Process 3.5 Million Books http://blog.gdeltproject.org/what-it-looks-like-to-process-3-5-million-books/ +* What It Looks Like to Process 3.5 Million Books + http://blog.gdeltproject.org/what-it-looks-like-to-process-3-5-million-books/ * Bug fixes and man page updates. diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/README new/parallel-20151122/README --- old/parallel-20151022/README 2015-10-21 23:26:58.000000000 +0200 +++ new/parallel-20151122/README 2015-11-22 19:47:28.000000000 +0100 @@ -40,9 +40,9 @@ Full installation of GNU Parallel is as simple as: - wget http://ftpmirror.gnu.org/parallel/parallel-20151022.tar.bz2 - bzip2 -dc parallel-20151022.tar.bz2 | tar xvf - - cd parallel-20151022 + wget http://ftpmirror.gnu.org/parallel/parallel-20151122.tar.bz2 + bzip2 -dc parallel-20151122.tar.bz2 | tar xvf - + cd parallel-20151122 ./configure && make && sudo make install @@ -51,9 +51,9 @@ If you are not root you can add ~/bin to your path and install in ~/bin and ~/share: - wget http://ftpmirror.gnu.org/parallel/parallel-20151022.tar.bz2 - bzip2 -dc parallel-20151022.tar.bz2 | tar xvf - - cd parallel-20151022 + wget http://ftpmirror.gnu.org/parallel/parallel-20151122.tar.bz2 + bzip2 -dc parallel-20151122.tar.bz2 | tar xvf - + cd parallel-20151122 ./configure --prefix=$HOME && make && make install Or if your system lacks 'make' you can simply copy src/parallel diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/configure new/parallel-20151122/configure --- old/parallel-20151022/configure 2015-10-21 23:27:08.000000000 +0200 +++ new/parallel-20151122/configure 2015-11-22 19:47:36.000000000 +0100 @@ -1,6 +1,6 @@ #! /bin/sh # Guess values for system-dependent variables and create Makefiles. -# Generated by GNU Autoconf 2.69 for parallel 20151022. +# Generated by GNU Autoconf 2.69 for parallel 20151122. # # Report bugs to <[email protected]>. # @@ -579,8 +579,8 @@ # Identity of this package. PACKAGE_NAME='parallel' PACKAGE_TARNAME='parallel' -PACKAGE_VERSION='20151022' -PACKAGE_STRING='parallel 20151022' +PACKAGE_VERSION='20151122' +PACKAGE_STRING='parallel 20151122' PACKAGE_BUGREPORT='[email protected]' PACKAGE_URL='' @@ -1203,7 +1203,7 @@ # Omit some internal or obsolete options to make the list less imposing. # This message is too long to be a string in the A/UX 3.1 sh. cat <<_ACEOF -\`configure' configures parallel 20151022 to adapt to many kinds of systems. +\`configure' configures parallel 20151122 to adapt to many kinds of systems. Usage: $0 [OPTION]... [VAR=VALUE]... @@ -1269,7 +1269,7 @@ if test -n "$ac_init_help"; then case $ac_init_help in - short | recursive ) echo "Configuration of parallel 20151022:";; + short | recursive ) echo "Configuration of parallel 20151122:";; esac cat <<\_ACEOF @@ -1345,7 +1345,7 @@ test -n "$ac_init_help" && exit $ac_status if $ac_init_version; then cat <<\_ACEOF -parallel configure 20151022 +parallel configure 20151122 generated by GNU Autoconf 2.69 Copyright (C) 2012 Free Software Foundation, Inc. @@ -1362,7 +1362,7 @@ This file contains any messages produced by compilers while running configure, to aid debugging if configure makes a mistake. -It was created by parallel $as_me 20151022, which was +It was created by parallel $as_me 20151122, which was generated by GNU Autoconf 2.69. Invocation command line was $ $0 $@ @@ -2225,7 +2225,7 @@ # Define the identity of the package. PACKAGE='parallel' - VERSION='20151022' + VERSION='20151122' cat >>confdefs.h <<_ACEOF @@ -2867,7 +2867,7 @@ # report actual input values of CONFIG_FILES etc. instead of their # values after options handling. ac_log=" -This file was extended by parallel $as_me 20151022, which was +This file was extended by parallel $as_me 20151122, which was generated by GNU Autoconf 2.69. Invocation command line was CONFIG_FILES = $CONFIG_FILES @@ -2929,7 +2929,7 @@ cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" ac_cs_version="\\ -parallel config.status 20151022 +parallel config.status 20151122 configured by $0, generated by GNU Autoconf 2.69, with options \\"\$ac_cs_config\\" diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/configure.ac new/parallel-20151122/configure.ac --- old/parallel-20151022/configure.ac 2015-10-21 23:26:58.000000000 +0200 +++ new/parallel-20151122/configure.ac 2015-11-22 19:47:28.000000000 +0100 @@ -1,4 +1,4 @@ -AC_INIT([parallel], [20151022], [[email protected]]) +AC_INIT([parallel], [20151122], [[email protected]]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) AC_CONFIG_HEADERS([config.h]) AC_CONFIG_FILES([ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/src/niceload new/parallel-20151122/src/niceload --- old/parallel-20151022/src/niceload 2015-10-21 23:26:58.000000000 +0200 +++ new/parallel-20151122/src/niceload 2015-11-22 19:47:28.000000000 +0100 @@ -24,7 +24,7 @@ use strict; use Getopt::Long; $Global::progname="niceload"; -$Global::version = 20151022; +$Global::version = 20151122; Getopt::Long::Configure("bundling","require_order"); get_options_from_array(\@ARGV) || die_usage(); if($opt::version) { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/src/parallel new/parallel-20151122/src/parallel --- old/parallel-20151022/src/parallel 2015-10-21 23:26:58.000000000 +0200 +++ new/parallel-20151122/src/parallel 2015-11-22 19:47:28.000000000 +0100 @@ -435,8 +435,8 @@ my $something_written = 0; for my $pid (keys %incomplete_jobs) { my $job = $incomplete_jobs{$pid}; - if($job->stdin_buffer_length()) { - $something_written += $job->non_block_write(); + if($job->block_length()) { + $something_written += $job->non_blocking_write(); } else { delete $incomplete_jobs{$pid} } @@ -509,7 +509,7 @@ # %Global::running # Returns: # $something_written = amount of bytes written - my ($header_ref,$block_ref,$recstart,$recend,$endpos) = @_; + my ($header_ref,$buffer_ref,$recstart,$recend,$endpos) = @_; my $something_written = 0; my $block_passed = 0; my $sleep = 1; @@ -522,13 +522,13 @@ values %Global::running); } while(my $job = shift @robin_queue) { - if($job->stdin_buffer_length() > 0) { - $something_written += $job->non_block_write(); + if($job->block_length() > 0) { + $something_written += $job->non_blocking_write(); } else { - $job->set_stdin_buffer($header_ref,$block_ref,$endpos,$recstart,$recend); + $job->set_block($header_ref,$buffer_ref,$endpos,$recstart,$recend); $block_passed = 1; $job->set_virgin(0); - $something_written += $job->non_block_write(); + $something_written += $job->non_blocking_write(); last; } } @@ -633,21 +633,21 @@ # Input: # $chunk_number = sequence number - to see if already run # $header_ref = reference to header string to prepend - # $record_ref = reference to record to write + # $buffer_ref = reference to record to write # $recstart = start string of record # $recend = end string of record - # $endpos = position in $record_ref where record ends + # $endpos = position in $buffer_ref where record ends # Uses: # $Global::job_already_run # $opt::roundrobin # @Global::virgin_jobs # Returns: # Number of chunks written (0 or 1) - my ($chunk_number,$header_ref,$record_ref,$recstart,$recend,$endpos) = @_; + my ($chunk_number,$header_ref,$buffer_ref,$recstart,$recend,$endpos) = @_; if($endpos == 0) { return 0; } if(vec($Global::job_already_run,$chunk_number,1)) { return 1; } if($opt::roundrobin) { - return round_robin_write($header_ref,$record_ref,$recstart,$recend,$endpos); + return round_robin_write($header_ref,$buffer_ref,$recstart,$recend,$endpos); } # If no virgin found, backoff my $sleep = 0.0001; # 0.01 ms - better performance on highend @@ -655,28 +655,31 @@ ::debug("pipe", "No virgin jobs"); $sleep = ::reap_usleep($sleep); # Jobs may not be started because of loadavg - # or too little time between each ssh login. + # or too little time between each ssh login + # or retrying failed jobs. start_more_jobs(); } my $job = shift @Global::virgin_jobs; # Job is no longer virgin $job->set_virgin(0); - # We ignore the removed rec_sep which is technically wrong. - $job->add_transfersize($endpos + length $$header_ref); - if(fork()) { - # Skip - } else { - # Chop of at $endpos as we do not know how many rec_sep will - # be removed. - substr($$record_ref,$endpos,length $$record_ref) = ""; - # Remove rec_sep - if($opt::remove_rec_sep) { - Job::remove_rec_sep($record_ref,$recstart,$recend); - } - $job->write($header_ref); - $job->write($record_ref); - close $job->fh(0,"w"); - exit(0); + if(1) { + # We ignore the removed rec_sep which is technically wrong. + $job->add_transfersize($endpos + length $$header_ref); + if(fork()) { + # Skip + } else { + # Chop of at $endpos as we do not know how many rec_sep will + # be removed. + substr($$buffer_ref,$endpos,length $$buffer_ref) = ""; + # Remove rec_sep + if($opt::remove_rec_sep) { + Job::remove_rec_sep($buffer_ref,$recstart,$recend); + } + $job->write($header_ref); + $job->write($buffer_ref); + close $job->fh(0,"w"); + exit(0); + } } close $job->fh(0,"w"); return 1; @@ -952,14 +955,7 @@ $opt::blocksize = 2**31-1; } $opt::memfree = multiply_binary_prefix($opt::memfree); - if(defined $opt::timeout and $opt::timeout !~ /^\d+(\.\d+)?%?$/) { - ::error("--timeout must be seconds or percentage."); - wait_and_exit(255); - } - if(defined $opt::fifo and $opt::cat) { - ::error("--fifo cannot be combined with --cat."); - ::wait_and_exit(255); - } + check_invalid_option_combinations(); if((defined $opt::fifo or defined $opt::cat) and not $opt::pipepart) { $opt::pipe = 1; @@ -1043,13 +1039,6 @@ # Default = {} $opt::tagstring = $Global::parensleft.$Global::parensright; } - if(defined $opt::pipepart and - (defined $opt::L or defined $opt::max_lines - or defined $opt::max_replace_args)) { - ::error("--pipepart is incompatible with --max-replace-args, ". - "--max-lines, and -L."); - wait_and_exit(255); - } if(grep /^$Global::arg_sep$|^$Global::arg_file_sep$/o, @ARGV) { # Deal with ::: and :::: @ARGV=read_args_from_command_line(); @@ -1062,23 +1051,6 @@ if(defined $opt::bar) { $opt::progress = $opt::bar; } - if(defined $opt::retired) { - ::error("-g has been retired. Use --group.", - "-B has been retired. Use --bf.", - "-T has been retired. Use --tty.", - "-U has been retired. Use --er.", - "-W has been retired. Use --wd.", - "-Y has been retired. Use --shebang.", - "-H has been retired. Use --halt.", - "--ctrlc has been retired.", - "--noctrlc has been retired."); - ::wait_and_exit(255); - } - if(defined $opt::tollef and not $opt::gnu) { - ::error("--tollef has been retired.","Remove --tollef or use --gnu to override --tollef."); - ::wait_and_exit(255); - } - citation_notice(); parse_halt(); @@ -1098,9 +1070,55 @@ open_joblog(); } +sub check_invalid_option_combinations { + if(defined $opt::timeout and $opt::timeout !~ /^\d+(\.\d+)?%?$/) { + ::error("--timeout must be seconds or percentage."); + wait_and_exit(255); + } + if(defined $opt::fifo and $opt::cat) { + ::error("--fifo cannot be combined with --cat."); + ::wait_and_exit(255); + } + if(defined $opt::pipepart and + (defined $opt::L or defined $opt::max_lines + or defined $opt::max_replace_args)) { + ::error("--pipepart is incompatible with --max-replace-args, ". + "--max-lines, and -L."); + wait_and_exit(255); + } + if(defined $opt::group and $opt::ungroup) { + ::error("--group cannot be combined with --ungroup."); + ::wait_and_exit(255); + } + if(defined $opt::group and $opt::linebuffer) { + ::error("--group cannot be combined with --line-buffer."); + ::wait_and_exit(255); + } + if(defined $opt::ungroup and $opt::linebuffer) { + ::error("--ungroup cannot be combined with --line-buffer."); + ::wait_and_exit(255); + } + if(defined $opt::tollef and not $opt::gnu) { + ::error("--tollef has been retired.","Remove --tollef or use --gnu to override --tollef."); + ::wait_and_exit(255); + } + if(defined $opt::retired) { + ::error("-g has been retired. Use --group.", + "-B has been retired. Use --bf.", + "-T has been retired. Use --tty.", + "-U has been retired. Use --er.", + "-W has been retired. Use --wd.", + "-Y has been retired. Use --shebang.", + "-H has been retired. Use --halt.", + "--ctrlc has been retired.", + "--noctrlc has been retired."); + ::wait_and_exit(255); + } +} + sub init_globals { # Defaults: - $Global::version = 20151022; + $Global::version = 20151122; $Global::progname = 'parallel'; $Global::infinity = 2**31; $Global::debug = 0; @@ -4229,7 +4247,8 @@ my $script = memfreescript(); # TODO add sshlogin and backgrounding - $self->{'memfree'} = qx{ $script }; + # Run it twice if it gives 0 + $self->{'memfree'} = qx{ $script } || qx{ $script }; if(not $self->{'memfree'}) { ::die_bug("Less than 1 byte free"); } @@ -6198,32 +6217,38 @@ } } -sub set_stdin_buffer { +sub set_block { # Copy stdin buffer from $block_ref up to $endpos - # Prepend with $header_ref + # Prepend with $header_ref if virgin (i.e. not --roundrobin) # Remove $recstart and $recend if needed # Input: # $header_ref = ref to $header to prepend - # $block_ref = ref to $block to pass on + # $buffer_ref = ref to $buffer containing the block # $endpos = length of $block to pass on # $recstart = --recstart regexp # $recend = --recend regexp # Returns: # N/A my $self = shift; - my ($header_ref,$block_ref,$endpos,$recstart,$recend) = @_; - $self->{'stdin_buffer'} = ($self->virgin() ? $$header_ref : "").substr($$block_ref,0,$endpos); + my ($header_ref,$buffer_ref,$endpos,$recstart,$recend) = @_; + $self->{'block'} = ($self->virgin() ? $$header_ref : "").substr($$buffer_ref,0,$endpos); if($opt::remove_rec_sep) { - remove_rec_sep(\$self->{'stdin_buffer'},$recstart,$recend); + remove_rec_sep(\$self->{'block'},$recstart,$recend); } - $self->{'stdin_buffer_length'} = length $self->{'stdin_buffer'}; - $self->{'stdin_buffer_pos'} = 0; - $self->add_transfersize($self->{'stdin_buffer_length'}); + $self->{'block_length'} = length $self->{'block'}; + $self->{'block_pos'} = 0; + $self->add_transfersize($self->{'block_length'}); } -sub stdin_buffer_length { +sub block_ref { + my $self = shift; + return \$self->{'block'}; +} + + +sub block_length { my $self = shift; - return $self->{'stdin_buffer_length'}; + return $self->{'block_length'}; } sub remove_rec_sep { @@ -6234,26 +6259,26 @@ $$block_ref =~ s/$recend$//os; } -sub non_block_write { +sub non_blocking_write { my $self = shift; my $something_written = 0; use POSIX qw(:errno_h); # for loop used to avoid copying substr: $buf will be an alias for the substr - for my $buf (substr($self->{'stdin_buffer'},$self->{'stdin_buffer_pos'})) { + for my $buf (substr($self->{'block'},$self->{'block_pos'})) { my $in = $self->fh(0,"w"); my $rv = syswrite($in, $buf); if (!defined($rv) && $! == EAGAIN) { # would block $something_written = 0; - } elsif ($self->{'stdin_buffer_pos'}+$rv != $self->{'stdin_buffer_length'}) { + } elsif ($self->{'block_pos'}+$rv != $self->{'block_length'}) { # incomplete write # Remove the written part - $self->{'stdin_buffer_pos'} += $rv; + $self->{'block_pos'} += $rv; $something_written = $rv; } else { # successfully wrote everything my $a = ""; - $self->set_stdin_buffer(\$a,\$a,0,"",""); + $self->set_block(\$a,\$a,0,"",""); $something_written = $rv; } } @@ -8001,11 +8026,13 @@ if($perlexpr =~ /^(\d+) /) { # Positional defined($record->[$1-1]) or next; - $self->{'len'}{$perlexpr} += length $record->[$1-1]->replace($perlexpr,$quote_arg,$self); + $self->{'len'}{$perlexpr} += + length $record->[$1-1]->replace($perlexpr,$quote_arg,$self); } else { for my $arg (@$record) { if(defined $arg) { - $self->{'len'}{$perlexpr} += length $arg->replace($perlexpr,$quote_arg,$self); + $self->{'len'}{$perlexpr} += + length $arg->replace($perlexpr,$quote_arg,$self); } } } @@ -8604,7 +8631,7 @@ if($opt::pipe or $opt::pipepart) { if($cmd_line->replaced() eq "") { # Empty command - pipe requires a command - ::error("--pipe must have a command to pipe into (e.g. 'cat')."); + ::error("--pipe/--pipepart must have a command to pipe into (e.g. 'cat')."); ::wait_and_exit(255); } } else { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/src/parallel.1 new/parallel-20151122/src/parallel.1 --- old/parallel-20151022/src/parallel.1 2015-10-21 23:27:12.000000000 +0200 +++ new/parallel-20151122/src/parallel.1 2015-10-22 00:06:45.000000000 +0200 @@ -533,8 +533,8 @@ string that is not in the command line. .Sp See also: \fB:::\fR. -.IP "\fB\-\-bar\fR (alpha testing)" 9 -.IX Item "--bar (alpha testing)" +.IP "\fB\-\-bar\fR (beta testing)" 9 +.IX Item "--bar (beta testing)" Show progress as a progress bar. In the bar is shown: % of jobs completed, estimated seconds left, and number of jobs started. .Sp @@ -618,8 +618,8 @@ \&\fIsize\fR defaults to 1M. .Sp See \fB\-\-pipe\fR and \fB\-\-pipepart\fR for use of this. -.IP "\fB\-\-cat\fR (alpha testing)" 9 -.IX Item "--cat (alpha testing)" +.IP "\fB\-\-cat\fR (beta testing)" 9 +.IX Item "--cat (beta testing)" Create a temporary file with content. Normally \fB\-\-pipe\fR/\fB\-\-pipepart\fR will give data to the program on stdin (standard input). With \fB\-\-cat\fR \&\s-1GNU \s0\fBparallel\fR will create a temporary file with the name in {}, so @@ -711,8 +711,8 @@ Delay starting next job \fIsecs\fR seconds. \s-1GNU \s0\fBparallel\fR will pause \&\fIsecs\fR seconds after starting each job. \fIsecs\fR can be less than 1 second. -.IP "\fB\-\-dry\-run\fR (alpha testing)" 9 -.IX Item "--dry-run (alpha testing)" +.IP "\fB\-\-dry\-run\fR (beta testing)" 9 +.IX Item "--dry-run (beta testing)" Print the job to run on stdout (standard output), but do not run the job. Use \fB\-v \-v\fR to include the wrapping that \s-1GNU\s0 Parallel generates (for remote jobs, \fB\-\-tmux\fR, \fB\-\-nice\fR, \fB\-\-pipe\fR, \fB\-\-pipepart\fR, @@ -767,8 +767,8 @@ and functions) use env_parallel as described under the option \fIcommand\fR. .Sp See also: \fB\-\-record\-env\fR. -.IP "\fB\-\-eta\fR (alpha testing)" 9 -.IX Item "--eta (alpha testing)" +.IP "\fB\-\-eta\fR (beta testing)" 9 +.IX Item "--eta (beta testing)" Show the estimated number of seconds before finishing. This forces \s-1GNU \&\s0\fBparallel\fR to read all jobs before starting to find the number of jobs. \s-1GNU \s0\fBparallel\fR normally only reads the next job to run. @@ -784,8 +784,8 @@ Implies \fB\-\-semaphore\fR. .Sp See also \fB\-\-bg\fR, \fBman sem\fR. -.IP "\fB\-\-fifo\fR (alpha testing)" 9 -.IX Item "--fifo (alpha testing)" +.IP "\fB\-\-fifo\fR (beta testing)" 9 +.IX Item "--fifo (beta testing)" Create a temporary fifo with content. Normally \fB\-\-pipe\fR and \&\fB\-\-pipepart\fR will give data to the program on stdin (standard input). With \fB\-\-fifo\fR \s-1GNU \s0\fBparallel\fR will create a temporary fifo @@ -810,8 +810,8 @@ Behave like \s-1GNU \s0\fBparallel\fR. This option historically took precedence over \fB\-\-tollef\fR. The \fB\-\-tollef\fR option is now retired, and therefore may not be used. \fB\-\-gnu\fR is kept for compatibility. -.IP "\fB\-\-group\fR (alpha testing)" 9 -.IX Item "--group (alpha testing)" +.IP "\fB\-\-group\fR (beta testing)" 9 +.IX Item "--group (beta testing)" Group output. Output from each jobs is grouped together and is only printed when the command is finished. stderr (standard error) first followed by stdout (standard output). This takes some \s-1CPU\s0 time. In @@ -830,11 +830,11 @@ .IX Item "-h" .PD Print a summary of the options to \s-1GNU \s0\fBparallel\fR and exit. -.IP "\fB\-\-halt\-on\-error\fR \fIval\fR (alpha testing)" 9 -.IX Item "--halt-on-error val (alpha testing)" +.IP "\fB\-\-halt\-on\-error\fR \fIval\fR (beta testing)" 9 +.IX Item "--halt-on-error val (beta testing)" .PD 0 -.IP "\fB\-\-halt\fR \fIval\fR (alpha testing)" 9 -.IX Item "--halt val (alpha testing)" +.IP "\fB\-\-halt\fR \fIval\fR (beta testing)" 9 +.IX Item "--halt val (beta testing)" .PD When should \s-1GNU \s0\fBparallel\fR terminate? In some situations it makes no sense to run all jobs. \s-1GNU \s0\fBparallel\fR should simply give up as soon @@ -1225,8 +1225,8 @@ performance is important use \fB\-\-pipepart\fR. .Sp See also: \fB\-\-recstart\fR, \fB\-\-recend\fR, \fB\-\-fifo\fR, \fB\-\-cat\fR, \fB\-\-pipepart\fR. -.IP "\fB\-\-pipepart\fR (alpha testing)" 9 -.IX Item "--pipepart (alpha testing)" +.IP "\fB\-\-pipepart\fR (beta testing)" 9 +.IX Item "--pipepart (beta testing)" Pipe parts of a physical file. \fB\-\-pipepart\fR works similar to \&\fB\-\-pipe\fR, but is much faster. It has a few limitations: .RS 9 @@ -1243,8 +1243,8 @@ Ignore any \fB\-\-profile\fR, \f(CW$PARALLEL\fR, and ~/.parallel/config to get full control on the command line (used by \s-1GNU \s0\fBparallel\fR internally when called with \fB\-\-sshlogin\fR). -.IP "\fB\-\-plus\fR (alpha testing)" 9 -.IX Item "--plus (alpha testing)" +.IP "\fB\-\-plus\fR (beta testing)" 9 +.IX Item "--plus (beta testing)" Activate additional replacement strings: {+/} {+.} {+..} {+...} {..} {...} {/..} {/...} {##}. The idea being that '{+foo}' matches the opposite of \&'{foo}' and {} = {+/}/{/} = {.}.{+.} = {+/}/{/.}.{+.} = {..}.{+..} = @@ -1323,8 +1323,8 @@ .IX Item "--no-keep-order" Overrides an earlier \fB\-\-keep\-order\fR (e.g. if set in \&\fB~/.parallel/config\fR). -.IP "\fB\-\-nice\fR \fIniceness\fR (alpha testing)" 9 -.IX Item "--nice niceness (alpha testing)" +.IP "\fB\-\-nice\fR \fIniceness\fR (beta testing)" 9 +.IX Item "--nice niceness (beta testing)" Run the command at this niceness. For simple commands you can just add \&\fBnice\fR in front of the command. But if the command consists of more sub commands (Like: ls|wc) then prepending \fBnice\fR will not always @@ -1776,8 +1776,8 @@ .IX Item "--shellquote" Does not run the command but quotes it. Useful for making quoted composed commands for \s-1GNU \s0\fBparallel\fR. -.IP "\fB\-\-shuf\fR (alpha testing)" 9 -.IX Item "--shuf (alpha testing)" +.IP "\fB\-\-shuf\fR (beta testing)" 9 +.IX Item "--shuf (beta testing)" Shuffle jobs. When having multiple input sources it is hard to randomize jobs. \-\-shuf will generate all jobs, and shuffle them before running them. This is useful to get a quick preview of the results @@ -1796,17 +1796,17 @@ Delay starting next ssh by \fIsecs\fR seconds. \s-1GNU \s0\fBparallel\fR will pause \&\fIsecs\fR seconds after starting each ssh. \fIsecs\fR can be less than 1 seconds. -.IP "\fB\-S\fR \fI[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]\fR (alpha testing)" 9 -.IX Item "-S [@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]] (alpha testing)" +.IP "\fB\-S\fR \fI[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]\fR (beta testing)" 9 +.IX Item "-S [@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]] (beta testing)" .PD 0 -.ie n .IP "\fB\-S\fR \fI\fI@hostgroup\fI\fR (alpha testing)" 9 -.el .IP "\fB\-S\fR \fI\f(CI@hostgroup\fI\fR (alpha testing)" 9 -.IX Item "-S @hostgroup (alpha testing)" -.IP "\fB\-\-sshlogin\fR \fI[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]\fR (alpha testing)" 9 -.IX Item "--sshlogin [@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]] (alpha testing)" -.ie n .IP "\fB\-\-sshlogin\fR \fI\fI@hostgroup\fI\fR (alpha testing)" 9 -.el .IP "\fB\-\-sshlogin\fR \fI\f(CI@hostgroup\fI\fR (alpha testing)" 9 -.IX Item "--sshlogin @hostgroup (alpha testing)" +.ie n .IP "\fB\-S\fR \fI\fI@hostgroup\fI\fR (beta testing)" 9 +.el .IP "\fB\-S\fR \fI\f(CI@hostgroup\fI\fR (beta testing)" 9 +.IX Item "-S @hostgroup (beta testing)" +.IP "\fB\-\-sshlogin\fR \fI[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]\fR (beta testing)" 9 +.IX Item "--sshlogin [@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]] (beta testing)" +.ie n .IP "\fB\-\-sshlogin\fR \fI\fI@hostgroup\fI\fR (beta testing)" 9 +.el .IP "\fB\-\-sshlogin\fR \fI\f(CI@hostgroup\fI\fR (beta testing)" 9 +.IX Item "--sshlogin @hostgroup (beta testing)" .PD Distribute jobs to remote computers. The jobs will be run on a list of remote computers. @@ -2107,11 +2107,11 @@ .IX Item "-V" .PD Print the version \s-1GNU \s0\fBparallel\fR and exit. -.IP "\fB\-\-workdir\fR \fImydir\fR (alpha testing)" 9 -.IX Item "--workdir mydir (alpha testing)" +.IP "\fB\-\-workdir\fR \fImydir\fR (beta testing)" 9 +.IX Item "--workdir mydir (beta testing)" .PD 0 -.IP "\fB\-\-wd\fR \fImydir\fR (alpha testing)" 9 -.IX Item "--wd mydir (alpha testing)" +.IP "\fB\-\-wd\fR \fImydir\fR (beta testing)" 9 +.IX Item "--wd mydir (beta testing)" .PD Files transferred using \fB\-\-transfer\fR and \fB\-\-return\fR will be relative to \fImydir\fR on remote computers, and the command will be executed in diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/src/parallel.html new/parallel-20151122/src/parallel.html --- old/parallel-20151022/src/parallel.html 2015-10-21 23:27:11.000000000 +0200 +++ new/parallel-20151122/src/parallel.html 2015-10-22 00:06:43.000000000 +0200 @@ -488,7 +488,7 @@ <p>See also: <b>:::</b>.</p> </dd> -<dt id="bar-alpha-testing"><b>--bar</b> (alpha testing)</dt> +<dt id="bar-beta-testing"><b>--bar</b> (beta testing)</dt> <dd> <p>Show progress as a progress bar. In the bar is shown: % of jobs completed, estimated seconds left, and number of jobs started.</p> @@ -571,7 +571,7 @@ <p>See <b>--pipe</b> and <b>--pipepart</b> for use of this.</p> </dd> -<dt id="cat-alpha-testing"><b>--cat</b> (alpha testing)</dt> +<dt id="cat-beta-testing"><b>--cat</b> (beta testing)</dt> <dd> <p>Create a temporary file with content. Normally <b>--pipe</b>/<b>--pipepart</b> will give data to the program on stdin (standard input). With <b>--cat</b> GNU <b>parallel</b> will create a temporary file with the name in {}, so you can do: <b>parallel --pipe --cat wc {}</b>.</p> @@ -661,7 +661,7 @@ <p>Delay starting next job <i>secs</i> seconds. GNU <b>parallel</b> will pause <i>secs</i> seconds after starting each job. <i>secs</i> can be less than 1 second.</p> </dd> -<dt id="dry-run-alpha-testing"><b>--dry-run</b> (alpha testing)</dt> +<dt id="dry-run-beta-testing"><b>--dry-run</b> (beta testing)</dt> <dd> <p>Print the job to run on stdout (standard output), but do not run the job. Use <b>-v -v</b> to include the wrapping that GNU Parallel generates (for remote jobs, <b>--tmux</b>, <b>--nice</b>, <b>--pipe</b>, <b>--pipepart</b>, <b>--fifo</b> and <b>--cat</b>). Do not count on this literaly, though, as the job may be scheduled on another computer or the local computer if : is in the list.</p> @@ -710,7 +710,7 @@ <p>See also: <b>--record-env</b>.</p> </dd> -<dt id="eta-alpha-testing"><b>--eta</b> (alpha testing)</dt> +<dt id="eta-beta-testing"><b>--eta</b> (beta testing)</dt> <dd> <p>Show the estimated number of seconds before finishing. This forces GNU <b>parallel</b> to read all jobs before starting to find the number of jobs. GNU <b>parallel</b> normally only reads the next job to run.</p> @@ -730,7 +730,7 @@ <p>See also <b>--bg</b>, <b>man sem</b>.</p> </dd> -<dt id="fifo-alpha-testing"><b>--fifo</b> (alpha testing)</dt> +<dt id="fifo-beta-testing"><b>--fifo</b> (beta testing)</dt> <dd> <p>Create a temporary fifo with content. Normally <b>--pipe</b> and <b>--pipepart</b> will give data to the program on stdin (standard input). With <b>--fifo</b> GNU <b>parallel</b> will create a temporary fifo with the name in {}, so you can do: <b>parallel --pipe --fifo wc {}</b>.</p> @@ -756,7 +756,7 @@ <p>Behave like GNU <b>parallel</b>. This option historically took precedence over <b>--tollef</b>. The <b>--tollef</b> option is now retired, and therefore may not be used. <b>--gnu</b> is kept for compatibility.</p> </dd> -<dt id="group-alpha-testing"><b>--group</b> (alpha testing)</dt> +<dt id="group-beta-testing"><b>--group</b> (beta testing)</dt> <dd> <p>Group output. Output from each jobs is grouped together and is only printed when the command is finished. stderr (standard error) first followed by stdout (standard output). This takes some CPU time. In rare situations GNU <b>parallel</b> takes up lots of CPU time and if it is acceptable that the outputs from different commands are mixed together, then disabling grouping with <b>-u</b> can speedup GNU <b>parallel</b> by a factor of 10.</p> @@ -776,11 +776,11 @@ <p>Print a summary of the options to GNU <b>parallel</b> and exit.</p> </dd> -<dt id="halt-on-error-val-alpha-testing"><b>--halt-on-error</b> <i>val</i> (alpha testing)</dt> +<dt id="halt-on-error-val-beta-testing"><b>--halt-on-error</b> <i>val</i> (beta testing)</dt> <dd> </dd> -<dt id="halt-val-alpha-testing"><b>--halt</b> <i>val</i> (alpha testing)</dt> +<dt id="halt-val-beta-testing"><b>--halt</b> <i>val</i> (beta testing)</dt> <dd> <p>When should GNU <b>parallel</b> terminate? In some situations it makes no sense to run all jobs. GNU <b>parallel</b> should simply give up as soon as a condition is met.</p> @@ -1207,7 +1207,7 @@ <p>See also: <b>--recstart</b>, <b>--recend</b>, <b>--fifo</b>, <b>--cat</b>, <b>--pipepart</b>.</p> </dd> -<dt id="pipepart-alpha-testing"><b>--pipepart</b> (alpha testing)</dt> +<dt id="pipepart-beta-testing"><b>--pipepart</b> (beta testing)</dt> <dd> <p>Pipe parts of a physical file. <b>--pipepart</b> works similar to <b>--pipe</b>, but is much faster. It has a few limitations:</p> @@ -1235,7 +1235,7 @@ <p>Ignore any <b>--profile</b>, $PARALLEL, and ~/.parallel/config to get full control on the command line (used by GNU <b>parallel</b> internally when called with <b>--sshlogin</b>).</p> </dd> -<dt id="plus-alpha-testing"><b>--plus</b> (alpha testing)</dt> +<dt id="plus-beta-testing"><b>--plus</b> (beta testing)</dt> <dd> <p>Activate additional replacement strings: {+/} {+.} {+..} {+...} {..} {...} {/..} {/...} {##}. The idea being that '{+foo}' matches the opposite of '{foo}' and {} = {+/}/{/} = {.}.{+.} = {+/}/{/.}.{+.} = {..}.{+..} = {+/}/{/..}.{+..} = {...}.{+...} = {+/}/{/...}.{+...}</p> @@ -1311,7 +1311,7 @@ <p>Overrides an earlier <b>--keep-order</b> (e.g. if set in <b>~/.parallel/config</b>).</p> </dd> -<dt id="nice-niceness-alpha-testing"><b>--nice</b> <i>niceness</i> (alpha testing)</dt> +<dt id="nice-niceness-beta-testing"><b>--nice</b> <i>niceness</i> (beta testing)</dt> <dd> <p>Run the command at this niceness. For simple commands you can just add <b>nice</b> in front of the command. But if the command consists of more sub commands (Like: ls|wc) then prepending <b>nice</b> will not always work. <b>--nice</b> will make sure all sub commands are niced - even on remote servers.</p> @@ -1725,7 +1725,7 @@ <p>Does not run the command but quotes it. Useful for making quoted composed commands for GNU <b>parallel</b>.</p> </dd> -<dt id="shuf-alpha-testing"><b>--shuf</b> (alpha testing)</dt> +<dt id="shuf-beta-testing"><b>--shuf</b> (beta testing)</dt> <dd> <p>Shuffle jobs. When having multiple input sources it is hard to randomize jobs. --shuf will generate all jobs, and shuffle them before running them. This is useful to get a quick preview of the results before running the full batch.</p> @@ -1749,19 +1749,19 @@ <p>Delay starting next ssh by <i>secs</i> seconds. GNU <b>parallel</b> will pause <i>secs</i> seconds after starting each ssh. <i>secs</i> can be less than 1 seconds.</p> </dd> -<dt id="S-hostgroups-ncpu-sshlogin-hostgroups-ncpu-sshlogin-...-alpha-testing"><b>-S</b> <i>[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]</i> (alpha testing)</dt> +<dt id="S-hostgroups-ncpu-sshlogin-hostgroups-ncpu-sshlogin-...-beta-testing"><b>-S</b> <i>[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]</i> (beta testing)</dt> <dd> </dd> -<dt id="S-hostgroup-alpha-testing"><b>-S</b> <i>@hostgroup</i> (alpha testing)</dt> +<dt id="S-hostgroup-beta-testing"><b>-S</b> <i>@hostgroup</i> (beta testing)</dt> <dd> </dd> -<dt id="sshlogin-hostgroups-ncpu-sshlogin-hostgroups-ncpu-sshlogin-...-alpha-testing"><b>--sshlogin</b> <i>[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]</i> (alpha testing)</dt> +<dt id="sshlogin-hostgroups-ncpu-sshlogin-hostgroups-ncpu-sshlogin-...-beta-testing"><b>--sshlogin</b> <i>[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]</i> (beta testing)</dt> <dd> </dd> -<dt id="sshlogin-hostgroup-alpha-testing"><b>--sshlogin</b> <i>@hostgroup</i> (alpha testing)</dt> +<dt id="sshlogin-hostgroup-beta-testing"><b>--sshlogin</b> <i>@hostgroup</i> (beta testing)</dt> <dd> <p>Distribute jobs to remote computers. The jobs will be run on a list of remote computers.</p> @@ -2055,11 +2055,11 @@ <p>Print the version GNU <b>parallel</b> and exit.</p> </dd> -<dt id="workdir-mydir-alpha-testing"><b>--workdir</b> <i>mydir</i> (alpha testing)</dt> +<dt id="workdir-mydir-beta-testing"><b>--workdir</b> <i>mydir</i> (beta testing)</dt> <dd> </dd> -<dt id="wd-mydir-alpha-testing"><b>--wd</b> <i>mydir</i> (alpha testing)</dt> +<dt id="wd-mydir-beta-testing"><b>--wd</b> <i>mydir</i> (beta testing)</dt> <dd> <p>Files transferred using <b>--transfer</b> and <b>--return</b> will be relative to <i>mydir</i> on remote computers, and the command will be executed in the dir <i>mydir</i>.</p> Files old/parallel-20151022/src/parallel.pdf and new/parallel-20151122/src/parallel.pdf differ diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/src/parallel.pod new/parallel-20151122/src/parallel.pod --- old/parallel-20151022/src/parallel.pod 2015-10-21 23:22:15.000000000 +0200 +++ new/parallel-20151122/src/parallel.pod 2015-10-21 23:56:29.000000000 +0200 @@ -426,7 +426,7 @@ See also: B<:::>. -=item B<--bar> (alpha testing) +=item B<--bar> (beta testing) Show progress as a progress bar. In the bar is shown: % of jobs completed, estimated seconds left, and number of jobs started. @@ -513,7 +513,7 @@ See B<--pipe> and B<--pipepart> for use of this. -=item B<--cat> (alpha testing) +=item B<--cat> (beta testing) Create a temporary file with content. Normally B<--pipe>/B<--pipepart> will give data to the program on stdin (standard input). With B<--cat> @@ -613,7 +613,7 @@ second. -=item B<--dry-run> (alpha testing) +=item B<--dry-run> (beta testing) Print the job to run on stdout (standard output), but do not run the job. Use B<-v -v> to include the wrapping that GNU Parallel generates @@ -672,7 +672,7 @@ See also: B<--record-env>. -=item B<--eta> (alpha testing) +=item B<--eta> (beta testing) Show the estimated number of seconds before finishing. This forces GNU B<parallel> to read all jobs before starting to find the number of @@ -693,7 +693,7 @@ See also B<--bg>, B<man sem>. -=item B<--fifo> (alpha testing) +=item B<--fifo> (beta testing) Create a temporary fifo with content. Normally B<--pipe> and B<--pipepart> will give data to the program on stdin (standard @@ -725,7 +725,7 @@ may not be used. B<--gnu> is kept for compatibility. -=item B<--group> (alpha testing) +=item B<--group> (beta testing) Group output. Output from each jobs is grouped together and is only printed when the command is finished. stderr (standard error) first @@ -747,9 +747,9 @@ Print a summary of the options to GNU B<parallel> and exit. -=item B<--halt-on-error> I<val> (alpha testing) +=item B<--halt-on-error> I<val> (beta testing) -=item B<--halt> I<val> (alpha testing) +=item B<--halt> I<val> (beta testing) When should GNU B<parallel> terminate? In some situations it makes no sense to run all jobs. GNU B<parallel> should simply give up as soon @@ -1172,7 +1172,7 @@ See also: B<--recstart>, B<--recend>, B<--fifo>, B<--cat>, B<--pipepart>. -=item B<--pipepart> (alpha testing) +=item B<--pipepart> (beta testing) Pipe parts of a physical file. B<--pipepart> works similar to B<--pipe>, but is much faster. It has a few limitations: @@ -1198,7 +1198,7 @@ called with B<--sshlogin>). -=item B<--plus> (alpha testing) +=item B<--plus> (beta testing) Activate additional replacement strings: {+/} {+.} {+..} {+...} {..} {...} {/..} {/...} {##}. The idea being that '{+foo}' matches the opposite of @@ -1288,7 +1288,7 @@ B<~/.parallel/config>). -=item B<--nice> I<niceness> (alpha testing) +=item B<--nice> I<niceness> (beta testing) Run the command at this niceness. For simple commands you can just add B<nice> in front of the command. But if the command consists of more @@ -1737,7 +1737,7 @@ composed commands for GNU B<parallel>. -=item B<--shuf> (alpha testing) +=item B<--shuf> (beta testing) Shuffle jobs. When having multiple input sources it is hard to randomize jobs. --shuf will generate all jobs, and shuffle them before @@ -1765,13 +1765,13 @@ seconds. -=item B<-S> I<[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]> (alpha testing) +=item B<-S> I<[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]> (beta testing) -=item B<-S> I<@hostgroup> (alpha testing) +=item B<-S> I<@hostgroup> (beta testing) -=item B<--sshlogin> I<[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]> (alpha testing) +=item B<--sshlogin> I<[@hostgroups/][ncpu/]sshlogin[,[@hostgroups/][ncpu/]sshlogin[,...]]> (beta testing) -=item B<--sshlogin> I<@hostgroup> (alpha testing) +=item B<--sshlogin> I<@hostgroup> (beta testing) Distribute jobs to remote computers. The jobs will be run on a list of remote computers. @@ -2088,9 +2088,9 @@ Print the version GNU B<parallel> and exit. -=item B<--workdir> I<mydir> (alpha testing) +=item B<--workdir> I<mydir> (beta testing) -=item B<--wd> I<mydir> (alpha testing) +=item B<--wd> I<mydir> (beta testing) Files transferred using B<--transfer> and B<--return> will be relative to I<mydir> on remote computers, and the command will be executed in diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/src/parallel.texi new/parallel-20151122/src/parallel.texi --- old/parallel-20151022/src/parallel.texi 2015-10-21 23:27:19.000000000 +0200 +++ new/parallel-20151122/src/parallel.texi 2015-10-22 00:06:54.000000000 +0200 @@ -527,8 +527,8 @@ See also: @strong{:::}. -@item @strong{--bar} (alpha testing) -@anchor{@strong{--bar} (alpha testing)} +@item @strong{--bar} (beta testing) +@anchor{@strong{--bar} (beta testing)} Show progress as a progress bar. In the bar is shown: % of jobs completed, estimated seconds left, and number of jobs started. @@ -621,8 +621,8 @@ See @strong{--pipe} and @strong{--pipepart} for use of this. -@item @strong{--cat} (alpha testing) -@anchor{@strong{--cat} (alpha testing)} +@item @strong{--cat} (beta testing) +@anchor{@strong{--cat} (beta testing)} Create a temporary file with content. Normally @strong{--pipe}/@strong{--pipepart} will give data to the program on stdin (standard input). With @strong{--cat} @@ -728,8 +728,8 @@ @emph{secs} seconds after starting each job. @emph{secs} can be less than 1 second. -@item @strong{--dry-run} (alpha testing) -@anchor{@strong{--dry-run} (alpha testing)} +@item @strong{--dry-run} (beta testing) +@anchor{@strong{--dry-run} (beta testing)} Print the job to run on stdout (standard output), but do not run the job. Use @strong{-v -v} to include the wrapping that GNU Parallel generates @@ -789,8 +789,8 @@ See also: @strong{--record-env}. -@item @strong{--eta} (alpha testing) -@anchor{@strong{--eta} (alpha testing)} +@item @strong{--eta} (beta testing) +@anchor{@strong{--eta} (beta testing)} Show the estimated number of seconds before finishing. This forces GNU @strong{parallel} to read all jobs before starting to find the number of @@ -810,8 +810,8 @@ See also @strong{--bg}, @strong{man sem}. -@item @strong{--fifo} (alpha testing) -@anchor{@strong{--fifo} (alpha testing)} +@item @strong{--fifo} (beta testing) +@anchor{@strong{--fifo} (beta testing)} Create a temporary fifo with content. Normally @strong{--pipe} and @strong{--pipepart} will give data to the program on stdin (standard @@ -842,8 +842,8 @@ over @strong{--tollef}. The @strong{--tollef} option is now retired, and therefore may not be used. @strong{--gnu} is kept for compatibility. -@item @strong{--group} (alpha testing) -@anchor{@strong{--group} (alpha testing)} +@item @strong{--group} (beta testing) +@anchor{@strong{--group} (beta testing)} Group output. Output from each jobs is grouped together and is only printed when the command is finished. stderr (standard error) first @@ -865,11 +865,11 @@ Print a summary of the options to GNU @strong{parallel} and exit. -@item @strong{--halt-on-error} @emph{val} (alpha testing) -@anchor{@strong{--halt-on-error} @emph{val} (alpha testing)} +@item @strong{--halt-on-error} @emph{val} (beta testing) +@anchor{@strong{--halt-on-error} @emph{val} (beta testing)} -@item @strong{--halt} @emph{val} (alpha testing) -@anchor{@strong{--halt} @emph{val} (alpha testing)} +@item @strong{--halt} @emph{val} (beta testing) +@anchor{@strong{--halt} @emph{val} (beta testing)} When should GNU @strong{parallel} terminate? In some situations it makes no sense to run all jobs. GNU @strong{parallel} should simply give up as soon @@ -1332,8 +1332,8 @@ See also: @strong{--recstart}, @strong{--recend}, @strong{--fifo}, @strong{--cat}, @strong{--pipepart}. -@item @strong{--pipepart} (alpha testing) -@anchor{@strong{--pipepart} (alpha testing)} +@item @strong{--pipepart} (beta testing) +@anchor{@strong{--pipepart} (beta testing)} Pipe parts of a physical file. @strong{--pipepart} works similar to @strong{--pipe}, but is much faster. It has a few limitations: @@ -1359,8 +1359,8 @@ control on the command line (used by GNU @strong{parallel} internally when called with @strong{--sshlogin}). -@item @strong{--plus} (alpha testing) -@anchor{@strong{--plus} (alpha testing)} +@item @strong{--plus} (beta testing) +@anchor{@strong{--plus} (beta testing)} Activate additional replacement strings: @{+/@} @{+.@} @{+..@} @{+...@} @{..@} @{...@} @{/..@} @{/...@} @{##@}. The idea being that '@{+foo@}' matches the opposite of @@ -1453,8 +1453,8 @@ Overrides an earlier @strong{--keep-order} (e.g. if set in @strong{~/.parallel/config}). -@item @strong{--nice} @emph{niceness} (alpha testing) -@anchor{@strong{--nice} @emph{niceness} (alpha testing)} +@item @strong{--nice} @emph{niceness} (beta testing) +@anchor{@strong{--nice} @emph{niceness} (beta testing)} Run the command at this niceness. For simple commands you can just add @strong{nice} in front of the command. But if the command consists of more @@ -1951,8 +1951,8 @@ Does not run the command but quotes it. Useful for making quoted composed commands for GNU @strong{parallel}. -@item @strong{--shuf} (alpha testing) -@anchor{@strong{--shuf} (alpha testing)} +@item @strong{--shuf} (beta testing) +@anchor{@strong{--shuf} (beta testing)} Shuffle jobs. When having multiple input sources it is hard to randomize jobs. --shuf will generate all jobs, and shuffle them before @@ -1979,17 +1979,17 @@ @emph{secs} seconds after starting each ssh. @emph{secs} can be less than 1 seconds. -@item @strong{-S} @emph{[@@hostgroups/][ncpu/]sshlogin[,[@@hostgroups/][ncpu/]sshlogin[,...]]} (alpha testing) -@anchor{@strong{-S} @emph{[@@hostgroups/][ncpu/]sshlogin[@comma{}[@@hostgroups/][ncpu/]sshlogin[@comma{}...]]} (alpha testing)} +@item @strong{-S} @emph{[@@hostgroups/][ncpu/]sshlogin[,[@@hostgroups/][ncpu/]sshlogin[,...]]} (beta testing) +@anchor{@strong{-S} @emph{[@@hostgroups/][ncpu/]sshlogin[@comma{}[@@hostgroups/][ncpu/]sshlogin[@comma{}...]]} (beta testing)} -@item @strong{-S} @emph{@@hostgroup} (alpha testing) -@anchor{@strong{-S} @emph{@@hostgroup} (alpha testing)} +@item @strong{-S} @emph{@@hostgroup} (beta testing) +@anchor{@strong{-S} @emph{@@hostgroup} (beta testing)} -@item @strong{--sshlogin} @emph{[@@hostgroups/][ncpu/]sshlogin[,[@@hostgroups/][ncpu/]sshlogin[,...]]} (alpha testing) -@anchor{@strong{--sshlogin} @emph{[@@hostgroups/][ncpu/]sshlogin[@comma{}[@@hostgroups/][ncpu/]sshlogin[@comma{}...]]} (alpha testing)} +@item @strong{--sshlogin} @emph{[@@hostgroups/][ncpu/]sshlogin[,[@@hostgroups/][ncpu/]sshlogin[,...]]} (beta testing) +@anchor{@strong{--sshlogin} @emph{[@@hostgroups/][ncpu/]sshlogin[@comma{}[@@hostgroups/][ncpu/]sshlogin[@comma{}...]]} (beta testing)} -@item @strong{--sshlogin} @emph{@@hostgroup} (alpha testing) -@anchor{@strong{--sshlogin} @emph{@@hostgroup} (alpha testing)} +@item @strong{--sshlogin} @emph{@@hostgroup} (beta testing) +@anchor{@strong{--sshlogin} @emph{@@hostgroup} (beta testing)} Distribute jobs to remote computers. The jobs will be run on a list of remote computers. @@ -2330,11 +2330,11 @@ Print the version GNU @strong{parallel} and exit. -@item @strong{--workdir} @emph{mydir} (alpha testing) -@anchor{@strong{--workdir} @emph{mydir} (alpha testing)} +@item @strong{--workdir} @emph{mydir} (beta testing) +@anchor{@strong{--workdir} @emph{mydir} (beta testing)} -@item @strong{--wd} @emph{mydir} (alpha testing) -@anchor{@strong{--wd} @emph{mydir} (alpha testing)} +@item @strong{--wd} @emph{mydir} (beta testing) +@anchor{@strong{--wd} @emph{mydir} (beta testing)} Files transferred using @strong{--transfer} and @strong{--return} will be relative to @emph{mydir} on remote computers, and the command will be executed in diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/src/sem new/parallel-20151122/src/sem --- old/parallel-20151022/src/sem 2015-10-21 23:26:58.000000000 +0200 +++ new/parallel-20151122/src/sem 2015-11-22 19:47:28.000000000 +0100 @@ -435,8 +435,8 @@ my $something_written = 0; for my $pid (keys %incomplete_jobs) { my $job = $incomplete_jobs{$pid}; - if($job->stdin_buffer_length()) { - $something_written += $job->non_block_write(); + if($job->block_length()) { + $something_written += $job->non_blocking_write(); } else { delete $incomplete_jobs{$pid} } @@ -509,7 +509,7 @@ # %Global::running # Returns: # $something_written = amount of bytes written - my ($header_ref,$block_ref,$recstart,$recend,$endpos) = @_; + my ($header_ref,$buffer_ref,$recstart,$recend,$endpos) = @_; my $something_written = 0; my $block_passed = 0; my $sleep = 1; @@ -522,13 +522,13 @@ values %Global::running); } while(my $job = shift @robin_queue) { - if($job->stdin_buffer_length() > 0) { - $something_written += $job->non_block_write(); + if($job->block_length() > 0) { + $something_written += $job->non_blocking_write(); } else { - $job->set_stdin_buffer($header_ref,$block_ref,$endpos,$recstart,$recend); + $job->set_block($header_ref,$buffer_ref,$endpos,$recstart,$recend); $block_passed = 1; $job->set_virgin(0); - $something_written += $job->non_block_write(); + $something_written += $job->non_blocking_write(); last; } } @@ -633,21 +633,21 @@ # Input: # $chunk_number = sequence number - to see if already run # $header_ref = reference to header string to prepend - # $record_ref = reference to record to write + # $buffer_ref = reference to record to write # $recstart = start string of record # $recend = end string of record - # $endpos = position in $record_ref where record ends + # $endpos = position in $buffer_ref where record ends # Uses: # $Global::job_already_run # $opt::roundrobin # @Global::virgin_jobs # Returns: # Number of chunks written (0 or 1) - my ($chunk_number,$header_ref,$record_ref,$recstart,$recend,$endpos) = @_; + my ($chunk_number,$header_ref,$buffer_ref,$recstart,$recend,$endpos) = @_; if($endpos == 0) { return 0; } if(vec($Global::job_already_run,$chunk_number,1)) { return 1; } if($opt::roundrobin) { - return round_robin_write($header_ref,$record_ref,$recstart,$recend,$endpos); + return round_robin_write($header_ref,$buffer_ref,$recstart,$recend,$endpos); } # If no virgin found, backoff my $sleep = 0.0001; # 0.01 ms - better performance on highend @@ -655,28 +655,31 @@ ::debug("pipe", "No virgin jobs"); $sleep = ::reap_usleep($sleep); # Jobs may not be started because of loadavg - # or too little time between each ssh login. + # or too little time between each ssh login + # or retrying failed jobs. start_more_jobs(); } my $job = shift @Global::virgin_jobs; # Job is no longer virgin $job->set_virgin(0); - # We ignore the removed rec_sep which is technically wrong. - $job->add_transfersize($endpos + length $$header_ref); - if(fork()) { - # Skip - } else { - # Chop of at $endpos as we do not know how many rec_sep will - # be removed. - substr($$record_ref,$endpos,length $$record_ref) = ""; - # Remove rec_sep - if($opt::remove_rec_sep) { - Job::remove_rec_sep($record_ref,$recstart,$recend); - } - $job->write($header_ref); - $job->write($record_ref); - close $job->fh(0,"w"); - exit(0); + if(1) { + # We ignore the removed rec_sep which is technically wrong. + $job->add_transfersize($endpos + length $$header_ref); + if(fork()) { + # Skip + } else { + # Chop of at $endpos as we do not know how many rec_sep will + # be removed. + substr($$buffer_ref,$endpos,length $$buffer_ref) = ""; + # Remove rec_sep + if($opt::remove_rec_sep) { + Job::remove_rec_sep($buffer_ref,$recstart,$recend); + } + $job->write($header_ref); + $job->write($buffer_ref); + close $job->fh(0,"w"); + exit(0); + } } close $job->fh(0,"w"); return 1; @@ -952,14 +955,7 @@ $opt::blocksize = 2**31-1; } $opt::memfree = multiply_binary_prefix($opt::memfree); - if(defined $opt::timeout and $opt::timeout !~ /^\d+(\.\d+)?%?$/) { - ::error("--timeout must be seconds or percentage."); - wait_and_exit(255); - } - if(defined $opt::fifo and $opt::cat) { - ::error("--fifo cannot be combined with --cat."); - ::wait_and_exit(255); - } + check_invalid_option_combinations(); if((defined $opt::fifo or defined $opt::cat) and not $opt::pipepart) { $opt::pipe = 1; @@ -1043,13 +1039,6 @@ # Default = {} $opt::tagstring = $Global::parensleft.$Global::parensright; } - if(defined $opt::pipepart and - (defined $opt::L or defined $opt::max_lines - or defined $opt::max_replace_args)) { - ::error("--pipepart is incompatible with --max-replace-args, ". - "--max-lines, and -L."); - wait_and_exit(255); - } if(grep /^$Global::arg_sep$|^$Global::arg_file_sep$/o, @ARGV) { # Deal with ::: and :::: @ARGV=read_args_from_command_line(); @@ -1062,23 +1051,6 @@ if(defined $opt::bar) { $opt::progress = $opt::bar; } - if(defined $opt::retired) { - ::error("-g has been retired. Use --group.", - "-B has been retired. Use --bf.", - "-T has been retired. Use --tty.", - "-U has been retired. Use --er.", - "-W has been retired. Use --wd.", - "-Y has been retired. Use --shebang.", - "-H has been retired. Use --halt.", - "--ctrlc has been retired.", - "--noctrlc has been retired."); - ::wait_and_exit(255); - } - if(defined $opt::tollef and not $opt::gnu) { - ::error("--tollef has been retired.","Remove --tollef or use --gnu to override --tollef."); - ::wait_and_exit(255); - } - citation_notice(); parse_halt(); @@ -1098,9 +1070,55 @@ open_joblog(); } +sub check_invalid_option_combinations { + if(defined $opt::timeout and $opt::timeout !~ /^\d+(\.\d+)?%?$/) { + ::error("--timeout must be seconds or percentage."); + wait_and_exit(255); + } + if(defined $opt::fifo and $opt::cat) { + ::error("--fifo cannot be combined with --cat."); + ::wait_and_exit(255); + } + if(defined $opt::pipepart and + (defined $opt::L or defined $opt::max_lines + or defined $opt::max_replace_args)) { + ::error("--pipepart is incompatible with --max-replace-args, ". + "--max-lines, and -L."); + wait_and_exit(255); + } + if(defined $opt::group and $opt::ungroup) { + ::error("--group cannot be combined with --ungroup."); + ::wait_and_exit(255); + } + if(defined $opt::group and $opt::linebuffer) { + ::error("--group cannot be combined with --line-buffer."); + ::wait_and_exit(255); + } + if(defined $opt::ungroup and $opt::linebuffer) { + ::error("--ungroup cannot be combined with --line-buffer."); + ::wait_and_exit(255); + } + if(defined $opt::tollef and not $opt::gnu) { + ::error("--tollef has been retired.","Remove --tollef or use --gnu to override --tollef."); + ::wait_and_exit(255); + } + if(defined $opt::retired) { + ::error("-g has been retired. Use --group.", + "-B has been retired. Use --bf.", + "-T has been retired. Use --tty.", + "-U has been retired. Use --er.", + "-W has been retired. Use --wd.", + "-Y has been retired. Use --shebang.", + "-H has been retired. Use --halt.", + "--ctrlc has been retired.", + "--noctrlc has been retired."); + ::wait_and_exit(255); + } +} + sub init_globals { # Defaults: - $Global::version = 20151022; + $Global::version = 20151122; $Global::progname = 'parallel'; $Global::infinity = 2**31; $Global::debug = 0; @@ -4229,7 +4247,8 @@ my $script = memfreescript(); # TODO add sshlogin and backgrounding - $self->{'memfree'} = qx{ $script }; + # Run it twice if it gives 0 + $self->{'memfree'} = qx{ $script } || qx{ $script }; if(not $self->{'memfree'}) { ::die_bug("Less than 1 byte free"); } @@ -6198,32 +6217,38 @@ } } -sub set_stdin_buffer { +sub set_block { # Copy stdin buffer from $block_ref up to $endpos - # Prepend with $header_ref + # Prepend with $header_ref if virgin (i.e. not --roundrobin) # Remove $recstart and $recend if needed # Input: # $header_ref = ref to $header to prepend - # $block_ref = ref to $block to pass on + # $buffer_ref = ref to $buffer containing the block # $endpos = length of $block to pass on # $recstart = --recstart regexp # $recend = --recend regexp # Returns: # N/A my $self = shift; - my ($header_ref,$block_ref,$endpos,$recstart,$recend) = @_; - $self->{'stdin_buffer'} = ($self->virgin() ? $$header_ref : "").substr($$block_ref,0,$endpos); + my ($header_ref,$buffer_ref,$endpos,$recstart,$recend) = @_; + $self->{'block'} = ($self->virgin() ? $$header_ref : "").substr($$buffer_ref,0,$endpos); if($opt::remove_rec_sep) { - remove_rec_sep(\$self->{'stdin_buffer'},$recstart,$recend); + remove_rec_sep(\$self->{'block'},$recstart,$recend); } - $self->{'stdin_buffer_length'} = length $self->{'stdin_buffer'}; - $self->{'stdin_buffer_pos'} = 0; - $self->add_transfersize($self->{'stdin_buffer_length'}); + $self->{'block_length'} = length $self->{'block'}; + $self->{'block_pos'} = 0; + $self->add_transfersize($self->{'block_length'}); } -sub stdin_buffer_length { +sub block_ref { + my $self = shift; + return \$self->{'block'}; +} + + +sub block_length { my $self = shift; - return $self->{'stdin_buffer_length'}; + return $self->{'block_length'}; } sub remove_rec_sep { @@ -6234,26 +6259,26 @@ $$block_ref =~ s/$recend$//os; } -sub non_block_write { +sub non_blocking_write { my $self = shift; my $something_written = 0; use POSIX qw(:errno_h); # for loop used to avoid copying substr: $buf will be an alias for the substr - for my $buf (substr($self->{'stdin_buffer'},$self->{'stdin_buffer_pos'})) { + for my $buf (substr($self->{'block'},$self->{'block_pos'})) { my $in = $self->fh(0,"w"); my $rv = syswrite($in, $buf); if (!defined($rv) && $! == EAGAIN) { # would block $something_written = 0; - } elsif ($self->{'stdin_buffer_pos'}+$rv != $self->{'stdin_buffer_length'}) { + } elsif ($self->{'block_pos'}+$rv != $self->{'block_length'}) { # incomplete write # Remove the written part - $self->{'stdin_buffer_pos'} += $rv; + $self->{'block_pos'} += $rv; $something_written = $rv; } else { # successfully wrote everything my $a = ""; - $self->set_stdin_buffer(\$a,\$a,0,"",""); + $self->set_block(\$a,\$a,0,"",""); $something_written = $rv; } } @@ -8001,11 +8026,13 @@ if($perlexpr =~ /^(\d+) /) { # Positional defined($record->[$1-1]) or next; - $self->{'len'}{$perlexpr} += length $record->[$1-1]->replace($perlexpr,$quote_arg,$self); + $self->{'len'}{$perlexpr} += + length $record->[$1-1]->replace($perlexpr,$quote_arg,$self); } else { for my $arg (@$record) { if(defined $arg) { - $self->{'len'}{$perlexpr} += length $arg->replace($perlexpr,$quote_arg,$self); + $self->{'len'}{$perlexpr} += + length $arg->replace($perlexpr,$quote_arg,$self); } } } @@ -8604,7 +8631,7 @@ if($opt::pipe or $opt::pipepart) { if($cmd_line->replaced() eq "") { # Empty command - pipe requires a command - ::error("--pipe must have a command to pipe into (e.g. 'cat')."); + ::error("--pipe/--pipepart must have a command to pipe into (e.g. 'cat')."); ::wait_and_exit(255); } } else { diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/src/sql new/parallel-20151122/src/sql --- old/parallel-20151022/src/sql 2015-10-21 23:26:58.000000000 +0200 +++ new/parallel-20151122/src/sql 2015-11-22 19:47:28.000000000 +0100 @@ -566,7 +566,7 @@ exit ($err); sub parse_options { - $Global::version = 20151022; + $Global::version = 20151122; $Global::progname = 'sql'; # This must be done first as this may exec myself diff -urN '--exclude=CVS' '--exclude=.cvsignore' '--exclude=.svn' '--exclude=.svnignore' old/parallel-20151022/src/sql.1 new/parallel-20151122/src/sql.1 --- old/parallel-20151022/src/sql.1 2015-10-21 23:27:11.000000000 +0200 +++ new/parallel-20151122/src/sql.1 2015-11-22 19:47:39.000000000 +0100 @@ -133,7 +133,7 @@ .\" ======================================================================== .\" .IX Title "SQL 1" -.TH SQL 1 "2015-10-21" "20151022" "parallel" +.TH SQL 1 "2015-11-22" "20151122" "parallel" .\" For nroff, turn off justification. Always turn off hyphenation; it makes .\" way too many mistakes in technical documents. .if n .ad l Files old/parallel-20151022/src/sql.pdf and new/parallel-20151122/src/sql.pdf differ ++++++ parallel-20151022.tar.bz2.sig -> parallel-20151122.tar.bz2.sig ++++++ --- /work/SRC/openSUSE:Factory/gnu_parallel/parallel-20151022.tar.bz2.sig 2015-10-25 14:18:14.000000000 +0100 +++ /work/SRC/openSUSE:Factory/.gnu_parallel.new/parallel-20151122.tar.bz2.sig 2015-11-26 17:03:45.000000000 +0100 @@ -2,7 +2,7 @@ # To check the signature run: # echo | gpg -# gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve parallel-20151022.tar.bz2.sig +# gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve parallel-20151122.tar.bz2.sig echo | gpg 2>/dev/null gpg --auto-key-locate keyserver --keyserver-options auto-key-retrieve $0 @@ -11,32 +11,32 @@ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 -iQTwBAABCgAGBQJWKAUwAAoJENGrRRaIiIiI/Jwmn0SvVu4qT0nKVBMqWh0+qQ1M -PDlxwOJKbEuHydsCiLg9I3nqvEFTAqq0rO5T5yBdw+D6ACPOF8MwId+lPATKr3sU -KthGAdMg/NimCeiARvbbuQsLzfd56hUtCCgyDv7aylgn8FBsA5HyE7c4G+32n/or -45QvfbpFCbLF4D9KrUXh0SkC8M+2aBa0sokK/a2odJefvqYNm1pSzNrWsXAEGIDs -zEgxXIvRiSnZUcguWVUFFwfoxJqGgY/nyDYDkmLHJU6gheX/M2pVhF2kwK7b3IBN -GVQqIN8/z1aQwqlFTRkVdNzphWGqoAmih5Ch5Kn5579jGVRRhF+rz2Y9oQUlm8gx -cmMH8ysEUM4HKwazS+ujaicBCSDuGMiJ7odwm7V1E8Mk3N9fem7Dm8bHAjlb0DYk -j5ICi3I9Mb8yzbsehF2crxS+nnfr4/T0tqkjFFJY2dKHEXVotiB0z3kUWPtcI863 -aPGxGJg3lJpfxond4A3Nd8fGmzHPHzE2h6swXF33+EJvFKyh7rqqb0jB6djAMThH -SCKEVIaHUCLxXAN5/Yf1lYJJ2/YEjuPpO8sqAZMKY+LI9YQoJ0n3Acx7J8KZf9oe -zPYnKI0rDNu2wYrme6nnbWY5A7erh5KB3OaslyA+fP9d/WVVeMK451r7nl0NHq1o -Nyhgpo1rEe1QeOvv5OuMeB5NVIYhgUWe7a896MasbDcC3TubxxwUwXnusTAadHH5 -m4cVcVr5GlfjWbzPPs0WQXmJpbSoX4X+EgnU2uQFQVe9z4KuugZ3/PM4DovzNH7R -Dk9lnaARSHdz7BReqli+gff5gGnqJ3/+3FbQsqoSO6FPpeBvBN1ohYTl1xxBceCH -92jysM0adF3N/5AamVfFFSHBfMYSjql2LgbRQXpeRl1aqTsftz0EsrW4qAe5cqEl -O7ilhzEj3+5orxwFB88JBf1BscD2ZI1irQi0UjlGplHZ/iJnNCjYewrj9Iu0665r -yB85SLA2RTwazmh4MU5bbxjNwy9CAbXS//wjj3+gc9y1wxS3BRO6wq9OBkGYy31g -TLDNYYDekwuBeICIzmPgXyRMluxq1HH4Regi8NzBvaZNQev7EAo0reBlbHCcVYjp -iOUaZ+I/a+c+XyMsYqgFe1ReKYoztQkGwVclk72ztP/kGowBROFpPMmG11ym0efu -oLMYfoQwBcAl23OtK8EXyrdk6ySSzuBejeRIOcfNsqGJjDqdKpSecQgF66bbTGyZ -MGz1rDvXSIzU8pcLae/xBbUouTji7JDeEplrOwiGAEcP6O64jhcLgIimLfZaGFel -VReXN3gA3loQWqpeY7ENrc1oAe0r60RR+emjrSklrcyPbLKV1liaKgQ5Ha9rX6oI -MafEwuN6GHSCwT6T1VCLHITRjAp1mrV6SmaFK5BF2Xo20QgrAPLj1WEw2jFbZ5bB -xApWndeJs1RhMKjhy6bF/kCnv2IkpPAj8en5zqpxlKbwuxXmdaNqNis+Sd/dscnf -CulJ/AKk6AuROKzbq2TwBm7CFDCmZDJPBiFre+qsKfbbxEhBi1bu9SCq+IQaU1XH -ulW/fd07W2oP4ze8DhJojQrxxIr5Itt9A1SSfstYp7GMsR3dZbRJedtsW0ilZ2zB -QqP7HPGu3153bZoU4F5o4iEZLg== -=78kD +iQTwBAABCgAGBQJWUg5pAAoJENGrRRaIiIiICMkmn288HZmNrrmTcYhwRqZ4NkeB ++2QibiyxnkLZCv3hyzhZ8UGJ9FYYzdxMq8bpsK7d6VnbwwcpGX91rwmsrFedA0yf +fUxRXQE0Ph9+6E5r0kqRP0wi4W5ZSpgqmGnOYyHZhy2mOQi1j70fvW70X4LRcrPg +pkyE++5abh7RRnrvRyW+BE91x5l0sk7deCiw/kRPPiZNgI4W389LhYwen8B6evCc +mfBTjJibvj93+58kbvDZPRjxFcpx2vq6E+jrquMfQiZ7JlkEF5ki9asNx9vqAFks +WnS3Id73ZJjazAFLA1aJB2aYcIRcby4g3xdMWhSWfCuuuqlZNWXnEhXWXvaY9Aad +DelrcTUceN6kdEKPv8/djq9iqmDneofQiu6h1osZ6bEvZ0Uncp+5Cr6RqeYO4+ZH +JAEmsDEAPadxxQ9Fzjo/fe0/u29LSF2/f9tR0Cg9G200TV9liVnZYlxjIYN6kfkh +8kvIAOofW1NXj0DSr3z9HqDCSZ5Nh+oRrl/s+DXQNjNfJEMp1uHWBbUbawiBBfQm +Y2Fs1CCeceD5oRwf5tGLFWlp62Xevflcbz6xmB1Zw528naHM6BkeofLSqrcjbKfA +KMpk3IE5lzdgSZkBqIh/eItm8yynijMe424bm818WD1ZWCMJYAmSZCSANHU13gd3 +8Sw5D1IAowBtiyWsBLBEun1u69XoY+D9GsxqsDbKREVmYzKrTn9pvDmh5/UBola1 +UUfMzT1Fe7RF9Te5U10MncppV8AE0IhzueJ4PUxwCQRJ/5eQqCviQal80PLIHH8f +ZXP3h41FGiuIafuoLH4/TqlpBzVX4gg2x32PYHeG34FipTTevACHtKM9ifqyeCO5 +PpYiYsCvFQnGexRPMqKNepUgCaq0N0+TO4+Orqxo9y7F8qFTBUYac4kux8nNGzdk +cf4I4or27Ux/XR1WhYPXJq03DrRPcjKKA5maW/gSiDIvUe198oMZwbVB0YoYiMQz +Wr9VKI4f6Wrnq2Mw1HaA5Dz2gL6gdCRvyZPqjqYSRk6OfgB3GohnGAIEwhFT5ipO +NoMplpRxAKNIyd8XxRWy20Pb7VQ27/JGKb0npkXglCk/ekcuuYW9e2U1+rKPjN3u +Tx/EQuv3UrDTpxs+V/Tj3/9ik//RgFLbx8s6yZt5zxjfV7ccVKCBZGvJsQ0IRZ1I +hAVQCDN2EwlpOE6RBw1cPkFTodSW3+I7OudYXTHdPtUQOLIzSf82CGBCL8RDyc+/ +8oDATnmGq4ewcDRRAlaND/V3JhnKA7VyVPgMqL5X5FKOV4QCY8+OnA7BqBmd+VUL +6CzoEw68sHkgcUCg+AmL2qm4g0NCk7qy37TDpJFClV6UHL4SwOg/ZCs53qgWWRPt +lE2YSc5LAv31Ne77LN/sAW44dnP+hwLP1xsfrZfePxKj6wltjK2/uctocVZUlUOC +NYLTlBUZfhki24pHQ8U6LdHZ1U6ol0JuHNrZA3eUpSWWrZdhDY5o6Ck+LlR7JpzK +Sr4X8IdFQ6LKuFsJlOP5cL6ytwjzXX3yiqvLaYgKFvtVYsNr+YX3lkdcUZXG8H0M +bx146xEiHrLMgF4oH4dCLQzS7diQn7lYyzUKDHoiJWPI6WXwjevakFYCgvihopM0 +tMAmoWJAC1oyx5tgX1Stt9VI2A== +=R6NL -----END PGP SIGNATURE-----
