Re: Example of use of GNU Parallel
Hi Nanditha, I'm not going to reply to what you have written, but note that it is considered bad netiquette to start a new topic by replying to an existing message, because it messes up the view in threaded messages. Next time, just write a new message to parallel@gnu.org. Furthermore, please reply to parallel@gnu.org (by hitting Reply to all) instead of replying to me in private. Regards, Shlomi Fish On Sat, 23 Mar 2013 17:31:45 +0530 Nanditha Rao nanditha...@gmail.com wrote: Sorry for the long mail, but I need to explain this strange problem [SNIPPED] -- - Shlomi Fish http://www.shlomifish.org/ Chuck Norris/etc. Facts - http://www.shlomifish.org/humour/bits/facts/ All language designers are arrogant. Goes with the territory… :-) — http://en.wikiquote.org/wiki/Larry_Wall Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: Creating directory with partial filename
Hi Mr. Thorsted,
On Mon, 3 Jun 2013 19:52:01 +
Tyler Thorsted t.thors...@ldschurch.org wrote:
Hello,
I am new to parallel and didn't see in the manual a way to use a portion of a
filename in creating a folder to move files into after processing.
I am using ImageMagick to convert images to another format and my naming
convention for files is static up until the last 6 characters, which are
unique to each image.
I would like to create a directory via the script to create a folder with the
static portion of the filename and process all the images with that static
portion to the that folder.
ie.
M1234_b_f0001_1.tif
M1234_b_f0001_2.tif
M1234_b_f0001_3.tif
M1234_b_f0001_4.tif
all jpg images would process to a directory named M1234_b_f0001
and
M1234_b_f0002_1.tif
M1234_b_f0002_2.tif
M1234_b_f0002_3.tif
M1234_b_f0002_4.tif
all jpg images would process to M1234_b_f0002
I have used the regex s/.\{6\}$// in other scripts to get the folder name,
but how do I use it in parallel.
I suggest you code a short script in shell, Perl, Python or whatever and use
parallel to call it, and this script will do the transformation you want with
its input and emit the result.
This is how I call parallel:
https://bitbucket.org/shlomif/black-hole-solitaire/src/fbc25e17627bd37a9756881fce7a5fe23f825c60/black-hole-solitaire/all-in-a-row-gnu-parallel-range-solving/range-solving.bash?at=default
(short URL - http://is.gd/8JGPst ).
Regards,
Shlomi Fish
--
-
Shlomi Fish http://www.shlomifish.org/
Understand what Open Source is - http://shlom.in/oss-fs
When Chuck Norris uses git, he takes a coffee break after initiating every git
commit. And then he waits for the commit to finish.
— http://www.shlomifish.org/humour/bits/facts/Chuck-Norris/
Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: Citation notice and funding for development - your feedback is needed
On Mon, 4 Nov 2013 00:13:22 +0100 Ole Tange ta...@gnu.org wrote: I have spent several man years on GNU Parallel. One of the things that helps fund this is me having a job. One of the things that helps me having a job is if my article on GNU Parallel is cited. The more citations the easier it is for me to find a job that allows for further development. GNU Parallel is used by more and more people for processing data that ends up in publications. That is good. But I have discovered that most of the time they forget to cite my article; which is bad. The problem seems to be that GNU Parallel is so easy to install and use that they never look at: The website, the manual, the --version, the --help, or the tutorial. Citations are primarily relevant in academia and research institutes, so the optimal would be if users in these organisations were reminded about the citation while everyone else is unaffected. The best idea I have come up with so far is printing a citation notice on STDERR if output is to a terminal when GNU Parallel starts. The notice will not be printed if STDERR is redirected (to a pipe or a file), it will also not be printed if --no-notice is given and it can be disabled completely by running --bibtex once: When using GNU Parallel to process data for publication please cite: O. Tange (2011): GNU Parallel - The Command-Line Power Tool, ;login: The USENIX Magazine, February 2011:42-47. This helps funding further development. To get rid of this notice run 'parallel --bibtex' once or use '--no-notice'. One problem I see with that is that (by default at least) it violates the UNIX’s Silence is Golden principle: http://catb.org/~esr/writings/taoup/html/ch11s09.html I.e: people may be annoyed from it, and may also lack the patience to read everything there, and conclude that the program is annoying (due to the users don't read principle of UI design). I should note that I violated this for http://fc-solve.shlomifish.org/ ’s fc-solve executable because people on Windows used to double click the executable and get an empty command prompt window (because fc-solve was waiting for standard input). So now: shlomif@telaviv1:~$ unset FREECELL_SOLVER_QUIET shlomif@telaviv1:~$ fc-solve Reading the board from the standard input. Please refer to the documentation for more usage information: http://fc-solve.shlomifish.org/docs/To cancel this message set the FREECELL_SOLVER_QUIET environment variable. It seems that with GNU parallel however, the reason is less justified and detracts from its general usability. Regards, Shlomi Fish -- - Shlomi Fish http://www.shlomifish.org/ Why I Love Perl - http://shlom.in/joy-of-perl There is no IGLU Cabal! None of them could pass the Turing test. But strangely enough a computer program they coded, could. Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: You all
Hi Mait, Please reply to list - http://shlom.in/reply . On Wed, 1 Jan 2014 10:36:06 +0900 Mait matte...@gmail.com wrote: bad people because didn't tell me about this crazy tool. You should spam about this regularly. Spamming is unethical, not to mention often illegal. However, if you care about GNU parallel, you might be able to donate some money for web advertising (Google AdSense/etc.). Will you? I've read tutorial and man page. It could be better if I read tutorial first. Yes, 2500 lines are not convenience. :-) First sentence of`man parallel` could be Read the tutorial. Don't read me first.. It should be You should probably read the tutorial first. (with a link). I can try submitting a patch for it. Anyway I'm happy to know parallela. It's parallel - not parallela (it is spelled without the trailing a). Very impressive. It just likely will be my shell. parallel cannot be made a shell (like bash, zsh, or ksh). If you mean that it will be part of your shell’s commands’ arsenal, then sure - it’s a welcome addition. One thing, Why the ing '--tollef' is in '/etc/parallela/config' by default? (Ubuntu 13.10) I've lost my two hours wondering why parallela ignore stdin. Is that upstream default? I do not see a file called /etc/parallel/config anywhere in Mageia Linux’s parallel-20130722-2.mga4 package. So I presume it’s an Ubuntu addition. What is tollef? The man page says: --tollef (obsolete - will be retired 20140222) Make GNU parallel behave more like Tollef's parallel command. It activates -u, -q, and --arg-sep --. It also causes -l to change meaning to --load. Not giving '--' is unsupported. Do not use --tollef unless you know what you are doing. To override use --gnu. I don't know what Tollef’s parallel command is. Regards, Shlomi Fish -- - Shlomi Fish http://www.shlomifish.org/ The Human Hacking Field Guide - http://shlom.in/hhfg In the Technion, there are many ways to get from one place to the other, but they are all the same length. Please reply to list if it's a mailing list post - http://shlom.in/reply .
1. Typo in https://www.gnu.org/software/parallel/ ; 2. Please add Mageia
Hi all, the page https://www.gnu.org/software/parallel/ reads: «Official packages exists for:» exists should be exist. Furthermore, please add the Mageia (= a community-maintained GNU/Linux distribution that started as a fork of Mandriva) package to the list there: * http://mageia.madb.org/package/show/application/0/name/parallel Regards, Shlomi Fish -- - Shlomi Fish http://www.shlomifish.org/ NSA Factoids - http://www.shlomifish.org/humour/bits/facts/NSA/ I invented the term Object‐Oriented, and I can tell you I did not have C++ in mind. — Alan Kay (Attributed) Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: 1. Typo in https://www.gnu.org/software/parallel/ ; 2. Please add Mageia
On Fri, 21 Feb 2014 12:38:59 +0100 Ole Tange o...@tange.dk wrote: Thanks. Fixed in next release. /Ole Thanks! Regards, Shlomi Fish -- - Shlomi Fish http://www.shlomifish.org/ Buffy Factoids - http://www.shlomifish.org/humour/bits/facts/Buffy/ The devil created a 10th circle of hell for the inventors of XSLT, because the first nine circles were too mild for them. — http://www.shlomifish.org/humour/bits/facts/XSLT/ Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: singleton for script with shebang
Hi Bradley, On Thu, 1 Jan 2015 13:54:57 -0800 Bradley Asztalos b...@customweather.com wrote: I have a simple script, foo.sh:: #!/bin/bash echo start $$ sleep 10 echo end $$ I can run make sure this runs as a singleton with sem --fg -u --id lock_id ./foo.sh But I'm wondering if it is possible to use shebang in this context as in: ---singleton.sh- #!/opt/local/bin/parallel --shebang-wrap /bin/bash --semaphore --fg -u --id lock_id echo start $$ sleep 10 echo end $$ - parallel is written in Perl 5 and uses «#!/usr/bin/perl» or whatever as a shebang. As a result it cannot be itself used as a she-bang processor. Furthermore, I should note that in programming, it is a good idea to stick to the https://en.wikipedia.org/wiki/KISS_principle and not try to be as clever as possible. Hope it helps. Regards, Shlomi Fish This does not work. Any ideas here? Bradley -- - Shlomi Fish http://www.shlomifish.org/ The Human Hacking Field Guide - http://shlom.in/hhfg A kid always wishes they were older until they are 18. Afterwards, they always wish they were younger. Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: Joblog Parsing
Hi Harry, On Wed, 1 Jul 2015 14:30:22 +0100 Harry Jubb harry.j...@gmail.com wrote: To all, If anyone is interested, I've written a small program to give some summary output for log files produced with the --joblog option. It's available here: https://github.com/harryjubb/joblog I'm not particularly interested in that, but let me thank you for sharing it, under a FOSS licence, and making sure that the repository contains README and LICENSE files. Thanks! Regards, Shlomi Fish All the best, Harry -- - Shlomi Fish http://www.shlomifish.org/ Humanity - Parody of Modern Life - http://shlom.in/humanity First the NSA ignores “silly” Internet memes, then they laugh at them, then they are unable to fight them, and then they lose. — http://www.shlomifish.org/humour/bits/facts/NSA Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: GNU Parallel 20150822 ('Hans Vibeke') released
Hi Hans,
On Sun, 23 Aug 2015 18:57:02 +0200
Hans Schou ch...@schou.dk wrote:
2015-08-23 13:11 GMT+02:00 Shlomi Fish shlo...@shlomifish.org:
Hi Ole,
On Sat, 22 Aug 2015 11:55:49 +0200
Ole Tange ta...@gnu.org wrote:
GNU Parallel 20150822 ('Hans Vibeke') has been released. It is
available for download at: http://ftp.gnu.org/gnu/parallel/
Pardon my ignorance, but who or what is Hans Vibeke? Neither
I am Hans.
Every 22th of the month Ole Tange releases a new version of GNU Parallel.
Each release has a name of an event which has just happen within this
month. You can read more about it in this thread:
http://lists.gnu.org/archive/html/parallel/2014-02/msg00010.html
It happens so that I have been married to Vibeke for 12½ years as of
2015-08-22 which also is the release date of GNU Parallel. Ole was invited
to the party, and named it after that. BTW in Denmark we call 12½ years of
wedding copper wedding.
I see. Congratulations and Mazal Tov!
Releases can not have special characters like '' because of Twitter etc,
so the release name was not Hans Vibeke 12½ years, which would be more
informative.
I'm pretty sure Twitter can handle and ½ and any other Unicode character
you throw at it fine. It has a character limit, though.
Did I miss anything?
Yes, the release/wedding party ;-)
:-).
Regards,
Shlomi Fish
./hans
--
-
Shlomi Fish http://www.shlomifish.org/
Beginners Site for the Vim text editor - http://vim.begin-site.org/
Objective jIronPerl++.NET Enterprise Edition♭ Professional Home Premium 64-bit
Single-user.
— based on a Freenode #perl conversation ( http://is.gd/cCUBY2 )
Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: GNU Parallel 20150822 ('Hans Vibeke') released
Hi Ole,
On Sat, 22 Aug 2015 11:55:49 +0200
Ole Tange ta...@gnu.org wrote:
GNU Parallel 20150822 ('Hans Vibeke') has been released. It is
available for download at: http://ftp.gnu.org/gnu/parallel/
Pardon my ignorance, but who or what is Hans Vibeke? Neither
https://encrypted.google.com/search?hl=enq=hans%20vibeke (whose results may
vary for you) nor https://duckduckgo.com/?q=hans+vibekeia=products find
anything definitive. Did I miss anything?
Regards,
Shlomi Fish
[SNIPPED]
--
-
Shlomi Fish http://www.shlomifish.org/
My Photos - http://www.flickr.com/photos/shlomif/
Chuck Norris does not code; when he sits at a computer, it just does whatever
he wants. — Kattana on Freenode’s #perl6 .
— http://www.shlomifish.org/humour/bits/facts/Chuck-Norris/
Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: GNU Parallel 20151022 ('Liquid Water') released
Hi Ole,
On Thu, 22 Oct 2015 00:13:23 +0200
Ole Tange <ta...@gnu.org> wrote:
> GNU Parallel 20151022 ('Liquid Water') has been released. It is
> available for download at: http://ftp.gnu.org/gnu/parallel/
>
I now packaged it for Mageia Cauldron - thanks! Just a question: is the 'Liquid
Water' nickname a reference to the water that were reportedly found on Mars?
You should document what the nickname refers to somewhere in the
announcement.
Regards,
Shlomi Fish
--
---------
Shlomi Fish http://www.shlomifish.org/
http://youtu.be/KxGRhd_iWuE - Never Give Up!!
The bad thing about hardware is that it sometimes works and it sometimes
doesn’t. The good thing about software is that it’s consistent: it always
does not work, and it always does not work in exactly the same way.
Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: GNU Parallel 20151022 ('Liquid Water') released
Hi Ole,
On Fri, 23 Oct 2015 00:16:56 +0200
Ole Tange <ta...@gnu.org> wrote:
> On Thu, Oct 22, 2015 at 2:03 PM, Shlomi Fish <shlo...@shlomifish.org> wrote:
> > Hi Ole,
> >
> > On Thu, 22 Oct 2015 00:13:23 +0200
> > Ole Tange <ta...@gnu.org> wrote:
> >
> >> GNU Parallel 20151022 ('Liquid Water') has been released.
> :
> > I now packaged it for Mageia Cauldron - thanks!
>
> Great. Can you email me a link that I can add to
> http://www.gnu.org/software/parallel/?
>
> I do not know how the packaging process is for Mageia Cauldron, but if
> you need a place to checkin packaging files, I will be happy to give
> you access to http://git.savannah.gnu.org/cgit/parallel.git/tree/packager
>
There is already a link to the Mageia package there, and Cauldron is the name of
its development branch (which eventually stabilises on the next release). The
current Mageia Cauldron is what is going to become Mageia version 6. The
Cauldron entry is easily reachable from the Mageia link.
I don't need a place to checkin packaging files because Mageia already provides
it for me, but thanks for the offer.
> > Just a question: is the 'Liquid Water' nickname a reference to the water
> > that were reportedly found on Mars?
>
> Maybe.
>
> > You should document what the nickname refers to somewhere in the
> > announcement.
>
> Nah. That will take all the fun out of people guessing.
>
I see.
Regards,
Shlomi Fish
>
> /Ole
>
--
-
Shlomi Fish http://www.shlomifish.org/
Star Trek: “We, the Living Dead” - http://shlom.in/st-wtld
Selina: How may I be of service?
Alan: Well, according to the Codex, we’re supposed to be slain by you.
— http://www.shlomifish.org/humour/Selina-Mandrake/
Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: GNU Parallel for multi threaded copy of 1GB file from linux machine to remote windows machine
Hi Pradeep, please reply to all recipients. See http://shlom.in/reply . On Mon, 19 Oct 2015 13:10:43 +0530 Pradeep Patra <smilesonisa...@gmail.com> wrote: > Hi all, >I am sorry I am beginner to use GNU parallel. I want to use GNU parallel > for multithreaded copy from linux machine to remote windows machine. GNU parallel performs multitasking using multiple processes - not using multiple threads. That put aside - a network copy is likely going to be https://en.wikipedia.org/wiki/I/O_bound and I suggest you take a look at tools like https://en.wikipedia.org/wiki/Rsync and http://zsync.moria.org.uk/ . > > Is it possible ? If so could anyone help me in providing the usage of it. You can read about how to use GNU parallel on its web site: * https://www.gnu.org/software/parallel/ There is a tutorial and if you find anything lacking you can tell us and hopefully it will be improved. But the gods helps them that help themselves. Regards, Shlomi Fish -- ----- Shlomi Fish http://www.shlomifish.org/ The Case for File Swapping - http://shlom.in/file-swap Forth - There are several ways to write it, but no way to read it. — http://www.shlomifish.org/humour/ways_to_do_it.html Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: GNU Parallel 20160322 ('Bruxelles') released
Hi Ole,
On Tue, 22 Mar 2016 21:49:11 +0100
Ole Tange <ta...@gnu.org> wrote:
> GNU Parallel 20160322 ('Bruxelles') has been released. It is available
> for download at: http://ftp.gnu.org/gnu/parallel/
>
thanks for this release. I have packaged it for Mageia Cauldron (= the Mageia
development branch, similar to Debian Testing).
One note: can the tarball also be supplied as tar.xz? I checked and it reduces
the size quite a bit.
Regards,
Shlomi Fish
--
---------
Shlomi Fish http://www.shlomifish.org/
“So, who the hell is Qoheleth?” - http://shlom.in/qoheleth
“Interesting” has a negative correlation with “successful”.
— Anno on Freenode’s #perl
Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: GNU Parallel 20160322 ('Bruxelles') released
Hi Ole,
thanks for your reply.
On Sun, 27 Mar 2016 20:00:57 +0200
Ole Tange <o...@tange.dk> wrote:
> All in all I come to the conclusion that the benefits are small and
> the downsides are just a bit bigger than the benefits. So for the
> foreseeable future there will not be an extra format.
>
I see. I understand and accept that.
Regards,
Shlomi Fish
>
> /Ole
>
--
---------
Shlomi Fish http://www.shlomifish.org/
Freecell Solver - http://fc-solve.shlomifish.org/
Chuck Norris helps God help those that help themselves.
— http://www.shlomifish.org/humour/bits/facts/Chuck-Norris/
Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: parallel error while parsing options
Hi, On Tue, 10 May 2016 16:28:45 +0200 Gabriel Ganne <gabriel.ga...@gmail.com> wrote: > Yes I am. > > Do tell me if you need me to test anything. > > Thanks, > I've now set up a CentOS 7.2.x x86-64 VBox VM and built and installed GNU parallel and it seems to run fine - tested the --help flag and a sample invocation - see the screenshot at: http://www.shlomifish.org/Files/files/images/centos-7.x-gnu-parallel--shlomif.png Some things you can try to diagnose the problem: 1. Devel-Trace - https://metacpan.org/search?q=devel%20trace=20 - you can use perl -d:Trace `which parallel` and see what it gives you. 2. strace. Regards, Shlomi Fish > On 10 May 2016 at 16:20, Shlomi Fish <shlo...@shlomifish.org> wrote: > > > Hi Gabriel, > > > > On Tue, 10 May 2016 14:02:33 +0200 > > Gabriel Ganne <gabriel.ga...@gmail.com> wrote: > > > > > Hi, > > > > > > I am running a redhat7 workstation. > > > > > > $ cat /etc/system-release > > > > > > > Thanks for the quick reply. I am going to try to reproduce it on a > > CentOS-7-x86_64-Minimal-1511.iso VM (I presume you're using the x86-64 > > architecture , right?) > > > > -- Shlomi > > > > > > > > Red Hat Enterprise Linux Workstation release 7.2 (Maipo) > > > > > > $ cat /etc/os-release > > > NAME="Red Hat Enterprise Linux Workstation" > > > VERSION="7.2 (Maipo)" > > > ID="rhel" > > > ID_LIKE="fedora" > > > VERSION_ID="7.2" > > > PRETTY_NAME="Red Hat Enterprise Linux Workstation 7.2 (Maipo)" > > > ANSI_COLOR="0;31" > > > CPE_NAME="cpe:/o:redhat:enterprise_linux:7.2:GA:workstation" > > > HOME_URL="https://www.redhat.com/; > > > BUG_REPORT_URL="https://bugzilla.redhat.com/; > > > > > > REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 7" > > > REDHAT_BUGZILLA_PRODUCT_VERSION=7.2 > > > REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux" > > > REDHAT_SUPPORT_PRODUCT_VERSION="7.2" > > > > > > Regards, > > > > > > > > > -- > > - > > Shlomi Fish http://www.shlomifish.org/ > > “So, who the hell is Qoheleth?” - http://shlom.in/qoheleth > > > > You gotta go out there, believe in the ball, and throw yourself. > > — The Wise Janitor in > > http://en.wikipedia.org/wiki/Not_another_teen_movie > > > > Please reply to list if it's a mailing list post - http://shlom.in/reply . > > > > > -- - Shlomi Fish http://www.shlomifish.org/ http://youtu.be/xZLwtc9x4yA - Anime in Real Life!! (Parody) We have a saying: “We survived Pharaoh - we’ll survive this.” I suppose it’s all cake after Pharaoh. — Freenode’s #reddit channel Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: parallel error while parsing options
Hi Gabriel, On Tue, 10 May 2016 10:28:58 +0200 Gabriel Ganne <gabriel.ga...@gmail.com> wrote: > Hi, > > I installed parallel from this link > http://ftp.gnu.org/gnu/parallel/parallel-20160422.tar.bz2 > I encounter the following error (or with any command) : > Thanks for the report. What is your OS/distribution/versions? Regards, Shlomi > $ parallel --help > parallel: This should not happen. You have found a bug. > Please contact <parallel@gnu.org> and include: > * The version number: 20160422 > * The bugid: pidtable format: 8022 1 > > * The command line being run > * The files being read (put the files on a webserver if they are big) > > If you get the error on smaller/fewer files, please include those instead. > > I most probably have something wrong in my environment, but I have no idea > what. > I am running rhel7. > > $ perl --version > > This is perl 5, version 16, subversion 3 (v5.16.3) built for > x86_64-linux-thread-multi > (with 29 registered patches, see perl -V for more detail) > > Copyright 1987-2012, Larry Wall > > Perl may be copied only under the terms of either the Artistic License or > the > GNU General Public License, which may be found in the Perl 5 source kit. > > Complete documentation for Perl, including FAQ lists, should be found on > this system using "man perl" or "perldoc perl". If you have access to the > Internet, point your browser at http://www.perl.org/, the Perl Home Page. > > Regards, > -- - Shlomi Fish http://www.shlomifish.org/ The Human Hacking Field Guide - http://shlom.in/hhfg At this point, I'd like to take a moment to speak to you about the Adobe PSD format. PSD is not a good format. PSD is not even a bad format. Calling it such would be an insult to other bad formats, such as PCX or JPEG. No, PSD is an abysmal format. — https://github.com/gco/xee/blob/master/XeePhotoshopLoader.m Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: Bug with --memfree
Hi Remi and all,
please reply to list.
On Wed, 24 Aug 2016 11:15:16 -0700
Rémi Matthey-Doret <matt...@zoology.ubc.ca> wrote:
> Hello,
>
> I received the following message
>
> parallel: This should not happen. You have found a bug. Please contact <
> parallel@gnu.org> [..]
>
Just for the record, with your example parallel-20160822 seems to be working
perfectly fine on Mageia Linux v6 x86-64:
«
shlomif@telaviv1:~$ /home/shlomif/apps/gnu-parallel/bin/parallel --memfree
5G example.txt && echo hello
Academic tradition requires you to cite works
you base your article on. When using programs that use GNU Parallel to process
data for publication please cite:
O. Tange (2011): GNU Parallel - The Command-Line Power Tool,
;login: The USENIX Magazine, February 2011:42-47.
This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 1 EUR you should feel free to use GNU Parallel without citing.
To silence the citation notice: run 'parallel --citation'.
Thank you
GNU parallel
hello
shlomif@telaviv1:~$
»
Can you try building parallel using a perl installed using
https://perlbrew.pl/ ?
Regards,
Shlomi Fish
> The version number:
>
> 20160822
>
> The bugid:
>
> Less than 1 byte free
>
> The command line being run:
>
> parallel --memfree 5G example.txt
>
> The files being read (put the files on a webserver if they are big)--
>
> See attachement ("example.txt")
>
> The issue is very specific to the use of the `memfree` flag. The command
> `parallel example.txt` works fine.
>
> For extra information: I have 24GB of RAM (Only 9GB currently used), 24
> logical cores and run on MAC OS X 10.11.3.
>
> Thanks a lot for your help!
>
> Remi
>
> Remi Matthey-Doret
>www.zoology.ubc.ca/~matthey/
> PhD Candidate, M. Whitlock Lab
> Beaty Biodiversity Research Center, room 250
> Dept. of Zoology, University of British Columbia
> 6270 University Blvd, Vancouver, BC, V6T 1Z4
> Phone: (604) 369-5929
Problem with parallel : it stops with a warning after several hours of operations
Hi all! With parallel-20170522-1.mga6 on Mageia v6 x86-64, I am getting this: 1,499,693,878.343569040 == 5,645,356,909 == 1,499,693,878.343580961 I could not solve this game. 1,499,693,878.343592882 Total number of states checked is 310,498. 1,499,693,878.343604088 [[== End 5,645,356,909 ==]] 1,499,693,890.266119957 == 5,645,357,053 == 1,499,693,890.266165018 Iterations count exceeded. 1,499,693,890.266180992 Total number of states checked is 2,400,000. 1,499,693,890.266192913 [[== End 5,645,357,053 ==]] 1,499,693,890.266206026 == 5,645,359,103 == 1,499,693,890.266216993 I could not solve this game. 1,499,693,890.266228914 Total number of states checked is 1,117,997. 1,499,693,890.266239882 [[== End 5,645,359,103 ==]] 1,499,693,891.484687090 == 5,645,384,850 == 1,499,693,891.484735966 Iterations count exceeded. 1,499,693,891.484750986 Total number of states checked is 2,400,000. 1,499,693,891.484764099 [[== End 5,645,384,850 ==]] parallel: Warning: No more file handles. parallel: Warning: Raising ulimit -n or /etc/security/limits.conf may help. and parallel stops after that. It is done after running it using https://github.com/shlomif/freecell-3fc-determine-solvability and with -j4 on a Core i3 machine. Is it a bug ? Can it be fixed? Regards, Shlomi Fish -- - Shlomi Fish http://www.shlomifish.org/ What does “Zionism” mean? - http://shlom.in/def-zionism Summer Glau can restore the people that Chuck killed, back to life. — http://www.shlomifish.org/humour/bits/facts/Summer-Glau/ Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: Problem with parallel : it stops with a warning after several hours of operations
Hi Hubert,
thanks for your reply.
On Tue, 11 Jul 2017 08:50:49 +0200 (CEST)
Hubert Kowalski <h.kowal...@hakger.pl> wrote:
> > parallel: Warning: No more file handles.
> > parallel: Warning: Raising ulimit -n or /etc/security/limits.conf may
> > help.
>
> Parallel runs out of file handles. try railing limit, for example
> https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
>
> hope it helps.
>
But why does it happen? I only run 4 jobs at the same time. Aren't file handles
getting closed and recycled? I don't think it should happen.
Regards,
Shlomi
> > Dnia 10 lipiec 2017 o 17:43 Shlomi Fish <shlo...@shlomifish.org> napisał(a):
> >
> >
> > Hi all!
> >
> > With parallel-20170522-1.mga6 on Mageia v6 x86-64, I am getting this:
> >
> > 1,499,693,878.343569040 == 5,645,356,909 ==
> > 1,499,693,878.343580961 I could not solve this game.
> > 1,499,693,878.343592882 Total number of states checked is 310,498.
> > 1,499,693,878.343604088 [[== End 5,645,356,909 ==]]
> > 1,499,693,890.266119957 == 5,645,357,053 ==
> > 1,499,693,890.266165018 Iterations count exceeded.
> > 1,499,693,890.266180992 Total number of states checked is 2,400,000.
> > 1,499,693,890.266192913 [[== End 5,645,357,053 ==]]
> > 1,499,693,890.266206026 == 5,645,359,103 ==
> > 1,499,693,890.266216993 I could not solve this game.
> > 1,499,693,890.266228914 Total number of states checked is 1,117,997.
> > 1,499,693,890.266239882 [[== End 5,645,359,103 ==]]
> > 1,499,693,891.484687090 == 5,645,384,850 ==
> > 1,499,693,891.484735966 Iterations count exceeded.
> > 1,499,693,891.484750986 Total number of states checked is 2,400,000.
> > 1,499,693,891.484764099 [[== End 5,645,384,850 ==]]
> > parallel: Warning: No more file handles.
> > parallel: Warning: Raising ulimit -n or /etc/security/limits.conf may help.
> >
> >
> >
> > and parallel stops after that.
> >
> > It is done after running it using
> > https://github.com/shlomif/freecell-3fc-determine-solvability and with -j4
> > on a Core i3 machine. Is it a bug ? Can it be fixed?
> >
> > Regards,
> >
> > Shlomi Fish
> >
> > --
> > -
> > Shlomi Fish http://www.shlomifish.org/
> > What does “Zionism” mean? - http://shlom.in/def-zionism
> >
> > Summer Glau can restore the people that Chuck killed, back to life.
> > — http://www.shlomifish.org/humour/bits/facts/Summer-Glau/
> >
> > Please reply to list if it's a mailing list post - http://shlom.in/reply .
> >
--
-
Shlomi Fish http://www.shlomifish.org/
http://youtu.be/xZLwtc9x4yA - Anime in Real Life!! (Parody)
“We are NO LONGER the knights who say ‘BitKeeper’. We are now
the knights who say ‘git, git, git, cogito — Linus!’.”
Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: how to build the parallel nature into the script?
Hi, On Wed, 13 Dec 2017 01:06:28 -0500 paralleluser <parallelu...@fastmail.net> wrote: > Dear Friends, > > Given a shell script, script.bash, it takes one argument, $1, which is a file > name to process, like this: script.bash myfile.txt > > To parallelize this I can do: > parallel script.bash ::: *.txt > > But what is the easiest way to build the parallel nature into the script so I > can do: script.bash *.txt > Why not write a wrapper script that does: #!/bin/bash parallel script.bash ::: "$@" Regards, Shlomi > and parallel does its magic behind the scenes? > > Thanks!... > -- ----- Shlomi Fish http://www.shlomifish.org/ Best Introductory Programming Language - http://shlom.in/intro-lang If a million Shakespeares had to write together, they would write like a monkey. — based on Stephen Wright, via Nadav Har’El. Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: parallel bug: Warning: No more file handles. when one job is delayed (reproducible with a test case)
On Sun, 29 Oct 2017 00:26:15 +0200 Ole Tange <o...@tange.dk> wrote: > On Sat, Oct 28, 2017 at 12:44 PM, Shlomi Fish <shlo...@shlomifish.org> wrote: > > > I see. So what you are saying is that parallel will work fine despite the > > warning and will continue running? > > Yep. But if you run into this problem, it might be a better idea to > remove -k and use something like --results instead. > > /Ole Thanks, Ole! I'll investigate. -- ----- Shlomi Fish http://www.shlomifish.org/ http://www.shlomifish.org/humour/ways_to_do_it.html If a million Shakespeares had to write together, they would write like a monkey. — based on Stephen Wright, via Nadav Har’El. Please reply to list if it's a mailing list post - http://shlom.in/reply .
parallel bug: Warning: No more file handles. when one job is delayed (reproducible with a test case)
Hi all!
Thanks for your work.
Attached are two files to reproduce a bug I ran into with GNU parallel
including the latest one on Mageia v7 x86-64:
shlomif@telaviv1:~$ bash run-range.bash
parallel: Warning: No more file handles.
parallel: Warning: Raising ulimit -n or /etc/security/limits.conf may help.
^CCompleted!
The run-single.bash script is delayed for n=1 and meanwhile other jobs
accumulate which may explain the problem. This problem caused me to lose one
night of uptime on an AWS instance because "parallel" got stuck, so I'd
appreciate an investigation and a fix.
Also see https://lists.gnu.org/archive/html/parallel/2017-07/msg6.html .
Regards,
Shlomi Fish
--
-----
Shlomi Fish http://www.shlomifish.org/
My Aphorisms - http://www.shlomifish.org/humour.html
Chuck Norris has 99 problems including a bitch.
— http://www.shlomifish.org/humour/bits/facts/Chuck-Norris/
Please reply to list if it's a mailing list post - http://shlom.in/reply .
run-range.bash
Description: Binary data
run-single.bash
Description: Binary data
Re: parallel bug: Warning: No more file handles. when one job is delayed (reproducible with a test case)
Hi Ole,
On Sat, 28 Oct 2017 00:28:02 +0200
Ole Tange <o...@tange.dk> wrote:
> On Fri, Oct 27, 2017 at 9:23 AM, Shlomi Fish <shlo...@shlomifish.org> wrote:
>
> > Thanks for your work.
>
> Good to know it is appreciated.
>
> > Attached are two files to reproduce a bug I ran into with GNU parallel
> > including the latest one on Mageia v7 x86-64:
> >
> > shlomif@telaviv1:~$ bash run-range.bash
> > parallel: Warning: No more file handles.
> > parallel: Warning: Raising ulimit -n or /etc/security/limits.conf may help.
> > ^CCompleted!
> >
> > The run-single.bash script is delayed for n=1 and meanwhile other jobs
> > accumulate which may explain the problem. This problem caused me to lose one
> > night of uptime on an AWS instance because "parallel" got stuck, so I'd
> > appreciate an investigation and a fix.
>
> Your problem can be illustrated with:
>
> seq 0 1000 | parallel -k -t sleep '{= $_ = $_ ? 0 : 10 =};echo {}'
>
> This will run 'sleep 10' followed by 1000 jobs of 'sleep 0'. -t causes
> the command to be printed as soon as it is started.
>
> Because of -k GNU Parallel must keep the order of the output. It does
> that by having open files to the temporary output files of jobs run.
> What happens here, is that before we can close any of the files, we
> will have to wait for the first job to complete. Because the other
> jobs are very fast to complete, then GNU Parallel runs out of file
> handles, and thus warns you:
>
> parallel: Warning: No more file handles.
> parallel: Warning: Raising ulimit -n or /etc/security/limits.conf may help.
>
> But it is just a warning: As soon as the first job completes, it
> completes the remaining jobs.
>
> > Also see
> > https://lists.gnu.org/archive/html/parallel/2017-07/msg6.html .
>
> If you use -k in that, then we have the explanation: GNU Parallel does
> not stop. It waits for one of the jobs to complete before it can close
> more filehandles.
>
I see. So what you are saying is that parallel will work fine despite the
warning and will continue running? Since it still got stuck, then the problem
is likely elsewhere. Thanks!
>
> /Ole
>
--
-
Shlomi Fish http://www.shlomifish.org/
Parody of "The Fountainhead" - http://shlom.in/towtf
A kid always wishes they were older until they are 18. Afterwards, they always
wish they were younger.
Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: GNU Parallel 20180222 ('Henrik') released
On Thu, 22 Feb 2018 23:48:24 +0100
Hans Schou <ch...@schou.dk> wrote:
> On Thu, Feb 22, 2018 at 10:55 PM, Ole Tange <ta...@gnu.org> wrote:
>
> > * --embed makes it possible to embed GNU parallel in a shell script.
> >
>
> Ain't that a dirty hack ;-)
>
> I got the speech prepared: Within this decade, GNU Parallel would be rolled
> out on all desktops. Not because it is easy, but because it is hard;
> because that goal will serve to organize and measure the best of our
> energies and skills, because that challenge is one that we are willing to
> accept, one we are unwilling to postpone, and one we intend to win.
>
> And to the foreigners who don't know the husband of Queen of Denmark has
> just died recently, he finally got the release name:
>
> GNU Parallel 20180222 ('Henrik') has been released. It is available
>
> for download at: http://ftpmirror.gnu.org/parallel/
>
>
> https://fr.wikipedia.org/wiki/Henri_de_Laborde_de_Monpezat
> Rest In Peace.
>
I see. Rest in peace.
Anyway, I packaged the new parallel for mageia v7 - see
http://pkgsubmit.mageia.org/ . Thanks!
> And my 15th wedding day:
> https://lists.gnu.org/archive/html/parallel/2015-08/msg00012.html
>
> ./hans
--
-
Shlomi Fish http://www.shlomifish.org/
https://github.com/shlomif/what-you-should-know-about-automated-testing
Buffy: OK, Xandoosh, can you perform a search to find where such a duplicator
is present?
Xander: Sure! Hey, Wil, can you use a spell for that?
— http://www.shlomifish.org/humour/Buffy/A-Few-Good-Slayers/
Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: Parallel script with variable name
Hi Ms. Fenderson, On Thu, 28 Feb 2019 12:20:13 +1030 Lindsey Fenderson wrote: > Hi, > > I'm very new to using GNU parallel, so this is probably a simple question > but I haven't been able to figure out from the resources online how to do > the following: > > I have a very long script that I am running over multiple input files and I > would like to parallelize this process. So for instance, > > ls *filepattern* | parallel script.sh > > seems to work as far as iterating over the files. However, I need to use > the filename as a variable. So in my serial script I was doing this: > > ls *filepattern* > filenames > while read filenames; do > lots_of_stuff_$filenames > done < filenames > > So how can I get gnu parallel to incorporate the current filename it is > using as a variable in my script? > See: * https://perl.plover.com/varvarname.html * https://en.wikipedia.org/wiki/Associative_array * https://en.wikipedia.org/wiki/Command-line_interface#Arguments Regards, Shlomi > Thanks -- ----- Shlomi Fish http://www.shlomifish.org/ “So, who the hell is Qoheleth?” - http://shlom.in/qoheleth Right to bear arms? In Soviet Russia, we have right to whole bear. — http://is.gd/EU4puV Please reply to list if it's a mailing list post - http://shlom.in/reply .
Re: Processing a big file using more CPUs
On Mon, 11 Feb 2019 23:54:43 +0100 Ole Tange wrote: > On Mon, Feb 4, 2019 at 10:19 PM Nio Wiklund wrote: > : > >cat bigfile | parallel --pipe --recend '' -k gzip -9 > bigfile.gz > : > > The reason why I want this is that I often create compressed images of > > the content of a drive, /dev/sdx, and I lose approximately half the > > compression improvement from gzip to xz, when using parallel. The > > improvement in speed is good, 2.5 times, but I think larger blocks would > > give xz a chance to get a compression much closer to what it can get > > without parallel. > > > > Is it possible with with the current code? In that case how? > > Since version 2016-07-22: > > parallel --pipepart -a bigfile --recend '' -k --block -1 xz > bigfile.xz > parallel --pipepart -a /dev/sdx --recend '' -k --block -1 xz > bigfile.xz > > Unfortunately the size computation of block devices only works under > GNU/Linux. > > (That said: pxz exists, and it may be more relevant to use here). > Hi Ole! https://jnovy.fedorapeople.org/pxz/node2.html - I see, but note that xz has a -T flag now as well - https://linux.die.net/man/1/xz . > > /Ole > -- - Shlomi Fish http://www.shlomifish.org/ NSA Factoids - http://www.shlomifish.org/humour/bits/facts/NSA/ Chuck Norris’ ciphers were once broken. He responded by breaking those individuals. (By sevvie: http://sevvie.github.io/ .) — http://www.shlomifish.org/humour/bits/facts/Chuck-Norris/ Please reply to list if it's a mailing list post - http://shlom.in/reply .
Can the automated tests be included in the source tarball?
Hi! Can the automated tests be included in the source tarball? It does not include them currently which makes assuring that it works properly harder. -- Shlomi Fish https://www.shlomifish.org/ Parody of "The Fountainhead" - https://shlom.in/towtf A positive attitude may not solve all your problems, but it will annoy enough people to make it worth the effort. — Herm Albright (via On Gossamer Wings) Please reply to list if it's a mailing list post - https://shlom.in/reply .
Re: Can the automated tests be included in the source tarball?
Hi Ole! Thanks for the reply. On Tue, 29 Sep 2020 01:18:31 +0200 Ole Tange wrote: > On Thu, Sep 24, 2020 at 8:37 AM Shlomi Fish wrote: > > > > Can the automated tests be included in the source tarball? It does not > > include them currently which makes assuring that it works properly harder. > > They are available in git. > > I am not sure they would make sense to normal users, thus I am not > convinced they should go into the tarball. > > Unfortunately some of them are heavily dependent on running on the > same equipment as mine due to timing issues (and every time I change > hardware or upgrade software, I have to tweak the tests or results a > little). > I see. I suppose you are aware of running on a pristine CI system, e.g: https://github.com/shlomif/Freenode-programming-channel-FAQ/blob/master/FAQ_with_ToC__generated.md#what-do-continuous-integration-ci-services-such-as-travis-ci-jenkins-or-appveyor-provide > But it would be a good idea to identify those that are not dependent > on this and make it clear that these should be able to run on all > systems. And if possible change more tests to be independent. Yes. > > If you want to help, a good start would be to have > testsuite/REQUIREMENTS ported so it would work on more systems. It > (tries to) install and set up all required software to run the > testsuite. Wow, it looks intimidating. And I thought some of my FOSS projects had too many prereqs! Of course, some users will still complain even if there is a moderate number of deps, which are even explicitly specified in the CI configuration (e.g.: https://github.com/shlomif/fortune-mod/issues/44 ). Anyway, one approach I used to avoid failing tests with certain configurations/missing-deps is to specify "tags" / "labels" in an environment variable and skip the test scripts (or individual assertions) if they are set or unset, e.g in: https://github.com/shlomif/fc-solve/blob/17426fe265a348fe1cd1df65d7fb1e1602711c86/fc-solve/source/t/t/fc-solve-output-Test-Trap.t#L207 (short url: https://is.gd/wyDrck ) > > GNU Parallel is pretty well tested on many systems. The testsuite is > not - not even close, so expect it to puke errors all over the place. > > > /Ole > Thanks! -- Shlomi Fish https://www.shlomifish.org/ Interview with Ben Collins-Sussman - https://shlom.in/sussman I’m trying to achieve world peace and this regex is the last thing standing in my way! ;) Please reply to list if it's a mailing list post - https://shlom.in/reply .
Re: Can the automated tests be included in the source tarball?
Hi Ole! While trying to get the test suite (or its "Portable.sh" subset) to pass on Fedora 32 x86-64, I ran into these grammar errors: https://www.shlomifish.org/Files/files/code/parallel-mak-grammar-v0.1.0.patch Patch is under CC0 / MITL/ExpatL - I didn't see an obvious way to submit patches or merge reqs on https://savannah.gnu.org/bugs/?group=parallel=additem . Otherwise note that what I tried ran "sudo" and asked for my password. :( > [snipped] -- Shlomi Fish https://www.shlomifish.org/ Parody of "The Fountainhead" - https://shlom.in/towtf Mastering ‘cat’ is almost as difficult as herding cats. — https://www.shlomifish.org/humour/bits/Mastering-Cat/ Please reply to list if it's a mailing list post - https://shlom.in/reply . diff --git a/testsuite/Makefile b/testsuite/Makefile index 43572e7..847329e 100644 --- a/testsuite/Makefile +++ b/testsuite/Makefile @@ -7,7 +7,7 @@ testsuite: 3 touch ~/.parallel/will-cite make stopvm -### Limited testsets - run once +### Limited test sets - run once # No 100s, 300s, mem, polarhome, tutorial 1: ../src/parallel tests-to-run/* wanted-results/* prereqlocal startdb prereqremote TRIES=1 time bash Start.sh '' '00s|mem|polarhome|tutorial' || true @@ -58,7 +58,7 @@ testlocalshort: ../src/parallel tests-to-run/*local* wanted-results/*local* prer testdb: ../src/parallel tests-to-run/*sql* wanted-results/*sql* prereqdb time bash Start.sh sql NONE -# Portable tests that runs everywhere +# Portable tests that run everywhere portable: # Cache sudo password sudo echo
Re: Inserting Parallel argument into command script
Hi Miriam,
Please reply to list:
https://www.shlomifish.org/philosophy/computers/netiquette/email/reply-to-list.html
On Fri, 19 Jun 2020 11:49:22 +1200
Miriam Dixon wrote:
> Hi,
>
> I am very new to Parallel, Linux, and computing in general but I need to
> use it for my Masters Thesis, so I am doing my best to learn quickly.
>
Welcome aboard!
> For my project, I am running a simulation using GATE, and I have a shell
> script to run the simulation for each of four different projection angles.
> I want to run each projection angle simulation on a different core, to
> speed things along and use the computer memory efficiently. I tried to do
> this by pulling projAngles out of my script and calling it as a parallel
> argument.
> parallel runSim.sh --projAngles {1} ::: 0 45 90 135
>
> This seemed to work, in that it started my runSim.sh, but it didn't use the
> value from projAngles within the script,
> time=$(bc -l <<< "$projAngles*$timePerProjection")
>
As far as I know "--projAngles" in the shell script's command line arguments
does not automatically propagate to a shell variable. E.g:
```
[shlomif@telaviv1 ~]$ cat miriam-test.bash
#! /bin/bash
#
# miriam-test.bash
# Copyright (C) 2020 Shlomi Fish
#
# Distributed under the terms of the MIT license.
#
echo "argv = < $@ >"
echo "projAngles = < $projAngles >"
[shlomif@telaviv1 ~]$ bash miriam-test.bash --projAngles 500
argv = < --projAngles 500 >
projAngles = < >
[shlomif@telaviv1 ~]$ projAngles=500 bash miriam-test.bash
argv = < >
projAngles = < 500 >
[shlomif@telaviv1 ~]$
```
#bash on freenode has these links in its topic:
Topic for #bash is: http://mywiki.wooledge.org/BashFAQ
| http://mywiki.wooledge.org/BashGuide | http://gnu.org/s/bash/manual |
http://wiki.bash-hackers.org/ | http://mywiki.wooledge.org/Quotes |
http://www.shellcheck.net/ | https://lists.gnu.org/mailman/listinfo/help-bash |
https://git.savannah.gnu.org/cgit/bash.git?h=devel
Jun 19 10:32:21 *
Topic for #bash set by izabera (Wed Dec 25 11:08:20 2019) Jun 19 10:32:21
-ChanServ- [#bash] Welcome to #bash, we discuss the GNU bash shell. Be
explicit if your question is about another shell. Don't trust the Internet on
bash, learn from the resources in the topic instead. You must be identified to
send to the channel.
You may wish to peruse some of them as well as
https://github.com/shlomif/Freenode-programming-channel-FAQ/blob/master/FAQ_with_ToC__generated.md
and https://perl-begin.org/FAQs/freenode-perl/ .
> Is there a way in the script or in parallel to insert the particular
> projAngles value for that run, and use it to calculate the next step?
>
Either parse the cmd line arguments (using $1 / shift / $@ / etc.) or pass them
to the script using https://en.wikipedia.org/wiki/Environment_variable .
> Cheers,
> Miriam
--
Shlomi Fish https://www.shlomifish.org/
Star Trek: “We, the Living Dead” - https://shlom.in/st-wtld
Absolutely. Also: a German dog barks “wau” and a cat meows “miau”
Su-Shee: German animals are true German patriots.
Please reply to list if it's a mailing list post - https://shlom.in/reply .
Re: New bug
Hi Evan! On Fri, 29 May 2020 21:20:33 + edowni...@protonmail.com wrote: > I have not tried a newer version yet, but I will. > > This version (20161222) is what comes with Debian buster > (https://packages.debian.org/buster/parallel). I suppose I was just assuming > that Debian had the most up-to-date version of parallel (or at least close > enough). > > Obviously I was wrong after examining https://ftp.gnu.org/gnu/parallel/ > Debian stable tends to have some old releases, so I'm not surprised. As the upstream maintainer of several FOSS packages, I have mixed feelings about Debian because while their packagers often provide me with quality patches or pull-requests, they sometimes tend to delay packaging the new releases (even in debian sid/unstable) for various reasons. See for instance: * https://packages.debian.org/unstable/web/wml vs. https://github.com/thewml/website-meta-language/releases * https://packages.debian.org/source/sid/freecell-solver vs. https://pkgs.org/download/freecell-solver * https://packages.debian.org/sid/fortune-mod vs. https://github.com/shlomif/fortune-mod/tags Regards, Shlomi > ~ Evan > > ‐‐‐ Original Message ‐‐‐ > On Friday, May 29, 2020 5:02 PM, Hubert Kowalski wrote: > > > your version number is ANCIENT. have you tried with newesr and see if bug's > > still there? > >> Dnia 29 maj 2020 o 19:55 edowni...@protonmail.com napisał(a): > > > Hello, > >> > > I've discovered a potential bug with parallel. > > Details: > * The > >> > > version number: 20161222 > > * The bugid: Less than 1 byte free > > * > >> > > The command line being run: $ time parallel --memfree 50G --retries 10 > >> > > -a commands.txt 2> parallel_stderr.txt > parallel_stdout.txt > > * The > >> > > specific commands in "commands.txt" are calls to a Python 3 script > >> > > which uses the SHAP library to explain Keras deep learning models. > >> > > https://github.com/slundberg/shap > > * The files being read (put the > >> > > files on a webserver if they are big) > > * Unfortunately I cannot > >> > > release the data files or code as they're for a private project I'm > >> > > working on. If need-be, I can try to replicate this error using SHAP > >> > > on an open dataset using a toy Python 3 script I can release. > > > >> > > Please let me know what you think. > > Cheers, > Evan -- Shlomi Fish https://www.shlomifish.org/ Freecell Solver - https://fc-solve.shlomifish.org/ If you ask Taylor Swift "What is the square root of 18.678?", she will answer "Why the hell does it matter?". — https://www.shlomifish.org/humour/bits/facts/Taylor-Swift/ Please reply to list if it's a mailing list post - https://shlom.in/reply .
Re: Can the automated tests be included in the source tarball?
Hi Ole!
Please acknowledge you have received this message.
On Thu, 1 Oct 2020 20:22:46 +0300
Shlomi Fish wrote:
> Hi Ole!
>
> While trying to get the test suite (or its "Portable.sh" subset) to pass on
> Fedora 32 x86-64, I ran into these grammar errors:
>
> https://www.shlomifish.org/Files/files/code/parallel-mak-grammar-v0.1.0.patch
>
> Patch is under CC0 / MITL/ExpatL - I didn't see an obvious way to submit
> patches or merge reqs on
> https://savannah.gnu.org/bugs/?group=parallel=additem .
>
> Otherwise note that what I tried ran "sudo" and asked for my password. :(
>
> > [snipped]
>
The corrections in the patch have not been applied yet:
«
[shlomif@localhost parallel]$ git show | head
commit e1a62d362d869822941b14a7d00fb0c59bff3109
Author: Ole Tange
Date: Sun Nov 22 16:22:54 2020 +0100
Released as 20201122 ('Biden')
diff --git a/NEWS b/NEWS
index eb4ad0dd..f1f89e40 100644
--- a/NEWS
+++ b/NEWS
»
I've been deadlocked waiting for this (while working on other projects), and
it's been over two months.
Why wasn't it applied?
--
Shlomi Fish https://www.shlomifish.org/
https://www.shlomifish.org/humour/Summerschool-at-the-NSA/
I’d rather have natural stupidity instead of artificial intelligence.
— Unknown
Please reply to list if it's a mailing list post - https://shlom.in/reply .
Re: GNU Parallel 20211022 ('Sinclair') released
On Fri, 22 Oct 2021 22:51:17 +0200
Ole Tange wrote:
> GNU Parallel 20211022 ('Sinclair') has been released. It is available
> for download at: lbry://@GnuParallel:4
>
> Quote of the month:
>
> GNU parallel is a severely underrated tool for just spawning a
> shitload of tasks. I use it every day and fucking love it.
> -- Jane @serialexpjane@twitter
>
> New in this release:
>
wrt the tagline, I thought it was about
https://en.wikipedia.org/wiki/Clive_Sinclair , but it may be about
https://www.msn.com/en-us/news/us/sinclair-broadcast-group-hit-by-ransomware-attack/ar-AAPFfTJ
.
--
Shlomi Fish https://www.shlomifish.org/
Summer Glau Facts - https://shlom.in/sglau-facts
Gödel’s Incompleteness Theorem is about to be replaced by the
[Clarissa] Darling “Like, Totally!” Completeness Theorem.
— https://www.shlomifish.org/humour/bits/facts/Clarissa/
Please reply to list if it's a mailing list post - https://shlom.in/reply .
Re: I think its not reading the number of cores right
Hi,
On Sun, 20 Aug 2023 20:42:53 -0400
Saint Michael wrote:
> My command line is
> parallel --gnu --no-run-if-empty -k --lb -j-5 --colsep ' ' processx "{1}
> {#}" ${FILE}
> and the file has 100.000 lines
> my box has 44 cores, minus 5, I should see 39 processes, but using ps ax |
> grep bash, I see only 10.
> How do I know how many cores parallel is seeing?
try editing parallel's Perl code and write the number of cores to a file. e.g:
{
open my $fh, ">", "$ENV{HOME}/par.cores.txt" or die;
print {$fh} "$num_cores\n";
close $fh;
}
https://blog.codinghorror.com/learn-to-read-the-source-luke/ .
--
Shlomi Fish https://www.shlomifish.org/
https://is.gd/MQHVF3 - The Atom Text Editor edits a 2,000,001B file
preaction: every mighty Klingon warrior uses Vim.
only green-blooded Romulan scum use Emacs
— #perl, https://is.gd/grdEqT
Please reply to list if it's a mailing list post - https://shlom.in/reply .
