Re: GNU Parallel 20240122 ('Frederik X') released

[Saint Michael]

If you export HOME="/tmp", then parallel always asks for a citation.
How do stop the citation from the command line?

On Sat, Jan 27, 2024 at 12:32 AM Ole Tange  wrote:
>
> On Thu, Jan 25, 2024 at 3:27 PM Saint Michael  wrote:
> >
> > I found a bug.
> > this script generates an error
> > 
> > #!/bin/bash
> > export HOME="/tmp"
> > process2(){
> > x="$1"
> > job="$2"
> > echo "$job, $x"
> >
> > }
> > export -f process2
> > seq 2 11|parallel --gnu --no-run-if-empty -k --lb -j+0 --colsep ' '
> > process2 "{1} {#}" 2>&1
> > 
> > but if you change
> > export HOME="/root"
> >
>
> I get no error when running that.
>
> This means your problem depends on your environment.
>
> Please follow 
> https://www.gnu.org/software/parallel/man.html#bug-dependent-on-environment
>
> /Ole
>

Fwd: GNU Parallel 20240122 ('Frederik X') released

[Saint Michael]

I found a bug.
this script generates an error

#!/bin/bash
export HOME="/tmp"
process2(){
x="$1"
job="$2"
echo "$job, $x"

}
export -f process2
seq 2 11|parallel --gnu --no-run-if-empty -k --lb -j+0 --colsep ' '
process2 "{1} {#}" 2>&1

but if you change
export HOME="/root"

then it works normally
The problem is that it defaults to /tmp. I spent two days to understand this.
It should work fine with /tmp as HOME




On Wed, Jan 24, 2024 at 1:58 PM Ole Tange  wrote:
>
> GNU Parallel 20240122 ('Frederik X') has been released. It is
> available for download at: lbry://@GnuParallel:4
>
> Quote of the month:
>
>   GNU Parallel alone provides more value than moreutils
> -- ferret7...@news.ycombinator.com
>
> New in this release:
>
> * --sshlogin supports ranges: server[01-12,15] 10.0.[1-10].[2-254]
> * --plus enables {slot-1} and {seq-1} = {%}-1 and {#}-1 to count from 0.
> * env_parallel.{sh,ash,dash,bash,ksh,zsh} are now the same script.
> * Bug fixes and man page updates.
>
> GNU Parallel - For people who live life in the parallel lane.
>
> If you like GNU Parallel record a video testimonial: Say who you are,
> what you use GNU Parallel for, how it helps you, and what you like
> most about it. Include a command that uses GNU Parallel if you feel
> like it.
>
>
> = About GNU Parallel =
>
> GNU Parallel is a shell tool for executing jobs in parallel using one
> or more computers. A job can be a single command or a small script
> that has to be run for each of the lines in the input. The typical
> input is a list of files, a list of hosts, a list of users, a list of
> URLs, or a list of tables. A job can also be a command that reads from
> a pipe. GNU Parallel can then split the input and pipe it into
> commands in parallel.
>
> If you use xargs and tee today you will find GNU Parallel very easy to
> use as GNU Parallel is written to have the same options as xargs. If
> you write loops in shell, you will find GNU Parallel may be able to
> replace most of the loops and make them run faster by running several
> jobs in parallel. GNU Parallel can even replace nested loops.
>
> GNU Parallel makes sure output from the commands is the same output as
> you would get had you run the commands sequentially. This makes it
> possible to use output from GNU Parallel as input for other programs.
>
> For example you can run this to convert all jpeg files into png and
> gif files and have a progress bar:
>
>   parallel --bar convert {1} {1.}.{2} ::: *.jpg ::: png gif
>
> Or you can generate big, medium, and small thumbnails of all jpeg
> files in sub dirs:
>
>   find . -name '*.jpg' |
> parallel convert -geometry {2} {1} {1//}/thumb{2}_{1/}  - ::: 50 100 
> 200
>
> You can find more about GNU Parallel at: http://www.gnu.org/s/parallel/
>
> You can install GNU Parallel in just 10 seconds with:
>
> $ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
>fetch -o - http://pi.dk/3 ) > install.sh
> $ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
> 12345678 883c667e 01eed62f 975ad28b 6d50e22a
> $ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
> cc21b4c9 43fd03e9 3ae1ae49 e28573c0
> $ sha512sum install.sh | grep ec113b49a54e705f86d51e784ebced224fdff3f52
> 79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
> fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
> $ bash install.sh
>
> Watch the intro video on 
> http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
>
> Walk through the tutorial (man parallel_tutorial). Your command line
> will love you for it.
>
> When using programs that use GNU Parallel to process data for
> publication please cite:
>
> O. Tange (2018): GNU Parallel 2018, March 2018,
> https://doi.org/10.5281/zenodo.1146014.
>
> If you like GNU Parallel:
>
> * Give a demo at your local user group/team/colleagues
> * Post the intro videos on Reddit/Diaspora*/forums/blogs/
> Identi.ca/Google+/Twitter/Facebook/Linkedin/mailing lists
> * Get the merchandise https://gnuparallel.threadless.com/designs/gnu-parallel
> * Request or write a review for your favourite blog or magazine
> * Request or build a package for your favourite distribution (if it is
> not already there)
> * Invite me for your next conference
>
> If you use programs that use GNU Parallel for research:
>
> * Please cite GNU Parallel in you publications (use --citation)
>
> If GNU Parallel saves you money:
>
> * (Have your company) donate to FSF https://my.fsf.org/donate/
>
>
> = About GNU SQL =
>
> GNU sql aims to give a simple, unified interface for accessing
> databases through all the different databases' command line clients.
> So far the focus has been on giving a common way to specify login
> information (protocol, username, password, hostname, and port number),
> size (database and table size), and running queries.
>
> The database is addressed using a DBURL. If commands are left out 

I think its not reading the number of cores right

[Saint Michael]

My command line is
parallel --gnu --no-run-if-empty -k --lb -j-5 --colsep ' '  processx "{1}
{#}"  ${FILE}
and the file has 100.000 lines
my box has 44 cores, minus 5, I should see 39 processes, but using ps ax |
grep bash, I see only 10.
How do I know how many cores parallel is seeing?

Bug found

[Saint Michael]

Line 62: awk -F, '{print $4}' goodcalls.csx
Line 62: parallel --gnu --no-run-if-empty -k --lb -j+0 --colsep ' '
process2 '{1} {#}'
parallel: This should not happen. You have found a bug. Please follow
https://www.gnu.org/software/parallel/man.html#REPORTING-BUGS

Include this in the report:
* The version number: 20220422
* The bugid: write_idfile: /root/.parallel/semaphores/id-mylock/id-mylock
* The command line being run
* The files being read (put the files on a webserver if they are big)

If you get the error on smaller/fewer files, please include those instead.
parallel: This should not happen. You have found a bug. Please follow
https://www.gnu.org/software/parallel/man.html#REPORTING-BUGS

Include this in the report:
* The version number: 20220422
* The bugid: write_idfile: /root/.parallel/semaphores/id-mylock/id-mylock
* The command line being run
* The files being read (put the files on a webserver if they are big)

If you get the error on smaller/fewer files, please include those instead.
parallel: This should not happen. You have found a bug. Please follow
https://www.gnu.org/software/parallel/man.html#REPORTING-BUGS

Include this in the report:
* The version number: 20220422
* The bugid: write_idfile: /root/.parallel/semaphores/id-mylock/id-mylock
* The command line being run
* The files being read (put the files on a webserver if they are big)

If you get the error on smaller/fewer files, please include those instead.
^CLine 67: exit
I have many threads writing in parallel to a single file:
sem --id mylock  grep -h "${callid}" goodcalls.csx >>
/usr/src/cdr/missing.csv

Re: processing csv files

[Saint Michael]

So how would I submit the contents of many files to parallel, without
concatenating them?
The function neds to process each file line by line.
I am sure there must be a better way. Why concatenate them at all? There is
no relationship between a line and the next line.
Maybe a new feature?


On Sun, Mar 6, 2022 at 4:19 PM Ole Tange  wrote:

> On Sat, Mar 5, 2022 at 2:46 AM Saint Michael  wrote:
> >
> > I have a bunch of *.csv files.
> > I need to process each line of the separately, so I do
> > function() { any process }
> > export -f function
> > cat *.csv | parallel --colsep ',' function  "{1} {2} {3} {4} {5} {6} {7}"
> >
> > The question is: is this the best possible way to do this?
>
> If the function can only read a single input per run, then yes.
>
> If the function can read more lines, look at --pipe.
>
> > I don't like to use "cat"
>
> cat is built for concatenating files (this is what the name comes from).
>
> /Ole
>

Question

[Saint Michael]

I have a case where I need the output to be written out to stdout as soon
as it's received, no ordering needed, just spit it out.
Is there an option to disable reordering or any manipulation of the output?

Question

[Saint Michael]

>
> I have a bunch of *.csv files.
I need to process each line of the separately, so I do
function() { any process }
export -f function
cat *.csv | parallel --colsep ',' function  "{1} {2} {3} {4} {5} {6} {7}"


The question is: is this the best possible way to do this? I don't like to
use "cat"

Re: VOTE: Changing the behaviour of --results

[Saint Michael]

[X] Change it

On Sat, Dec 19, 2020 at 12:29 PM  wrote:

> Am 19.12.2020 um 18:00 schrieb gnu.org :
> > [x] I don't care
>
>
>
>

Re: GNU Parallel 20190622 ('HongKong') released

[Saint Michael]

It's working now

On Sun, Jun 23, 2019 at 5:41 AM Ole Tange  wrote:

> On Sun, Jun 23, 2019 at 6:29 AM Saint Michael  wrote:
> >
> > it fails to install
>
> Thank for giving it a go.
>
> > The signature on parallel-20190622.tar.bz2 is wrong. This may indicate
> that a criminal has changed the code.
> > THIS IS BAD AND THE CODE WILL NOT BE INSTALLED.
>
> I cannot reproduce that. So this probably means that
> http://ftpmirror.gnu.org/parallel/ redirects you to a different server
> than I am being directed to.
>
> In theory this server _could_ be compromised (in which case the error
> is correct), but most likely there is another cause (maybe the
> mirroring failed half way - in which case it should fix itself within
> a day).
>
> Can you tell me which server http://ftpmirror.gnu.org/parallel/
> redirects you to? And do you still see the issue?
>
>
> /Ole
>

Install in 10 sec fails

[Saint Michael]

This does not work
You can install GNU Parallel in just 10 seconds with:
(wget -O -pi.dk/3 || curl pi.dk/3/) | bash

(wget -O -pi.dk/3 || curl pi.dk/3/) | bash  [ OK ]
wget: missing URL
Usage: wget [OPTION]... [URL]...
Try `wget --help' for more options.
  % Total% Received % Xferd  Average Speed   TimeTime Time
Current
 Dload  Upload   Total   SpentLeft
Speed
100  5201  100  52010 0   2931  0  0:00:01  0:00:01 --:--:--
2936
main: line 34: fetch: command not found
  % Total% Received % Xferd  Average Speed   TimeTime Time
Current
 Dload  Upload   Total   SpentLeft
Speed
  0 00 00 0  0  0 --:--:-- --:--:--
--:--:-- 0
100   185  100   1850 0364  0 --:--:-- --:--:-- --:--:--
364
100 215570 215570 0  34601  0 --:--:-- --:--:-- --:--:--
34601
  % Total% Received % Xferd  Average Speed   TimeTime Time
Current
 Dload  Upload   Total   SpentLeft
Speed
  0 00 00 0  0  0 --:--:-- --:--:--
--:--:-- 0
100 1755k  100 1755k0 0   670k  0  0:00:02  0:00:02 --:--:--
1018k
  % Total% Received % Xferd  Average Speed   TimeTime Time
Current
 Dload  Upload   Total   SpentLeft
Speed
  0 00 00 0  0  0 --:--:-- --:--:--
--:--:-- 0
100  2050  100  20500 0   3988  0 --:--:-- --:--:-- --:--:--
3988
gpg: connecting dirmngr at '/root/.gnupg/S.dirmngr' failed: IPC connect
call failed
gpg: keyserver receive failed: No dirmngr
gpg: keyserver receive failed: Connection refused
Cannot fetch keyID 0xFFF1, so the signature cannot be checked.
This means that if the code has been changed by criminals, you will not
discover that!
Continue anyway? (y/n)

Citation issue

[Saint Michael]

>
> I installed the latest version, parallel-20180922

but I keep getting this, in spite of having done the citation
"Come on: You have run parallel 32 times. Isn't it about time you run
'parallel --citation' once to silence the citation notice? "
Also I am using CentOS Linux release 7.5.1804 (Core), it has a package
called moreutils, that has Parallel. But how do I install the latest
version without having uninstall moreutils, which has other utilities that
I use often?
I tried and if I simply compile, make and make install Parallel, I am still
using the old version. I had to manually erase the old executable before
typing make install. Now I have

parallel --version
GNU parallel 20180922
Copyright (C) 2007-2018 Ole Tange and Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later 
This is free software: you are free to change and redistribute it.
GNU parallel comes with no warranty.

Web site: http://www.gnu.org/software/parallel

When using programs that use GNU Parallel to process data for publication
please cite as described in 'parallel --citation'.


Yours

Federico



>