Fwd: Multiple sqlworkers

[Thomas Bereknyei]

[forgot to cc the mailing list]
Ole,

Running your example with sqlite gives me the same result. I suspect
sqlite enforces a bit more serialization when it comes to the results,
while something like postgres allows multiple queries in flight. This
example gives me numbers like 187, 201, 193, 190

seq 100 | parallel --sqlmaster postgresql:///tom/tabl 'sleep 1; echo {}'
(
parallel -uj30 --sqlworker postgresql:///tom/tabl &
parallel -uj30 --sqlworker postgresql:///tom/tabl &
parallel -uj30 --sqlworker postgresql:///tom/tabl &
parallel -uj30 --sqlworker postgresql:///tom/tabl & wait ) | wc -l

So perhaps with postgres the queries don't do quite the same thing? Or
its "too fast" and the other queries in flight are given the same row?


On Sat, Jun 6, 2020 at 4:03 PM Ole Tange  wrote:
>
> On Mon, Apr 6, 2020 at 5:22 AM Thomas Bereknyei  wrote:
>
> > Running some experiments with the sqlmaster/sqlworker system shows
> > that running multiple independent workers results in many jobs being
> > done multiple times. This is noted in the manual as
> >
> > "If you have more than one --sqlworker jobs may be run more than once."
> >
> > In simple testing, this is not an occasional thing, but results in
> > nearly all jobs being performed N times.
>
> So I tested:
>
> seq 100 | parallel --sqlmaster sqlite3:///fisk/fisk 'sleep 4;echo {}'
> (
> parallel -uj30 --sqlworker sqlite3:///fisk/fisk &
> parallel -uj30 --sqlworker sqlite3:///fisk/fisk &
> parallel -uj30 --sqlworker sqlite3:///fisk/fisk &
> parallel -uj30 --sqlworker sqlite3:///fisk/fisk &
> wait
> )  | wc -l
>
> This gives 100..103 on my machine. Thus 0..3 jobs were run twice.
>
> So I cannot reproduce the error you see.
>
> That is not to say that you have not found a bug, but I cannot
> reproduce it on my system.
>
>
> /Ole

Re: New bug

[Shlomi Fish]

Hi Evan!

On Fri, 29 May 2020 21:20:33 +
edowni...@protonmail.com wrote:

> I have not tried a newer version yet, but I will.
> 
> This version (20161222) is what comes with Debian buster
> (https://packages.debian.org/buster/parallel). I suppose I was just assuming
> that Debian had the most up-to-date version of parallel (or at least close
> enough).
> 
> Obviously I was wrong after examining https://ftp.gnu.org/gnu/parallel/
> 

Debian stable tends to have some old releases, so I'm not surprised. As the
upstream maintainer of several FOSS packages, I have mixed feelings about
Debian because while their packagers often provide me with quality patches or
pull-requests, they sometimes tend to delay packaging the new releases (even in
debian sid/unstable) for various reasons. See for instance:

* https://packages.debian.org/unstable/web/wml vs.
https://github.com/thewml/website-meta-language/releases

* https://packages.debian.org/source/sid/freecell-solver vs.
https://pkgs.org/download/freecell-solver

* https://packages.debian.org/sid/fortune-mod vs.
https://github.com/shlomif/fortune-mod/tags

Regards,

Shlomi

> ~ Evan
> 
> ‐‐‐ Original Message ‐‐‐
> On Friday, May 29, 2020 5:02 PM, Hubert Kowalski  wrote:
> 
> > your version number is ANCIENT. have you tried with newesr and see if bug's
> > still there?  
> >> Dnia 29 maj 2020 o 19:55 edowni...@protonmail.com napisał(a): > > > Hello,
> >> > > I've discovered a potential bug with parallel. > > Details: > * The
> >> > > version number: 20161222 > > * The bugid: Less than 1 byte free > > *
> >> > > The command line being run: $ time parallel --memfree 50G --retries 10
> >> > > -a commands.txt 2> parallel_stderr.txt > parallel_stdout.txt > > * The
> >> > > specific commands in "commands.txt" are calls to a Python 3 script
> >> > > which uses the SHAP library to explain Keras deep learning models.
> >> > > https://github.com/slundberg/shap > > * The files being read (put the
> >> > > files on a webserver if they are big) > > * Unfortunately I cannot
> >> > > release the data files or code as they're for a private project I'm
> >> > > working on. If need-be, I can try to replicate this error using SHAP
> >> > > on an open dataset using a toy Python 3 script I can release. > >
> >> > > Please let me know what you think. > > Cheers, > Evan   



-- 

Shlomi Fish   https://www.shlomifish.org/
Freecell Solver - https://fc-solve.shlomifish.org/

If you ask Taylor Swift "What is the square root of 18.678?", she will answer
"Why the hell does it matter?".
— https://www.shlomifish.org/humour/bits/facts/Taylor-Swift/

Please reply to list if it's a mailing list post - https://shlom.in/reply .