Here is more concrete example:
$ wc -l data.txt
1000000 data.txt
$ cat data.txt | awk -v path=dataSerial '{print $0 > (path "/" $1 ".txt")}'
$ cat dataSerial/*txt | wc -l
1000000
$ cat data.txt | parallel --pipe -- awk -v path=dataParallel -f script.awk
edd@max:/tmp/parallel$ cat dataParallel/*txt | wc -l
50040
edd@max:/tmp/parallel$
edd@max:/tmp/parallel$ cat script.awk
{
print $0 > (path "/" $1 ".txt")
}
edd@max:/tmp/parallel$
Now, am I being silly and just falling into the trap of a race condition
where my data is being lost when written?
Dirk
