On Fri, May 9, 2014 at 11:44 AM, V A Ramesh <varavind...@yahoo.com> wrote:
> I am running a job like this
>
> parallel --eta -S .. --trc {.}.out < Testing_Para
>
> Testing_para:
:
> svm-train -c 0.125 -g 0.015625 -v 10 -m 2000 Combined_Full.scaled >&
> Combined_Full.scaled_0.125_0.015625.out
> svm-train -c 0.125 -g 0.03125 -v 10 -m 2000 Combined_Full.scaled >&
> Combined_Full.scaled_0.125_0.03125.out
:
> svm-train -c 0.25 -g 0.00048828125 -v 10 -m 2000 Combined_Full.scaled >&
> Combined_Full.scaled_0.25_0.00048828125.out
> svm-train -c 0.25 -g 0.0009765625 -v 10 -m 2000 Combined_Full.scaled >&
> Combined_Full.scaled_0.25_0.0009765625.out
> Problem 1:
>
> I want '.out' files after completion of the job. I realized that --trc
> {.}.out is not the appropriate way to use and I manually retrieved the
> results back from the remote computer.
When you pass the whole command as argument {.} will contain the whole
command. That is clearly not what you want.
It seems you want to run 'svm-train' on different values and want an
output file to depend on this. To simplify your command let us assume
you want the output from:
svm-train a 1
svm-train a 2
svm-train a 3
svm-train a 4
svm-train a 5
svm-train a 6
svm-train a 7
svm-train b 1
svm-train b 2
svm-train b 3
You can the generate the commands using GNU Parallel:
parallel --xapply svm-train {1} {2} ::: a a a a a a a b b b ::: 1
2 3 4 5 6 7 1 2 3
If you would rather have the 'a a a a a a a b b b' and '1 2 3 4 5 6 7
1 2 3' in files, use :::: possibly with --colsep.
Now you want the output put in files depending on the arguments:
parallel --xapply svm-train {1} {2} ">&" out.{1}.{2} ::: a a a a a
a a b b b ::: 1 2 3 4 5 6 7 1 2 3
To return these back and remove them from the remote site you need
--return --cleanup:
parallel --return out.{1}.{2} --cleanup --xapply svm-train {1} {2}
">&" out.{1}.{2} ::: a a a a a a a b b b ::: 1 2 3 4 5 6 7 1 2 3
But if you are simply redirecting output into a file, why not use
--results which is made for this kind of problem:
parallel --results outdir --xapply svm-train {1} {2} ::: a a a a a
a a b b b ::: 1 2 3 4 5 6 7 1 2 3 >/dev/null
Now the output is neatly stored in a structured way in 'outdir/*/*/*/*'
> Problem 2:
>
> I also had to transfer input files from host to remote computer manually.
> How to solve this two problems.
If the argument is the inputfile, then you use --transfer. I do not
see an inputfile in your example.
If the inputfile is more like a read-only database that does not
change, but remains the same for every job, use --basefile.
> Vakkalagadda A Ramesh
> Jr. Research Fellow
> Lab of Computational Biology
> Centre for DNA Fingerprinting and Diagnostics (CDFD)
I am hoping your research results in published articles.
/Ole
