good to know it's working.
you might want to try out the new feature caused you trouble:
Time taken
Peak disk usage (GB) argument to train-model.perl
Baseline
02:56:05
61.2
New
02:51:00
36.5
+4 cores
02:30:09
37.1 -cores 4
+compress intermediate sort files
02:10:11
18.7 -cores 4 -sort-compress gzip
+optimized
01:37:00
18.7
-cores 4 -sort-compress gzip -sort-buffer-size 200M -sort-batch-size
253 -sort-parallel 4
On 26/05/2012 18:12, Tomas Hudik wrote:
Hi There,
Finally I got it. Basically, there is no bug J
The problem was the older (svn) moses version didn't require absolute
path for train-model.perl for parameter "--corpus". When I put the
absolute path to the new moses version -- the path was correctly hand
out to extract-parallel and then to ln.
The conclusion is -- *if a problem with path arises always try to put
absolute paths to all possible parameters* J
Cheers, Tomas
*From:*Tomas Hudik [mailto:[email protected]]
*Sent:* Saturday, May 26, 2012 5:27 PM
*To:* Hieu Hoang; [email protected]
*Subject:* Re: [Moses-support] new PhraseExtractor -- segmentation
faul; the old one run smooth
Hi guys,
Phil -- thanks.
Hieu -- you are correct (dir is created). The problem is in command ln
in extract_parallel (lines: 70 and 2 more below 70).
$cmd = "ln -s $target $TMPDIR/target.$numStr";
ln creates symbolic link (no error message is generated), however, the
link is invalid. It is due to variable $target which contains only
file name (no path). But ln for the 1st argument requires a path.
e.g.:
ln -s lotus.tok.rem.clean.lw.cs ./model/tmp.23799/target.00003
model/tmp.23799/target.00003 is created but it points to
lotus.tok.rem.clean.lw.cs instead of ../../lotus.tok.rem.clean.lw.cs,
therefore path is needed:
ln -s /home/mee/lotus.tok.rem.clean.lw.cs ./model/tmp.23799/target.00003
creates valid link.
extract_parallel needs to be fixed (to work with the path), or
train-model.perl needs to provide full path
Cheers, Tomas
Ps Just for curiosity: for what reason are the links created?
Wouldn't be better to work with original files?
*From:*Hieu Hoang [mailto:[email protected]]
<mailto:[mailto:[email protected]]>
*Sent:* Saturday, May 26, 2012 2:50 PM
*To:* [email protected] <mailto:[email protected]>; Tomas Hudik
*Subject:* Re: [Moses-support] new PhraseExtractor -- segmentation
faul; the old one run smooth
hi tomas
1. thanks. Phil fixed it
https://github.com/moses-smt/mosesdecoder/commit/82580280bc0b30607b00a55ffe0f22d5665269a3
2. The temp directory is created in
extract-parallel.perl line 34
Do you get an error when running the extract or scoring?
On 26/05/2012 12:33, Tomas Hudik wrote:
Hi Hieu,
Thanks for your reply. Yep - I downloaded latest git version
(de8a2e7667fe2bde9df0ef5a32b3b85b6469eb0f) and found out following
problems:
1.During compilation; gcc version 4.6.3. produces error in:
gcc.compile.c++
scripts/training/phrase-extract/pcfg-common/bin/gcc-4.6.3/release/debug-symbols-on/link-static/threading-multi/xml_tree_parser.o
In file included from
scripts/training/phrase-extract/pcfg-common/pcfg_tree.h:24:0,
from
scripts/training/phrase-extract/pcfg-common/xml_tree_parser.h:24,
from
scripts/training/phrase-extract/pcfg-common/xml_tree_parser.cc:20:
scripts/training/phrase-extract/pcfg-common/syntax_tree.h: In
destructor 'virtual Moses::PCFG::SyntaxTreeBase<T,
DerivedType>::~SyntaxTreeBase()':
scripts/training/phrase-extract/pcfg-common/syntax_tree.h:83:8: error:
'size_t' was not declared in this scope
scripts/training/phrase-extract/pcfg-common/syntax_tree.h:83:8: note:
suggested alternatives:
/usr/lib/gcc/x86_64-redhat-linux/4.6.3/../../../../include/c++/4.6.3/x86_64-redhat-linux/bits/c++config.h:1737:26:
note: 'std::size_t'
/usr/lib/gcc/x86_64-redhat-linux/4.6.3/../../../../include/c++/4.6.3/x86_64-redhat-linux/bits/c++config.h:1737:26:
note: 'std::size_t'
scripts/training/phrase-extract/pcfg-common/syntax_tree.h:83:15:
error: expected ';' before 'i'
scripts/training/phrase-extract/pcfg-common/syntax_tree.h:83:22:
error: 'i' was not declared in this scope
It means, size_t should be replaced by std::size_t. After this,
compilation works fine. I would do this but I'm not with familiar with
git ...
2. during training(extract phrases ;step 5) there is a command:
/home/mee/moses/scripts//generic/extract-parallel.perl 1 split
"sort "
/home/mee/new/moses/scripts//training/phrase-extract/extract
lotus.tok.rem.clean.lw.cs lotus.tok.rem.clean.lw.en
./model/aligned.grow-diag-final-and ./model/extract 7 orientation
--model wbe-msd --GZOutput
Which executes commands like:
ln -s lotus.tok.rem.clean.lw.en
/home/mee/lotus/train/model/tmp.22477/source.00000
ln: failed to create symbolic link
`/home/mee/lotus/train/model/tmp.22477/source.00000': No such file or
directory
The problem is that directory tmp.22477 is not created yet.
Presumable, some mkdir command should predecease symbolic link creation.
Are my assumptions correct?
Thx, Tomas
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
