yes, it does tree-to-tree, string-to-tree, tree-to-string. I also think
it does SAMT. The basic stuff are there for people to play with & extend
the filter script
filter-model-given-input.pl -Hierarchical
works with the new file format when u use the -Hierarchical argument
On 22/05/2010 11:24, dongxinghua0213 wrote:
thank you for your reply, the problem is solved. there are many
translation models on tree-string , string-based ,or tree-tree, on
which moses implemented is based ? Could you give me some details,or
some papersto refer to ?
another problem,
When I train the hierarchical phrase-based model using the following
command:
tools/moses-scripts/scripts-20100517-1641/training/train-model.perl
-scripts-root-dir tools/moses-scripts/scripts-20100517-1641 -root-dir
hier-dir -corpus hier-dir/corpus/raw -f ch -e uy -alignment
grow-diag-final-and -parallel -hierarchical -glue-grammar
-max-phrase-length 5 -lm
0:3:/home/top/programming/language_model/hier-dir/lm/raw.uy.lm:0
errors occurred,seeming this:
......
(6.2) creating table half hier-dir/model/rule-table.half.f2e @ Thu May
20 17:08:43 CST 2010
tools/moses-scripts/scripts-20100517-1641/training/phrase-extract/score hier-dir/model/extract.sorted
hier-dir/model/lex.f2e hier-dir/model/rule-table.half.f2e --Hierarchical
Executing:
tools/moses-scripts/scripts-20100517-1641/training/phrase-extract/score hier-dir/model/extract.sorted
hier-dir/model/lex.f2e hier-dir/model/rule-table.half.f2e --Hierarchical
Score v2.0 written by Philipp Koehn
scoring methods for extracted rules
processing hierarchical rules
Loading lexical translation table from hier-dir/model/lex.f2e.
.score: score.cpp:454: void
outputPhrasePair(std::vector<PhraseAlignment*,
std::allocator<PhraseAlignment*> >&, float): Assertion
`bestAlignment->alignedToT[ j ].size() == 1' failed.
ERROR: Execution of:
tools/moses-scripts/scripts-20100517-1641/training/phrase-extract/score hier-dir/model/extract.sorted
hier-dir/model/lex.f2e hier-dir/model/rule-table.half.f2e --Hierarchical
died with signal 6, without coredump
I have no idear about it , but when I used the clean script to filter
my corpus ,it made sentence length shorter than 15,it worked! Are the
corpus too big to fit into the RAM of the machine,or the problem my
corpus itself ? is there another script to filter it ?
thank you again ! best wishes!
??2010-05-22 09:58:13??"Hieu Hoang" <[email protected]
<mailto:[email protected]>> ??????
you should binarise your phrase table, otherwise it takes up too
much memory. This is described at the bottom of this page
http://www.statmt.org/moses/?n=Moses.SyntaxTutorial
eg.
~/CreateOnDiskPt 1 1 5 100 0 rule-table.gz rules.bin
then change the ini file so the rules are read from the binary
phrase table
[ttable-file]
2 0 0 5
/home/top/programming/language_model/working-dir/model/rules.bin
6 0 0 1
/home/top/programming/language_model/working-dir/model/glue-grammar
On 22/05/2010 02:47, dongxinghua0213 wrote:
the contents in moses.ini are as follows,but we can successed
when decoding in sample-models !
#########################
### MOSES CONFIG FILE ###
#########################
# input factors
[input-factors]
0
# mapping steps
[mapping]
0 T 0
1 T 1
# translation tables: source-factors, target-factors, number of
scores, file
[ttable-file]
6 0 0 5
/home/top/programming/language_model/working-dir/model/rule-table.gz
6 0 0 1
/home/top/programming/language_model/working-dir/model/glue-grammar
# no generation models, no generation-file section
# language models: type(srilm/irstlm), factors, order, file
[lmodel-file]
0 0 3 /home/top/programming/language_model/working-dir/lm/raw.ch.lm
# limit on how many phrase translations e for each phrase f are
loaded
# 0 = all elements loaded
[ttable-limit]
20
# language model weights
[weight-l]
0.5000
# translation model weights
[weight-t]
0.2
0.2
0.2
0.2
0.2
1.0
# no generation models, no weight-generation section
# word penalty
[weight-w]
-1
[cube-pruning-pop-limit]
1000
[glue-rule-type]
0
[non-terminals]
X
[search-algorithm]
3
[inputtype]
3
[max-chart-span]
20
1000
??2010-05-21 23:28:48??"Hieu Hoang" <[email protected]> ??????
can you send me your ini file, a few line from each of your
phrase tables and a few lines from you input. I'll try &
figure out what the problem is
2010/5/21 dongxinghua0213 <[email protected]
<mailto:[email protected]>>
when I use the following command to decoder, an error
occurred !
moses1/moses-chart-cmd/src/moses_chart -config
working-dir/tuning/filtered/moses.ini -input-file
working-dir/tuning/input> hier-dir/tuning/output
Defined parameters (per moses.ini or switch):
config: working-dir/tuning/filtered/moses.ini
cube-pruning-pop-limit: 1000
glue-rule-type: 0
input-factors: 0
input-file: working-dir/tuning/input
inputtype: 3
lmodel-file: 0 0 3
/home/top/programming/language_model/working-dir/lm/raw.ch.lm
mapping: 0 T 0 1 T 1
max-chart-span: 20 1000
non-terminals: X
search-algorithm: 3
ttable-file: 6 0 0 5
/home/top/programming/language_model/working-dir/tuning/filtered/phrase-table.0-0.1.1
6 0 0 1
/home/top/programming/language_model/working-dir/tuning/filtered/phrase-table.0-0.2.1
ttable-limit: 20
weight-l: 0.5000
weight-t: 0.2 0.2 0.2 0.2 0.2 1.0
weight-w: -1
Loading lexical distortion models...have 0 models
Start loading LanguageModel
/home/top/programming/language_model/working-dir/lm/raw.ch.lm
: [0.000] seconds
Finished loading LanguageModels : [0.000] seconds
Using uniform ttable-limit of 20 for all translation tables.
Start loading PhraseTable
/home/top/programming/language_model/working-dir/tuning/filtered/phrase-table.0-0.1.1
: [0.000] seconds
filePath:
/home/top/programming/language_model/working-dir/tuning/filtered/phrase-table.0-0.1.1
Start loading new format pt model : [0.000] seconds
moses_chart: Phrase.cpp:246: void
Moses::Phrase::CreateFromStringNewFormat(Moses::FactorDirection,
const std::vector<unsigned int, std::allocator<unsigned
int> >&, const std::string&, const std::string&,
Moses::Word&): Assertion `nextPos != string::npos' failed.
thank you for your attention!
------------------------------------------------------------------------
??????????????????????????????????????????
<http://ym.163.com/?from=od3>
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
------------------------------------------------------------------------
??????????????????????????????????????????
<http://ym.163.com/?from=od3>
------------------------------------------------------------------------
?????????????????????????????????????????? <http://ym.163.com/?from=od3>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
