Hi Barry,
sorry, my mistake ...
Here comes the experiment with the *same* string:
1. COMMAND-LINE:
% echo 'das ist ein haus' | moses -f string-to-tree/moses.ini
Defined parameters (per moses.ini or switch):
config: string-to-tree/moses.ini
cube-pruning-pop-limit: 1000
feature: KENLM name=LM factor=0 order=3 num-features=1
path=lm/europarl.srilm.gz WordPenalty UnknownWordPenalty
PhraseDictionaryMemory input-factor=0 output-factor=0
path=string-to-tree/rule-table num-features=1 table-limit=20
input-factors: 0
inputtype: 3
mapping: 0 T 0
max-chart-span: 20 1000
non-terminals: X S
search-algorithm: 3
translation-details: translation-details.log
weight: WordPenalty0= 0 LM= 0.5 PhraseDictionaryMemory0= 0.5
line=KENLM name=LM factor=0 order=3 num-features=1
path=lm/europarl.srilm.gz
Loading the LM will be faster if you build a binary file.
Reading lm/europarl.srilm.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
**The ARPA file is missing <unk>. Substituting log10 probability
-100.000.
**************************************************************************************************
FeatureFunction: LM start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 2 end: 2
line=PhraseDictionaryMemory input-factor=0 output-factor=0
path=string-to-tree/rule-table num-features=1 table-limit=20
FeatureFunction: PhraseDictionaryMemory0 start: 3 end: 3
Loading LM
Loading WordPenalty0
Loading UnknownWordPenalty0
Loading PhraseDictionaryMemory0
Start loading text phrase table. Moses format : [0.773] seconds
Reading string-to-tree/rule-table
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
max-chart-span: 20
Created input-output object : [0.773] seconds
Line 0: Initialize search took 0.000 seconds total
Translating: <s> das ist ein haus </s> ||| [0,0]=X (1) [0,1]=X
(1) [0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1)
[1,2]=X (1) [1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1)
[2,3]=X (1) [2,4]=X (1) [2,5]=X (1) [3,3]=X (1) [3,4]=X (1)
[3,5]=X (1) [4,4]=X (1) [4,5]=X (1) [5,5]=X (1)
0 1 2 3 4 5
0 3 2 2 1 0
0 0 0 2 0
0 0 4 0
0 8 0
0 0
1
BEST TRANSLATION: 31 TOP -> <s> S </s> :1-1 : term=: nonterm=1-1
: c=-3.206 core=(-6.413,-2.000,0.000,0.000) [0..5] 15
[total=-12.332] core=(-22.361,-6.000,0.000,-2.303)
this is a house
Line 0: Additional reporting took 0.000 seconds total
Line 0: Translation took 0.001 seconds total
Translation took 0.010 seconds
Name:moses VmPeak:79168 kB VmRSS:7432 kB RSSMax:35968 kB
user:0.758 sys:0.021 CPU:0.780 real:0.776
2.) SERVER:
% /home/maba/bin/moses --server --config string-to-tree/moses.ini
Defined parameters (per moses.ini or switch):
config: string-to-tree/moses.ini
cube-pruning-pop-limit: 1000
feature: KENLM name=LM factor=0 order=3 num-features=1
path=lm/europarl.srilm.gz WordPenalty UnknownWordPenalty
PhraseDictionaryMemory input-factor=0 output-factor=0
path=string-to-tree/rule-table num-features=1 table-limit=20
input-factors: 0
inputtype: 3
mapping: 0 T 0
max-chart-span: 20 1000
non-terminals: X S
search-algorithm: 3
server:
translation-details: translation-details.log
weight: WordPenalty0= 0 LM= 0.5 PhraseDictionaryMemory0= 0.5
line=KENLM name=LM factor=0 order=3 num-features=1
path=lm/europarl.srilm.gz
Loading the LM will be faster if you build a binary file.
Reading lm/europarl.srilm.gz
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
**The ARPA file is missing <unk>. Substituting log10 probability
-100.000.
**************************************************************************************************
FeatureFunction: LM start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 2 end: 2
line=PhraseDictionaryMemory input-factor=0 output-factor=0
path=string-to-tree/rule-table num-features=1 table-limit=20
FeatureFunction: PhraseDictionaryMemory0 start: 3 end: 3
Loading LM
Loading WordPenalty0
Loading UnknownWordPenalty0
Loading PhraseDictionaryMemory0
Start loading text phrase table. Moses format : [0.747] seconds
Reading string-to-tree/rule-table
----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
****************************************************************************************************
max-chart-span: 20
RUN SERVER at pid 0
[moses/server/Server.cpp:49] Listening on port 8080
[moses/server/TranslationRequest.cpp:315] Input: das ist ein haus
Translating: <s> das ist ein haus </s>
0 1 2 3 4 5
0 3 2 2 1 0
0 0 0 0 0
0 0 0 0
0 0 0
0 0
0
Translation took 0.000 seconds
=> XML-RPC result after calling (see moses server output just above) ...
% curl --data @rpc.xml '192.168.178.14:8080/RPC2
<http://192.168.178.14:8080/RPC2>'
<?xml version="1.0" encoding="UTF-8"?>
<methodResponse>
<params>
<param><value><struct>
<member><name>text</name>
<value><string></string></value></member>
</struct></value></param>
</params>
</methodResponse>
... with rpc.xml ...
<?xml version="1.0" encoding="UTF-8"?>
<methodCall>
<methodName>translate</methodName>
<params><param><value>
<struct>
<member>
<name>text</name>
<value>
<string>das ist ein haus</string>
</value>
</member>
</struct></value></param>
</params>
</methodCall>
Cheers,
Martin
Am 01.02.2016 um 14:16 schrieb Barry Haddow:
Hi Martin
In your original mail, you passed different inputs to the server and
command-line version. The following are extracted from your debug:
command-line:
Translating: <s> das ist ein haus </s>
server:
Translating: <s> dies ist ein haus . </s>
(Although you specify the command-line in the first case as "echo
'dies ist ein haus' | moses -f string-to-tree/moses.ini", which
contradicts the debug. I assume you spliced together different runs.)
Could you show an example where you pass the *same* string to the
two different Moses versions and get different outputs?
cheers - Barry
On 01/02/16 12:47, Martin Baumgärtner wrote:
With the current fix the crash is gone, but there is still an
unexpected difference between server mode and command line moses:
the first produces an empty string, now, whereas the latter
produces "this is a house". I tried another engine
(english->chinese) and got the same behaviour - empty string vs.
correct translation.
Cheers,
Martin
Am 29.01.2016 um 22:37 schrieb Matthias Huck:
On Fri, 2016-01-29 at 21:26 +0000, Hieu Hoang wrote:
The decoder should handle no translation without falling over. But
yes, the model is too toy
Normally the decoder would always produce some translation. (The translation
could be an empty sentence, of course.) If it's misconfigured, it should tell
you about it. But maybe not with a segmentation fault. :-)
On 29 Jan 2016 9:15 pm, "Matthias Huck"<mh...@inf.ed.ac.uk> wrote:
Hi,
It seems to me that this toy string-to-tree setup is either
outdated,
or it always had issues. It should be replaced.
Under real-world conditions, the decoder should always be able to
produce some hypothesis. We would therefore usually extract a whole
set
of glue rules. And we would typically also add an [unknown-lhs]
section
to the moses.ini that would tell the decoder which left-hand side
non
-terminal labels to use for out-of-vocabulary words. To my
knowledge,
these two techniques are crucial for being able to parse any input
sentence provided to the chart decoder in syntax-based translation.
So, in my opinion, the problem is most likely neither the server
implementation nor the syntax-based decoder, but a problematic
setup.
I would consider it okay for the server to crash (or at least print
a
warning) under such circumstances. You don't want it to silently
not
translate complete sentences.
(I must admit that I didn't look into it in too much detail, but it
sho
uld be easy to confirm.)
Cheers,
Matthias
On Fri, 2016-01-29 at 20:28 +0000, Barry Haddow wrote:
Hi All
I think I see what happened now.
When you give the input "dies ist ein haus" to the sample model,
the
"dies" is unknown, and there is no translation. The server did
not check
for this condition, and got a seg fault. I have added a check, so
if you
pull and try again it should not crash.
In the log pasted by Martin, he passed "das ist ein haus" to
command-line Moses, which works, and gives a translation.
I think ideally the sample models should handle unknown words,
and give
a translation. Maybe adding a glue rule would be sufficient?
cheers - Barry
On 29/01/16 11:13, Barry Haddow wrote:
Hi
When I run command-line Moses, I get the output below - i.e. no
best
translation. The server crashes for me since it does not check
for the
null pointer, but the command-line version does.
I think there should be a translation for this example.
cheers - Barry
[gna]bhaddow: echo 'dies ist ein haus' | ~/moses.new/bin/moses
-f
string-to-tree/moses.ini
Defined parameters (per moses.ini or switch):
config: string-to-tree/moses.ini
cube-pruning-pop-limit: 1000
feature: KENLM name=LM factor=0 order=3 num
-features=1
path=lm/europarl.srilm.gz WordPenalty UnknownWordPenalty
PhraseDictionaryMemory input-factor=0 output-factor=0
path=string-to-tree/rule-table num-features=1 table-limit=20
input-factors: 0
inputtype: 3
mapping: 0 T 0
max-chart-span: 20 1000
non-terminals: X S
search-algorithm: 3
translation-details: translation-details.log
weight: WordPenalty0= 0 LM= 0.5
PhraseDictionaryMemory0= 0.5
line=KENLM name=LM factor=0 order=3 num-features=1
path=lm/europarl.srilm.gz
Loading the LM will be faster if you build a binary file.
Reading lm/europarl.srilm.gz
----5---10---15---20---25---30---35---40---45---50---55---60--
-65---70---75---80---85---90---95--100
**The ARPA file is missing . Substituting log10 probability
-100.000.
*******************************************************************
*******************************
FeatureFunction: LM start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 2 end: 2
line=PhraseDictionaryMemory input-factor=0 output-factor=0
path=string-to-tree/rule-table num-features=1 table-limit=20
FeatureFunction: PhraseDictionaryMemory0 start: 3 end: 3
Loading LM
Loading WordPenalty0
Loading UnknownWordPenalty0
Loading PhraseDictionaryMemory0
Start loading text phrase table. Moses format : [3.038] seconds
Reading string-to-tree/rule-table
----5---10---15---20---25---30---35---40---45---50---55---60--
-65---70---75---80---85---90---95--100
*******************************************************************
*********************************
max-chart-span: 20
Created input-output object : [3.041] seconds
Line 0: Initialize search took 0.000 seconds total
Translating: dies ist ein haus ||| [0,0]=X (1) [0,1]=X (1)
[0,2]=X (1) [0,3]=X (1) [0,4]=X (1) [0,5]=X (1) [1,1]=X (1)
[1,2]=X (1)
[1,3]=X (1) [1,4]=X (1) [1,5]=X (1) [2,2]=X (1) [2,3]=X (1)
[2,4]=X (1)
[2,5]=X (1) [3,3]=X (1) [3,4]=X (1) [3,5]=X (1) [4,4]=X (1)
[4,5]=X (1)
[5,5]=X (1)
0 1 2 3 4 5
0 1 2 2 1 0
0 0 0 2 0
0 0 4 0
0 0 0
0 0
0
Line 0: Additional reporting took 0.000 seconds total
Line 0: Translation took 0.002 seconds total
Translation took 0.000 seconds
Name:moses VmPeak:74024 kB VmRSS:11084 kB RSSMax:36832 kB
user:2.972 sys:0.048 CPU:3.020 real:3.058
On 29/01/16 00:40, Hieu Hoang wrote:
If it works ok on the command line but crashes when using the
server,
then that suggest a server issue.
I don't know much about the server code, to be honest.
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
--
*STAR Group* <http://www.star-group.net>
<http://www.star-group.net/>
*Martin Baumgärtner*
STAR Language Technology & Solutions GmbH
Umberto-Nobile-Straße 19 | 71063 Sindelfingen | Germany
Tel. +49 70 31-4 10 92-0
<mailto:martin.baumgaert...@star-group.net>martin.baumgaert...@star-group.net
Fax +49 70 31-4 10 92-70 www.star-group.net
<http://www.star-group.net/>
Geschäftsführer: Oliver Rau, Bernd Barth
Handelsregister Stuttgart HRB 245654 | St.-Nr. 56098/11677
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
--
*STAR Group* <http://www.star-group.net>
<http://www.star-group.net/>
*Martin Baumgärtner*
STAR Language Technology & Solutions GmbH
Umberto-Nobile-Straße 19 | 71063 Sindelfingen | Germany
Tel. +49 70 31-4 10 92-0
<mailto:martin.baumgaert...@star-group.net>martin.baumgaert...@star-group.net
Fax +49 70 31-4 10 92-70 www.star-group.net
<http://www.star-group.net/>
Geschäftsführer: Oliver Rau, Bernd Barth
Handelsregister Stuttgart HRB 245654 | St.-Nr. 56098/11677
