test

[Matt Post]

matt (from my phone)

test incubator

[Matt Post]

matt (from my phone)

[VOTE] Release Apache Joshua 6.1 RC#2

[lewis john mcgibbney]

Hello user@ and dev,
Please VOTE on the Apache Joshua 6.1 Release Candidate #2.

We solved 50 issues: https://s.apache.org/joshua6.1

Git source tag (29c8be650d53216f779a340d33f8f61af4d45629):
https://s.apache.org/pk2t 

Staging repo:
https://repository.apache.org/content/repositories/orgapachejoshua-1001/


Source Release Artifacts: https://dist.apache.org/repos/
dist/dev/incubator/joshua/

PGP release keys (signed using 48BAEBF6): https://dist.apache.org/repos/
dist/release/incubator/joshua/KEYS

Vote will be open for 72 hours.
Thank you to everyone that is able to VOTE as well as everyone that
contributed to Apache Joshua 6.1.

[ ] +1, let's get it released!!!
[ ] +/-0, fine, but consider to fix few issues before...
[ ] -1, nope, because... (and please explain why)

P.S. here is my +1

-- 
http://home.apache.org/~lewismc/
@hectorMcSpector
http://www.linkedin.com/in/lmcgibbney

Re: Error while running the tutorial

[Fernando E Alva Manchego]

Hi,

The number is 0. The corpus I'm using is the one provided with the
download: ASR.

Well, I tried with Hadoop 2.7.3, 2.6.5 and 2.5.2 and I get the exact same
error. What could be wrong with its setup? It's just adding the
$HADOOP_HOME/bin to the PATH.

By the way, I really appreciate all the help you're giving.

Cheers,
Fernando

On 22 November 2016 at 22:30, Matt Post  wrote:

> It looks like you have a very small corpus. Can you tell me what number
> this command reports?
>
> gzip -cd grammar.gz | grep Infinity | wc -l
>
> matt
>
> On Nov 22, 2016, at 5:28 PM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
> Hello,
>
> I'm using Hadoop 2.7.3 and Java 8. Apparently, the Hadoop setup is OK,
> according to the instructions given in:
>
> https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/
> SingleCluster.html#Standalone_Operation
>
> I'll try and earlier version of Hadoop and see how it goes.
>
> Cheers,
> Fernando
>
> On 22 November 2016 at 19:06, John Hewitt  wrote:
>
>> Grepping through the log file, I found the following problem:
>>
>> class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob
>> FAILED
>>
>> This is a prereq of OutputJob, hence OutputJob failed.
>>
>> Here's a link to a useful closed issue with an almost identical problem.
>> https://issues.apache.org/jira/browse/JOSHUA-297
>>
>> +1 on the hadoop setup question, as well as the version of Java you're
>> using, for good measure.
>>
>> -John
>>
>> On Tue, Nov 22, 2016 at 1:28 PM, Fernando E Alva Manchego <
>> fealvamanche...@sheffield.ac.uk> wrote:
>>
>>> I'm attaching the file because it's big to paste all its content here.
>>> The size of data/train/thrax-input-file is 4.9M. I'll check the hadoop
>>> setup.
>>>
>>> Cheers,
>>> Fernando
>>>
>>> On 22 November 2016 at 18:15, Matt Post  wrote:
>>>
 Okay, that is the size of a compressed empty file. So the grammar did
 not extract properly. Did you setup Hadoop properly? Can you paste the
 contents of thrax.log? What is the file size of 
 data/train/thrax-input-file?




 On Nov 22, 2016, at 1:12 PM, Fernando E Alva Manchego <
 fealvamanche...@sheffield.ac.uk> wrote:

 Hello,

 It's 20 Bytes.

 Best,
 Fernando

 On 22 November 2016 at 18:00, Matt Post  wrote:

> eigen3 is not necessary. What is the file size of grammar.gz?
>
>
> On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
> Hello,
>
> Well, I ran that command and it went fine: build 100%
>
> However, now I ran the tutorial command again and I get:
>
> * Packing grammar at "grammar.gz" to "../joshua-tutorial/runs/1/tun
> e/model/grammar.gz.packed"
> * Running the grammar-packer.pl script with the command:
> $JOSHUA/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o
> ../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
> Exception in thread "main" java.util.NoSuchElementException
> at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
> at org.apache.joshua.tools.GrammarPacker.getGrammarReader(Gramm
> arPacker.java:239)
> at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
> at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCl
> i.java:120)
> at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerC
> li.java:137)
> * FATAL: Couldn't pack the grammar.
> * Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
> * __init__() takes at least 3 arguments (2 given)
>
> One thing I noticed is this "error" message when compiling:
>
> -- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR
> EIGEN3_VERSION_OK) (Required is at least version "2.91.0")
> CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
>   Not building interpolation.  Eigen3 was not found.
>
> Is Eigen3 really necessary?
>
> Cheers,
> Fernando
>
> On 18 November 2016 at 18:15, Matt Post  wrote:
>
>> Okay, it looks like KenLM is not building. This is a perennial pain.
>> You can see the KenLM build lines in download_deps.sh. What is output 
>> when
>> you run
>>
>> ./jni/build_kenlm.sh
>>
>> matt
>>
>>
>>
>> On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego <
>> fealvamanche...@sheffield.ac.uk> wrote:
>>
>> Hello,
>>
>> UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that 
>> libken.so
>> is there. Now, I run the command again and what I get is the same error
>> that Lewis pointed out:
>>
>> [lm-sort-uniq] rebuilding...
>>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
>>   dep= 

Re: Error while running the tutorial

[Matt Post]

It looks like you have a very small corpus. Can you tell me what number this 
command reports?

gzip -cd grammar.gz | grep Infinity | wc -l

matt

> On Nov 22, 2016, at 5:28 PM, Fernando E Alva Manchego 
>  wrote:
> 
> Hello,
> 
> I'm using Hadoop 2.7.3 and Java 8. Apparently, the Hadoop setup is OK, 
> according to the instructions given in:
> 
> https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html#Standalone_Operation
>  
> 
> 
> I'll try and earlier version of Hadoop and see how it goes.
> 
> Cheers,
> Fernando 
> 
> On 22 November 2016 at 19:06, John Hewitt  > wrote:
> Grepping through the log file, I found the following problem: 
> 
> class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob FAILED
> 
> This is a prereq of OutputJob, hence OutputJob failed. 
> 
> Here's a link to a useful closed issue with an almost identical problem. 
> https://issues.apache.org/jira/browse/JOSHUA-297 
> 
> 
> +1 on the hadoop setup question, as well as the version of Java you're using, 
> for good measure. 
> 
> -John
> 
> On Tue, Nov 22, 2016 at 1:28 PM, Fernando E Alva Manchego 
> > 
> wrote:
> I'm attaching the file because it's big to paste all its content here. The 
> size of data/train/thrax-input-file is 4.9M. I'll check the hadoop setup.
> 
> Cheers,
> Fernando 
> 
> On 22 November 2016 at 18:15, Matt Post  > wrote:
> Okay, that is the size of a compressed empty file. So the grammar did not 
> extract properly. Did you setup Hadoop properly? Can you paste the contents 
> of thrax.log? What is the file size of data/train/thrax-input-file?
> 
> 
> 
> 
>> On Nov 22, 2016, at 1:12 PM, Fernando E Alva Manchego 
>> > 
>> wrote:
>> 
>> Hello,
>> 
>> It's 20 Bytes.
>> 
>> Best,
>> Fernando
>> 
>> On 22 November 2016 at 18:00, Matt Post > > wrote:
>> eigen3 is not necessary. What is the file size of grammar.gz?
>> 
>> 
>>> On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego 
>>> > 
>>> wrote:
>>> 
>>> Hello,
>>> 
>>> Well, I ran that command and it went fine: build 100%
>>> 
>>> However, now I ran the tutorial command again and I get:
>>> 
>>> * Packing grammar at "grammar.gz" to 
>>> "../joshua-tutorial/runs/1/tune/model/grammar.gz.packed"
>>> * Running the grammar-packer.pl  script with the 
>>> command: $JOSHUA/scripts/support/grammar-packer.pl 
>>>  -a -T /tmp -g grammar.gz -o 
>>> ../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
>>> Exception in thread "main" java.util.NoSuchElementException
>>> at org.apache.joshua.util.io 
>>> .LineReader.next(LineReader.java:276)
>>> at 
>>> org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
>>> at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>>> at 
>>> org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
>>> at 
>>> org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
>>> * FATAL: Couldn't pack the grammar.
>>> * Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
>>> * __init__() takes at least 3 arguments (2 given)
>>> 
>>> One thing I noticed is this "error" message when compiling:
>>> 
>>> -- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR EIGEN3_VERSION_OK) 
>>> (Required is at least version "2.91.0")
>>> CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
>>>   Not building interpolation.  Eigen3 was not found.
>>> 
>>> Is Eigen3 really necessary? 
>>> 
>>> Cheers,
>>> Fernando 
>>> 
>>> On 18 November 2016 at 18:15, Matt Post >> > wrote:
>>> Okay, it looks like KenLM is not building. This is a perennial pain. You 
>>> can see the KenLM build lines in download_deps.sh. What is output when you 
>>> run 
>>> 
>>> ./jni/build_kenlm.sh
>>> 
>>> matt
>>> 
>>> 
>>> 
 On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego 
 > 
 wrote:
 
 Hello,
 
 UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that 
 libken.so is there. Now, I run the command again and what I get is the 
 same error that Lewis pointed out:
 
 [lm-sort-uniq] rebuilding...
   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
   dep= 

Re: Error while running the tutorial

[Fernando E Alva Manchego]

Hello,

I'm using Hadoop 2.7.3 and Java 8. Apparently, the Hadoop setup is OK,
according to the instructions given in:

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html#Standalone_Operation

I'll try and earlier version of Hadoop and see how it goes.

Cheers,
Fernando

On 22 November 2016 at 19:06, John Hewitt  wrote:

> Grepping through the log file, I found the following problem:
>
> class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob FAILED
>
> This is a prereq of OutputJob, hence OutputJob failed.
>
> Here's a link to a useful closed issue with an almost identical problem.
> https://issues.apache.org/jira/browse/JOSHUA-297
>
> +1 on the hadoop setup question, as well as the version of Java you're
> using, for good measure.
>
> -John
>
> On Tue, Nov 22, 2016 at 1:28 PM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
>> I'm attaching the file because it's big to paste all its content here.
>> The size of data/train/thrax-input-file is 4.9M. I'll check the hadoop
>> setup.
>>
>> Cheers,
>> Fernando
>>
>> On 22 November 2016 at 18:15, Matt Post  wrote:
>>
>>> Okay, that is the size of a compressed empty file. So the grammar did
>>> not extract properly. Did you setup Hadoop properly? Can you paste the
>>> contents of thrax.log? What is the file size of data/train/thrax-input-file?
>>>
>>>
>>>
>>>
>>> On Nov 22, 2016, at 1:12 PM, Fernando E Alva Manchego <
>>> fealvamanche...@sheffield.ac.uk> wrote:
>>>
>>> Hello,
>>>
>>> It's 20 Bytes.
>>>
>>> Best,
>>> Fernando
>>>
>>> On 22 November 2016 at 18:00, Matt Post  wrote:
>>>
 eigen3 is not necessary. What is the file size of grammar.gz?


 On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego <
 fealvamanche...@sheffield.ac.uk> wrote:

 Hello,

 Well, I ran that command and it went fine: build 100%

 However, now I ran the tutorial command again and I get:

 * Packing grammar at "grammar.gz" to "../joshua-tutorial/runs/1/tun
 e/model/grammar.gz.packed"
 * Running the grammar-packer.pl script with the command:
 $JOSHUA/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o
 ../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
 Exception in thread "main" java.util.NoSuchElementException
 at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
 at org.apache.joshua.tools.GrammarPacker.getGrammarReader(Gramm
 arPacker.java:239)
 at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
 at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCl
 i.java:120)
 at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerC
 li.java:137)
 * FATAL: Couldn't pack the grammar.
 * Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
 * __init__() takes at least 3 arguments (2 given)

 One thing I noticed is this "error" message when compiling:

 -- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR
 EIGEN3_VERSION_OK) (Required is at least version "2.91.0")
 CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
   Not building interpolation.  Eigen3 was not found.

 Is Eigen3 really necessary?

 Cheers,
 Fernando

 On 18 November 2016 at 18:15, Matt Post  wrote:

> Okay, it looks like KenLM is not building. This is a perennial pain.
> You can see the KenLM build lines in download_deps.sh. What is output when
> you run
>
> ./jni/build_kenlm.sh
>
> matt
>
>
>
> On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
> Hello,
>
> UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that 
> libken.so
> is there. Now, I run the command again and what I get is the same error
> that Lewis pointed out:
>
> [lm-sort-uniq] rebuilding...
>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en.uniq [NOT FOUND]
>   cmd= $JOSHUA/scripts/training/scat /export/data/falva/joshua-tuto
> rial/runs/1/data/train/corpus.en | sort -u -T /tmp -S 8G | gzip -9n
> >.../joshua-tutorial/runs/1/data/train/corpus.en.uniq
>   took 1 seconds (1s)
> * FATAL: $JOSHUA/bin/lmplz (for building LMs) does not exist.
>   This is often a problem with the boost libraries (particularly
> threaded
>   versus unthreaded).
>
> Cheers,
> Fernando
>
> On 18 November 2016 at 16:40, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
>> Hello,
>>
>> Sorry for the late reply. I have downloaded joshua again and followed
>> the updated procedure, but I still get the same error when running the
>> following command:
>>
>> 

Re: Error while running the tutorial

[John Hewitt]

Grepping through the log file, I found the following problem:

class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob FAILED

This is a prereq of OutputJob, hence OutputJob failed.

Here's a link to a useful closed issue with an almost identical problem.
https://issues.apache.org/jira/browse/JOSHUA-297

+1 on the hadoop setup question, as well as the version of Java you're
using, for good measure.

-John

On Tue, Nov 22, 2016 at 1:28 PM, Fernando E Alva Manchego <
fealvamanche...@sheffield.ac.uk> wrote:

> I'm attaching the file because it's big to paste all its content here. The
> size of data/train/thrax-input-file is 4.9M. I'll check the hadoop setup.
>
> Cheers,
> Fernando
>
> On 22 November 2016 at 18:15, Matt Post  wrote:
>
>> Okay, that is the size of a compressed empty file. So the grammar did not
>> extract properly. Did you setup Hadoop properly? Can you paste the contents
>> of thrax.log? What is the file size of data/train/thrax-input-file?
>>
>>
>>
>>
>> On Nov 22, 2016, at 1:12 PM, Fernando E Alva Manchego <
>> fealvamanche...@sheffield.ac.uk> wrote:
>>
>> Hello,
>>
>> It's 20 Bytes.
>>
>> Best,
>> Fernando
>>
>> On 22 November 2016 at 18:00, Matt Post  wrote:
>>
>>> eigen3 is not necessary. What is the file size of grammar.gz?
>>>
>>>
>>> On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego <
>>> fealvamanche...@sheffield.ac.uk> wrote:
>>>
>>> Hello,
>>>
>>> Well, I ran that command and it went fine: build 100%
>>>
>>> However, now I ran the tutorial command again and I get:
>>>
>>> * Packing grammar at "grammar.gz" to "../joshua-tutorial/runs/1/tun
>>> e/model/grammar.gz.packed"
>>> * Running the grammar-packer.pl script with the command:
>>> $JOSHUA/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o
>>> ../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
>>> Exception in thread "main" java.util.NoSuchElementException
>>> at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
>>> at org.apache.joshua.tools.GrammarPacker.getGrammarReader(Gramm
>>> arPacker.java:239)
>>> at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>>> at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCl
>>> i.java:120)
>>> at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerC
>>> li.java:137)
>>> * FATAL: Couldn't pack the grammar.
>>> * Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
>>> * __init__() takes at least 3 arguments (2 given)
>>>
>>> One thing I noticed is this "error" message when compiling:
>>>
>>> -- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR
>>> EIGEN3_VERSION_OK) (Required is at least version "2.91.0")
>>> CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
>>>   Not building interpolation.  Eigen3 was not found.
>>>
>>> Is Eigen3 really necessary?
>>>
>>> Cheers,
>>> Fernando
>>>
>>> On 18 November 2016 at 18:15, Matt Post  wrote:
>>>
 Okay, it looks like KenLM is not building. This is a perennial pain.
 You can see the KenLM build lines in download_deps.sh. What is output when
 you run

 ./jni/build_kenlm.sh

 matt



 On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego <
 fealvamanche...@sheffield.ac.uk> wrote:

 Hello,

 UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that 
 libken.so
 is there. Now, I run the command again and what I get is the same error
 that Lewis pointed out:

 [lm-sort-uniq] rebuilding...
   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
   dep= ../joshua-tutorial/runs/1/data/train/corpus.en.uniq [NOT FOUND]
   cmd= $JOSHUA/scripts/training/scat /export/data/falva/joshua-tuto
 rial/runs/1/data/train/corpus.en | sort -u -T /tmp -S 8G | gzip -9n
 >.../joshua-tutorial/runs/1/data/train/corpus.en.uniq
   took 1 seconds (1s)
 * FATAL: $JOSHUA/bin/lmplz (for building LMs) does not exist.
   This is often a problem with the boost libraries (particularly
 threaded
   versus unthreaded).

 Cheers,
 Fernando

 On 18 November 2016 at 16:40, Fernando E Alva Manchego <
 fealvamanche...@sheffield.ac.uk> wrote:

> Hello,
>
> Sorry for the late reply. I have downloaded joshua again and followed
> the updated procedure, but I still get the same error when running the
> following command:
>
> $JOSHUA/bin/pipeline.pl \
>  --rundir 1 \
>  --readme "Baseline Hiero run" \
>  --source es \
>  --target en \
>  --type hiero \
>  --corpus $FISHER/corpus/asr/fisher_train \
>  --tune $FISHER/corpus/asr/fisher_dev \
>  --test $FISHER/corpus/asr/fisher_dev2 \
>  --maxlen 11 \
>  --maxlen-tune 11 \
>  --maxlen-test 11 \
>  --tuner-iterations 1 \
>  --lm-order 3
>
> The error is still:
> [pack-grammar] rebuilding...
>   dep= 

Re: Error while running the tutorial

[Matt Post]

eigen3 is not necessary. What is the file size of grammar.gz?


> On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego 
>  wrote:
> 
> Hello,
> 
> Well, I ran that command and it went fine: build 100%
> 
> However, now I ran the tutorial command again and I get:
> 
> * Packing grammar at "grammar.gz" to 
> "../joshua-tutorial/runs/1/tune/model/grammar.gz.packed"
> * Running the grammar-packer.pl  script with the 
> command: $JOSHUA/scripts/support/grammar-packer.pl 
>  -a -T /tmp -g grammar.gz -o 
> ../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
> Exception in thread "main" java.util.NoSuchElementException
>   at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
>   at 
> org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
>   at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>   at 
> org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
>   at 
> org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
> * FATAL: Couldn't pack the grammar.
> * Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
> * __init__() takes at least 3 arguments (2 given)
> 
> One thing I noticed is this "error" message when compiling:
> 
> -- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR EIGEN3_VERSION_OK) 
> (Required is at least version "2.91.0")
> CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
>   Not building interpolation.  Eigen3 was not found.
> 
> Is Eigen3 really necessary? 
> 
> Cheers,
> Fernando 
> 
> On 18 November 2016 at 18:15, Matt Post  > wrote:
> Okay, it looks like KenLM is not building. This is a perennial pain. You can 
> see the KenLM build lines in download_deps.sh. What is output when you run 
> 
>   ./jni/build_kenlm.sh
> 
> matt
> 
> 
> 
>> On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego 
>> > 
>> wrote:
>> 
>> Hello,
>> 
>> UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that 
>> libken.so is there. Now, I run the command again and what I get is the same 
>> error that Lewis pointed out:
>> 
>> [lm-sort-uniq] rebuilding...
>>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
>>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en.uniq [NOT FOUND]
>>   cmd= $JOSHUA/scripts/training/scat 
>> /export/data/falva/joshua-tutorial/runs/1/data/train/corpus.en | sort -u -T 
>> /tmp -S 8G | gzip -9n >.../joshua-tutorial/runs/1/data/train/corpus.en.uniq
>>   took 1 seconds (1s)
>> * FATAL: $JOSHUA/bin/lmplz (for building LMs) does not exist.
>>   This is often a problem with the boost libraries (particularly threaded
>>   versus unthreaded).
>> 
>> Cheers,
>> Fernando 
>> 
>> On 18 November 2016 at 16:40, Fernando E Alva Manchego 
>> > 
>> wrote:
>> Hello,
>> 
>> Sorry for the late reply. I have downloaded joshua again and followed the 
>> updated procedure, but I still get the same error when running the following 
>> command:
>> 
>> $JOSHUA/bin/pipeline.pl  \
>>--rundir 1 \
>>--readme "Baseline Hiero run" \
>>--source es \
>>--target en \
>>--type hiero \
>>--corpus $FISHER/corpus/asr/fisher_train \
>>--tune $FISHER/corpus/asr/fisher_dev \
>>--test $FISHER/corpus/asr/fisher_dev2 \
>>--maxlen 11 \
>>--maxlen-tune 11 \
>>--maxlen-test 11 \
>>--tuner-iterations 1 \
>>--lm-order 3
>> 
>> The error is still:
>> [pack-grammar] rebuilding...
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/vocabulary [NOT FOUND]
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/encoding [NOT FOUND]
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/slice_0.source [NOT 
>> FOUND]
>>   cmd= $JOSHUA/scripts/support/grammar-packer.pl  
>> -a -T /tmp -m 8g -g grammar.gz -o $HOME/joshua-tutorial/runs/1/grammar.packed
>>   JOB FAILED (return code 1)
>> Exception in thread "main" java.util.NoSuchElementException
>>  at org.apache.joshua.util.io 
>> .LineReader.next(LineReader.java:276)
>>  at 
>> org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
>>  at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>>  at 
>> org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
>>  at 
>> org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
>> * FATAL: Couldn't pack the grammar.
>> * Copying sorted grammars (/tmp/grammar.gzTQzG) to current directory.
>> 
>> What I have noticed now is that, when running the tests after compilation, 
>> this error message appears:
>> 
>> ERROR - Can't find 

Re: Unable to run the language packs - facing some errors

[Matt Post]

Hi,

Please send support requests to u...@joshua.apache.org 
. 

You need to increase the memory. Edit the joshua script and try doubling it. 
Let us know. 

matt (from my phone)



> On Nov 21, 2016, at 7:41 AM, Dixon Daniel  wrote:
> 
> Hi,
> 
> I am trying to run the German to English Language pack but I get the error 
> shown below:
> 
> dixon@HOME:~/Joshua/apache-joshua-de-en-2016-11-18$ cat example.de 
>  | ./prepare.sh | ./joshua
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'LanguageModel -lm_type berkeleylm -lm_order 4 -lm_file 
> model/lm.berkeleylm'!
> at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:632)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:394)
> at org.apache.joshua.decoder.Decoder.(Decoder.java:128)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:628)
> ... 3 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.lang.reflect.Array.newArray(Native Method)
> at java.lang.reflect.Array.newInstance(Array.java:75)
> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1678)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
> at edu.berkeley.nlp.lm.io.IOUtils.readObjFile(IOUtils.java:139)
> at edu.berkeley.nlp.lm.io.IOUtils.readObjFileHard(IOUtils.java:164)
> at edu.berkeley.nlp.lm.io.IOUtils.readObjFileHard(IOUtils.java:159)
> at edu.berkeley.nlp.lm.io.LmReaders.readLmBinary(LmReaders.java:337)
> at 
> org.apache.joshua.decoder.ff.lm.berkeley_lm.LMGrammarBerkeley.(LMGrammarBerkeley.java:87)
> at 
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.initializeLM(LanguageModelFF.java:158)
> at 
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:132)
> ... 8 more
> 
> Could you please help me resolve this error?
> 
> Thanks,
> Dixon
> 
> Have a great day!

Re: Error while running the tutorial

[Fernando E Alva Manchego]

Hello,

Well, I ran that command and it went fine: build 100%

However, now I ran the tutorial command again and I get:

* Packing grammar at "grammar.gz" to
"../joshua-tutorial/runs/1/tune/model/grammar.gz.packed"
* Running the grammar-packer.pl script with the command:
$JOSHUA/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o
../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
Exception in thread "main" java.util.NoSuchElementException
at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
at
org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
* FATAL: Couldn't pack the grammar.
* Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
* __init__() takes at least 3 arguments (2 given)

One thing I noticed is this "error" message when compiling:

-- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR EIGEN3_VERSION_OK)
(Required is at least version "2.91.0")
CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
  Not building interpolation.  Eigen3 was not found.

Is Eigen3 really necessary?

Cheers,
Fernando

On 18 November 2016 at 18:15, Matt Post  wrote:

> Okay, it looks like KenLM is not building. This is a perennial pain. You
> can see the KenLM build lines in download_deps.sh. What is output when you
> run
>
> ./jni/build_kenlm.sh
>
> matt
>
>
>
> On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
> Hello,
>
> UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that libken.so
> is there. Now, I run the command again and what I get is the same error
> that Lewis pointed out:
>
> [lm-sort-uniq] rebuilding...
>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en.uniq [NOT FOUND]
>   cmd= $JOSHUA/scripts/training/scat /export/data/falva/joshua-
> tutorial/runs/1/data/train/corpus.en | sort -u -T /tmp -S 8G | gzip -9n
> >.../joshua-tutorial/runs/1/data/train/corpus.en.uniq
>   took 1 seconds (1s)
> * FATAL: $JOSHUA/bin/lmplz (for building LMs) does not exist.
>   This is often a problem with the boost libraries (particularly threaded
>   versus unthreaded).
>
> Cheers,
> Fernando
>
> On 18 November 2016 at 16:40, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
>> Hello,
>>
>> Sorry for the late reply. I have downloaded joshua again and followed the
>> updated procedure, but I still get the same error when running the
>> following command:
>>
>> $JOSHUA/bin/pipeline.pl \
>>  --rundir 1 \
>>  --readme "Baseline Hiero run" \
>>  --source es \
>>  --target en \
>>  --type hiero \
>>  --corpus $FISHER/corpus/asr/fisher_train \
>>  --tune $FISHER/corpus/asr/fisher_dev \
>>  --test $FISHER/corpus/asr/fisher_dev2 \
>>  --maxlen 11 \
>>  --maxlen-tune 11 \
>>  --maxlen-test 11 \
>>  --tuner-iterations 1 \
>>  --lm-order 3
>>
>> The error is still:
>> [pack-grammar] rebuilding...
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/vocabulary [NOT FOUND]
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/encoding [NOT FOUND]
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/slice_0.source
>> [NOT FOUND]
>>   cmd= $JOSHUA/scripts/support/grammar-packer.pl -a -T /tmp -m 8g -g
>> grammar.gz -o $HOME/joshua-tutorial/runs/1/grammar.packed
>>   JOB FAILED (return code 1)
>> Exception in thread "main" java.util.NoSuchElementException
>> at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
>> at org.apache.joshua.tools.GrammarPacker.getGrammarReader(Gramm
>> arPacker.java:239)
>> at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>> at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCl
>> i.java:120)
>> at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerC
>> li.java:137)
>> * FATAL: Couldn't pack the grammar.
>> * Copying sorted grammars (/tmp/grammar.gzTQzG) to current directory.
>>
>> What I have noticed now is that, when running the tests after
>> compilation, this error message appears:
>>
>> ERROR - Can't find libken.so (libken.dylib on OS X) on the Java library
>> path.
>> WARN - No glue grammar found! Creating dummy glue grammar.
>>
>> Could that be the source of the error? Thank you.
>>
>> @Lewis: I'll make sure to given them your regards.
>>
>> Best
>> Fernando
>>
>> On 18 November 2016 at 13:42, Matt Post  wrote:
>>
>>> I just updated that page to use "mvn package" instead of the old "mvn
>>> compile assembly:single". So Fernando, please make sure you follow the
>>> updated instructions.
>>>
>>>
>>>
>>>
>>> On Nov 17, 2016, at 10:10 PM, lewis john mcgibbney 
>>> wrote:
>>>
>>> Hi Fernando,
>>> First and foremost please give y regards to the GATE team at