Re: Language packs version 3

[Łukasz Sieńko STUD]

​Hello,
I am a happy user of Apache Joshua Language packs! Thank you for your work!
I would like to ask you for language packs, in particular polish - english
version.
*Where could I find version 3 of language packs of any language pair?*
I would like to run a language pack with "Improved Translation With KenLM"
so I followed the instruction here: https://cwiki.apache.org/
confluence/display/JOSHUA/Language+Packs and I realized that there is no
language pair with version 3 that would have "joshua.config.kenlm" file
inside.

Cheers,
Lukasz
​

[RESULT] WAS Re: [VOTE] Graduate the Apache Joshua (Incubating) Project

[lewis john mcgibbney]

Hi Folks,
72 hours has come and gone. I am closing off this thread. Thank you to
everyone that VOTE'd. RESULT is below

[7] +1 Graduate the Apache Joshua (Incubating) Project
Rajesh Dharmadhikari
Lewis John McGibbney*
Tommaso Teofili*
Thamme Gowda*
Tom Barber*
Chris A. Mattmann*
kellen sunderland*

[0] -1 NO NOT Graduate the Apache Joshua (Incubating) Project... please
provide reasoning

* Joshua PPMC Binding

The VOTE therefore passes :)
I'll continue with the Graduation as described in the hyperlink below.
Thanks
Lewis

On Tue, Apr 24, 2018 at 10:02 PM, lewis john mcgibbney 
wrote:

> Hi Folks,
> I would like to open a VOTE for graduating the Apache Joshua (Incubating)
> project.
> For those that are interested, the Incubator guidelines on graduation can
> be found at [0].
> Joshua has been reporting to the IPMC since 16th March 2016 and made one
> Incubating release.
>
> Joshua Basics
>
>- Podling Proposal 
>- Status: current
>- Established: 2016-02-13
>- Incubating for 802 days
>- Prior Board Reports 
>
> There are a few issues to resolve before drafting the graduation
> resolution however this community VOTE is timely. The VOTE will be open at
> least 72 hours and will pass if 3 +1's are received from the Joshua PPMC.
>
> [ ] +1 Graduate the Apache Joshua (Incubating) Project
> [ ] -1 NO NOT Graduate the Apache Joshua (Incubating) Project... please
> provide reasoning
>
> P.S. Here is my binding +1
>
> [0] https://incubator.apache.org/guides/graduation.html#the_
> graduation_process
>
>
> --
> http://home.apache.org/~lewismc/
> http://people.apache.org/keys/committer/lewismc
>



-- 
http://home.apache.org/~lewismc/
http://people.apache.org/keys/committer/lewismc

Re: [VOTE] Graduate the Apache Joshua (Incubating) Project

[kellen sunderland]

+1 (binding)

On Thu, Apr 26, 2018 at 7:06 PM, Mattmann, Chris A (1761) <
chris.a.mattm...@jpl.nasa.gov> wrote:

> +1!
>
> Sent from my iPhone
>
> On Apr 26, 2018, at 5:04 PM, Tom Barber  spicule.co.uk>> wrote:
>
> +1
>
> On Fri, 27 Apr 2018, 00:58 Thamme Gowda,  d...@gmail.com>> wrote:
> +1 (binding)
>
>
> Cheers,
> TG
>
> --
> *Thamme Gowda *
> @thammegowda  | https://isi.edu/~tg
> ~Sent via somebody's Webmail server
>
> 2018-04-24 22:02 GMT-07:00 lewis john mcgibbney  >:
>
> > Hi Folks,
> > I would like to open a VOTE for graduating the Apache Joshua (Incubating)
> > project.
> > For those that are interested, the Incubator guidelines on graduation can
> > be found at [0].
> > Joshua has been reporting to the IPMC since 16th March 2016 and made one
> > Incubating release.
> >
> > Joshua Basics
> >
> >- Podling Proposal 
> >- Status: current
> >- Established: 2016-02-13
> >- Incubating for 802 days
> >- Prior Board Reports  >
> >
> > There are a few issues to resolve before drafting the graduation
> resolution
> > however this community VOTE is timely. The VOTE will be open at least 72
> > hours and will pass if 3 +1's are received from the Joshua PPMC.
> >
> > [ ] +1 Graduate the Apache Joshua (Incubating) Project
> > [ ] -1 NO NOT Graduate the Apache Joshua (Incubating) Project... please
> > provide reasoning
> >
> > P.S. Here is my binding +1
> >
> > [0]
> > https://incubator.apache.org/guides/graduation.html#the_
> graduation_process
> >
> >
> > --
> > http://home.apache.org/~lewismc/
> > http://people.apache.org/keys/committer/lewismc
> >
>
>
> Spicule Limited is registered in England & Wales. Company Number:
> 09954122. Registered office: First Floor, Telecom House, 125-135 Preston
> Road, Brighton, England, BN1 6AF. VAT No. 251478891.
>
>
> All engagements are subject to Spicule Terms and Conditions of Business.
> This email and its contents are intended solely for the individual to whom
> it is addressed and may contain information that is confidential,
> privileged or otherwise protected from disclosure, distributing or copying.
> Any views or opinions presented in this email are solely those of the
> author and do not necessarily represent those of Spicule Limited. The
> company accepts no liability for any damage caused by any virus transmitted
> by this email. If you have received this message in error, please notify us
> immediately by reply email before deleting it from your system. Service of
> legal notice cannot be effected on Spicule Limited by email.
>

Re: [VOTE] Graduate the Apache Joshua (Incubating) Project

[Mattmann, Chris A (1761)]

+1!

Sent from my iPhone

On Apr 26, 2018, at 5:04 PM, Tom Barber 
> wrote:

+1

On Fri, 27 Apr 2018, 00:58 Thamme Gowda, 
> wrote:
+1 (binding)


Cheers,
TG

--
*Thamme Gowda *
@thammegowda  | https://isi.edu/~tg
~Sent via somebody's Webmail server

2018-04-24 22:02 GMT-07:00 lewis john mcgibbney 
>:

> Hi Folks,
> I would like to open a VOTE for graduating the Apache Joshua (Incubating)
> project.
> For those that are interested, the Incubator guidelines on graduation can
> be found at [0].
> Joshua has been reporting to the IPMC since 16th March 2016 and made one
> Incubating release.
>
> Joshua Basics
>
>- Podling Proposal 
>- Status: current
>- Established: 2016-02-13
>- Incubating for 802 days
>- Prior Board Reports 
>
> There are a few issues to resolve before drafting the graduation resolution
> however this community VOTE is timely. The VOTE will be open at least 72
> hours and will pass if 3 +1's are received from the Joshua PPMC.
>
> [ ] +1 Graduate the Apache Joshua (Incubating) Project
> [ ] -1 NO NOT Graduate the Apache Joshua (Incubating) Project... please
> provide reasoning
>
> P.S. Here is my binding +1
>
> [0]
> https://incubator.apache.org/guides/graduation.html#the_graduation_process
>
>
> --
> http://home.apache.org/~lewismc/
> http://people.apache.org/keys/committer/lewismc
>


Spicule Limited is registered in England & Wales. Company Number: 09954122. 
Registered office: First Floor, Telecom House, 125-135 Preston Road, Brighton, 
England, BN1 6AF. VAT No. 251478891.


All engagements are subject to Spicule Terms and Conditions of Business. This 
email and its contents are intended solely for the individual to whom it is 
addressed and may contain information that is confidential, privileged or 
otherwise protected from disclosure, distributing or copying. Any views or 
opinions presented in this email are solely those of the author and do not 
necessarily represent those of Spicule Limited. The company accepts no 
liability for any damage caused by any virus transmitted by this email. If you 
have received this message in error, please notify us immediately by reply 
email before deleting it from your system. Service of legal notice cannot be 
effected on Spicule Limited by email.

Re: [VOTE] Graduate the Apache Joshua (Incubating) Project

[Thamme Gowda]

+1 (binding)


Cheers,
TG

--
*Thamme Gowda *
@thammegowda  | https://isi.edu/~tg
~Sent via somebody's Webmail server

2018-04-24 22:02 GMT-07:00 lewis john mcgibbney :

> Hi Folks,
> I would like to open a VOTE for graduating the Apache Joshua (Incubating)
> project.
> For those that are interested, the Incubator guidelines on graduation can
> be found at [0].
> Joshua has been reporting to the IPMC since 16th March 2016 and made one
> Incubating release.
>
> Joshua Basics
>
>- Podling Proposal 
>- Status: current
>- Established: 2016-02-13
>- Incubating for 802 days
>- Prior Board Reports 
>
> There are a few issues to resolve before drafting the graduation resolution
> however this community VOTE is timely. The VOTE will be open at least 72
> hours and will pass if 3 +1's are received from the Joshua PPMC.
>
> [ ] +1 Graduate the Apache Joshua (Incubating) Project
> [ ] -1 NO NOT Graduate the Apache Joshua (Incubating) Project... please
> provide reasoning
>
> P.S. Here is my binding +1
>
> [0]
> https://incubator.apache.org/guides/graduation.html#the_graduation_process
>
>
> --
> http://home.apache.org/~lewismc/
> http://people.apache.org/keys/committer/lewismc
>

Re: Hadoop tmp Directory Issue

[lewis john mcgibbney]

Hi Mozhdeh,
As you can see from the pipeline.pl source [0], you are required to
override the '-tmp' flag [1] on the pipeline.pl execution.
I would highly encourage you to also scope out the entire parameter and
flag options to geta better feel for the pipeline execution.
HTH
Lewis

[0]
https://github.com/apache/incubator-joshua/blob/master/scripts/training/pipeline.pl#L228-L229
[1]
https://github.com/apache/incubator-joshua/blob/master/scripts/training/pipeline.pl#L305

On Sat, Feb 3, 2018 at 10:53 AM, <
user-digest-h...@joshua.incubator.apache.org> wrote:

>
> From: Mozhdeh Gheini 
> To: "user@joshua.incubator.apache.org" 
> Cc:
> Bcc:
> Date: Sat, 3 Feb 2018 18:52:55 +
> Subject: Hadoop tmp Directory Issue
>
> Hi,
>
>
> I'm working with joshua on a grid computing system. The problem I face is
> that although I modify hadoop.tmp.dir and set it to the temporary directory
> assigned to me, Hadoop still ends up in /tmp. This causes serious problems
> because the next user that wants to run joshua and is assigned to the same
> machine that I was on, is not the owner of /tmp/hadoop which was created
> during my last run and cannot proceed. In a nutshell, Hadoop doesn't end up
> where it's supposed to.
>
>
> This is the command that gets run:
>
>
> hadoop jar /nas/home/gheini/libs/joshua/thrax/bin/thrax.jar -D
> hadoop.tmp.dir=/lfs_ssd/uge-tmpdir/138005.1.all.q -D
> mapreduce.task.timeout=0 -D mapreduce.child.java.opts='-
> Djava.io.tmpdir=/lfs_ssd/uge-tmpdir/138005.1.all.q' -D
> mapreduce.map.java.opts='-Xmx4g' -D mapreduce.reduce.java.opts='-Xmx4g'
> thrax-phrase.conf 
> pipeline-es-en-phrase-_nas_home_gheini_experiments_joshua_tmp_setting_test_2
> > thrax.log 2>&1; rm -f grammar grammar.gz; hadoop fs -cat
> pipeline-es-en-phrase-_nas_home_gheini_experiments_
> joshua_tmp_setting_test_2/final/* | gzip -cd |
> /nas/home/gheini/libs/joshua/scripts/training/filter-rules.pl -t 100 |
> gzip -9n > grammar.gz
>
>
> I'm using Hadoop 3.0.0. I think something is happening within Thrax but
> cannot figure it out. I'd appreciate it if anyone could help.
>
>
> Thanks,
>
> Mozhdeh
>
>
>


-- 
http://home.apache.org/~lewismc/
http://people.apache.org/keys/committer/lewismc

Re: pipeline.pl failed

[lewis john mcgibbney]

Hi Marco,
I've retained the lines of logging which indicate that your source and
target corpus' are not available or are not found by the pipeline.
Please make sure that the file are available at
/home/joshua/joshua/LRTenit/RUN7/data/train/train
Lewis

On Mon, Oct 23, 2017 at 8:14 AM, <
user-digest-h...@joshua.incubator.apache.org> wrote:

>
> From: marco garzuoli 
> To: user@joshua.incubator.apache.org
> Cc:
> Bcc:
> Date: Mon, 23 Oct 2017 17:14:08 +0200
> Subject: pipeline.pl failed
>
> Hi,
>
> I've downloaded and succcessfully installed Joshua from GIT (git clone
> https://github.com/apache/incubator-joshua joshua) on an Ubuntu server
>
> ...
>
>dep=/home/joshua/joshua/LRTenit/RUN7/data/train/train.en [NOT FOUND]
>
>dep=/home/joshua/joshua/LRTenit/RUN7/data/train/train.it [NOT FOUND]
>
>
>
-- 
http://home.apache.org/~lewismc/
@hectorMcSpector
http://www.linkedin.com/in/lmcgibbney

Re: use the en to de language pack

[Matt Post]

maybe it was forgotten. just try any english sentence then 

matt (from my phone)

> Le 13 oct. 2017 à 17:17, blackthirt33n .  a écrit :
> 
> in the instructions for the language pack it states the following:
> # SRC and TRG are the two-character ISO 639-1 language codes
> tar xzf apache-joshua-SRC-TRG--MM-DD.tgz
> cd apache-joshua-SRC-TRG--MM-DD
> cat example.SRC | ./prepare.sh | ./joshua
> 
> when following the above for the language pack
> http://cs.jhu.edu/~post/language-packs/apache-joshua-en-de-2017-01-31-phrase.tgz
> there is no "example.en" 
> is this something that I need to create.
> Also the language pack seems to contain a complete version of 
> joshua and kenlm which should you use the joshua built on your personal
> system or this in the language pack?
> 
> Regards,
> James

Re: how can i demonstrate joshua

[Matt Post]

you have to install libbz2. this will have a package name of something like 
“bzip2-devel”. 

matt (from my phone)

> Le 10 oct. 2017 à 17:08, blackthirt33n .  a écrit :
> 
> i have been following that but have hit a snag:
> 
> vagrant@ubuntu:~/joshua$ ./jni/build_kenlm.sh
> Using JOSHUA=/home/vagrant/joshua
> -- The C compiler identification is GNU 6.3.0
> -- The CXX compiler identification is GNU 6.3.0
> -- Check for working C compiler: /usr/bin/cc
> -- Check for working C compiler: /usr/bin/cc -- broken
> CMake Error at /usr/share/cmake-3.7/Modules/CMakeTestCCompiler.cmake:51 
> (message):
>   The C compiler "/usr/bin/cc" is not able to compile a simple test program.
> 
>   It fails with the following output:
> 
>Change Dir: /home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeTmp
> 
> 
> 
>   Run Build Command:"/usr/bin/make" "cmTC_44219/fast"
> 
>   /usr/bin/make -f CMakeFiles/cmTC_44219.dir/build.make
>   CMakeFiles/cmTC_44219.dir/build
> 
>   make[1]: Entering directory
>   '/home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeTmp'
> 
>   Building C object CMakeFiles/cmTC_44219.dir/testCCompiler.c.o
> 
>   /usr/bin/cc -o CMakeFiles/cmTC_44219.dir/testCCompiler.c.o -c
>   /home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeTmp/testCCompiler.c
> 
>   Linking C executable cmTC_44219
> 
>   /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_44219.dir/link.txt
>   --verbose=1
> 
>   /usr/bin/cc -lz -lbz2 -llzma CMakeFiles/cmTC_44219.dir/testCCompiler.c.o -o
>   cmTC_44219 -rdynamic
> 
>   /usr/bin/ld: cannot find -lbz2
> 
>   collect2: error: ld returned 1 exit status
> 
>   CMakeFiles/cmTC_44219.dir/build.make:97: recipe for target 'cmTC_44219'
>   failed
> 
>   make[1]: *** [cmTC_44219] Error 1
> 
>   make[1]: Leaving directory
>   '/home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeTmp'
> 
>   Makefile:126: recipe for target 'cmTC_44219/fast' failed
> 
>   make: *** [cmTC_44219/fast] Error 2
> 
> 
> 
> 
> 
>   CMake will not be able to correctly generate this project.
> Call Stack (most recent call first):
>   CMakeLists.txt:4 (project)
> 
> 
> -- Configuring incomplete, errors occurred!
> See also "/home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeOutput.log".
> See also "/home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeError.log".
> make: *** No targets specified and no makefile found.  Stop.
> cp: cannot stat 'bin/query': No such file or directory
> cp: cannot stat 'bin/lmplz': No such file or directory
> cp: cannot stat 'bin/build_binary': No such file or directory
> g++: error: lm/CMakeFiles/kenlm.dir/*.o: No such file or directory
> g++: error: util/CMakeFiles/kenlm_util.dir/*.o: No such file or directory
> g++: error: util/CMakeFiles/kenlm_util.dir/double-conversion/*.o: No such 
> file or directory
> 
>> On Sun, Oct 8, 2017 at 10:14 PM, John Hewitt  wrote:
>> Hi James,
>> 
>> The instructions should be relatively self-contained with in the following 
>> tutorial: 
>> 
>> https://cwiki.apache.org/confluence/display/JOSHUA/Language+Packs#LanguagePacks-UsingLanguagePacks
>> 
>> The first few commands there should start you on your way; let us know if 
>> you have more specific questions.
>> 
>> -John
>> 
>>> On Sat, Oct 7, 2017 at 6:50 PM, blackthirt33n .  wrote:
>>> how can I a demonstrate joshua i want to use the english - german language 
>>> pack.
>>> James
>> 
> 

Re: how can i demonstrate joshua

[blackthirt33n .]

i have been following that but have hit a snag:

vagrant@ubuntu:~/joshua$ ./jni/build_kenlm.sh
Using JOSHUA=/home/vagrant/joshua
-- The C compiler identification is GNU 6.3.0
-- The CXX compiler identification is GNU 6.3.0
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- broken
CMake Error at /usr/share/cmake-3.7/Modules/CMakeTestCCompiler.cmake:51
(message):
  The C compiler "/usr/bin/cc" is not able to compile a simple test program.

  It fails with the following output:

   Change Dir: /home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeTmp



  Run Build Command:"/usr/bin/make" "cmTC_44219/fast"

  /usr/bin/make -f CMakeFiles/cmTC_44219.dir/build.make
  CMakeFiles/cmTC_44219.dir/build

  make[1]: Entering directory
  '/home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeTmp'

  Building C object CMakeFiles/cmTC_44219.dir/testCCompiler.c.o

  /usr/bin/cc -o CMakeFiles/cmTC_44219.dir/testCCompiler.c.o -c
  /home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeTmp/testCCompiler.c

  Linking C executable cmTC_44219

  /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_44219.dir/link.txt
  --verbose=1

  /usr/bin/cc -lz -lbz2 -llzma CMakeFiles/cmTC_44219.dir/testCCompiler.c.o
-o
  cmTC_44219 -rdynamic

  /usr/bin/ld: cannot find -lbz2

  collect2: error: ld returned 1 exit status

  CMakeFiles/cmTC_44219.dir/build.make:97: recipe for target 'cmTC_44219'
  failed

  make[1]: *** [cmTC_44219] Error 1

  make[1]: Leaving directory
  '/home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeTmp'

  Makefile:126: recipe for target 'cmTC_44219/fast' failed

  make: *** [cmTC_44219/fast] Error 2





  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:4 (project)


-- Configuring incomplete, errors occurred!
See also "/home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeOutput.log".
See also "/home/vagrant/joshua/ext/kenlm/build/CMakeFiles/CMakeError.log".
make: *** No targets specified and no makefile found.  Stop.
cp: cannot stat 'bin/query': No such file or directory
cp: cannot stat 'bin/lmplz': No such file or directory
cp: cannot stat 'bin/build_binary': No such file or directory
g++: error: lm/CMakeFiles/kenlm.dir/*.o: No such file or directory
g++: error: util/CMakeFiles/kenlm_util.dir/*.o: No such file or directory
g++: error: util/CMakeFiles/kenlm_util.dir/double-conversion/*.o: No such
file or directory

On Sun, Oct 8, 2017 at 10:14 PM, John Hewitt  wrote:

> Hi James,
>
> The instructions should be relatively self-contained with in the following
> tutorial:
>
> https://cwiki.apache.org/confluence/display/JOSHUA/
> Language+Packs#LanguagePacks-UsingLanguagePacks
>
> The first few commands there should start you on your way; let us know if
> you have more specific questions.
>
> -John
>
> On Sat, Oct 7, 2017 at 6:50 PM, blackthirt33n . 
> wrote:
>
>> how can I a demonstrate joshua i want to use the english - german
>> language pack.
>> James
>>
>
>

Re: how can i demonstrate joshua

[John Hewitt]

Hi James,

The instructions should be relatively self-contained with in the following
tutorial:

https://cwiki.apache.org/confluence/display/JOSHUA/Language+Packs#LanguagePacks-UsingLanguagePacks

The first few commands there should start you on your way; let us know if
you have more specific questions.

-John

On Sat, Oct 7, 2017 at 6:50 PM, blackthirt33n .  wrote:

> how can I a demonstrate joshua i want to use the english - german language
> pack.
> James
>

Re: Download links

[Tom Barber]

Also from a graduation pov, we need to get off of Matt hosting downloads in
his .edu space.

:)

On Fri, Aug 25, 2017 at 3:25 PM, Tom Barber  wrote:

> Hi folks
>
> You probably know, your download links are empty. If you don't, you do now.
>
> Tom
>



-- 
Tom Barber
CTO Spicule LTD
t...@spicule.co.uk

http://spicule.co.uk

@spiculeim 

Schedule a meeting with me 

GB: +44(0)5603641316
US: +18448141689

Re: Joshua

[John Hewitt]

By deleting grammar.gz and re-running the pipeline script, Joshua will
recognize that it needs to restart from the grammar step.

If the grammar.gz file is something like 24 bytes, then it's an empty
gzipped file.

-John

On Mon, Aug 14, 2017 at 5:57 AM, Arezoo Arjomand <arezooarjom...@yahoo.com>
wrote:

> Hi,
> i want to run Joshua on a server. how much disk space should i get to run
> the ldc corpus which is in the fisher folder?
> thank you
>

Re: Joshua

[Arezoo Arjomand]

Hi, 
i want to run Joshua on a server. how much disk space should i get to run the 
ldc corpus which is in the fisher folder?thank you

Re: Joshua

[Arezoo Arjomand]

how can i run from the grammar step?it could be from disk space? disk space is 
68 GB... 

Re: Joshua

[Arezoo Arjomand]

 how can i run form the grammar step?

 

On Monday, August 14, 2017 4:58 AM, Matt Post  wrote:
 

 It looks like grammar sorting is failing. Check the logs to see why. Delete 
grammar.* and try again from that step.


On Aug 14, 2017, at 10:49 AM, Arezoo Arjomand  wrote:
Hi,I add "--alignergiza" to terminal command. It seems the alignment error is 
fixed  but the grammer error is still remain both for berkeley aligner and 
giza. grammar.gz is empty and the runing dir is attached. 
 
 

On Monday, August 14, 2017 2:08 AM, Matt Post  wrote:
 

 It looks like alignment failed. Is there a file alignments/training.align? 
That is build from the two pieces, under alignments/0/giza.SRC-TRG (and 
TRG-SRC) that failed.


On Aug 13, 2017, at 7:21 PM, Arezoo Arjomand  wrote:
Hi,When I run the pipleline the following error is shown. The previous error , 
write in the previous email, is shown when i run the same dir for second time 
and grammar.gz is empty. 
 how can i fix the folloewing error? 

[source-numlines] rebuilding...
  dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es [CHANGED]
  cmd=cat /home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es | wc -l
  took 0 seconds (0s)
[source-numlines] retrieved cached result => 77457
[giza-0] rebuilding...
  dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus.es 
[CHANGED]
  dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus.en 
[CHANGED]
  dep=alignments/0/model/aligned.grow-diag-final [NOT FOUND]
  cmd=rm -f alignments/0/corpus.0-0.*; 
/home/arezoo1/joshua-tutorial/joshua/scripts/training/run-giza.pl --root-dir 
alignments/0 -e en -f es -corpus 
/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus -merge 
grow-diag-final  > alignments/0/giza.log 2>&1
*** Error in `/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal': double 
free or corruption (out): 0x556a69b42160 ***
=== Backtrace: =
/lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7f91d0fb908b]
/lib/x86_64-linux-gnu/libc.so.6(+0x826fa)[0x7f91d0fc26fa]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f91d0fc612c]
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal(+0x2b5a)[0x556a6993ab5a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f91d0f603f1]
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal(+0x5f4a)[0x556a6993df4a]
=== Memory map: 
556a69938000-556a69941000 r-xp  08:0a 1051501    
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
556a69b41000-556a69b42000 r--p 9000 08:0a 1051501    
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
556a69b42000-556a69b43000 rw-p a000 08:0a 1051501    
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
556a69b43000-556a69b45000 rw-p  00:00 0 
556a6af09000-556a6afbf000 rw-p  00:00 0  [heap]
7f91cc00-7f91cc021000 rw-p  00:00 0 
7f91cc021000-7f91d000 ---p  00:00 0 
7f91d0c37000-7f91d0d3f000 r-xp  08:0a 1708999    
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0d3f000-7f91d0f3e000 ---p 00108000 08:0a 1708999    
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0f3e000-7f91d0f3f000 r--p 00107000 08:0a 1708999    
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0f3f000-7f91d0f4 rw-p 00108000 08:0a 1708999    
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0f4-7f91d10fd000 r-xp  08:0a 1708931    
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d10fd000-7f91d12fd000 ---p 001bd000 08:0a 1708931    
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d12fd000-7f91d1301000 r--p 001bd000 08:0a 1708931    
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d1301000-7f91d1303000 rw-p 001c1000 08:0a 1708931    
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d1303000-7f91d1307000 rw-p  00:00 0 
7f91d1307000-7f91d131d000 r-xp  08:0a 1708971    
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d131d000-7f91d151c000 ---p 00016000 08:0a 1708971    
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d151c000-7f91d151d000 r--p 00015000 08:0a 1708971    
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d151d000-7f91d151e000 rw-p 00016000 08:0a 1708971    
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d151e000-7f91d1697000 r-xp  08:0a 1976366    
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d1697000-7f91d1896000 ---p 00179000 08:0a 1976366    
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d1896000-7f91d18a r--p 00178000 08:0a 1976366    
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d18a-7f91d18a2000 rw-p 00182000 08:0a 1976366    
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d18a2000-7f91d18a6000 rw-p  00:00 0 
7f91d18a6000-7f91d18cb000 r-xp 

Re: Joshua

[Arezoo Arjomand]

how can i run from the grammar step? 
 --
Best Regards
Arezoo Arjomandzadeh
MSc student in Artificial Intelligence
Computer & IT engineering 
Shahrood University of Technology, Iran
 

On Monday, August 14, 2017 4:58 AM, Matt Post  wrote:
 

 It looks like grammar sorting is failing. Check the logs to see why. Delete 
grammar.* and try again from that step.


On Aug 14, 2017, at 10:49 AM, Arezoo Arjomand  wrote:
Hi,I add "--alignergiza" to terminal command. It seems the alignment error is 
fixed  but the grammer error is still remain both for berkeley aligner and 
giza. grammar.gz is empty and the runing dir is attached. 
 
 

On Monday, August 14, 2017 2:08 AM, Matt Post  wrote:
 

 It looks like alignment failed. Is there a file alignments/training.align? 
That is build from the two pieces, under alignments/0/giza.SRC-TRG (and 
TRG-SRC) that failed.


On Aug 13, 2017, at 7:21 PM, Arezoo Arjomand  wrote:
Hi,When I run the pipleline the following error is shown. The previous error , 
write in the previous email, is shown when i run the same dir for second time 
and grammar.gz is empty. 
 how can i fix the folloewing error? 

[source-numlines] rebuilding...
  dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es [CHANGED]
  cmd=cat /home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es | wc -l
  took 0 seconds (0s)
[source-numlines] retrieved cached result => 77457
[giza-0] rebuilding...
  dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus.es 
[CHANGED]
  dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus.en 
[CHANGED]
  dep=alignments/0/model/aligned.grow-diag-final [NOT FOUND]
  cmd=rm -f alignments/0/corpus.0-0.*; 
/home/arezoo1/joshua-tutorial/joshua/scripts/training/run-giza.pl --root-dir 
alignments/0 -e en -f es -corpus 
/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus -merge 
grow-diag-final  > alignments/0/giza.log 2>&1
*** Error in `/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal': double 
free or corruption (out): 0x556a69b42160 ***
=== Backtrace: =
/lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7f91d0fb908b]
/lib/x86_64-linux-gnu/libc.so.6(+0x826fa)[0x7f91d0fc26fa]
/lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f91d0fc612c]
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal(+0x2b5a)[0x556a6993ab5a]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f91d0f603f1]
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal(+0x5f4a)[0x556a6993df4a]
=== Memory map: 
556a69938000-556a69941000 r-xp  08:0a 1051501    
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
556a69b41000-556a69b42000 r--p 9000 08:0a 1051501    
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
556a69b42000-556a69b43000 rw-p a000 08:0a 1051501    
/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
556a69b43000-556a69b45000 rw-p  00:00 0 
556a6af09000-556a6afbf000 rw-p  00:00 0  [heap]
7f91cc00-7f91cc021000 rw-p  00:00 0 
7f91cc021000-7f91d000 ---p  00:00 0 
7f91d0c37000-7f91d0d3f000 r-xp  08:0a 1708999    
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0d3f000-7f91d0f3e000 ---p 00108000 08:0a 1708999    
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0f3e000-7f91d0f3f000 r--p 00107000 08:0a 1708999    
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0f3f000-7f91d0f4 rw-p 00108000 08:0a 1708999    
/lib/x86_64-linux-gnu/libm-2.24.so
7f91d0f4-7f91d10fd000 r-xp  08:0a 1708931    
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d10fd000-7f91d12fd000 ---p 001bd000 08:0a 1708931    
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d12fd000-7f91d1301000 r--p 001bd000 08:0a 1708931    
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d1301000-7f91d1303000 rw-p 001c1000 08:0a 1708931    
/lib/x86_64-linux-gnu/libc-2.24.so
7f91d1303000-7f91d1307000 rw-p  00:00 0 
7f91d1307000-7f91d131d000 r-xp  08:0a 1708971    
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d131d000-7f91d151c000 ---p 00016000 08:0a 1708971    
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d151c000-7f91d151d000 r--p 00015000 08:0a 1708971    
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d151d000-7f91d151e000 rw-p 00016000 08:0a 1708971    
/lib/x86_64-linux-gnu/libgcc_s.so.1
7f91d151e000-7f91d1697000 r-xp  08:0a 1976366    
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d1697000-7f91d1896000 ---p 00179000 08:0a 1976366    
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d1896000-7f91d18a r--p 00178000 08:0a 1976366    
/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
7f91d18a-7f91d18a2000 rw-p 00182000 08:0a 1976366  

Re: Joshua

[Matt Post]

It looks like grammar sorting is failing. Check the logs to see why. Delete 
grammar.* and try again from that step.


> On Aug 14, 2017, at 10:49 AM, Arezoo Arjomand  > wrote:
> 
> Hi,
> I add "--aligner giza" to terminal command. It seems the alignment error is 
> fixed  but the grammer error is still remain both for berkeley aligner and 
> giza. grammar.gz is empty and the runing dir is attached. 
>  
> 
> 
> 
> On Monday, August 14, 2017 2:08 AM, Matt Post  > wrote:
> 
> 
> It looks like alignment failed. Is there a file alignments/training.align? 
> That is build from the two pieces, under alignments/0/giza.SRC-TRG (and 
> TRG-SRC) that failed.
> 
> 
>> On Aug 13, 2017, at 7:21 PM, Arezoo Arjomand > > wrote:
>> 
>> Hi,
>> When I run the pipleline the following error is shown. The previous error , 
>> write in the previous email, is shown when i run the same dir for second 
>> time and grammar.gz is empty. 
>>  how can i fix the folloewing error? 
>> 
>> [source-numlines] rebuilding...
>>   dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es 
>>  [CHANGED]
>>   cmd=cat /home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es 
>>  | wc -l
>>   took 0 seconds (0s)
>> [source-numlines] retrieved cached result => 77457
>> [giza-0] rebuilding...
>>   dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus.es 
>>  [CHANGED]
>>   dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus.en 
>> [CHANGED]
>>   dep=alignments/0/model/aligned.grow-diag-final [NOT FOUND]
>>   cmd=rm -f alignments/0/corpus.0-0.*; 
>> /home/arezoo1/joshua-tutorial/joshua/scripts/training/run-giza.pl --root-dir 
>> alignments/0 -e en -f es -corpus 
>> /home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus -merge 
>> grow-diag-final  > alignments/0/giza.log 2>&1
>> *** Error in `/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal': double 
>> free or corruption (out): 0x556a69b42160 ***
>> === Backtrace: =
>> /lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7f91d0fb908b]
>> /lib/x86_64-linux-gnu/libc.so.6(+0x826fa)[0x7f91d0fc26fa]
>> /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f91d0fc612c]
>> /home/arezoo1/joshua-tutorial/joshua/ext/symal/symal(+0x2b5a)[0x556a6993ab5a]
>> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f91d0f603f1]
>> /home/arezoo1/joshua-tutorial/joshua/ext/symal/symal(+0x5f4a)[0x556a6993df4a]
>> === Memory map: 
>> 556a69938000-556a69941000 r-xp  08:0a 1051501
>> /home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
>> 556a69b41000-556a69b42000 r--p 9000 08:0a 1051501
>> /home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
>> 556a69b42000-556a69b43000 rw-p a000 08:0a 1051501
>> /home/arezoo1/joshua-tutorial/joshua/ext/symal/symal
>> 556a69b43000-556a69b45000 rw-p  00:00 0 
>> 556a6af09000-556a6afbf000 rw-p  00:00 0  
>> [heap]
>> 7f91cc00-7f91cc021000 rw-p  00:00 0 
>> 7f91cc021000-7f91d000 ---p  00:00 0 
>> 7f91d0c37000-7f91d0d3f000 r-xp  08:0a 1708999
>> /lib/x86_64-linux-gnu/libm-2.24.so
>> 7f91d0d3f000-7f91d0f3e000 ---p 00108000 08:0a 1708999
>> /lib/x86_64-linux-gnu/libm-2.24.so
>> 7f91d0f3e000-7f91d0f3f000 r--p 00107000 08:0a 1708999
>> /lib/x86_64-linux-gnu/libm-2.24.so
>> 7f91d0f3f000-7f91d0f4 rw-p 00108000 08:0a 1708999
>> /lib/x86_64-linux-gnu/libm-2.24.so
>> 7f91d0f4-7f91d10fd000 r-xp  08:0a 1708931
>> /lib/x86_64-linux-gnu/libc-2.24.so
>> 7f91d10fd000-7f91d12fd000 ---p 001bd000 08:0a 1708931
>> /lib/x86_64-linux-gnu/libc-2.24.so
>> 7f91d12fd000-7f91d1301000 r--p 001bd000 08:0a 1708931
>> /lib/x86_64-linux-gnu/libc-2.24.so
>> 7f91d1301000-7f91d1303000 rw-p 001c1000 08:0a 1708931
>> /lib/x86_64-linux-gnu/libc-2.24.so
>> 7f91d1303000-7f91d1307000 rw-p  00:00 0 
>> 7f91d1307000-7f91d131d000 r-xp  08:0a 1708971
>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>> 7f91d131d000-7f91d151c000 ---p 00016000 08:0a 1708971
>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>> 7f91d151c000-7f91d151d000 r--p 00015000 08:0a 1708971
>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>> 7f91d151d000-7f91d151e000 rw-p 00016000 08:0a 1708971
>> /lib/x86_64-linux-gnu/libgcc_s.so.1
>> 7f91d151e000-7f91d1697000 r-xp  08:0a 1976366
>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
>> 7f91d1697000-7f91d1896000 ---p 00179000 08:0a 1976366
>> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22
>> 

Re:

[Arezoo Arjomand]

2.24.so
7f23cd6c-7f23cd6c1000 rw-p 00026000 08:0a 1708903    
/lib/x86_64-linux-gnu/ld-2.24.so
7f23cd6c1000-7f23cd6c2000 rw-p  00:00 0 
7ffea6a36000-7ffea6a58000 rw-p  00:00 0  [stack]
7ffea6b27000-7ffea6b29000 r--p  00:00 0  [vvar]
7ffea6b29000-7ffea6b2b000 r-xp  00:00 0  [vdso]
ff60-ff601000 r-xp  00:00 0  
[vsyscall]
  JOB FAILED (return code 2)
[aligner-combine] rebuilding...
  dep=alignments/0/model/aligned.grow-diag-final [CHANGED]
  dep=alignments/training.align [NOT FOUND]
  cmd=cat alignments/0/model/aligned.grow-diag-final > alignments/training.align
  took 0 seconds (0s)
[thrax-input-file] rebuilding...
  dep=/home/arezoo1/joshua-tutorial/runs/267/data/train/corpus.es [CHANGED]
  dep=/home/arezoo1/joshua-tutorial/runs/267/data/train/corpus.en [CHANGED]
  dep=alignments/training.align [CHANGED]
  dep=/home/arezoo1/joshua-tutorial/runs/267/data/train/thrax-input-file [NOT 
FOUND]
  cmd=/home/arezoo1/joshua/scripts/training/paste 
/home/arezoo1/joshua-tutorial/runs/267/data/train/corpus.es 
/home/arezoo1/joshua-tutorial/runs/267/data/train/corpus.en 
alignments/training.align | perl -pe 's/\t/ ||| /g' | grep -v '()' | grep -v 
'||| \+$' > /home/arezoo1/joshua-tutorial/runs/267/data/train/thrax-input-file
  took 1 seconds (1s)
[thrax-prep] rebuilding...
  dep=/home/arezoo1/joshua-tutorial/runs/267/data/train/thrax-input-file 
[CHANGED]
  dep=grammar.gz [NOT FOUND]
  cmd=hadoop fs -rm -r 
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_267; hadoop fs -mkdir 
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_267; hadoop fs -put 
/home/arezoo1/joshua-tutorial/runs/267/data/train/thrax-input-file 
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_267/input-file
  took 3 seconds (3s)
[thrax-run] rebuilding...
  dep=/home/arezoo1/joshua-tutorial/runs/267/data/train/thrax-input-file 
[CHANGED]
  dep=thrax-phrase.conf [CHANGED]
  dep=grammar.gz [NOT FOUND]
  cmd=hadoop jar /home/arezoo1/joshua/thrax/bin/thrax.jar -D 
mapreduce.task.timeout=0 -D mapreduce.map.java.opts='-Xmx4g' -D 
mapreduce.reduce.java.opts='-Xmx4g' -D hadoop.tmp.dir=/tmp thrax-phrase.conf 
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_267 > thrax.log 2>&1; 
rm -f grammar grammar.gz; hadoop fs -cat 
pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_267/final/* | gzip -cd 
| /home/arezoo1/joshua/scripts/training/filter-rules.pl -t 100 | gzip -9n > 
grammar.gz
  took 29 seconds (29s)
17/08/07 07:20:53 INFO Configuration.deprecation: io.bytes.per.checksum is 
deprecated. Instead, use dfs.bytes-per-checksum
17/08/07 07:20:53 INFO fs.TrashPolicyDefault: Namenode trash configuration: 
Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_267
[pack-grammar] rebuilding...
  dep=/home/arezoo1/joshua-tutorial/runs/267/grammar.packed/vocabulary [NOT 
FOUND]
  dep=/home/arezoo1/joshua-tutorial/runs/267/grammar.packed/encoding [NOT FOUND]
  dep=/home/arezoo1/joshua-tutorial/runs/267/grammar.packed/slice_0.source 
[NOT FOUND]
  cmd=/home/arezoo1/joshua/scripts/support/grammar-packer.pl -a -T /tmp -m 8g 
-g grammar.gz -o /home/arezoo1/joshua-tutorial/runs/267/grammar.packed
  JOB FAILED (return code 1)
Exception in thread "main" java.util.NoSuchElementException
    at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
    at 
org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
    at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
    at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
    at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
* FATAL: Couldn't pack the grammar.
* Copying sorted grammars (/tmp/grammar.gzYYV8) to current directory.

 
 

On Sunday, August 6, 2017 6:20 PM, Matt Post <p...@cs.jhu.edu> wrote:
 

 It looks like the alignments didn't combine, which probably means alignment 
failed, which probably means the binaries didn't build and aren't in 
$JOSHUA/bin. These are built as one of the preprocessing steps. Did you run 
download-deps.sh?
matt




On Aug 6, 2017, at 9:19 PM, Arezoo Arj <arezooarjom...@yahoo.com> wrote:

Hi
Thank you for your reply. I'll be really greatfull if you send me  Matt's email 
address? 
From: John Hewitt
Sent: ‎8/‎6/‎2017 11:21 PM
To: user@joshua.incubator.apache.org; Arezoo Arjomand
Subject: Re:

Hi Azeroo,
I'm replying to you and to the user@joshua mailing list. Matt is on this list, 
the old Google Group isn't used any more, and Hieu is not involved in Joshua 
development. 
Looks like the job died at

  cmd=rm -f alignments/0/corpus.0-0.*; /joshua/scripts/training/run- giza.pl 
--root-dir alignments/0 -e en -f es -corpus /home/arezoo1/joshua-tutorial/ 
runs/data/train/splits/0

Re:

[John Hewitt]

Hi Azeroo,

I'm replying to you and to the user@joshua mailing list. Matt is on this
list, the old Google Group isn't used any more, and Hieu is not involved in
Joshua development.

Looks like the job died at


*  cmd=rm -f alignments/0/corpus.0-0.*;
/joshua/scripts/training/run-giza.pl  --root-dir
alignments/0 -e en -f es -corpus
/home/arezoo1/joshua-tutorial/runs/data/train/splits/0/corpus -merge
grow-diag-final  > alignments/0/giza.log 2>&1  JOB FAILED (return code 127)*


You'll note that stdout and stderr have both been piped to
*alignments/0/giza.log.*
I'd suggest looking there first; you might be able to figure out the issue
based on the error log there. If not, please post its contents in a reply.

Could you let us know if you're using the 6.1 release?

-John



On Sun, Aug 6, 2017 at 7:01 AM, Arezoo Arjomand 
wrote:

> I download  Apache Joshua (Incubating) and compile it successfully. but
> when i run it as the instruction https://cwiki.apache.org/
> confluence/display/JOSHUA/Joshua+Tutorial the following error is appeared.
> I'm really appreciate if you tell me how can i fix it.
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> *arezoo1@arezoo1-X450CC:~/joshua-tutorial/runs$ $JOSHUA/bin/pipeline.pl
>  \ --rundir tessttt \> --readme "Baseline Hiero run" \>
> --source es \> --target en \> --type hiero \> --corpus
> $FISHER/corpus/asr/fisher_train \> --tune $FISHER/corpus/asr/fisher_dev \>
> --test $FISHER/corpus/asr/fisher_dev2 \> --maxlen 11 \> --maxlen-tune 11 \>
> --maxlen-test 11 \> --tuner-iterations 1 \> --lm-order
> 3[train-copy-and-filter] rebuilding...
> dep=/home/arezoo1/joshua-tutorial/fisher-callhome-corpus-master/corpus/asr/fisher_train.es
>  [CHANGED]
> dep=/home/arezoo1/joshua-tutorial/fisher-callhome-corpus-master/corpus/asr/fisher_train.en
> [CHANGED]  dep=/home/arezoo1/joshua-tutorial/runs/data/train/train.es
>  [NOT FOUND]
> dep=/home/arezoo1/joshua-tutorial/runs/data/train/train.en [NOT FOUND]
> cmd=/joshua/scripts/training/paste <(cat
> /home/arezoo1/joshua-tutorial/fisher-callhome-corpus-master/corpus/asr/fisher_train.es
> ) <(cat
> /home/arezoo1/joshua-tutorial/fisher-callhome-corpus-master/corpus/asr/fisher_train.en)
> | /joshua/scripts/training/filter-empty-lines.pl
>  | /joshua/scripts/support/split2files
> /home/arezoo1/joshua-tutorial/runs/data/train/train.es 
> /home/arezoo1/joshua-tutorial/runs/data/train/train.en  took 1 seconds
> (1s)[train-tokenize-es] rebuilding...
> dep=/home/arezoo1/joshua-tutorial/runs/data/train/train.es
>  [CHANGED]
> dep=/home/arezoo1/joshua-tutorial/runs/data/train/train.tok.es
>  [NOT FOUND]  cmd=/joshua/scripts/training/scat
> /home/arezoo1/joshua-tutorial/runs/data/train/train.es  |
> /joshua/scripts/preparation/normalize.pl  es |
> /joshua/scripts/preparation/tokenize.pl  -l es 2>
> /dev/null > /home/arezoo1/joshua-tutorial/runs/data/train/train.tok.es
>   took 7 seconds (7s)[train-tokenize-en]
> rebuilding...  dep=/home/arezoo1/joshua-tutorial/runs/data/train/train.en
> [CHANGED]  dep=/home/arezoo1/joshua-tutorial/runs/data/train/train.tok.en
> [NOT FOUND]  cmd=/joshua/scripts/training/scat
> /home/arezoo1/joshua-tutorial/runs/data/train/train.en |
> /joshua/scripts/preparation/normalize.pl  en |
> /joshua/scripts/preparation/tokenize.pl  -l en 2>
> /dev/null > /home/arezoo1/joshua-tutorial/runs/data/train/train.tok.en
> took 9 seconds (9s)[train-trim] rebuilding...
> dep=/home/arezoo1/joshua-tutorial/runs/data/train/train.tok.es
>  [CHANGED]
> dep=/home/arezoo1/joshua-tutorial/runs/data/train/train.tok.en [CHANGED]
> dep=/home/arezoo1/joshua-tutorial/runs/data/train/train.tok.11.es
>  [NOT FOUND]
> dep=/home/arezoo1/joshua-tutorial/runs/data/train/train.tok.11.en [NOT
> FOUND]  cmd=/joshua/scripts/training/paste
> /home/arezoo1/joshua-tutorial/runs/data/train/train.tok.es
> 
> /home/arezoo1/joshua-tutorial/runs/data/train/train.tok.en |
> /joshua/scripts/training/trim_parallel_corpus.pl
>  11 | /joshua/scripts/support/split2files
> /home/arezoo1/joshua-tutorial/runs/data/train/train.tok.11.es
> 

Re: joshua prints hyphens instead of translation

[Matt Post]

Hi Nicoara,

I finally had a minute to look at this, and it seems to me that everything is 
working fine, and this is just the normal kind of noise you might expect from 
MT systems when tested on data that is different from what they are trained on. 
You have somehow picked two sentences ("hello" and "how are you") that are not 
translated well, but others seem to work fine:

$ pwd
$HOME/language-packs/apache-joshua-en-de-2017-01-31
$ cat example.en 
hello
how are you
this is a test
This is a test .
Those who hurt others hurt themselves .
I think this event is best described as the state enforcing surveillance as the 
normative form of care
$ cat example.en  | ./joshua
-
-
Dies ist ein test
Dies ist ein test.
Verletzt die selbst verletzt worden sein.
Das ereignis dagegen, da der staat als form der versorgung normativen 
durchsetzung überwachung

The model we have provided is a relatively small phrase-based model trained 
mostly on news data. One would hope that it would get sentences like you 
provided, but I am not too surprised it didn't do very well.

matt


> On Jul 11, 2017, at 5:09 PM, Nicoara Talpes  > wrote:
> 
> Hello,
> 
> I think this is an important issue to solve for the following reasons:
> 
> 1) this is one of only two language packs that have both *-en and en-*. 
> 2) there seems to be no solution to running the language pack on Windows
> 3) German is a very circulated language
> 
> Please let me know when this direction en-de is resolved or if it is running 
> on any other machine (maybe mine has an issue).
> I have a project where I am attempting to make use of this language pack 
> specifically.
> 
> Thank you,
> Nicoara
> 
> On Thu, Jul 6, 2017 at 5:19 PM, Matt Post  > wrote:
> Hi,
> 
> Something is clearly wrong but it is not obvious from the output. I hope to 
> look into this soon. You might try another en-* language pack to see if that 
> has the same problem in the meantime which would help isolate this.
> 
> matt
> 
>> On Jul 5, 2017, at 12:02 PM, Nicoara Talpes > > wrote:
>> 
>> Hello,
>> Is the response ok ?
>> Thank you
>> 
>> 
>> 
>> On July 4, 2017, at 7:59 PM, Nicoara Talpes > > wrote:
>> 
>> 
>> Hello,
>> 
>> Here it is:
>> 
>> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head 
>> example.SRC | ./prepare.sh  | ./joshua -v 1
>> INFO - Parameters read from configuration file: joshua.config
>> INFO - tm = 'moses -path model/grammar.gz.packed -maxspan 0 -owner pt'
>> INFO - defaultnonterminal = 'X'
>> INFO - goalsymbol = 'GOAL'
>> INFO - markoovs = 'false'
>> INFO - search = 'stack'
>> INFO - pop-limit: 100
>> INFO - poplimit = '100'
>> INFO - topn = '1'
>> INFO - useuniquenbest = 'true'
>> INFO - outputformat = '%S'
>> INFO - includealignindex = 'false'
>> INFO - featurefunction = 'OOVPenalty'
>> INFO - featurefunction = 'WordPenalty'
>> INFO - featurefunction = 'PhrasePenalty'
>> INFO - featurefunction = 'Distortion'
>> INFO - featurefunction = 'LanguageModel -lm_type berkeleylm -lm_order 4 
>> -lm_file model/lm.berkeleylm'
>> INFO - lowercase = 'true'
>> INFO - projectcase = 'true'
>> INFO - c = 'joshua.config'
>> INFO - v = '0'
>> INFO - v = '1'
>> INFO - Read 9 weights (0 of them dense)
>> INFO - Reading vocabulary: model/grammar.gz.packed/vocabulary
>> INFO - Read 1404929 entries from the vocabulary
>> INFO - Reading packed config: model/grammar.gz.packed/config
>> 102030405060708090.100%
>> INFO - Reading encoder configuration: model/grammar.gz.packed/encoding
>> INFO - Loaded 64487199 rules
>> INFO - Memory used 2360.054904 MB
>> INFO - Grammar loading took: 151 seconds.
>> INFO - Stateful object with state index 0
>> INFO - Loading Berkeley LM from binary model/lm.berkeleylm
>> INFO - FEATURE: tm_pt (weight 0.000)
>> INFO - FEATURE: OOVPenalty (weight 0.016)
>> INFO - FEATURE: WordPenalty (weight -0.279)
>> INFO - FEATURE: PhrasePenalty (weight 0.001)
>> INFO - FEATURE: Distortion (weight 0.123)
>> INFO - FEATURE: lm_0, order 4 (weight 0.314), classLm=false
>> INFO - Grammar sorting happening lazily on-demand.
>> INFO - Model loading took 176 seconds
>> INFO - Memory used 2823.330152 MB
>> INFO - Input 0:  hello 
>> INFO - Input 0: Collecting options took 0.0 seconds
>> INFO - Input 0: Search took 0.073 seconds
>> INFO - Input 0: Translation took 0.824 seconds
>> INFO - Input 0: Memory used is 2828.316456 MB
>> INFO - Input 0: 1-best extraction took 0.172 seconds
>> -
>> INFO - Input 1:  how are you 
>> INFO - Input 1: Collecting options took 0.0 seconds
>> INFO - Input 1: Search took 0.94 seconds
>> INFO - Input 1: Translation took 5.397 seconds
>> INFO - 

Re: joshua prints hyphens instead of translation

[Nicoara Talpes]

Hello,

I think this is an important issue to solve for the following reasons:

1) this is one of only two language packs that have both *-en and en-*.
2) there seems to be no solution to running the language pack on Windows
3) German is a very circulated language

Please let me know when this direction en-de is resolved or if it is
running on any other machine (maybe mine has an issue).
I have a project where I am attempting to make use of this language pack
specifically.

Thank you,
Nicoara

On Thu, Jul 6, 2017 at 5:19 PM, Matt Post  wrote:

> Hi,
>
> Something is clearly wrong but it is not obvious from the output. I hope
> to look into this soon. You might try another en-* language pack to see if
> that has the same problem in the meantime which would help isolate this.
>
> matt
>
> On Jul 5, 2017, at 12:02 PM, Nicoara Talpes 
> wrote:
>
> Hello,
> Is the response ok ?
> Thank you
>
>
> On July 4, 2017, at 7:59 PM, Nicoara Talpes 
> wrote:
>
>
> Hello,
>
> Here it is:
>
> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head
> example.SRC | ./prepare.sh | ./joshua -v 1
> INFO - Parameters read from configuration file: joshua.config
> INFO - tm = 'moses -path model/grammar.gz.packed -maxspan 0 -owner pt'
> INFO - defaultnonterminal = 'X'
> INFO - goalsymbol = 'GOAL'
> INFO - markoovs = 'false'
> INFO - search = 'stack'
> INFO - pop-limit: 100
> INFO - poplimit = '100'
> INFO - topn = '1'
> INFO - useuniquenbest = 'true'
> INFO - outputformat = '%S'
> INFO - includealignindex = 'false'
> INFO - featurefunction = 'OOVPenalty'
> INFO - featurefunction = 'WordPenalty'
> INFO - featurefunction = 'PhrasePenalty'
> INFO - featurefunction = 'Distortion'
> INFO - featurefunction = 'LanguageModel -lm_type berkeleylm -lm_order
> 4 -lm_file model/lm.berkeleylm'
> INFO - lowercase = 'true'
> INFO - projectcase = 'true'
> INFO - c = 'joshua.config'
> INFO - v = '0'
> INFO - v = '1'
> INFO - Read 9 weights (0 of them dense)
> INFO - Reading vocabulary: model/grammar.gz.packed/vocabulary
> INFO - Read 1404929 entries from the vocabulary
> INFO - Reading packed config: model/grammar.gz.packed/config
> 1020304050
> 60708090.100%
> INFO - Reading encoder configuration: model/grammar.gz.packed/encoding
> INFO - Loaded 64487199 rules
> INFO - Memory used 2360.054904 MB
> INFO - Grammar loading took: 151 seconds.
> INFO - Stateful object with state index 0
> INFO - Loading Berkeley LM from binary model/lm.berkeleylm
> INFO - FEATURE: tm_pt (weight 0.000)
> INFO - FEATURE: OOVPenalty (weight 0.016)
> INFO - FEATURE: WordPenalty (weight -0.279)
> INFO - FEATURE: PhrasePenalty (weight 0.001)
> INFO - FEATURE: Distortion (weight 0.123)
> INFO - FEATURE: lm_0, order 4 (weight 0.314), classLm=false
> INFO - Grammar sorting happening lazily on-demand.
> INFO - Model loading took 176 seconds
> INFO - Memory used 2823.330152 MB
> INFO - Input 0:  hello 
> INFO - Input 0: Collecting options took 0.0 seconds
> INFO - Input 0: Search took 0.073 seconds
> INFO - Input 0: Translation took 0.824 seconds
> INFO - Input 0: Memory used is 2828.316456 MB
> INFO - Input 0: 1-best extraction took 0.172 seconds
> -
> INFO - Input 1:  how are you 
> INFO - Input 1: Collecting options took 0.0 seconds
> INFO - Input 1: Search took 0.94 seconds
> INFO - Input 1: Translation took 5.397 seconds
> INFO - Input 1: Memory used is 2858.289328 MB
> INFO - Input 1: 1-best extraction took 0.468 seconds
> -
> INFO - Decoding completed.
> INFO - Memory used 2858.289328 MB
> INFO - Total running time: 183 seconds
>
> Did this output the translation anywhere?
>
> Also , can you tell me what is the command corresponding to this on
> Windows, I could try there.
>
> Thank you,
>
> Nicoara
>
>
> On Tue, Jul 4, 2017 at 7:33 PM, Matt Post  wrote:
>
>> Sorry, that should be "joshua -v 1", can you show that, please?
>>
>>
>> On Jul 4, 2017, at 12:25 PM, Nicoara Talpes 
>> wrote:
>>
>> Hello,
>>
>> Thanks for responding .
>>
>> Here it is the output, I hope it clarifies a little:
>>
>> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head
>> example.SRC
>> hello
>> how are you
>> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head
>> example.SRC | ./prepare.sh
>> hello
>> how are you
>> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head
>> example.SRC | ./prepare.sh | ./joshua -v
>> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 5
>> at org.apache.joshua.decoder.ArgsParser.(ArgsParser.java:60)
>> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.
>> java:61)
>> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ cat
>> example.SRC | ./prepare.sh | ./joshua
>> -
>> -
>>
>> How to proceed?
>>
>> 

Re: joshua prints hyphens instead of translation

[Nicoara Talpes]

Hello,
Is the response ok ?
Thank you

On July 4, 2017, at 7:59 PM, Nicoara Talpes  wrote:

Hello,

Here it is:

nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head 
example.SRC | ./prepare.sh | ./joshua -v 1
INFO - Parameters read from configuration file: joshua.config
INFO - tm = 'moses -path model/grammar.gz.packed -maxspan 0 -owner pt'
INFO - defaultnonterminal = 'X'
INFO - goalsymbol = 'GOAL'
INFO - markoovs = 'false'
INFO - search = 'stack'
INFO - pop-limit: 100
INFO - poplimit = '100'
INFO - topn = '1'
INFO - useuniquenbest = 'true'
INFO - outputformat = '%S'
INFO - includealignindex = 'false'
INFO - featurefunction = 'OOVPenalty'
INFO - featurefunction = 'WordPenalty'
INFO - featurefunction = 'PhrasePenalty'
INFO - featurefunction = 'Distortion'
INFO - featurefunction = 'LanguageModel -lm_type berkeleylm -lm_order 4 
-lm_file model/lm.berkeleylm'
INFO - lowercase = 'true'
INFO - projectcase = 'true'
INFO - c = 'joshua.config'
INFO - v = '0'
INFO - v = '1'
INFO - Read 9 weights (0 of them dense)
INFO - Reading vocabulary: model/grammar.gz.packed/vocabulary
INFO - Read 1404929 entries from the vocabulary
INFO - Reading packed config: model/grammar.gz.packed/config
102030405060708090.100%
INFO - Reading encoder configuration: model/grammar.gz.packed/encoding
INFO - Loaded 64487199 rules
INFO - Memory used 2360.054904 MB
INFO - Grammar loading took: 151 seconds.
INFO - Stateful object with state index 0
INFO - Loading Berkeley LM from binary model/lm.berkeleylm
INFO - FEATURE: tm_pt (weight 0.000)
INFO - FEATURE: OOVPenalty (weight 0.016)
INFO - FEATURE: WordPenalty (weight -0.279)
INFO - FEATURE: PhrasePenalty (weight 0.001)
INFO - FEATURE: Distortion (weight 0.123)
INFO - FEATURE: lm_0, order 4 (weight 0.314), classLm=false
INFO - Grammar sorting happening lazily on-demand.
INFO - Model loading took 176 seconds
INFO - Memory used 2823.330152 MB
INFO - Input 0:  hello 
INFO - Input 0: Collecting options took 0.0 seconds
INFO - Input 0: Search took 0.073 seconds
INFO - Input 0: Translation took 0.824 seconds
INFO - Input 0: Memory used is 2828.316456 MB
INFO - Input 0: 1-best extraction took 0.172 seconds
-
INFO - Input 1:  how are you 
INFO - Input 1: Collecting options took 0.0 seconds
INFO - Input 1: Search took 0.94 seconds
INFO - Input 1: Translation took 5.397 seconds
INFO - Input 1: Memory used is 2858.289328 MB
INFO - Input 1: 1-best extraction took 0.468 seconds
-
INFO - Decoding completed.
INFO - Memory used 2858.289328 MB
INFO - Total running time: 183 seconds

Did this output the translation anywhere?

Also , can you tell me what is the command corresponding to this on Windows, I 
could try there.

Thank you,

Nicoara



On Tue, Jul 4, 2017 at 7:33 PM, Matt Post  wrote:

Sorry, that should be "joshua -v 1", can you show that, please?



On Jul 4, 2017, at 12:25 PM, Nicoara Talpes  wrote:


Hello,

Thanks for responding .

Here it is the output, I hope it clarifies a little:

nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head example.SRC
hello 
how are you
nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head 
example.SRC | ./prepare.sh 
hello
how are you
nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head 
example.SRC | ./prepare.sh | ./joshua -v
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 5
    at org.apache.joshua.decoder.ArgsParser.(ArgsParser.java:60)
    at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:61)
nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ cat example.SRC 
| ./prepare.sh | ./joshua
-
-

How to proceed? 

Thank you


On Tue, Jul 4, 2017 at 4:31 PM, Matt Post  wrote:

Hi Nicoara,


Sorry, I seem to have missed your followup question. 


Can you please debug this a little? What do you get from the following commands?


head example.SRC

head example.SRC | ./prepare.sh

head example.SRC | ./prepare.sh | ./joshua -v


matt



On Jul 3, 2017, at 5:30 PM, Nicoara Talpes  wrote:


Hello Joshua Community,


I am running "cat example.SRC | ./prepare.sh | ./joshua" in a terminal on a 
Ubuntu machine, but for every line in the example.SRC, there is a hyphen shown 
on the terminal. No translation seems to be happening. I am using the 
English-German pack.


How to fix this?


Is there a FAQ that I missed where this is addressed?


Apologies for asking this question again in a matter of a few days, but I am in 
need of a response to move forward.


Thank you,


Nicoara

Re: joshua prints hyphens instead of translation

[Matt Post]

Sorry, that should be "joshua -v 1", can you show that, please?


> On Jul 4, 2017, at 12:25 PM, Nicoara Talpes  > wrote:
> 
> Hello,
> 
> Thanks for responding .
> 
> Here it is the output, I hope it clarifies a little:
> 
> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head 
> example.SRC
> hello 
> how are you
> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head 
> example.SRC | ./prepare.sh 
> hello
> how are you
> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head 
> example.SRC | ./prepare.sh | ./joshua -v
> Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 5
> at org.apache.joshua.decoder.ArgsParser.(ArgsParser.java:60)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:61)
> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ cat 
> example.SRC | ./prepare.sh | ./joshua
> -
> -
> 
> How to proceed? 
> 
> Thank you
> 
> On Tue, Jul 4, 2017 at 4:31 PM, Matt Post  > wrote:
> Hi Nicoara,
> 
> Sorry, I seem to have missed your followup question. 
> 
> Can you please debug this a little? What do you get from the following 
> commands?
> 
>   head example.SRC
>   head example.SRC | ./prepare.sh
>   head example.SRC | ./prepare.sh | ./joshua -v
> 
> matt
> 
> 
>> On Jul 3, 2017, at 5:30 PM, Nicoara Talpes > > wrote:
>> 
>> Hello Joshua Community,
>> 
>> I am running "cat example.SRC | ./prepare.sh | ./joshua" in a terminal on a 
>> Ubuntu machine, but for every line in the example.SRC, there is a hyphen 
>> shown on the terminal. No translation seems to be happening. I am using the 
>> English-German pack.
>> 
>> How to fix this?
>> 
>> Is there a FAQ that I missed where this is addressed?
>> 
>> Apologies for asking this question again in a matter of a few days, but I am 
>> in need of a response to move forward.
>> 
>> Thank you,
>> 
>> Nicoara
> 
> 

Re: joshua prints hyphens instead of translation

[Nicoara Talpes]

Hello,

Thanks for responding .

Here it is the output, I hope it clarifies a little:

nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head
example.SRC
hello
how are you
nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head
example.SRC | ./prepare.sh
hello
how are you
nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ head
example.SRC | ./prepare.sh | ./joshua -v
Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 5
at org.apache.joshua.decoder.ArgsParser.(ArgsParser.java:60)
at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:61)
nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ cat
example.SRC | ./prepare.sh | ./joshua
-
-

How to proceed?

Thank you

On Tue, Jul 4, 2017 at 4:31 PM, Matt Post  wrote:

> Hi Nicoara,
>
> Sorry, I seem to have missed your followup question.
>
> Can you please debug this a little? What do you get from the following
> commands?
>
> head example.SRC
> head example.SRC | ./prepare.sh
> head example.SRC | ./prepare.sh | ./joshua -v
>
> matt
>
>
> On Jul 3, 2017, at 5:30 PM, Nicoara Talpes 
> wrote:
>
> Hello Joshua Community,
>
> I am running "cat example.SRC | ./prepare.sh | ./joshua" in a terminal on
> a Ubuntu machine, but for every line in the example.SRC, there is a
> hyphen shown on the terminal. No translation seems to be happening. I am
> using the English-German pack.
>
> How to fix this?
>
> Is there a FAQ that I missed where this is addressed?
>
> Apologies for asking this question again in a matter of a few days, but I
> am in need of a response to move forward.
>
> Thank you,
>
> Nicoara
>
>
>

Re: joshua prints hyphens instead of translation

[Matt Post]

Hi Nicoara,

Sorry, I seem to have missed your followup question. 

Can you please debug this a little? What do you get from the following commands?

head example.SRC
head example.SRC | ./prepare.sh
head example.SRC | ./prepare.sh | ./joshua -v

matt


> On Jul 3, 2017, at 5:30 PM, Nicoara Talpes  > wrote:
> 
> Hello Joshua Community,
> 
> I am running "cat example.SRC | ./prepare.sh | ./joshua" in a terminal on a 
> Ubuntu machine, but for every line in the example.SRC, there is a hyphen 
> shown on the terminal. No translation seems to be happening. I am using the 
> English-German pack.
> 
> How to fix this?
> 
> Is there a FAQ that I missed where this is addressed?
> 
> Apologies for asking this question again in a matter of a few days, but I am 
> in need of a response to move forward.
> 
> Thank you,
> 
> Nicoara

Re: cannot allocate memory

[Matt Post]

Yes, the default memory is not set very well on many of these language packs. 
John's suggestions here should solve the problem. Please do let us know!

matt


> On Jun 26, 2017, at 2:09 PM, John Hewitt  > wrote:
> 
> Hi Nicoara,
> 
> Machine translation is a memory-intensive task, with large amounts of memory 
> used for storing translation information and language modeling information. 
> 
> From the language packs page 
> : add an 
> option to the decoder (e.g. in this case the ./joshua command) 
> 
> "-m XXg" — increase the amount of memory provided to Joshua. The default is 
> 8g, but for the larger language packs, you will want 16 or 24. In general, 
> 50% more memory than the raw model size should be more than sufficient.
> 
> Since you're using the English-German pack, which is 4.5GB, I'd suggest about 
> 8GB of RAM. Also make sure your VM has been allocated that much RAM from the 
> host system. 
> 
> Best,
> John
> 
> 
> 
> On Mon, Jun 26, 2017 at 12:00 PM, Nicoara Talpes  > wrote:
> Hello Joshua Community,
> 
> When running in an Ubuntu VM "cat example.SRC | ./prepare.sh | ./joshua" as 
> per instructions here 
> https://cwiki.apache.org/confluence/display/JOSHUA/Language+Packs 
>   the jvm 
> goes quickly to 4 gb and the process crashes.
> 
> The input is a file with one line to translate: "Hello".
> 
> What is the amount of RAM required by Joshua? How to limit the amount of RAM 
> required?
> 
> The error is this:
> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ cat 
> example.SRC | ./prepare.sh | ./joshua
> Java HotSpot(TM) 64-Bit Server VM warning: INFO: 
> os::commit_memory(0x0005cbf0, 76021760, 0) failed; error='Cannot 
> allocate memory' (errno=12)
> #
> # There is insufficient memory for the Java Runtime Environment to continue.
> # Native memory allocation (mmap) failed to map 76x021760 bytes for 
> committing reserved memory.
> # An error report file with more information is saved as:
> # 
> /home/nicoara/Desktop/joshua/apache-joshua-en-de-2017-01-31/hs_err_pid10488.log
> 
> Thank you,
> Nicoara
> 

Re: cannot allocate memory

[John Hewitt]

Hi Nicoara,

Machine translation is a memory-intensive task, with large amounts of
memory used for storing translation information and language modeling
information.

>From the language packs page
: add an
option to the decoder (e.g. in this case the ./joshua command)


   - "-m XXg" — increase the amount of memory provided to Joshua. The
   default is 8g, but for the larger language packs, you will want 16 or 24.
   In general, 50% more memory than the raw model size should be more than
   sufficient.


Since you're using the English-German pack, which is 4.5GB, I'd suggest
about 8GB of RAM. Also make sure your VM has been allocated that much RAM
from the host system.

Best,
John



On Mon, Jun 26, 2017 at 12:00 PM, Nicoara Talpes 
wrote:

> Hello Joshua Community,
>
> When running in an Ubuntu VM "cat example.SRC | ./prepare.sh | ./joshua"
> as per instructions here https://cwiki.apache.org/
> confluence/display/JOSHUA/Language+Packs  the jvm goes quickly to 4 gb
> and the process crashes.
>
> The input is a file with one line to translate: "Hello".
>
> What is the amount of RAM required by Joshua? How to limit the amount of
> RAM required?
>
> The error is this:
> nicoara@ubuntu:~/Desktop/joshua/apache-joshua-en-de-2017-01-31$ cat
> example.SRC | ./prepare.sh | ./joshua
> Java HotSpot(TM) 64-Bit Server VM warning: INFO:
> os::commit_memory(0x0005cbf0, 76021760, 0) failed; error='Cannot
> allocate memory' (errno=12)
> #
> # There is insufficient memory for the Java Runtime Environment to
> continue.
> # Native memory allocation (mmap) failed to map 76x021760 bytes
> for committing reserved memory.
> # An error report file with more information is saved as:
> # /home/nicoara/Desktop/joshua/apache-joshua-en-de-2017-01-31/
> hs_err_pid10488.log
>
> Thank you,
> Nicoara
>

Re: Error while running the tutorial

[Fernando E Alva Manchego]

Hi!

I used your files and I got the same error. I'll check the hadoop
environment again. Apparently, it's all pointing out to that as the source
of the problem. Thank you!

Best,
Fernando

On 29 November 2016 at 14:11, Matt Post  wrote:

> I just tried this on my machine and everything worked fine (using your
> exact command). I suspect something is wrong with your Hadoop environment.
> Also, however, my files have fewer lines:
>
> $ wc -l alignments/training.align data/train/corpus.{en,es}
>   76690 alignments/training.align
>   76690 data/train/corpus.en
>   76690 data/train/corpus.es
>
> If you want to share your three files, I could take a look. Maybe you're
> getting a weird tokenization.
>
> You can also try my files:
>
>   http://cs.jhu.edu/~post/tmp/training.align
>   http://cs.jhu.edu/~post/tmp/corpus.en
>   http://cs.jhu.edu/~post/tmp/corpus.es
>
> Then modify your pipeline command like this:
>
> $JOSHUA/bin/pipeline.pl --rundir 1 --readme "Baseline Hiero run" --source
> es --target en --type hiero --lm-order 3 --first-step thrax --last-step
> thrax --corpus /path/to/my/corpus --alignment /path/to/training.align
>
>
>
>
> On Nov 29, 2016, at 7:14 AM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
> Hello,
>
> The output is:
>
>77457 data/train/corpus.en
>77457 data/train/corpus.es
>77457 alignments/training.align
>  232371 total
>
> Best,
> Fernando
>
> On 28 November 2016 at 16:13, Matt Post  wrote:
>
>> This is strange — I'm not sure why the AnnotationJob would fail.
>>
>> What is the output of
>>
>> wc -l data/train/corpus.* alignments/training.align
>>
>> matt
>>
>>
>> On Nov 22, 2016, at 6:34 PM, Fernando E Alva Manchego <
>> fealvamanche...@sheffield.ac.uk> wrote:
>>
>> Hi,
>>
>> The number is 0. The corpus I'm using is the one provided with the
>> download: ASR.
>>
>> Well, I tried with Hadoop 2.7.3, 2.6.5 and 2.5.2 and I get the exact same
>> error. What could be wrong with its setup? It's just adding the
>> $HADOOP_HOME/bin to the PATH.
>>
>> By the way, I really appreciate all the help you're giving.
>>
>> Cheers,
>> Fernando
>>
>> On 22 November 2016 at 22:30, Matt Post  wrote:
>>
>>> It looks like you have a very small corpus. Can you tell me what number
>>> this command reports?
>>>
>>> gzip -cd grammar.gz | grep Infinity | wc -l
>>>
>>> matt
>>>
>>> On Nov 22, 2016, at 5:28 PM, Fernando E Alva Manchego <
>>> fealvamanche...@sheffield.ac.uk> wrote:
>>>
>>> Hello,
>>>
>>> I'm using Hadoop 2.7.3 and Java 8. Apparently, the Hadoop setup is OK,
>>> according to the instructions given in:
>>>
>>> https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/ha
>>> doop-common/SingleCluster.html#Standalone_Operation
>>>
>>> I'll try and earlier version of Hadoop and see how it goes.
>>>
>>> Cheers,
>>> Fernando
>>>
>>> On 22 November 2016 at 19:06, John Hewitt 
>>> wrote:
>>>
 Grepping through the log file, I found the following problem:

 class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob
 FAILED

 This is a prereq of OutputJob, hence OutputJob failed.

 Here's a link to a useful closed issue with an almost identical
 problem. https://issues.apache.org/jira/browse/JOSHUA-297

 +1 on the hadoop setup question, as well as the version of Java you're
 using, for good measure.

 -John

 On Tue, Nov 22, 2016 at 1:28 PM, Fernando E Alva Manchego <
 fealvamanche...@sheffield.ac.uk> wrote:

> I'm attaching the file because it's big to paste all its content here.
> The size of data/train/thrax-input-file is 4.9M. I'll check the
> hadoop setup.
>
> Cheers,
> Fernando
>
> On 22 November 2016 at 18:15, Matt Post  wrote:
>
>> Okay, that is the size of a compressed empty file. So the grammar did
>> not extract properly. Did you setup Hadoop properly? Can you paste the
>> contents of thrax.log? What is the file size of 
>> data/train/thrax-input-file?
>>
>>
>>
>>
>> On Nov 22, 2016, at 1:12 PM, Fernando E Alva Manchego <
>> fealvamanche...@sheffield.ac.uk> wrote:
>>
>> Hello,
>>
>> It's 20 Bytes.
>>
>> Best,
>> Fernando
>>
>> On 22 November 2016 at 18:00, Matt Post  wrote:
>>
>>> eigen3 is not necessary. What is the file size of grammar.gz?
>>>
>>>
>>> On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego <
>>> fealvamanche...@sheffield.ac.uk> wrote:
>>>
>>> Hello,
>>>
>>> Well, I ran that command and it went fine: build 100%
>>>
>>> However, now I ran the tutorial command again and I get:
>>>
>>> * Packing grammar at "grammar.gz" to "../joshua-tutorial/runs/1/tun
>>> e/model/grammar.gz.packed"
>>> * Running the grammar-packer.pl script with the command:
>>> $JOSHUA/scripts/support/grammar-packer.pl 

Re: [VOTE] Release Apache Joshua 6.1 RC#2

[Tommaso Teofili]

+1

Tommaso

Il giorno mer 23 nov 2016 alle ore 15:25 kellen sunderland <
kellen.sunderl...@gmail.com> ha scritto:

> +1, many thanks Lewis.
>
> On Wed, Nov 23, 2016 at 2:34 PM, Matt Post  wrote:
>
> > +1 Thanks, Lewis!
> >
> >
> > > On Nov 23, 2016, at 12:15 AM, lewis john mcgibbney  >
> > wrote:
> > >
> > > Hello user@ and dev,
> > > Please VOTE on the Apache Joshua 6.1 Release Candidate #2.
> > >
> > > We solved 50 issues: https://s.apache.org/joshua6.1
> > >
> > > Git source tag (29c8be650d53216f779a340d33f8f61af4d45629):
> > > https://s.apache.org/pk2t 
> > >
> > > Staging repo:
> > >
> https://repository.apache.org/content/repositories/orgapachejoshua-1001/
> > >  > orgapachejoshua-1000/>
> > >
> > > Source Release Artifacts: https://dist.apache.org/repos/
> > > dist/dev/incubator/joshua/
> > >
> > > PGP release keys (signed using 48BAEBF6):
> https://dist.apache.org/repos/
> > > dist/release/incubator/joshua/KEYS
> > >
> > > Vote will be open for 72 hours.
> > > Thank you to everyone that is able to VOTE as well as everyone that
> > > contributed to Apache Joshua 6.1.
> > >
> > > [ ] +1, let's get it released!!!
> > > [ ] +/-0, fine, but consider to fix few issues before...
> > > [ ] -1, nope, because... (and please explain why)
> > >
> > > P.S. here is my +1
> > >
> > > --
> > > http://home.apache.org/~lewismc/
> > > @hectorMcSpector
> > > http://www.linkedin.com/in/lmcgibbney
> >
> >
>

Re: [VOTE] Release Apache Joshua 6.1 RC#2

[Matt Post]

+1 Thanks, Lewis!


> On Nov 23, 2016, at 12:15 AM, lewis john mcgibbney  wrote:
> 
> Hello user@ and dev,
> Please VOTE on the Apache Joshua 6.1 Release Candidate #2.
> 
> We solved 50 issues: https://s.apache.org/joshua6.1
> 
> Git source tag (29c8be650d53216f779a340d33f8f61af4d45629):
> https://s.apache.org/pk2t 
> 
> Staging repo:
> https://repository.apache.org/content/repositories/orgapachejoshua-1001/
> 
> 
> Source Release Artifacts: https://dist.apache.org/repos/
> dist/dev/incubator/joshua/
> 
> PGP release keys (signed using 48BAEBF6): https://dist.apache.org/repos/
> dist/release/incubator/joshua/KEYS
> 
> Vote will be open for 72 hours.
> Thank you to everyone that is able to VOTE as well as everyone that
> contributed to Apache Joshua 6.1.
> 
> [ ] +1, let's get it released!!!
> [ ] +/-0, fine, but consider to fix few issues before...
> [ ] -1, nope, because... (and please explain why)
> 
> P.S. here is my +1
> 
> -- 
> http://home.apache.org/~lewismc/
> @hectorMcSpector
> http://www.linkedin.com/in/lmcgibbney

Re: Apache Joshua Project

[Matt Post]

1. Yes, you can translate as much as you'd like. Do you mean lots of sentences 
or long sentences?

2. Yes, that is what it does. It even offers (in server mode) a 
Google-translate-style API.

3. There may be someone interested in helping you. What exactly are you trying 
to do? What do you mean "all" language pairs?

> On Nov 19, 2016, at 2:08 PM, Aliaksei Rudak  wrote:
> 
> Hi Matt,
> Can you help me and ask several questions about  Joshua project ?
> 
> 1) Is it possible to translate big amounts of text with Joshua ? ( For 
> example 1000 characters per transaction)
> 2) Does Joshua works like Google Translate ? So you can put sentence in one 
> language and get translated in another language ?
> 3) Can you  (or your teammates ) help me with deployment Joshua on my server 
> and setup all language pairs ? I will pay you.
> 
> Regards,
> Alexei
> 

Re: test incubator

[Henri Yandell]

received.

On Tue, Nov 22, 2016 at 11:05 PM, Matt Post  wrote:

>
>
> matt (from my phone)
>

Re: Error while running the tutorial

[Fernando E Alva Manchego]

Hi,

The number is 0. The corpus I'm using is the one provided with the
download: ASR.

Well, I tried with Hadoop 2.7.3, 2.6.5 and 2.5.2 and I get the exact same
error. What could be wrong with its setup? It's just adding the
$HADOOP_HOME/bin to the PATH.

By the way, I really appreciate all the help you're giving.

Cheers,
Fernando

On 22 November 2016 at 22:30, Matt Post  wrote:

> It looks like you have a very small corpus. Can you tell me what number
> this command reports?
>
> gzip -cd grammar.gz | grep Infinity | wc -l
>
> matt
>
> On Nov 22, 2016, at 5:28 PM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
> Hello,
>
> I'm using Hadoop 2.7.3 and Java 8. Apparently, the Hadoop setup is OK,
> according to the instructions given in:
>
> https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/
> SingleCluster.html#Standalone_Operation
>
> I'll try and earlier version of Hadoop and see how it goes.
>
> Cheers,
> Fernando
>
> On 22 November 2016 at 19:06, John Hewitt  wrote:
>
>> Grepping through the log file, I found the following problem:
>>
>> class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob
>> FAILED
>>
>> This is a prereq of OutputJob, hence OutputJob failed.
>>
>> Here's a link to a useful closed issue with an almost identical problem.
>> https://issues.apache.org/jira/browse/JOSHUA-297
>>
>> +1 on the hadoop setup question, as well as the version of Java you're
>> using, for good measure.
>>
>> -John
>>
>> On Tue, Nov 22, 2016 at 1:28 PM, Fernando E Alva Manchego <
>> fealvamanche...@sheffield.ac.uk> wrote:
>>
>>> I'm attaching the file because it's big to paste all its content here.
>>> The size of data/train/thrax-input-file is 4.9M. I'll check the hadoop
>>> setup.
>>>
>>> Cheers,
>>> Fernando
>>>
>>> On 22 November 2016 at 18:15, Matt Post  wrote:
>>>
 Okay, that is the size of a compressed empty file. So the grammar did
 not extract properly. Did you setup Hadoop properly? Can you paste the
 contents of thrax.log? What is the file size of 
 data/train/thrax-input-file?




 On Nov 22, 2016, at 1:12 PM, Fernando E Alva Manchego <
 fealvamanche...@sheffield.ac.uk> wrote:

 Hello,

 It's 20 Bytes.

 Best,
 Fernando

 On 22 November 2016 at 18:00, Matt Post  wrote:

> eigen3 is not necessary. What is the file size of grammar.gz?
>
>
> On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
> Hello,
>
> Well, I ran that command and it went fine: build 100%
>
> However, now I ran the tutorial command again and I get:
>
> * Packing grammar at "grammar.gz" to "../joshua-tutorial/runs/1/tun
> e/model/grammar.gz.packed"
> * Running the grammar-packer.pl script with the command:
> $JOSHUA/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o
> ../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
> Exception in thread "main" java.util.NoSuchElementException
> at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
> at org.apache.joshua.tools.GrammarPacker.getGrammarReader(Gramm
> arPacker.java:239)
> at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
> at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCl
> i.java:120)
> at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerC
> li.java:137)
> * FATAL: Couldn't pack the grammar.
> * Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
> * __init__() takes at least 3 arguments (2 given)
>
> One thing I noticed is this "error" message when compiling:
>
> -- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR
> EIGEN3_VERSION_OK) (Required is at least version "2.91.0")
> CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
>   Not building interpolation.  Eigen3 was not found.
>
> Is Eigen3 really necessary?
>
> Cheers,
> Fernando
>
> On 18 November 2016 at 18:15, Matt Post  wrote:
>
>> Okay, it looks like KenLM is not building. This is a perennial pain.
>> You can see the KenLM build lines in download_deps.sh. What is output 
>> when
>> you run
>>
>> ./jni/build_kenlm.sh
>>
>> matt
>>
>>
>>
>> On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego <
>> fealvamanche...@sheffield.ac.uk> wrote:
>>
>> Hello,
>>
>> UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that 
>> libken.so
>> is there. Now, I run the command again and what I get is the same error
>> that Lewis pointed out:
>>
>> [lm-sort-uniq] rebuilding...
>>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
>>   dep= 

Re: Error while running the tutorial

[Matt Post]

It looks like you have a very small corpus. Can you tell me what number this 
command reports?

gzip -cd grammar.gz | grep Infinity | wc -l

matt

> On Nov 22, 2016, at 5:28 PM, Fernando E Alva Manchego 
>  wrote:
> 
> Hello,
> 
> I'm using Hadoop 2.7.3 and Java 8. Apparently, the Hadoop setup is OK, 
> according to the instructions given in:
> 
> https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html#Standalone_Operation
>  
> 
> 
> I'll try and earlier version of Hadoop and see how it goes.
> 
> Cheers,
> Fernando 
> 
> On 22 November 2016 at 19:06, John Hewitt  > wrote:
> Grepping through the log file, I found the following problem: 
> 
> class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob FAILED
> 
> This is a prereq of OutputJob, hence OutputJob failed. 
> 
> Here's a link to a useful closed issue with an almost identical problem. 
> https://issues.apache.org/jira/browse/JOSHUA-297 
> 
> 
> +1 on the hadoop setup question, as well as the version of Java you're using, 
> for good measure. 
> 
> -John
> 
> On Tue, Nov 22, 2016 at 1:28 PM, Fernando E Alva Manchego 
> > 
> wrote:
> I'm attaching the file because it's big to paste all its content here. The 
> size of data/train/thrax-input-file is 4.9M. I'll check the hadoop setup.
> 
> Cheers,
> Fernando 
> 
> On 22 November 2016 at 18:15, Matt Post  > wrote:
> Okay, that is the size of a compressed empty file. So the grammar did not 
> extract properly. Did you setup Hadoop properly? Can you paste the contents 
> of thrax.log? What is the file size of data/train/thrax-input-file?
> 
> 
> 
> 
>> On Nov 22, 2016, at 1:12 PM, Fernando E Alva Manchego 
>> > 
>> wrote:
>> 
>> Hello,
>> 
>> It's 20 Bytes.
>> 
>> Best,
>> Fernando
>> 
>> On 22 November 2016 at 18:00, Matt Post > > wrote:
>> eigen3 is not necessary. What is the file size of grammar.gz?
>> 
>> 
>>> On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego 
>>> > 
>>> wrote:
>>> 
>>> Hello,
>>> 
>>> Well, I ran that command and it went fine: build 100%
>>> 
>>> However, now I ran the tutorial command again and I get:
>>> 
>>> * Packing grammar at "grammar.gz" to 
>>> "../joshua-tutorial/runs/1/tune/model/grammar.gz.packed"
>>> * Running the grammar-packer.pl  script with the 
>>> command: $JOSHUA/scripts/support/grammar-packer.pl 
>>>  -a -T /tmp -g grammar.gz -o 
>>> ../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
>>> Exception in thread "main" java.util.NoSuchElementException
>>> at org.apache.joshua.util.io 
>>> .LineReader.next(LineReader.java:276)
>>> at 
>>> org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
>>> at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>>> at 
>>> org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
>>> at 
>>> org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
>>> * FATAL: Couldn't pack the grammar.
>>> * Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
>>> * __init__() takes at least 3 arguments (2 given)
>>> 
>>> One thing I noticed is this "error" message when compiling:
>>> 
>>> -- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR EIGEN3_VERSION_OK) 
>>> (Required is at least version "2.91.0")
>>> CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
>>>   Not building interpolation.  Eigen3 was not found.
>>> 
>>> Is Eigen3 really necessary? 
>>> 
>>> Cheers,
>>> Fernando 
>>> 
>>> On 18 November 2016 at 18:15, Matt Post >> > wrote:
>>> Okay, it looks like KenLM is not building. This is a perennial pain. You 
>>> can see the KenLM build lines in download_deps.sh. What is output when you 
>>> run 
>>> 
>>> ./jni/build_kenlm.sh
>>> 
>>> matt
>>> 
>>> 
>>> 
 On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego 
 > 
 wrote:
 
 Hello,
 
 UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that 
 libken.so is there. Now, I run the command again and what I get is the 
 same error that Lewis pointed out:
 
 [lm-sort-uniq] rebuilding...
   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
   dep= 

Re: Error while running the tutorial

[Fernando E Alva Manchego]

Hello,

I'm using Hadoop 2.7.3 and Java 8. Apparently, the Hadoop setup is OK,
according to the instructions given in:

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html#Standalone_Operation

I'll try and earlier version of Hadoop and see how it goes.

Cheers,
Fernando

On 22 November 2016 at 19:06, John Hewitt  wrote:

> Grepping through the log file, I found the following problem:
>
> class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob FAILED
>
> This is a prereq of OutputJob, hence OutputJob failed.
>
> Here's a link to a useful closed issue with an almost identical problem.
> https://issues.apache.org/jira/browse/JOSHUA-297
>
> +1 on the hadoop setup question, as well as the version of Java you're
> using, for good measure.
>
> -John
>
> On Tue, Nov 22, 2016 at 1:28 PM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
>> I'm attaching the file because it's big to paste all its content here.
>> The size of data/train/thrax-input-file is 4.9M. I'll check the hadoop
>> setup.
>>
>> Cheers,
>> Fernando
>>
>> On 22 November 2016 at 18:15, Matt Post  wrote:
>>
>>> Okay, that is the size of a compressed empty file. So the grammar did
>>> not extract properly. Did you setup Hadoop properly? Can you paste the
>>> contents of thrax.log? What is the file size of data/train/thrax-input-file?
>>>
>>>
>>>
>>>
>>> On Nov 22, 2016, at 1:12 PM, Fernando E Alva Manchego <
>>> fealvamanche...@sheffield.ac.uk> wrote:
>>>
>>> Hello,
>>>
>>> It's 20 Bytes.
>>>
>>> Best,
>>> Fernando
>>>
>>> On 22 November 2016 at 18:00, Matt Post  wrote:
>>>
 eigen3 is not necessary. What is the file size of grammar.gz?


 On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego <
 fealvamanche...@sheffield.ac.uk> wrote:

 Hello,

 Well, I ran that command and it went fine: build 100%

 However, now I ran the tutorial command again and I get:

 * Packing grammar at "grammar.gz" to "../joshua-tutorial/runs/1/tun
 e/model/grammar.gz.packed"
 * Running the grammar-packer.pl script with the command:
 $JOSHUA/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o
 ../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
 Exception in thread "main" java.util.NoSuchElementException
 at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
 at org.apache.joshua.tools.GrammarPacker.getGrammarReader(Gramm
 arPacker.java:239)
 at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
 at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCl
 i.java:120)
 at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerC
 li.java:137)
 * FATAL: Couldn't pack the grammar.
 * Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
 * __init__() takes at least 3 arguments (2 given)

 One thing I noticed is this "error" message when compiling:

 -- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR
 EIGEN3_VERSION_OK) (Required is at least version "2.91.0")
 CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
   Not building interpolation.  Eigen3 was not found.

 Is Eigen3 really necessary?

 Cheers,
 Fernando

 On 18 November 2016 at 18:15, Matt Post  wrote:

> Okay, it looks like KenLM is not building. This is a perennial pain.
> You can see the KenLM build lines in download_deps.sh. What is output when
> you run
>
> ./jni/build_kenlm.sh
>
> matt
>
>
>
> On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
> Hello,
>
> UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that 
> libken.so
> is there. Now, I run the command again and what I get is the same error
> that Lewis pointed out:
>
> [lm-sort-uniq] rebuilding...
>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en.uniq [NOT FOUND]
>   cmd= $JOSHUA/scripts/training/scat /export/data/falva/joshua-tuto
> rial/runs/1/data/train/corpus.en | sort -u -T /tmp -S 8G | gzip -9n
> >.../joshua-tutorial/runs/1/data/train/corpus.en.uniq
>   took 1 seconds (1s)
> * FATAL: $JOSHUA/bin/lmplz (for building LMs) does not exist.
>   This is often a problem with the boost libraries (particularly
> threaded
>   versus unthreaded).
>
> Cheers,
> Fernando
>
> On 18 November 2016 at 16:40, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
>> Hello,
>>
>> Sorry for the late reply. I have downloaded joshua again and followed
>> the updated procedure, but I still get the same error when running the
>> following command:
>>
>> 

Re: Error while running the tutorial

[John Hewitt]

Grepping through the log file, I found the following problem:

class edu.jhu.thrax.hadoop.features.annotation.AnnotationFeatureJob FAILED

This is a prereq of OutputJob, hence OutputJob failed.

Here's a link to a useful closed issue with an almost identical problem.
https://issues.apache.org/jira/browse/JOSHUA-297

+1 on the hadoop setup question, as well as the version of Java you're
using, for good measure.

-John

On Tue, Nov 22, 2016 at 1:28 PM, Fernando E Alva Manchego <
fealvamanche...@sheffield.ac.uk> wrote:

> I'm attaching the file because it's big to paste all its content here. The
> size of data/train/thrax-input-file is 4.9M. I'll check the hadoop setup.
>
> Cheers,
> Fernando
>
> On 22 November 2016 at 18:15, Matt Post  wrote:
>
>> Okay, that is the size of a compressed empty file. So the grammar did not
>> extract properly. Did you setup Hadoop properly? Can you paste the contents
>> of thrax.log? What is the file size of data/train/thrax-input-file?
>>
>>
>>
>>
>> On Nov 22, 2016, at 1:12 PM, Fernando E Alva Manchego <
>> fealvamanche...@sheffield.ac.uk> wrote:
>>
>> Hello,
>>
>> It's 20 Bytes.
>>
>> Best,
>> Fernando
>>
>> On 22 November 2016 at 18:00, Matt Post  wrote:
>>
>>> eigen3 is not necessary. What is the file size of grammar.gz?
>>>
>>>
>>> On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego <
>>> fealvamanche...@sheffield.ac.uk> wrote:
>>>
>>> Hello,
>>>
>>> Well, I ran that command and it went fine: build 100%
>>>
>>> However, now I ran the tutorial command again and I get:
>>>
>>> * Packing grammar at "grammar.gz" to "../joshua-tutorial/runs/1/tun
>>> e/model/grammar.gz.packed"
>>> * Running the grammar-packer.pl script with the command:
>>> $JOSHUA/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o
>>> ../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
>>> Exception in thread "main" java.util.NoSuchElementException
>>> at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
>>> at org.apache.joshua.tools.GrammarPacker.getGrammarReader(Gramm
>>> arPacker.java:239)
>>> at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>>> at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCl
>>> i.java:120)
>>> at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerC
>>> li.java:137)
>>> * FATAL: Couldn't pack the grammar.
>>> * Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
>>> * __init__() takes at least 3 arguments (2 given)
>>>
>>> One thing I noticed is this "error" message when compiling:
>>>
>>> -- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR
>>> EIGEN3_VERSION_OK) (Required is at least version "2.91.0")
>>> CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
>>>   Not building interpolation.  Eigen3 was not found.
>>>
>>> Is Eigen3 really necessary?
>>>
>>> Cheers,
>>> Fernando
>>>
>>> On 18 November 2016 at 18:15, Matt Post  wrote:
>>>
 Okay, it looks like KenLM is not building. This is a perennial pain.
 You can see the KenLM build lines in download_deps.sh. What is output when
 you run

 ./jni/build_kenlm.sh

 matt



 On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego <
 fealvamanche...@sheffield.ac.uk> wrote:

 Hello,

 UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that 
 libken.so
 is there. Now, I run the command again and what I get is the same error
 that Lewis pointed out:

 [lm-sort-uniq] rebuilding...
   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
   dep= ../joshua-tutorial/runs/1/data/train/corpus.en.uniq [NOT FOUND]
   cmd= $JOSHUA/scripts/training/scat /export/data/falva/joshua-tuto
 rial/runs/1/data/train/corpus.en | sort -u -T /tmp -S 8G | gzip -9n
 >.../joshua-tutorial/runs/1/data/train/corpus.en.uniq
   took 1 seconds (1s)
 * FATAL: $JOSHUA/bin/lmplz (for building LMs) does not exist.
   This is often a problem with the boost libraries (particularly
 threaded
   versus unthreaded).

 Cheers,
 Fernando

 On 18 November 2016 at 16:40, Fernando E Alva Manchego <
 fealvamanche...@sheffield.ac.uk> wrote:

> Hello,
>
> Sorry for the late reply. I have downloaded joshua again and followed
> the updated procedure, but I still get the same error when running the
> following command:
>
> $JOSHUA/bin/pipeline.pl \
>  --rundir 1 \
>  --readme "Baseline Hiero run" \
>  --source es \
>  --target en \
>  --type hiero \
>  --corpus $FISHER/corpus/asr/fisher_train \
>  --tune $FISHER/corpus/asr/fisher_dev \
>  --test $FISHER/corpus/asr/fisher_dev2 \
>  --maxlen 11 \
>  --maxlen-tune 11 \
>  --maxlen-test 11 \
>  --tuner-iterations 1 \
>  --lm-order 3
>
> The error is still:
> [pack-grammar] rebuilding...
>   dep= 

Re: Error while running the tutorial

[Matt Post]

eigen3 is not necessary. What is the file size of grammar.gz?


> On Nov 22, 2016, at 7:54 AM, Fernando E Alva Manchego 
>  wrote:
> 
> Hello,
> 
> Well, I ran that command and it went fine: build 100%
> 
> However, now I ran the tutorial command again and I get:
> 
> * Packing grammar at "grammar.gz" to 
> "../joshua-tutorial/runs/1/tune/model/grammar.gz.packed"
> * Running the grammar-packer.pl  script with the 
> command: $JOSHUA/scripts/support/grammar-packer.pl 
>  -a -T /tmp -g grammar.gz -o 
> ../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
> Exception in thread "main" java.util.NoSuchElementException
>   at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
>   at 
> org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
>   at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>   at 
> org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
>   at 
> org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
> * FATAL: Couldn't pack the grammar.
> * Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
> * __init__() takes at least 3 arguments (2 given)
> 
> One thing I noticed is this "error" message when compiling:
> 
> -- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR EIGEN3_VERSION_OK) 
> (Required is at least version "2.91.0")
> CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
>   Not building interpolation.  Eigen3 was not found.
> 
> Is Eigen3 really necessary? 
> 
> Cheers,
> Fernando 
> 
> On 18 November 2016 at 18:15, Matt Post  > wrote:
> Okay, it looks like KenLM is not building. This is a perennial pain. You can 
> see the KenLM build lines in download_deps.sh. What is output when you run 
> 
>   ./jni/build_kenlm.sh
> 
> matt
> 
> 
> 
>> On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego 
>> > 
>> wrote:
>> 
>> Hello,
>> 
>> UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that 
>> libken.so is there. Now, I run the command again and what I get is the same 
>> error that Lewis pointed out:
>> 
>> [lm-sort-uniq] rebuilding...
>>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
>>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en.uniq [NOT FOUND]
>>   cmd= $JOSHUA/scripts/training/scat 
>> /export/data/falva/joshua-tutorial/runs/1/data/train/corpus.en | sort -u -T 
>> /tmp -S 8G | gzip -9n >.../joshua-tutorial/runs/1/data/train/corpus.en.uniq
>>   took 1 seconds (1s)
>> * FATAL: $JOSHUA/bin/lmplz (for building LMs) does not exist.
>>   This is often a problem with the boost libraries (particularly threaded
>>   versus unthreaded).
>> 
>> Cheers,
>> Fernando 
>> 
>> On 18 November 2016 at 16:40, Fernando E Alva Manchego 
>> > 
>> wrote:
>> Hello,
>> 
>> Sorry for the late reply. I have downloaded joshua again and followed the 
>> updated procedure, but I still get the same error when running the following 
>> command:
>> 
>> $JOSHUA/bin/pipeline.pl  \
>>--rundir 1 \
>>--readme "Baseline Hiero run" \
>>--source es \
>>--target en \
>>--type hiero \
>>--corpus $FISHER/corpus/asr/fisher_train \
>>--tune $FISHER/corpus/asr/fisher_dev \
>>--test $FISHER/corpus/asr/fisher_dev2 \
>>--maxlen 11 \
>>--maxlen-tune 11 \
>>--maxlen-test 11 \
>>--tuner-iterations 1 \
>>--lm-order 3
>> 
>> The error is still:
>> [pack-grammar] rebuilding...
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/vocabulary [NOT FOUND]
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/encoding [NOT FOUND]
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/slice_0.source [NOT 
>> FOUND]
>>   cmd= $JOSHUA/scripts/support/grammar-packer.pl  
>> -a -T /tmp -m 8g -g grammar.gz -o $HOME/joshua-tutorial/runs/1/grammar.packed
>>   JOB FAILED (return code 1)
>> Exception in thread "main" java.util.NoSuchElementException
>>  at org.apache.joshua.util.io 
>> .LineReader.next(LineReader.java:276)
>>  at 
>> org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
>>  at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>>  at 
>> org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
>>  at 
>> org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
>> * FATAL: Couldn't pack the grammar.
>> * Copying sorted grammars (/tmp/grammar.gzTQzG) to current directory.
>> 
>> What I have noticed now is that, when running the tests after compilation, 
>> this error message appears:
>> 
>> ERROR - Can't find 

Re: Unable to run the language packs - facing some errors

[Matt Post]

Hi,

Please send support requests to u...@joshua.apache.org 
. 

You need to increase the memory. Edit the joshua script and try doubling it. 
Let us know. 

matt (from my phone)



> On Nov 21, 2016, at 7:41 AM, Dixon Daniel  wrote:
> 
> Hi,
> 
> I am trying to run the German to English Language pack but I get the error 
> shown below:
> 
> dixon@HOME:~/Joshua/apache-joshua-de-en-2016-11-18$ cat example.de 
>  | ./prepare.sh | ./joshua
> Exception in thread "main" java.lang.RuntimeException: Unable to instantiate 
> feature function 'LanguageModel -lm_type berkeleylm -lm_order 4 -lm_file 
> model/lm.berkeleylm'!
> at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:632)
> at org.apache.joshua.decoder.Decoder.initialize(Decoder.java:394)
> at org.apache.joshua.decoder.Decoder.(Decoder.java:128)
> at org.apache.joshua.decoder.JoshuaDecoder.main(JoshuaDecoder.java:69)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.joshua.decoder.Decoder.initializeFeatureFunctions(Decoder.java:628)
> ... 3 more
> Caused by: java.lang.OutOfMemoryError: Java heap space
> at java.lang.reflect.Array.newArray(Native Method)
> at java.lang.reflect.Array.newInstance(Array.java:75)
> at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1678)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at 
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2018)
> at 
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1942)
> at 
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1808)
> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1353)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:373)
> at edu.berkeley.nlp.lm.io.IOUtils.readObjFile(IOUtils.java:139)
> at edu.berkeley.nlp.lm.io.IOUtils.readObjFileHard(IOUtils.java:164)
> at edu.berkeley.nlp.lm.io.IOUtils.readObjFileHard(IOUtils.java:159)
> at edu.berkeley.nlp.lm.io.LmReaders.readLmBinary(LmReaders.java:337)
> at 
> org.apache.joshua.decoder.ff.lm.berkeley_lm.LMGrammarBerkeley.(LMGrammarBerkeley.java:87)
> at 
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.initializeLM(LanguageModelFF.java:158)
> at 
> org.apache.joshua.decoder.ff.lm.LanguageModelFF.(LanguageModelFF.java:132)
> ... 8 more
> 
> Could you please help me resolve this error?
> 
> Thanks,
> Dixon
> 
> Have a great day!

Re: Error while running the tutorial

[Fernando E Alva Manchego]

Hello,

Well, I ran that command and it went fine: build 100%

However, now I ran the tutorial command again and I get:

* Packing grammar at "grammar.gz" to
"../joshua-tutorial/runs/1/tune/model/grammar.gz.packed"
* Running the grammar-packer.pl script with the command:
$JOSHUA/scripts/support/grammar-packer.pl -a -T /tmp -g grammar.gz -o
../joshua-tutorial/runs/1/tune/model/grammar.gz.packed
Exception in thread "main" java.util.NoSuchElementException
at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
at
org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
* FATAL: Couldn't pack the grammar.
* Copying sorted grammars (/tmp/grammar.gzR7NI) to current directory.
* __init__() takes at least 3 arguments (2 given)

One thing I noticed is this "error" message when compiling:

-- Could NOT find Eigen3 (missing:  EIGEN3_INCLUDE_DIR EIGEN3_VERSION_OK)
(Required is at least version "2.91.0")
CMake Warning at lm/interpolate/CMakeLists.txt:65 (message):
  Not building interpolation.  Eigen3 was not found.

Is Eigen3 really necessary?

Cheers,
Fernando

On 18 November 2016 at 18:15, Matt Post  wrote:

> Okay, it looks like KenLM is not building. This is a perennial pain. You
> can see the KenLM build lines in download_deps.sh. What is output when you
> run
>
> ./jni/build_kenlm.sh
>
> matt
>
>
>
> On Nov 18, 2016, at 12:24 PM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
> Hello,
>
> UPDATE:  I added $JOSHUA/lib to LD_LIBRARY_PATH  because I saw that libken.so
> is there. Now, I run the command again and what I get is the same error
> that Lewis pointed out:
>
> [lm-sort-uniq] rebuilding...
>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en [CHANGED]
>   dep= ../joshua-tutorial/runs/1/data/train/corpus.en.uniq [NOT FOUND]
>   cmd= $JOSHUA/scripts/training/scat /export/data/falva/joshua-
> tutorial/runs/1/data/train/corpus.en | sort -u -T /tmp -S 8G | gzip -9n
> >.../joshua-tutorial/runs/1/data/train/corpus.en.uniq
>   took 1 seconds (1s)
> * FATAL: $JOSHUA/bin/lmplz (for building LMs) does not exist.
>   This is often a problem with the boost libraries (particularly threaded
>   versus unthreaded).
>
> Cheers,
> Fernando
>
> On 18 November 2016 at 16:40, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
>> Hello,
>>
>> Sorry for the late reply. I have downloaded joshua again and followed the
>> updated procedure, but I still get the same error when running the
>> following command:
>>
>> $JOSHUA/bin/pipeline.pl \
>>  --rundir 1 \
>>  --readme "Baseline Hiero run" \
>>  --source es \
>>  --target en \
>>  --type hiero \
>>  --corpus $FISHER/corpus/asr/fisher_train \
>>  --tune $FISHER/corpus/asr/fisher_dev \
>>  --test $FISHER/corpus/asr/fisher_dev2 \
>>  --maxlen 11 \
>>  --maxlen-tune 11 \
>>  --maxlen-test 11 \
>>  --tuner-iterations 1 \
>>  --lm-order 3
>>
>> The error is still:
>> [pack-grammar] rebuilding...
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/vocabulary [NOT FOUND]
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/encoding [NOT FOUND]
>>   dep= $HOME/joshua-tutorial/runs/1/grammar.packed/slice_0.source
>> [NOT FOUND]
>>   cmd= $JOSHUA/scripts/support/grammar-packer.pl -a -T /tmp -m 8g -g
>> grammar.gz -o $HOME/joshua-tutorial/runs/1/grammar.packed
>>   JOB FAILED (return code 1)
>> Exception in thread "main" java.util.NoSuchElementException
>> at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
>> at org.apache.joshua.tools.GrammarPacker.getGrammarReader(Gramm
>> arPacker.java:239)
>> at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
>> at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCl
>> i.java:120)
>> at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerC
>> li.java:137)
>> * FATAL: Couldn't pack the grammar.
>> * Copying sorted grammars (/tmp/grammar.gzTQzG) to current directory.
>>
>> What I have noticed now is that, when running the tests after
>> compilation, this error message appears:
>>
>> ERROR - Can't find libken.so (libken.dylib on OS X) on the Java library
>> path.
>> WARN - No glue grammar found! Creating dummy glue grammar.
>>
>> Could that be the source of the error? Thank you.
>>
>> @Lewis: I'll make sure to given them your regards.
>>
>> Best
>> Fernando
>>
>> On 18 November 2016 at 13:42, Matt Post  wrote:
>>
>>> I just updated that page to use "mvn package" instead of the old "mvn
>>> compile assembly:single". So Fernando, please make sure you follow the
>>> updated instructions.
>>>
>>>
>>>
>>>
>>> On Nov 17, 2016, at 10:10 PM, lewis john mcgibbney 
>>> wrote:
>>>
>>> Hi Fernando,
>>> First and foremost please give y regards to the GATE team at 

Re: API for Language Pack Server

[Matt Post]

We will document the HTTP/REST server soon. In the meantime you can see how the 
ajax queries are formed in the web demo (under web/). But that is limited to 
one sentence per connection, so it will be lots faster if you use the TCP 
server mode, which is documented in the README.

./joshua -server-port 5674 -server-type tcp

You will also likely want to increase memory (-m, must be the first argument) 
and to use multiple threads (-threads N, where N is no more than the number of 
cores on your machine, and should probably be one or two less).

matt


> On Nov 21, 2016, at 2:14 AM, Siddharth Dinesh  wrote:
> 
> Hi, 
> 
> I am working on translating 2-3 million documents. For this, I have set up 
> the language pack, and ran Joshua in server mode as well.
> 
> Can I have some information about the API to use to translate using the HTTP 
> server mode?
> 
> Thanks,
> Siddharth

Re: [RESULTS] WAS Re: [VOTE] Release Apache Joshua (Incubating) 6.1

[lewis john mcgibbney]

Hi Matt,
replies inline
On Fri, Nov 18, 2016 at 5:54 PM, <
user-digest-h...@joshua.incubator.apache.org> wrote:

>
> From: Matt Post <p...@cs.jhu.edu>
> To: user@joshua.incubator.apache.org
> Cc: d...@joshua.incubator.apache.org
> Date: Fri, 18 Nov 2016 17:10:17 -0500
> Subject: Re: [RESULTS] WAS Re: [VOTE] Release Apache Joshua (Incubating)
> 6.1
> Lewis,
>
> Thanks for heading this up. Is it okay to use the latest master for the
> 6.1 release?
>


The RC is based off of the tag which was cut at
https://git-wip-us.apache.org/repos/asf?p=incubator-joshua.git;a=commit;h=167489bbd78526b9833fe7c88646bf96101d5d2b
So whatever was in the code at that stage will be in the release candidate
being reviewed by general@incubator.


> Or would that require a new vote?
>

yes it would require a new VOTE as essentially we are VOTE'ing off of a
different codebase (albeit a very similar one). If needs be, we can easily
roll a 6.1.1... we will see how things go over on general@, we may end up
having to implement changes to the RC anyways... in which case I'll cut it
off of master branch again and your changes will be incorporated Matt.


> I think I'm done making changes in the near term, now that the language
> packs are out.
>
>
> Yes, excellent job from you and your colleagues at JHU. What a sterling
job. Some really interesting language combinations there. I noticed you
didn't have Gaelic ;) I'll need to work on that for my next adventure :)

Enjoy the weekend team.. over and out.
Lewis

Re: [VOTE] Release Apache Joshua (Incubating) 6.1

[David Meikle]

+1 

Cheers,
Dave

> On 14 Nov 2016, at 17:16, lewis john mcgibbney  wrote:
> 
> [ ] +1, let's get it released!!!
> [ ] +/-0, fine, but consider to fix few issues before...
> [ ] -1, nope, because... (and please explain why)

Re: [RESULTS] WAS Re: [VOTE] Release Apache Joshua (Incubating) 6.1

[Mattmann, Chris A (3010)]

I will try and VOTE today sorry for the delay great job Lewis.

++
Chris Mattmann, Ph.D.
Principal Data Scientist, Engineering Administrative Office (3010)
Manager, Open Source Projects Formulation and Development Office (8212)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 180-503E, Mailstop: 180-503
Email: chris.a.mattm...@nasa.gov<mailto:chris.a.mattm...@nasa.gov>
WWW:  http://sunset.usc.edu/~mattmann/
++
Director, Information Retrieval and Data Science Group (IRDS)
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
WWW: http://irds.usc.edu/
++


From: lewis john mcgibbney <lewi...@apache.org>
Reply-To: "user@joshua.incubator.apache.org" <user@joshua.incubator.apache.org>
Date: Friday, November 18, 2016 at 2:09 PM
To: "d...@joshua.incubator.apache.org" <d...@joshua.incubator.apache.org>, 
"user@joshua.incubator.apache.org" <user@joshua.incubator.apache.org>
Subject: [RESULTS] WAS Re: [VOTE] Release Apache Joshua (Incubating) 6.1

Hi Team,
OK so I am going to bring this VOTE to a close... 72 hours has come and gone.
The VOTE's are in and are as follows

[8] +1, let's get it released!!!
Lewis John McGibbney**
Paul M. Ramirez**
Matt Post*
John Hewitt*
Tommaso Teofili**
Thamme Gowda*
Kellen Sunderland*
Felix Hieber*

[0] +/-0, fine, but consider to fix few issues before...
[0] -1, nope, because... (and please explain why)
* = Joshua PPMC
** Joshua PPMC + Incubator PMC
Thank you to everyone that was able to VOTE, much appreciated. Also thank you 
everyone that was able to contribute towards Joshua in the past. The project is 
looking in great shape and the release of the language packs in just killer.
I'll progress with the VOTE over on general@incubator.
See you there!
Lewis

On Mon, Nov 14, 2016 at 9:16 AM, lewis john mcgibbney 
<lewi...@apache.org<mailto:lewi...@apache.org>> wrote:
Hi Folks,
Please VOTE on the Apache Joshua 6.1 Release Candidate #1.

We solved 44 issues: https://s.apache.org/joshua6.1

Git source tag (167489bbd78526b9833fe7c88646bf96101d5d2b): 
https://s.apache.org/joshua6.1tag

Staging repo: 
https://repository.apache.org/content/repositories/orgapachejoshua-1000/

Source Release Artifacts: 
https://dist.apache.org/repos/dist/dev/incubator/joshua/

PGP release keys (signed using 48BAEBF6): 
https://dist.apache.org/repos/dist/release/incubator/joshua/KEYS

Vote will be open for 72 hours.
Thank you to everyone that is able to VOTE as well as everyone that contributed 
to Apache Joshua 6.1.

[ ] +1, let's get it released!!!
[ ] +/-0, fine, but consider to fix few issues before...
[ ] -1, nope, because... (and please explain why)
P.S. here is my +1

--
http://home.apache.org/~lewismc/<http://home.apache.org/%7Elewismc/>
@hectorMcSpector
http://www.linkedin.com/in/lmcgibbney



--
http://home.apache.org/~lewismc/
@hectorMcSpector
http://www.linkedin.com/in/lmcgibbney

Re: [VOTE] Release Apache Joshua (Incubating) 6.1

[Thamme Gowda]

+1

Thamme

On Nov 15, 2016 6:18 AM, "Felix Hieber"  wrote:

> +1
>
> Felix
>
> On Tue, Nov 15, 2016 at 10:55 AM Tommaso Teofili <
> tommaso.teof...@gmail.com>
> wrote:
>
> > +1
> >
> > Tommaso
> >
> > Il giorno lun 14 nov 2016 alle ore 19:16 John Hewitt <
> > john...@seas.upenn.edu>
> > ha scritto:
> >
> > > +1 Let's do it.
> > >
> > > -John
> > >
> > > On Mon, Nov 14, 2016 at 1:13 PM, kellen sunderland <
> > > kellen.sunderl...@gmail.com> wrote:
> > >
> > > +1 .  Thanks to Lewis and Matt for all the recent work.
> > >
> > > On Nov 14, 2016 7:11 PM, "Matt Post"  wrote:
> > >
> > > +1
> > >
> > > Thanks for starting this off, Lewis!
> > >
> > >
> > > > On Nov 14, 2016, at 12:54 PM, Ramirez, Paul M (398M) <
> > > paul.m.rami...@jpl.nasa.gov> wrote:
> > > >
> > > > +1, let's get it released!!!
> > > >
> > > > --Paul
> > > >
> > > > 
> ==
> > > > Paul Ramirez - Group Supervisor
> > > > Computer Science for Data Intensive Applications (398M)
> > > > NASA - Jet Propulsion Laboratory
> > > > 4800 Oak Grove Dr.
> > > > Pasadena, CA 91109 USA
> > > > Mailstop: 158-242
> > > > Office: 818-354-1015
> > > > Cell: 818-395-8194
> > > > 
> ==
> > > >
> > > > On 11/14/16, 9:16 AM, "lewis john mcgibbney" 
> > wrote:
> > > >
> > > >Hi Folks,
> > > >Please VOTE on the Apache Joshua 6.1 Release Candidate #1.
> > > >
> > > >We solved 44 issues: https://s.apache.org/joshua6.1
> > > >
> > > >Git source tag (167489bbd78526b9833fe7c88646bf96101d5d2b):
> > > >https://s.apache.org/joshua6.1tag
> > > >
> > > >Staging repo:
> > > >https://repository.apache.org/content/repositories/
> > > orgapachejoshua-1000/
> > > >
> > > >Source Release Artifacts:
> > > >https://dist.apache.org/repos/dist/dev/incubator/joshua/
> > > >
> > > >PGP release keys (signed using 48BAEBF6):
> > > >https://dist.apache.org/repos/dist/release/incubator/joshua/KEYS
> > > >
> > > >Vote will be open for 72 hours.
> > > >Thank you to everyone that is able to VOTE as well as everyone
> that
> > > >contributed to Apache Joshua 6.1.
> > > >
> > > >[ ] +1, let's get it released!!!
> > > >[ ] +/-0, fine, but consider to fix few issues before...
> > > >[ ] -1, nope, because... (and please explain why)
> > > >
> > > >P.S. here is my +1
> > > >
> > > >--
> > > >http://home.apache.org/~lewismc/
> > > >@hectorMcSpector
> > > >http://www.linkedin.com/in/lmcgibbney
> > > >
> > > >
> > >
> > >
> > >
> >
>

Re: Error while running the tutorial

[lewis john mcgibbney]

Hi Folks,
I submitted a PR to implement OS compatibility running on Mac OSX.
https://github.com/joshua-decoder/fisher-callhome-corpus/pull/1
@Fernando I cannot reproduce your issue...
I do however encounter the following issue

lmcgibbn@LMC-056430 /usr/local/joshua_resources/joshua-tutorial $
$JOSHUA/bin/pipeline.pl --rundir 1 --readme "Baseline Hiero run" --source
es --target en --type hiero --corpus $FISHER/corpus/ldc/fisher_train --tune
$FISHER/corpus/ldc/fisher_dev --test $FISHER/corpus/ldc/fisher_dev2
--maxlen 11 --maxlen-tune 11 --maxlen-test 11 --tuner-iterations 1
--lm-order 3
...
[thrax-run] rebuilding...

dep=/usr/local/joshua_resources/joshua-tutorial/1/data/train/thrax-input-file
[CHANGED]
  dep=thrax-hiero.conf [CHANGED]
  dep=grammar.gz [NOT FOUND]
  cmd=hadoop jar /usr/local/incubator-joshua/thrax/bin/thrax.jar -D
mapreduce.task.timeout=0 -D mapreduce.map.java.opts='-Xmx4g' -D
mapreduce.reduce.java.opts='-Xmx4g' -D hadoop.tmp.dir=/tmp thrax-hiero.conf
pipeline-es-en-hiero-_usr_local_joshua_resources_joshua-tutorial_1 >
thrax.log 2>&1; rm -f grammar grammar.gz; hadoop fs -cat
pipeline-es-en-hiero-_usr_local_joshua_resources_joshua-tutorial_1/final/*
| gzip -cd | /usr/local/incubator-joshua/scripts/training/filter-rules.pl
-t 100 | gzip -9n > grammar.gz
  took 201 seconds (3m21s)
16/11/17 20:09:04 WARN util.NativeCodeLoader: Unable to load native-hadoop
library for your platform... using builtin-java classes where applicable
16/11/17 20:09:05 INFO Configuration.deprecation: io.bytes.per.checksum is
deprecated. Instead, use dfs.bytes-per-checksum
16/11/17 20:09:05 INFO fs.TrashPolicyDefault: Namenode trash configuration:
Deletion interval = 0 minutes, Emptier interval = 0 minutes.
Deleted pipeline-es-en-hiero-_usr_local_joshua_resources_joshua-tutorial_1
[pack-grammar] rebuilding...

dep=/usr/local/joshua_resources/joshua-tutorial/1/grammar.packed/vocabulary
[NOT FOUND]
  dep=/usr/local/joshua_resources/joshua-tutorial/1/grammar.packed/encoding
[NOT FOUND]

dep=/usr/local/joshua_resources/joshua-tutorial/1/grammar.packed/slice_0.source
[NOT FOUND]
  cmd=/usr/local/incubator-joshua/scripts/support/grammar-packer.pl -a -T
/tmp -m 8g -g grammar.gz -o
/usr/local/joshua_resources/joshua-tutorial/1/grammar.packed
  took 80 seconds (1m20s)
[lm-sort-uniq] rebuilding...
  dep=/usr/local/joshua_resources/joshua-tutorial/1/data/train/corpus.en
[CHANGED]

dep=/usr/local/joshua_resources/joshua-tutorial/1/data/train/corpus.en.uniq
[NOT FOUND]
  cmd=/usr/local/incubator-joshua/scripts/training/scat
/usr/local/joshua_resources/joshua-tutorial/1/data/train/corpus.en | sort
-u -T /tmp -S 8G | gzip -9n >
/usr/local/joshua_resources/joshua-tutorial/1/data/train/corpus.en.uniq
  took 1 seconds (1s)
* FATAL: /usr/local/incubator-joshua/bin/lmplz (for building LMs) does not
exist.
  This is often a problem with the boost libraries (particularly threaded
  versus unthreaded).

We no longer have the $JOSHUA/bin/lmplz script... so we need to address
this.

On Thu, Nov 17, 2016 at 7:10 PM, lewis john mcgibbney 
wrote:

> Hi Fernando,
> First and foremost please give y regards to the GATE team at Sheffield. I
> spent a great week down there a number of years back and I am fond of the
> place.
> Are you following the tutorial at https://cwiki.apache.org/
> confluence/display/JOSHUA/Joshua+Tutorial ?
> If so then I'll try it out and see if I can reproduce.
> Lewis
>
> On Thu, Nov 17, 2016 at 9:38 AM,  incubator.apache.org> wrote:
>
>> From: Fernando E Alva Manchego 
>> To: user@joshua.incubator.apache.org
>> Cc:
>> Date: Thu, 17 Nov 2016 17:37:53 +
>> Subject: Error while running the tutorial
>> Hello!
>>
>> I'm running the tutorial (phrase) and the following error came up:
>>
>> Error: Could not find or load main class org.apache.joshua.tools.Gramma
>> rPackerCli
>>
>> When I installed Joshua, I ran the tests and everything was OK. Do you
>> have any idea what might be happening? Thank you.
>>
>>
>>


-- 
http://home.apache.org/~lewismc/
@hectorMcSpector
http://www.linkedin.com/in/lmcgibbney

Re: Error while running the tutorial

[Fernando E Alva Manchego]

Hello,

1. Here's the list requested:

archive-tmp
classes
generated-sources
joshua-6.2-SNAPSHOT.jar
joshua-6.2-SNAPSHOT-jar-with-dependencies.jar
joshua-6.2-SNAPSHOT-src.tar.gz
joshua-6.2-SNAPSHOT-src.zip
maven-archiver
maven-shared-archive-resources
rat.txt
surefire-reports
test-classes

2. After running that command the error message changes:
[pack-grammar] rebuilding...
  dep=.../joshua-tutorial/runs/1/grammar.packed/vocabulary [NOT FOUND]
  dep=.../joshua-tutorial/runs/1/grammar.packed/encoding [NOT FOUND]
  dep=.../joshua-tutorial/runs/1/grammar.packed/slice_0.source [NOT
FOUND]
  cmd=/joshua/scripts/support/grammar-packer.pl -a -T /tmp -m 8g -g
grammar.gz -o /joshua-tutorial/runs/1/grammar.packed
  JOB FAILED (return code 1)
Exception in thread "main" java.util.NoSuchElementException
at org.apache.joshua.util.io.LineReader.next(LineReader.java:276)
at
org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239)
at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184)
at org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120)
at org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137)
* FATAL: Couldn't pack the grammar.
* Copying sorted grammars (/tmp/grammar.gze1qd) to current directory.



Best,
Fernando Alva Manchego (PhD Candidate)
NLP Research Group - Department of Computer Science
University of Sheffield

On 17 November 2016 at 18:09, Matt Post  wrote:

> Please send a listing of the contents of the "$JOSHUA/target" directory.
>
> Try running "mvn package" in $JOSHUA, and see if that helps.
>
>
> On Nov 17, 2016, at 12:59 PM, Fernando E Alva Manchego <
> fealvamanche...@sheffield.ac.uk> wrote:
>
> Hello,
>
> Thanks for the quick reply:
>
> (a) I followed the instructions in the Getting Started page. I didn't have
> Maven in my computer, so I downloaded Maven 3.3.9. When I ran mvn test the
> message I got at the end was:
>
> Tests run: 151, Failures: 0, Errors: 0, Skipped: 0
>
> So that's why I assume everything was OK with the installation.
>
> (b) The command I typed was:
>
> $JOSHUA/bin/pipeline.pl \
>   --rundir 2 \
>   --readme "Baseline phrase run" \
>   --source es \
>   --target en \
>   --type phrase \
>   --corpus $FISHER/corpus/asr/fisher_train \
>   --tune $FISHER/corpus/asr/fisher_dev \
>   --test $FISHER/corpus/asr/fisher_dev2 \
>   --maxlen 11 \
>   --maxlen-tune 11 \
>   --maxlen-test 11 \
>   --tuner-iterations 1 \
>   --lm-order 3
>
> It's exactly as shown in the Tutorial page, except for the ldc folder
> which I changed to asr. After running that command, I get (at the end of a
> long list of other commandas):
>
> dep= .../joshua-tutorial/runs/1/grammar.packed/vocabulary [NOT FOUND]
>   dep= .../joshua-tutorial/runs/1/grammar.packed/encoding [NOT FOUND]
>   dep= ... /joshua-tutorial/runs/1/grammar.packed/slice_0.source [NOT
> FOUND]
>   cmd= .../joshua/scripts/support/grammar-packer.pl -a -T /tmp -m 8g -g
> grammar.gz -o /export/data/falva/joshua-tutorial/runs/1/grammar.packed
>   JOB FAILED (return code 1)
> Error: Could not find or load main class org.apache.joshua.tools.
> GrammarPackerCli
> * FATAL: Couldn't pack the grammar.
> * Copying sorted grammars (/tmp/grammar.gzL3kB) to current directory
>
> Any suggestion is more than welcomed. Thanks.
>
> Best,
> Fernando
>
> On 17 November 2016 at 17:50, Matt Post  wrote:
>
>> Hello,
>>
>> Thanks for the note. Can you please tell us:
>>
>> (a) how you installed Joshua?
>>
>> (b) the command you typed that failed?
>>
>> matt
>>
>>
>> On Nov 17, 2016, at 12:37 PM, Fernando E Alva Manchego <
>> fealvamanche...@sheffield.ac.uk> wrote:
>>
>> Hello!
>>
>> I'm running the tutorial (phrase) and the following error came up:
>>
>> Error: Could not find or load main class org.apache.joshua.tools.Gramma
>> rPackerCli
>>
>> When I installed Joshua, I ran the tests and everything was OK. Do you
>> have any idea what might be happening? Thank you.
>>
>> Best,
>> Fernando
>>
>>
>>
>
>

Re: Building Language Pack's without Mandatory Arguments

[Mattmann, Chris A (3980)]

Matt you had built me a specialized one and I think it lives on 
our xdata machine somewhere I’ll take a look but if you still have
it that would be grand

++
Chris Mattmann, Ph.D.
Chief Architect
Instrument Software and Science Data Systems Section (398)
NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
Office: 168-519, Mailstop: 168-527
Email: chris.a.mattm...@nasa.gov
WWW:  http://sunset.usc.edu/~mattmann/
++
Director, Information Retrieval and Data Science Group (IRDS)
Adjunct Associate Professor, Computer Science Department
University of Southern California, Los Angeles, CA 90089 USA
WWW: http://irds.usc.edu/
++









On 4/5/16, 12:46 PM, "Matt Post"  wrote:

>
>
>
>Can you not use the posted Spanish → English language pack?
>
>
>
>
>On Apr 5, 2016, at 12:34 PM, Lewis John Mcgibbney  
>wrote:
>
>That was my concern Matt. Thanks for the explanation.
>Basically I've been passed a packed grammar for Spanish --> English 
>translation (possibly one you built for Chris M. While back) and wondered if I 
>could package it into a language pack. It seems that without the accompanying 
>parameters we've discussed
> its usefulness is questionable.
>
>On Tuesday, April 5, 2016, Matt Post  wrote:
>
>Hi Lewis,
>
>
>I would suggest taking another language pack and just copying that over. You 
>can delete the language model and grammars from it, and then edit the 
>joshua.config to do the following:
>
>
>- Remove the language model lines: 
>
>
>feature-function = StateMinimizingLanguageModel ...
>feature-function = LanguageModel ...
>
>
>- Modify the ^tm lines to point to your new grammar. You will want to keep the 
>glue grammar. Also, the tm specification should match the type of grammar you 
>learned (set to "thrax" if it was a hiero grammar, or to "phrase" or "moses" 
>if a phrase
> or moses one).
>
>
>That said, what are you doing this for? It's unlikely to work very well 
>without a language model, and without tuned parameters.
>
>
>matt
>
>
>
>
>
>On Apr 5, 2016, at 12:54 AM, Lewis John Mcgibbney > wrote:
>
>Hi Folks,
>
>Does anyone know if it is possible to bundle a language pack without input for 
>the mandatory arguments as documented at [0]?
>
>Say for example I was passed a packed grammar which was built from a pipeline 
>execution however no accompanying configuration file, the tuned configuration 
>file, etc. was provided as well.
>
>Any help is appreciated.
>
>Thanks
>
>Lewis
>
>[0] 
>http://joshua-decoder.org/6.0/bundle.html 
>
>
>-- 
>Lewis
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>-- 
>Lewis
>
>
>
>
>
>
>

Re: Building Language Pack's without Mandatory Arguments

[Lewis John Mcgibbney]

Yep, I sure can. That's the one built in to the brew package as well.
This email was just a curiosity. It looks like it would be very difficult
to transform a packed grammar to a language pack without the accompanying
parameters.thanks for the heads up Matt.

On Tuesday, April 5, 2016, Matt Post  wrote:

> Can you not use the posted Spanish → English language pack?
>
>
> On Apr 5, 2016, at 12:34 PM, Lewis John Mcgibbney <
> lewis.mcgibb...@gmail.com
> > wrote:
>
> That was my concern Matt. Thanks for the explanation.
> Basically I've been passed a packed grammar for Spanish --> English
> translation (possibly one you built for Chris M. While back) and wondered
> if I could package it into a language pack. It seems that without the
> accompanying parameters we've discussed its usefulness is questionable.
>
> On Tuesday, April 5, 2016, Matt Post  > wrote:
>
>> Hi Lewis,
>>
>> I would suggest taking another language pack and just copying that over.
>> You can delete the language model and grammars from it, and then edit the
>> joshua.config to do the following:
>>
>> - Remove the language model lines:
>>
>> feature-function = StateMinimizingLanguageModel ...
>> feature-function = LanguageModel ...
>>
>> - Modify the ^tm lines to point to your new grammar. You will want to
>> keep the glue grammar. Also, the tm specification should match the type of
>> grammar you learned (set to "thrax" if it was a hiero grammar, or to
>> "phrase" or "moses" if a phrase or moses one).
>>
>> That said, what are you doing this for? It's unlikely to work very well
>> without a language model, and without tuned parameters.
>>
>> matt
>>
>>
>> On Apr 5, 2016, at 12:54 AM, Lewis John Mcgibbney <
>> lewis.mcgibb...@gmail.com> wrote:
>>
>> Hi Folks,
>> Does anyone know if it is possible to bundle a language pack without
>> input for the mandatory arguments as documented at [0]?
>> Say for example I was passed a packed grammar which was built from a
>> pipeline execution however no accompanying configuration file, the tuned
>> configuration file, etc. was provided as well.
>> Any help is appreciated.
>> Thanks
>> Lewis
>>
>> [0] http://joshua-decoder.org/6.0/bundle.html
>>
>> --
>> *Lewis*
>>
>>
>>
>
> --
> *Lewis*
>
>
>

-- 
*Lewis*

Re: Building Language Pack's without Mandatory Arguments

[Matt Post]

Can you not use the posted Spanish → English language pack?


> On Apr 5, 2016, at 12:34 PM, Lewis John Mcgibbney  
> wrote:
> 
> That was my concern Matt. Thanks for the explanation.
> Basically I've been passed a packed grammar for Spanish --> English 
> translation (possibly one you built for Chris M. While back) and wondered if 
> I could package it into a language pack. It seems that without the 
> accompanying parameters we've discussed its usefulness is questionable.
> 
> On Tuesday, April 5, 2016, Matt Post  > wrote:
> Hi Lewis,
> 
> I would suggest taking another language pack and just copying that over. You 
> can delete the language model and grammars from it, and then edit the 
> joshua.config to do the following:
> 
> - Remove the language model lines: 
> 
>   feature-function = StateMinimizingLanguageModel ...
>   feature-function = LanguageModel ...
> 
> - Modify the ^tm lines to point to your new grammar. You will want to keep 
> the glue grammar. Also, the tm specification should match the type of grammar 
> you learned (set to "thrax" if it was a hiero grammar, or to "phrase" or 
> "moses" if a phrase or moses one).
> 
> That said, what are you doing this for? It's unlikely to work very well 
> without a language model, and without tuned parameters.
> 
> matt
> 
> 
>> On Apr 5, 2016, at 12:54 AM, Lewis John Mcgibbney > > wrote:
>> 
>> Hi Folks,
>> Does anyone know if it is possible to bundle a language pack without input 
>> for the mandatory arguments as documented at [0]?
>> Say for example I was passed a packed grammar which was built from a 
>> pipeline execution however no accompanying configuration file, the tuned 
>> configuration file, etc. was provided as well.
>> Any help is appreciated.
>> Thanks
>> Lewis
>> 
>> [0] http://joshua-decoder.org/6.0/bundle.html 
>> 
>> 
>> -- 
>> Lewis 
> 
> 
> 
> -- 
> Lewis 
>