It looks like grammar sorting is failing. Check the logs to see why. Delete grammar.* and try again from that step.
> On Aug 14, 2017, at 10:49 AM, Arezoo Arjomand <arezooarjom...@yahoo.com > <mailto:arezooarjom...@yahoo.com>> wrote: > > Hi, > I add "--aligner giza" to terminal command. It seems the alignment error is > fixed but the grammer error is still remain both for berkeley aligner and > giza. grammar.gz is empty and the runing dir is attached. > > > > > On Monday, August 14, 2017 2:08 AM, Matt Post <p...@cs.jhu.edu > <mailto:p...@cs.jhu.edu>> wrote: > > > It looks like alignment failed. Is there a file alignments/training.align? > That is build from the two pieces, under alignments/0/giza.SRC-TRG (and > TRG-SRC) that failed. > > >> On Aug 13, 2017, at 7:21 PM, Arezoo Arjomand <arezooarjom...@yahoo.com >> <mailto:arezooarjom...@yahoo.com>> wrote: >> >> Hi, >> When I run the pipleline the following error is shown. The previous error , >> write in the previous email, is shown when i run the same dir for second >> time and grammar.gz is empty. >> how can i fix the folloewing error? >> >> [source-numlines] rebuilding... >> dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es >> <http://corpus.es/> [CHANGED] >> cmd=cat /home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es >> <http://corpus.es/> | wc -l >> took 0 seconds (0s) >> [source-numlines] retrieved cached result => 77457 >> [giza-0] rebuilding... >> dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus.es >> <http://corpus.es/> [CHANGED] >> dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus.en >> [CHANGED] >> dep=alignments/0/model/aligned.grow-diag-final [NOT FOUND] >> cmd=rm -f alignments/0/corpus.0-0.*; >> /home/arezoo1/joshua-tutorial/joshua/scripts/training/run-giza.pl --root-dir >> alignments/0 -e en -f es -corpus >> /home/arezoo1/joshua-tutorial/runs/02/data/train/splits/0/corpus -merge >> grow-diag-final > alignments/0/giza.log 2>&1 >> *** Error in `/home/arezoo1/joshua-tutorial/joshua/ext/symal/symal': double >> free or corruption (out): 0x0000556a69b42160 *** >> ======= Backtrace: ========= >> /lib/x86_64-linux-gnu/libc.so.6(+0x7908b)[0x7f91d0fb908b] >> /lib/x86_64-linux-gnu/libc.so.6(+0x826fa)[0x7f91d0fc26fa] >> /lib/x86_64-linux-gnu/libc.so.6(cfree+0x4c)[0x7f91d0fc612c] >> /home/arezoo1/joshua-tutorial/joshua/ext/symal/symal(+0x2b5a)[0x556a6993ab5a] >> /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7f91d0f603f1] >> /home/arezoo1/joshua-tutorial/joshua/ext/symal/symal(+0x5f4a)[0x556a6993df4a] >> ======= Memory map: ======== >> 556a69938000-556a69941000 r-xp 00000000 08:0a 1051501 >> /home/arezoo1/joshua-tutorial/joshua/ext/symal/symal >> 556a69b41000-556a69b42000 r--p 00009000 08:0a 1051501 >> /home/arezoo1/joshua-tutorial/joshua/ext/symal/symal >> 556a69b42000-556a69b43000 rw-p 0000a000 08:0a 1051501 >> /home/arezoo1/joshua-tutorial/joshua/ext/symal/symal >> 556a69b43000-556a69b45000 rw-p 00000000 00:00 0 >> 556a6af09000-556a6afbf000 rw-p 00000000 00:00 0 >> [heap] >> 7f91cc000000-7f91cc021000 rw-p 00000000 00:00 0 >> 7f91cc021000-7f91d0000000 ---p 00000000 00:00 0 >> 7f91d0c37000-7f91d0d3f000 r-xp 00000000 08:0a 1708999 >> /lib/x86_64-linux-gnu/libm-2.24.so >> 7f91d0d3f000-7f91d0f3e000 ---p 00108000 08:0a 1708999 >> /lib/x86_64-linux-gnu/libm-2.24.so >> 7f91d0f3e000-7f91d0f3f000 r--p 00107000 08:0a 1708999 >> /lib/x86_64-linux-gnu/libm-2.24.so >> 7f91d0f3f000-7f91d0f40000 rw-p 00108000 08:0a 1708999 >> /lib/x86_64-linux-gnu/libm-2.24.so >> 7f91d0f40000-7f91d10fd000 r-xp 00000000 08:0a 1708931 >> /lib/x86_64-linux-gnu/libc-2.24.so >> 7f91d10fd000-7f91d12fd000 ---p 001bd000 08:0a 1708931 >> /lib/x86_64-linux-gnu/libc-2.24.so >> 7f91d12fd000-7f91d1301000 r--p 001bd000 08:0a 1708931 >> /lib/x86_64-linux-gnu/libc-2.24.so >> 7f91d1301000-7f91d1303000 rw-p 001c1000 08:0a 1708931 >> /lib/x86_64-linux-gnu/libc-2.24.so >> 7f91d1303000-7f91d1307000 rw-p 00000000 00:00 0 >> 7f91d1307000-7f91d131d000 r-xp 00000000 08:0a 1708971 >> /lib/x86_64-linux-gnu/libgcc_s.so.1 >> 7f91d131d000-7f91d151c000 ---p 00016000 08:0a 1708971 >> /lib/x86_64-linux-gnu/libgcc_s.so.1 >> 7f91d151c000-7f91d151d000 r--p 00015000 08:0a 1708971 >> /lib/x86_64-linux-gnu/libgcc_s.so.1 >> 7f91d151d000-7f91d151e000 rw-p 00016000 08:0a 1708971 >> /lib/x86_64-linux-gnu/libgcc_s.so.1 >> 7f91d151e000-7f91d1697000 r-xp 00000000 08:0a 1976366 >> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22 >> 7f91d1697000-7f91d1896000 ---p 00179000 08:0a 1976366 >> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22 >> 7f91d1896000-7f91d18a0000 r--p 00178000 08:0a 1976366 >> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22 >> 7f91d18a0000-7f91d18a2000 rw-p 00182000 08:0a 1976366 >> /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22 >> 7f91d18a2000-7f91d18a6000 rw-p 00000000 00:00 0 >> 7f91d18a6000-7f91d18cb000 r-xp 00000000 08:0a 1708903 >> /lib/x86_64-linux-gnu/ld-2.24.so >> 7f91d1aa6000-7f91d1aaa000 rw-p 00000000 00:00 0 >> 7f91d1ac7000-7f91d1acb000 rw-p 00000000 00:00 0 >> 7f91d1acb000-7f91d1acc000 r--p 00025000 08:0a 1708903 >> /lib/x86_64-linux-gnu/ld-2.24.so >> 7f91d1acc000-7f91d1acd000 rw-p 00026000 08:0a 1708903 >> /lib/x86_64-linux-gnu/ld-2.24.so >> 7f91d1acd000-7f91d1ace000 rw-p 00000000 00:00 0 >> 7ffc675c9000-7ffc675ea000 rw-p 00000000 00:00 0 >> [stack] >> 7ffc675f4000-7ffc675f6000 r--p 00000000 00:00 0 >> [vvar] >> 7ffc675f6000-7ffc675f8000 r-xp 00000000 00:00 0 >> [vdso] >> ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 >> [vsyscall] >> JOB FAILED (return code 2) >> [aligner-combine] rebuilding... >> dep=alignments/0/model/aligned.grow-diag-final [CHANGED] >> dep=alignments/training.align [NOT FOUND] >> cmd=cat alignments/0/model/aligned.grow-diag-final > >> alignments/training.align >> took 0 seconds (0s) >> [thrax-input-file] rebuilding... >> dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es >> <http://corpus.es/> [CHANGED] >> dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.en [CHANGED] >> dep=alignments/training.align [CHANGED] >> dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/thrax-input-file [NOT >> FOUND] >> cmd=/home/arezoo1/joshua-tutorial/joshua/scripts/training/paste >> /home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.es >> <http://corpus.es/> >> /home/arezoo1/joshua-tutorial/runs/02/data/train/corpus.en >> alignments/training.align | perl -pe 's/\t/ ||| /g' | grep -v '()' | grep -v >> '||| \+$' > /home/arezoo1/joshua-tutorial/runs/02/data/train/thrax-input-file >> took 1 seconds (1s) >> [thrax-prep] rebuilding... >> dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/thrax-input-file >> [CHANGED] >> dep=grammar.gz [NOT FOUND] >> cmd=hadoop fs -rm -r >> pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02; hadoop fs >> -mkdir pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02; hadoop >> fs -put /home/arezoo1/joshua-tutorial/runs/02/data/train/thrax-input-file >> pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02/input-file >> took 4 seconds (4s) >> [thrax-run] rebuilding... >> dep=/home/arezoo1/joshua-tutorial/runs/02/data/train/thrax-input-file >> [CHANGED] >> dep=thrax-phrase.conf [CHANGED] >> dep=grammar.gz [NOT FOUND] >> cmd=hadoop jar /home/arezoo1/joshua-tutorial/joshua/thrax/bin/thrax.jar -D >> mapreduce.task.timeout=0 -D mapreduce.map.java.opts='-Xmx4g' -D >> mapreduce.reduce.java.opts='-Xmx4g' -D hadoop.tmp.dir=/tmp thrax-phrase.conf >> pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02 > thrax.log >> 2>&1; rm -f grammar grammar.gz; hadoop fs -cat >> pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02/final/* | gzip >> -cd | /home/arezoo1/joshua-tutorial/joshua/scripts/training/filter-rules.pl >> -t 100 | gzip -9n > grammar.gz >> took 28 seconds (28s) >> 17/08/13 13:16:06 INFO Configuration.deprecation: io.bytes.per.checksum is >> deprecated. Instead, use dfs.bytes-per-checksum >> 17/08/13 13:16:06 INFO fs.TrashPolicyDefault: Namenode trash configuration: >> Deletion interval = 0 minutes, Emptier interval = 0 minutes. >> Deleted pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_02 >> [pack-grammar] rebuilding... >> dep=/home/arezoo1/joshua-tutorial/runs/02/grammar.packed/vocabulary [NOT >> FOUND] >> dep=/home/arezoo1/joshua-tutorial/runs/02/grammar.packed/encoding [NOT >> FOUND] >> >> dep=/home/arezoo1/joshua-tutorial/runs/02/grammar.packed/slice_00000.source >> [NOT FOUND] >> cmd=/home/arezoo1/joshua-tutorial/joshua/scripts/support/grammar-packer.pl >> -a -T /tmp -m 8g -g grammar.gz -o >> /home/arezoo1/joshua-tutorial/runs/02/grammar.packed >> JOB FAILED (return code 1) >> Exception in thread "main" java.util.NoSuchElementException >> at org.apache.joshua.util.io.LineReader.next(LineReader.java:276) >> at >> org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239) >> at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184) >> at >> org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120) >> at >> org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137) >> * FATAL: Couldn't pack the grammar. >> * Copying sorted grammars (/tmp/grammar.gzAte7) to current directory. >> >> >> >> On Saturday, August 12, 2017 2:03 AM, Matt Post <p...@cs.jhu.edu >> <mailto:p...@cs.jhu.edu>> wrote: >> >> >> You probably have an empty grammar. What's the file size of grammar.gz? >> >> >>> On Aug 8, 2017, at 12:40 PM, Arezoo Arjomand <arezooarjom...@yahoo.com >>> <mailto:arezooarjom...@yahoo.com>> wrote: >>> >>> Hi, >>> >>> When I run the pipeline the following error was accured. how can i fix it? >>> >>> Deleted pipeline-es-en-phrase-_home_arezoo1_joshua-tutorial_runs_2 >>> [pack-grammar] rebuilding... >>> dep=/home/arezoo1/joshua-tutorial/runs/2/grammar.packed/vocabulary [NOT >>> FOUND] >>> dep=/home/arezoo1/joshua-tutorial/runs/2/grammar.packed/encoding [NOT >>> FOUND] >>> >>> dep=/home/arezoo1/joshua-tutorial/runs/2/grammar.packed/slice_00000.source >>> [NOT FOUND] >>> >>> cmd=/home/arezoo1/joshua-tutorial/joshua/scripts/support/grammar-packer.pl >>> -a -T /tmp -m 8g -g grammar.gz -o >>> /home/arezoo1/joshua-tutorial/runs/2/grammar.packed >>> JOB FAILED (return code 1) >>> Exception in thread "main" java.util.NoSuchElementException >>> at org.apache.joshua.util.io.LineReader.next(LineReader.java:276) >>> at >>> org.apache.joshua.tools.GrammarPacker.getGrammarReader(GrammarPacker.java:239) >>> at org.apache.joshua.tools.GrammarPacker.pack(GrammarPacker.java:184) >>> at >>> org.apache.joshua.tools.GrammarPackerCli.run(GrammarPackerCli.java:120) >>> at >>> org.apache.joshua.tools.GrammarPackerCli.main(GrammarPackerCli.java:137) >>> * FATAL: Couldn't pack the grammar. >>> * Copying sorted grammars (/tmp/grammar.gzegdu) to current directory. >>> >> >> >> > > > > <920.zip>
