in case you didn't know, you should use the extract-rule argument
--GZOutput
this keeps disk io from maxing out which i've found is the limiting factor.
if your sort version also supports it,
-S 10G --batch-size 253 --compress-program gzip --parallel 10
my EMS executes this:
extract-parallel.perl 20 split "sort -S 10G --batch-size 253
--compress-program gzip " extract-rules corpus.3.cs corpus.3.en
aligned.3.grow-diag-final-and extract.3 --GlueGrammar glue-grammar.3
--MaxSpan 10 --GZOutput
this keeps
On 03/07/2012 21:43, Alexander Fraser wrote:
> On Tue, Jul 3, 2012 at 4:43 PM, Alex Fraser <[email protected]> wrote:
>> With Ken's changes, it builds with 1 50 (thanks!).
>>
>> But it now hangs as with 1 49.
>>
>> I guess it has something to do with threading. Running it through gdb,
>> I can see that a thread pool is started, despite the fact that threads
>> defaults to 1, this triggers a thread according to GDB. GDB also
>> claims that this thread exits. The hang occurs after the last line of
>> main.
>>
>> Cheers, Alex
>>
>>
>> On Tue, Jul 3, 2012 at 12:22 PM, Hieu Hoang <[email protected]> wrote:
>>> it builds and runs ok on boost v1.49 for me:
>>> $cat source target align
>>> a
>>> b
>>> 0-0
>>>
>>> $ ./extract-rules target source align extract --GlueGrammar
>>> glue-grammar --MaxSpan 10
>>>
>>> $cat extract extract.inv
>>> a [X] ||| b [X] ||| 0-0 ||| 1 |||
>>> b [X] ||| a [X] ||| 0-0 ||| 1
>>>
>>> $cat glue-grammar
>>> <s> [X] ||| <s> [S] ||| 1 ||| ||| 0
>>> [X][S] </s> [X] ||| [X][S] </s> [S] ||| 1 ||| 0-0 ||| 0
>>> [X][S] [X][X] [X] ||| [X][S] [X][X] [S] ||| 2.718 ||| 0-0 1-1 ||| 0
>>>
>>> On 03/07/2012 16:46, Alexander Fraser wrote:
>>>> Hi Folks,
>>>>
>>>> I've been having trouble with extract-rules. The build fails with
>>>> Boost 1 50, and with Boost 1 49 it does build but doesn't terminate
>>>> (just create a trivial training example - source: "1", target "1",
>>>> alignment "0-0").
>>>>
>>>> Anyone run into this?
>>>>
>>>> Thanks, Alex
>>>>
>>>> PS here are the first lines of the 1 50 build fail:
>>>>
>>>>
>>>> gcc.link
>>>> phrase-extract/bin/gcc-4.4.5/release/debug-symbols-on/link-static/threading-multi/extract-rules
>>>> phrase-extract/bin/gcc-4.4.5/release/debug-symbols-on/link-static/threading-multi/extract-rules.o:
>>>> In function `__static_ini\
>>>> tialization_and_destruction_0':
>>>> /export/ws12/damt/tools/boost/include/boost/system/error_code.hpp:214:
>>>> undefined reference to `boost::system::generic_catego\
>>>> ry()'
>>>> /export/ws12/damt/tools/boost/include/boost/system/error_code.hpp:215:
>>>> undefined reference to `boost::system::generic_catego\
>>>> ry()'
>>>> /export/ws12/damt/tools/boost/include/boost/system/error_code.hpp:216:
>>>> undefined reference to `boost::system::system_categor\
>>>> y()'
>>>> phrase-extract/bin/gcc-4.4.5/release/debug-symbols-on/link-static/threading-multi/extract-rules.o:
>>>> In function `thread_excep\
>>>> tion':
>>>> /export/ws12/damt/tools/boost/include/boost/thread/exceptions.hpp:49:
>>>> undefined reference to `boost::system::system_category\
>>>> ()'
>>>> phrase-extract/bin/gcc-4.4.5/release/debug-symbols-on/link-static/threading-multi/ThreadPool.o:
>>>> In function `__static_initia\
>>>> lization_and_destruction_0':
>>>> /export/ws12/damt/tools/boost/include/boost/system/error_code.hpp:214:
>>>> undefined reference to `boost::system::generic_catego\
>>>> ry()'
>>>> /export/ws12/damt/tools/boost/include/boost/system/error_code.hpp:215:
>>>> undefined reference to `boost::system::generic_catego\
>>>> ry()'
>>>> /export/ws12/damt/tools/boost/include/boost/system/error_code.hpp:216:
>>>> undefined reference to `boost::system::system_categor\
>>>> y()'
>>>> phrase-extract/bin/gcc-4.4.5/release/debug-symbols-on/link-static/threading-multi/ThreadPool.o:
>>>> In function `thread_exceptio\
>>>> n':
>>>> /export/ws12/damt/tools/boost/include/boost/thread/exceptions.hpp:49:
>>>> undefined reference to `boost::system::system_category\
>>>> ()'
>>>> phrase-extract/bin/gcc-4.4.5/release/debug-symbols-on/link-static/threading-multi/ThreadPool.o:
>>>> In function `condition_error\
>>>> ':
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> [email protected]
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> [email protected]
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
