Hi Kenneth,
it is me again, but I have aproblem to compile Moses with KenLM.
I run:
./regenerate-makefiles.sh
and
./configure --with-boost=./boost/boost_1_47_0/
because I had an old version of the boost lib.
when I run
make j 16
I got this error compiling kenlm:
if /bin/sh ../libtool --tag=CXX --mode=compile g++ -DHAVE_CONFIG_H -I. -I.
-I.. -W -Wall -ffor-scope -D_FILE_OFFSET_BITS=64 -D_LARGE_FILES
-I/nfs/staging/turchmo/moses27_10_2011/moses-smt-mosesdecoder-694d62c/boost/boost_1_47_0/
-DTRACE_ENABLE=1
-I/nfs/staging/turchmo/moses27_10_2011/moses-smt-mosesdecoder-694d62c/kenlm
-g -O2 -MT mmap.lo -MD -MP -MF ".deps/mmap.Tpo" -c -o mmap.lo `test -f
'util/mmap.cc' || echo './'`util/mmap.cc; \
then mv -f ".deps/mmap.Tpo" ".deps/mmap.Plo"; else rm -f
".deps/mmap.Tpo"; exit 1; fi
mv: cannot stat `.deps/binary_format.Tpo': No such file or directory
Do you have any ideas about it?
thanks a lot
Marco
On Mon, Oct 10, 2011 at 7:22 PM, marco turchi <[email protected]>wrote:
> I see, so no chances of translating without rebuild everything :-(
>
> Thanks a lot
> Marco
>
>
> On Mon, Oct 10, 2011 at 7:05 PM, Kenneth Heafield <[email protected]>wrote:
>
>> **
>> Ah. I've changed the version number to force people to rebuild their
>> binary files when the format changes. Old versions produced that error
>> message when the version number changed. Quantization wasn't even
>> implemented in kenlm until some time in 2011, so you're not going to get a
>> Moses from 2010 to work with it.
>>
>> As to the compiler errors on the tests, you're probably using an old
>> version of Boost not supported by the tests.
>>
>> Kenneth
>>
>>
>> On 10/10/11 17:57, marco turchi wrote:
>>
>> Hi
>> what I have done is to download the last version of kenlm, built it and
>> run the build_binary. Then I call a 2010 version of moses using the lm.
>>
>> I run the ./test.sh and I got these messages:
>> util/bit_packing_test.cc:14: error: expected constructor, destructor, or
>> type conversion before â(â token
>> util/bit_packing_test.cc:21: error: expected constructor, destructor, or
>> type conversion before â(â token
>> util/bit_packing_test.cc:59: error: expected `}' at end of input
>> util/bit_packing_test.cc:59: error: expected `}' at end of input
>>
>> I guess, I need to reinstall the full package moses+kenlm and biuld
>> everything together.
>>
>> Thanks a lot
>> Marco
>>
>> On Mon, Oct 10, 2011 at 6:46 PM, Kenneth Heafield <[email protected]>wrote:
>>
>>> Hi,
>>>
>>> Number 8 means prefault and number 9 means lazy mmap. It's an
>>> option and orthogonal to the data structure.
>>>
>>> Since the binary file is the in-memory representation, I do paranoid
>>> checks to make sure your machine represents floats, 64-bit integers, and
>>> such in the same way. For example a 32-bit build will have different
>>> alignment than a 64-bit build. This check is complaining.
>>>
>>> Please try build_binary and moses from the same build. If that
>>> doesn't work, please send me the first kilobyte of your binary file.
>>>
>>> Also, if you have Boost, can you cd kenlm && make clean && ./test.sh
>>> and complain if there are any test failures?
>>>
>>> Kenneth
>>>
>>>
>>> On 10/10/11 17:31, marco turchi wrote:
>>>
>>> Hi Kenneth,
>>> which number shall I use in the moses.ini 8 or 9 if i build my lm with
>>> these parameters -q 8 -b 8?
>>>
>>> I got this error when I run moses:
>>> In LanguageModelKen::Load: nGramOrder = 5 will be ignored. Using
>>> whatever the file has.
>>> terminate called after throwing an instance of 'lm::FormatLoadException'
>>> what(): File looks like it should be loaded with mmap, but the test
>>> values don't match. Was it built on a different machine or with a
>>> different compiler?
>>>
>>> I have the feeling that my moses version needs to be updated!
>>>
>>> Thanks a lot
>>> Marco
>>>
>>> On Sat, Oct 8, 2011 at 1:02 PM, marco turchi <[email protected]>wrote:
>>>
>>>> Thanks!
>>>> I'm going to update my version.
>>>>
>>>> Cheers
>>>> Marco
>>>>
>>>>
>>>> On Sat, Oct 8, 2011 at 1:01 PM, Kenneth Heafield
>>>> <[email protected]>wrote:
>>>>
>>>>> Fixed in revision 4314. There's still an issue with some SRILM
>>>>> models failing to build that I'll get to soon.
>>>>>
>>>>> On 10/08/11 11:52, marco turchi wrote:
>>>>>
>>>>> Hi,
>>>>> thanks a lot for the answer.
>>>>> Great, so I can use -m 2048 to build it. Do you think it is enough?
>>>>>
>>>>> Thanks again
>>>>> Marco
>>>>>
>>>>> On Sat, Oct 8, 2011 at 12:46 PM, Kenneth Heafield <[email protected]
>>>>> > wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> This looks like a bug in the trie implementation due to some
>>>>>> recent changes I made for left state minimization. I'll fix it soon. A
>>>>>> workaround is to pass a large -m option to build_binary.
>>>>>>
>>>>>> Sorry,
>>>>>>
>>>>>> Kenneth
>>>>>>
>>>>>>
>>>>>> On 10/08/11 11:34, marco turchi wrote:
>>>>>>
>>>>>> Dear All,
>>>>>> I'm trying to build a lm using a large dataset (> 11 M sentences). I
>>>>>> have generated the Arpa format with irstlm and now I'd like to binarize
>>>>>> it
>>>>>> using kenlm.
>>>>>>
>>>>>> I have called the build_binary to estimate memory usage, and I got
>>>>>> this
>>>>>>
>>>>>> Memory estimate:
>>>>>> type MB
>>>>>> probing 16129 assuming -p 1.5
>>>>>> trie 7462 without quantization
>>>>>> trie 4361 assuming -q 8 -b 8 quantization
>>>>>> trie 6440 assuming -a 22 array pointer compression
>>>>>> trie 3339 assuming -a 22 -q 8 -b 8 array pointer compression and
>>>>>> quantization
>>>>>>
>>>>>> then I run the binarization in this way:
>>>>>>
>>>>>> /nfs/staging/turchmo/moses/kenlmNew/build_binary -i -t /tmp/ -q 8 -b
>>>>>> 8 trie irstLM.ARPA.txt irstLanguageModel.binary.lm
>>>>>>
>>>>>> but I got this error:
>>>>>>
>>>>>> lm/search_trie.cc:409 in void
>>>>>> lm::ngram::trie::<unnamed>::SanityCheckCounts(const std::vector<long
>>>>>> unsigned int, std::allocator<long unsigned int> >&, const
>>>>>> std::vector<long
>>>>>> unsigned int, std::allocator<long unsigned int> >&) threw
>>>>>> util::Exception'.
>>>>>> Longest count should be constant but it changed from 289546423 to
>>>>>> 289546405 Byte: 37297517525
>>>>>>
>>>>>> I have had a look into the mailing list, but I do not find any post
>>>>>> with the same error.
>>>>>>
>>>>>> Any ideas?
>>>>>>
>>>>>> Thanks a lot
>>>>>> Marco
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing
>>>>>> [email protected]http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> [email protected]
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>>
>>
>>
>
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
