ok, i can't reproduce your error
FUnction not implemented
you should find out exactly how lmplz is being run, it may be that you
have a slightly older version and doesn't know all the arguments you've
given it.
On 26/11/2013 06:47, Prasanth K wrote:
Hello Hieu,
My first attempt was to specify the absolute amount of memory (10G)
but that gave an error saying function not implemented. Later, when I
tried specifying the relative size (80%), I got a similar parse error
to what you have given above. Strange that it should
@Kenneth, thanks for the code to estimate physical memory. I am going
to give it a shot and let you know how it goes.
- Regards,
Prasanth
On Mon, Nov 25, 2013 at 9:20 PM, Hieu Hoang <[email protected]
<mailto:[email protected]>> wrote:
Prasanth - what is the exact lmplz command that was ran by the EMS?
This works
.../lmplz --order 5 --text lm/europarl.lowercased.1 --arpa
lm/europarl.lmplz -T /tmp -S 1G
This doesn't
.../lmplz --order 5 --text lm/europarl.lowercased.1 --arpa
lm/europarl.lmplz -T /tmp -S 80%
it give the error
util/usage.cc:220 in uint64_t util::<anonymous
namespace>::ParseNum(const std::string &) [Num = double] threw
SizeParseError because `!mem'.
Failed to parse 80% into a memory size because % was specified but the
physical memory size could not be determined.
However, it worked even with the source code from 4 days ago.
On 25/11/2013 19:07, Kenneth Heafield wrote:
> Hi,
>
> I've taken a shot in the dark based on physmem.c to
support physical
> memory estimation on BSD and OS X. Please clone
>
> github.com/kpu/kenlm <http://github.com/kpu/kenlm>
>
> and compile with
>
> ./bjam
>
> If that fails, please let Hieu and I know (maybe Hieu can help
since he
> has OS X). If it doesn't fail, run
>
> bin/lmplz
>
> with no argument. The help message will include a line e.g.
>
> "This machine has 135224176640 bytes of memory."
>
> or
>
> "Unable to determine the amount of memory on this machine."
>
> If it works, then I'll push to Moses. Trying to not break Moses
master
> for OS X.
>
> Kenneth
>
> On 11/24/13 22:40, Prasanth K wrote:
>> Hi Kenneth,
>>
>> Thanks for the clarification w.r.t. calculating the memory
size. But I
>> am running these on a Mac (10.9 Mavericks). Do you think I
should still
>> port the lmplz code to Mac for the estimation of probabilities?
>>
>> One thing though, I did change the default clang compiler that
comes
>> with this new Mac to a gcc-4.8 (not sure that changes anything
in this
>> context).
>>
>> - Prasanth
>>
>>
>>
>>
>> On Fri, Nov 22, 2013 at 6:50 PM, Kenneth Heafield
<[email protected] <mailto:[email protected]>
>> <mailto:[email protected] <mailto:[email protected]>>> wrote:
>>
>> Hi,
>>
>> What OS are you on? Cygwin? Apparently every OS
reports
>> memory size
>> in a different way:
>>
>>
http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=lib/physmem.c;h=2629936146e3042f927523322f18aca76996cd7f;hb=HEAD
>>
>> The good news is that the above code is LGPLv2:
>>
>>
http://git.savannah.gnu.org/gitweb/?p=gnulib.git;a=blob;f=modules/physmem;h=9644522e0493a85a9fb4ae7c4449741c2c1500ea;hb=HEAD
>>
>> But currently I'm just using this short function that will
fail on some
>> platforms:
>>
>> uint64_t GuessPhysicalMemory() {
>> #if defined(_WIN32) || defined(_WIN64)
>> return 0;
>> #elif defined(_SC_PHYS_PAGES) && defined(_SC_PAGESIZE)
>> long pages = sysconf(_SC_PHYS_PAGES);
>> if (pages == -1) return 0;
>> long page_size = sysconf(_SC_PAGESIZE);
>> if (page_size == -1) return 0;
>> return static_cast<uint64_t>(pages) *
>> static_cast<uint64_t>(page_size);
>> #else
>> return 0;
>> #endif
>> }
>>
>> If it fails, I just don't let users specify memory as a
percentage. So
>> one thing thing to fix is putting physmem.{h,c} in util
then changing
>> calls to GuessPhysicalMemory. But I'm also not a fan of
the way the GNU
>> code gives up and makes up a number at the end.
>>
>> The second porting issue is that lmplz makes parallel use
of pread,
>> pwrite, and write. Windows is unsafe in this regard
(POSIX requires
>> that pread/pwrite not change the file pointer; Windows has
no way to
>> implement that atomically). To fix this, we'll always
specify the file
>> offset in cases that happen concurrently. Extend
util/stream/io.* with
>> a PWrite class based on PWriteOrThrow then change
FileBuffer to use
>> PWrite. Then I guess one should rename
PReadOrThrow/PWriteOrThrow to
>> something that indicates they're not-quite-POSIX on
windows. Also, the
>> macros in these functions should detect cygwin, bypassing
cygwin's
>> "Function not implemented" and calling Windows APIs
directly (they're
>> already there for _WIN32).
>>
>> I don't have a windows box so I can say what should be
changed at a high
>> level, but need an actual user to ensure it compiles and
runs correctly.
>>
>> Kenneth
>>
>> On 11/22/13 06:49, Prasanth K wrote:
>> > Hi,
>> >
>> > I am trying to use KenLM for building a language model
on the Europarl
>> > corpus. Following the instructions in
>> >
>>
(http://www.statmt.org/moses/?n=FactoredTraining.BuildingLanguageModel#ntoc19),
>> > I added the few lines for getting KenLM to estimate the LM
>> probabilities
>> > (order/n=5) to my config file to the EMS. The language
model dies down
>> > during training saying that the "Function not
implemented" at counting
>> > and sorting n-grams stage (the first stage itself). Does
this mean
>> there
>> > is something wrong with my installation? Or is just
insufficient
>> memory?
>> >
>> > Incidentally, when I started giving the amount of memory
in terms of %
>> > (80%) there was an error "Failed to parse .. into memory
size because
>> > physical memory size could not be determined". I am also
curious why
>> > this happens?
>> >
>> > Kenneth, can you shed some light on this? Thanks.
>> >
>> > - Regards,
>> > Prasanth
>> >
>> >
>> >
>> > --
>> > "Theories have four stages of acceptance. i) this is
worthless
>> nonsense;
>> > ii) this is an interesting, but perverse, point of view,
iii) this is
>> > true, but quite unimportant; iv) I always said so."
>> >
>> > --- J.B.S. Haldane
>> >
>> >
>> > _______________________________________________
>> > Moses-support mailing list
>> > [email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
>> > http://mailman.mit.edu/mailman/listinfo/moses-support
>> >
>> _______________________________________________
>> Moses-support mailing list
>> [email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> --
>> "Theories have four stages of acceptance. i) this is worthless
nonsense;
>> ii) this is an interesting, but perverse, point of view, iii)
this is
>> true, but quite unimportant; iv) I always said so."
>>
>> --- J.B.S. Haldane
> _______________________________________________
> Moses-support mailing list
> [email protected] <mailto:[email protected]>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
--
"Theories have four stages of acceptance. i) this is worthless
nonsense; ii) this is an interesting, but perverse, point of view,
iii) this is true, but quite unimportant; iv) I always said so."
--- J.B.S. Haldane
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
