err, i thought every ff uses (const Factor*), but perhaps not now that i
look at the code. Probing pt uses it
ProbingPt.h line 43-47
probing pt doesn't support multiple factors.
Most other pt supprts supports multiple factors but does it
inefficiently by getting tokenizing the string into multiple factors.
Factor class also has a unique id which KenLM and IRSTLM uses to map to
their internal id. I would prefer everyone (const Factor*) as the unique
id, but it's no big deal
On 09/10/2015 23:59, Marcin Junczys-Dowmunt wrote:
oh, I didn't know that. Is any feature function actually using that?
W dniu 10.10.2015 o 00:54, Hieu Hoang pisze:
const Factor* is the vocab id. It's guaranteed to be unique for each
unique string. You can map directly to the string using
factor->GetString()
On 09/10/2015 22:55, Lane Schwartz wrote:
Thanks, Marcin.
So when the various components of Moses pass words back and forth,
what do they send each other? std::string? StringPiece?
On Fri, Oct 9, 2015 at 4:28 PM, Marcin Junczys-Dowmunt
<[email protected]> wrote:
For instance in my phrase table that would be
mosesdecoder/moses/TranslationModel/CompactPT/PhraseDecoder.h
StringVector<unsigned char, unsigned, std::allocator>
m_sourceSymbols;
StringVector<unsigned char, unsigned, std::allocator>
m_targetSymbols;
That's a memory-mapped vector of strings.
W dniu 09.10.2015 o 23:22, Lane Schwartz pisze:
Seriously? That sounds inefficient.
I've found code in KenLM that maps from strings to integers,
but not the other way around.
Marcin, do you know, for example, where any Moses code is for
doing the mapping for any data structure?
On Fri, Oct 9, 2015 at 4:14 PM, Marcin Junczys-Dowmunt
<[email protected]> wrote:
Hi,
This would only be a simple thing if there was a common
framework for that, but there isn't. Each datastructure
implements its own vocabularies and look-up tables. There
is no common set of integers.
Best,
Marcin
W dniu 09.10.2015 o 23:11, Lane Schwartz pisze:
Hey,
I know this should be a simple thing to find, but what
code in Moses is responsible for mapping back and forth
between strings and integers?
Thanks,
Lane
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
_______________________________________________
Moses-support mailing list
[email protected] <mailto:[email protected]>
http://mailman.mit.edu/mailman/listinfo/moses-support
--
When a place gets crowded enough to require ID's, social
collapse is not
far away. It is time to go elsewhere. The best thing about
space travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
--
When a place gets crowded enough to require ID's, social collapse is not
far away. It is time to go elsewhere. The best thing about space
travel
is that it made it possible to go elsewhere.
-- R.A. Heinlein, "Time Enough For Love"
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
--
Hieu Hoang
http://www.hoang.co.uk/hieu
--
Hieu Hoang
http://www.hoang.co.uk/hieu
_______________________________________________
Moses-support mailing list
[email protected]
http://mailman.mit.edu/mailman/listinfo/moses-support
