Mitchell Jeffrey <[email protected]> writes:
[...] > ## Synopsis > Disambiguation is an essential part of the MT process, the aim of > which is to correctly identify the meaning and function of each word > of input text. Apertium currently uses a bigram/trigram > part-of-speech tagger, which means any given word is disambiguated > entirely on the categorisation of the two or three words preceding it. > This project seeks to implement a more flexible and comprehensive > disambiguation framework, based on a language-specific constraint > grammar and a language-independent finite state transducer for > processing text. Not quite… this sounds like the CG should replace apertium-tagger. It is meant to run before it in the pipeline, in order to take care of the cases where rules (where contexts may span until either boundary of the sentence) are more suited than bigram/trigram statistics. > ## Benefits to Apertium > Introducing the option of constraint grammar disambiguation benefits > the entire Apertium community as it is language-independent (although > language-specific grammars will need to be written). CG > disambiguation would allow the addition of new language pairs which > are currently unsuitable for parsing with the current bigram/trigram > disambiguation methods. …and makes it possible to disambiguate based on the whole sentence instead of just the nearest couple of words without running into this problem: http://wiki.apertium.org/wiki/Apertium_and_Constraint_Grammar#Performance > ## Details > The aim of the CG parser is to take natural language input text and > functionally identify (disambiguate) each word of input. An entire CG > parser processes text in the following sequence: > 1. Preprocessing - case conversion, sentence delimitation This is handled by lttoolbox. > 2. Lexicon updating - identification of unknown words This is not the responsibility of the CG. > 3. Morphological analysis - attaching a list of possible > morphological readings to each wordform lttoolbox again. > 4. Local morphological disambiguation - some readings may be > discarded with a simple inspection, perhaps employing the current > bigram/trigram methods The current bigram tagger goes _after_ the CG, since it is guaranteed to leave only one reading (and if the current tagger went before the CG, the CG would be useless for disambiguation). > 5. Constraint Grammar Parsing, in three simultaneous phases: (1) > application of disambiguation constraints; (2) assignment of clause > boundaries; (3) assignment of grammatical labels such as ‘finite main > verb’ or ‘subject’. Clause boundaries are identified and > disambiguated recursively throughout. Not recursively… CG iterates through rule sections until there are no more rules making any changes. Within a rule section, vislcg3 iterates through rules, but a finite state CG would not be deterministic in that way. Have you read http://wiki.apertium.org/wiki/Constraint_Grammar/Optimisation ? > There are several design considerations and goals to take into account > when planning to implement a CG parser. Speed and accuracy are the > obvious ones, but there are also questions regarding the open-ended > nature of natural language. Any constraint grammar provided to the > parser will be incomplete in the sense that there will almost > certainly be a given sentence that a native speaker would accept but > the grammar would not. Similarly, there may be cases where the input > text is ungrammatical in the first place; this should not result in a > disproportionate decrease in accuracy. The proposal is to write a parser, not a grammar, so this is not quite relevant here (though you could have a short line in the introduction on how CG's, though rule-based, are robust in the sense that they are meant for open-ended text which may include grammatical errors). > Additionally, some items in > the grammar will be rule-based while others may (perhaps in future) be > probabilistic. I would leave out that sentence. The bigram tagger does statistical disambiguation, I don't see how anything probabilistic within the CG module could be more than a hack. > Consideration will need to be given to how the parser design affects > end-user usage for users as well as grammar writers, and should > include facilities for testing, debugging and optimisation of > constraint grammar definitions. > > ## Deliverables & Schedule > The primary focus of this project will be to design, implement (in > C++), integrate and document a language-independent CG parser, and > also to specify the XML grammar standard it will accept. Most of the > time will be spent on detail item (5), with the expectation that much > of the other functionality will already be available elsewhere within > Apertium. > > 4 week community Bonding: Reading around the subject area and > acquiring specific skills such as C++, parser design, language > processing and computational linguistics > 12 week coding period: pursuant primarily to detail item (5) > 1 week sprint: final polish, debugging and documentation effort I'd like to see a more detailed plan, especially wrt. which features should be implemented and prioritised. Some of the CG functions implemented by e.g. vislcg3[1] are a lot more important than others, so think about the feature set and test cases for that. E.g. LIST, SELECT/REMOVE, star (*), BARRIER, Careful (C) are important. Things like spanning window boundaries, setting marks or making dependency trees should be deferred until much later. Unification is possible to avoid by just writing more rules. [...] > Recently I’ve been working on an online chessboard (jQuery/node.js), Include the URL in your proposal, if you can ;) -- Kevin Brubeck Unhammer ------------------------------------------------------------------------------ Create and publish websites with WebMatrix Use the most popular FREE web apps or write code yourself; WebMatrix provides all the features you need to develop and publish your website. http://p.sf.net/sfu/ms-webmatrix-sf _______________________________________________ Apertium-stuff mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/apertium-stuff
