Hi all,
Just leaving a draft proposal for the crows to pick at. I'd really
appreciate some feedback from the more knowledgeable subscribers,
particularly with regards to technical accuracy and the proposed
deliverables/implementation schedule.
Many thanks, Mitchell
----------
RULE BASED FINITE STATE DISAMBIGUATION FOR APERTIUM
## Introduction
Machine learning facilitates international communication, and is
especially useful in supporting the continued usage of minority
languages for which human translators are in short supply. Apertium
represents both an academic exercise and a useful software development
project whose goal it is to provide an accurate, universal automated
translation engine and accompanying language-specific datasets.
My bilingual experience has taught me that each language provides an
insightful, valuable and most importantly, unique way of interpreting
and conceptualising the world around us. The subtle tools of
expression encapsulated within a language are extremely valuable both
in terms of cultural heritage and linguistic history. A sad fact of
the modern world is that less frequently spoken minority languages are
slowly dying off in response to the convenience and necessity required
by increasingly globalised communication. Language extinction is
occurring all around the world, including my native Australia which
once boasted almost 1000 indigenous languages.
Machine translation is one tool of many which could help slow the
decline of minority languages not only as a stand-in replacement for
human translators, but also as a educational tool. Being an open
source project, Apertium is in a position to support minority
languages which aren’t financially viable to maintain in the
commercial sense.
On a personal level, machine translation projects such as this
represent a unique opportunity to combine my interests in computer
science, natural language and mathematics/logic. I am interested in
how humans communicate and convey meaning; investigating how machines
could process human language (for instance with a translation package
such as Apertium) would not only be an insightful venture in itself,
but would also inform my understanding of the condition of human
communication.
## Synopsis
Disambiguation is an essential part of the MT process, the aim of
which is to correctly identify the meaning and function of each word
of input text. Apertium currently uses a bigram/trigram
part-of-speech tagger, which means any given word is disambiguated
entirely on the categorisation of the two or three words preceding it.
This project seeks to implement a more flexible and comprehensive
disambiguation framework, based on a language-specific constraint
grammar and a language-independent finite state transducer for
processing text.
## Benefits to Apertium
Introducing the option of constraint grammar disambiguation benefits
the entire Apertium community as it is language-independent (although
language-specific grammars will need to be written). CG
disambiguation would allow the addition of new language pairs which
are currently unsuitable for parsing with the current bigram/trigram
disambiguation methods.
## Details
The aim of the CG parser is to take natural language input text and
functionally identify (disambiguate) each word of input. An entire CG
parser processes text in the following sequence:
1. Preprocessing - case conversion, sentence delimitation
2. Lexicon updating - identification of unknown words
3. Morphological analysis - attaching a list of possible
morphological readings to each wordform
4. Local morphological disambiguation - some readings may be
discarded with a simple inspection, perhaps employing the current
bigram/trigram methods
5. Constraint Grammar Parsing, in three simultaneous phases: (1)
application of disambiguation constraints; (2) assignment of clause
boundaries; (3) assignment of grammatical labels such as ‘finite main
verb’ or ‘subject’. Clause boundaries are identified and
disambiguated recursively throughout.
There are several design considerations and goals to take into account
when planning to implement a CG parser. Speed and accuracy are the
obvious ones, but there are also questions regarding the open-ended
nature of natural language. Any constraint grammar provided to the
parser will be incomplete in the sense that there will almost
certainly be a given sentence that a native speaker would accept but
the grammar would not. Similarly, there may be cases where the input
text is ungrammatical in the first place; this should not result in a
disproportionate decrease in accuracy. Additionally, some items in
the grammar will be rule-based while others may (perhaps in future) be
probabilistic.
Consideration will need to be given to how the parser design affects
end-user usage for users as well as grammar writers, and should
include facilities for testing, debugging and optimisation of
constraint grammar definitions.
## Deliverables & Schedule
The primary focus of this project will be to design, implement (in
C++), integrate and document a language-independent CG parser, and
also to specify the XML grammar standard it will accept. Most of the
time will be spent on detail item (5), with the expectation that much
of the other functionality will already be available elsewhere within
Apertium.
4 week community Bonding: Reading around the subject area and
acquiring specific skills such as C++, parser design, language
processing and computational linguistics
12 week coding period: pursuant primarily to detail item (5)
1 week sprint: final polish, debugging and documentation effort
The three-month coding period would suggest that an agile approach
would be best, with short (perhaps weekly) cycles of reading,
research, planning, implementation, testing and documentation - in
approximately that order. I will have part-time coursework
commitments at university for the first half of development, but can
reasonably expect that time requirements will be minimal and that I
will be able to commit to this project on a full-time (30-40 hours per
week) basis throughout.
## Bio
I am based in Melbourne, Australia and currently completing course
requirements for a B.Sc degree on a part-time basis with majors in
mathematics and physics, as well as a minor in computer science.
Current language skills include Java, Python, Javascript, HTML/CSS and
LaTeX - as mentioned above, I expect to pick up C++ quickly before the
coding period begins. Previous project experience has seen me learn
and use new languages, tools and APIs as required (namely the creation
of a data analysis suite for a research physics laboratory built with
Enthought Python).
One of the most interesting courses in my degree has been one on the
theory of computation, which covered topics such as finite state
machines, formal grammar, parsing and compiler design. It would be an
absolute delight to deepen and broaden my knowledge along these lines
by producing a practical codebase for processing natural language.
Recently I’ve been working on an online chessboard (jQuery/node.js),
meeting with my Japanese-English language exchange partner and playing
sports such as Ultimate and Rogaining.
------------------------------------------------------------------------------
Create and publish websites with WebMatrix
Use the most popular FREE web apps or write code yourself;
WebMatrix provides all the features you need to develop and
publish your website. http://p.sf.net/sfu/ms-webmatrix-sf
_______________________________________________
Apertium-stuff mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/apertium-stuff
