[
https://issues.apache.org/jira/browse/UIMA-4286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14377835#comment-14377835
]
Mario Juric commented on UIMA-4286:
-----------------------------------
Here is my initial feedback to the tests:
1) Setting HtmlConverter.PARAM_PROCESS_ALL to true appears to work as expected
and solves our must important issue.
2) Unless I am reading this wrong then the gap text now gets included as part
of the tag’s offsets but I think I would actually prefer they are not since
they are not part of the original text. However, after applying the same
settings to one of our articles then I can see that the gap is actually not
included as part of the offsets so I think I am interpreting the test
incorrectly.
3) The HtmlConverter.PARAM_EXPAND_OFFSETS feature seems to apply to tags that
don’t have any content. Its difficult to understand what it exactly does
without looking at the implementation, also because the documentation seems to
have been copied from HtmlConverter.PARAM_PROCESS_ALL without being updated. It
seems that these tags gets the offsets of either the first or the last
annotation depending on what it is closest to. I can’t see from just reading
the test and the implementation code inside the converter whether its just
within the same subtree or the whole document. I am not yet totally into
details of how the annotation index iterator exactly works in the different
cases and I would have to play around with that some more. Either way would
probably be acceptable since we would not display this type of “invisible"
metadata with annotations. Its different with body text or citations where we
want to show evidence in the full text.
I also experimented with the new features on a few different articles. Here is
some initial feedback and thoughts on this:
4) Spaces are ignored in HtmlConverter.PARAM_GAP_TEXT e.g. " . " produces the
same result as "." . In many cases the sentences are not separated properly
for NLP even when HtmlConverter.PARAM_SKIP_WHITESPACES is set to false. I
thought that this is acceptable to do since the extra whitespace should not
affect the parser output but I do see wrong sentence segmentation in the form
of tag content being eagerly joined into the same sentence. Its too early to
say how big an issue this is since the important sentences seem to be good but
I need to experiment with this some more as well.
5) The HtmlConverter.PARAM_GAP_INDUCING_TAGS allows me to whitelist what tags
should induce gaps like some HTML tags can induce line breaks. I think this is
in line with how the HTMLConverter otherwise is configurable and it gives us
some detailed control, which I would want at some point. I am then speculating
whether I would rather like a map to allow different gap text for different
tags. I might want “given-name" and “surname” tags to be separated by space but
citations be separated by “.“ or similar. However, it adds the burden to build
this list/map for every XML type. I also need to consider that we often process
mixed article jobs in our pipeline since this is operationally simpler to work
with for us. This requires that we probably start using some UIMA flow control
in the pipeline. Until now we just passed metadata about the article type
through the pipeline with the CAS and use this information to dynamically adapt
processing inside the analysis engine accordingly where needed but I wonder if
we can do the samme high-level “switch" with the UIMA flow control although we
haven't tried this yet.
6) I think this is not related to this feature in particular and its probably a
known issue unless I interpret what I see incorrectly: Spaces seem to be lost
around italic or bold tagged text.
Generally I am very positive to the new features but we will need some time to
play with fine tuning the parameters to see whether we could overcome any
critical sentence segmentation issues that might appear since this is most
important to us.
> Ruta: HTMLConverter: Option to convert tags outside body tags
> -------------------------------------------------------------
>
> Key: UIMA-4286
> URL: https://issues.apache.org/jira/browse/UIMA-4286
> Project: UIMA
> Issue Type: Improvement
> Components: ruta
> Affects Versions: 2.2.1ruta
> Reporter: Mario Juric
> Assignee: Peter Klügl
> Fix For: 2.3.0ruta
>
>
> The HTML converter only converts tags that are found inside the body tag.
> Therefore some information carrying tags like citations get left out when
> applying the converter to XML articles with many metadata. It would be useful
> to add the option to have all tags converted since this would allow content
> outside the body to be parsed by natural language analysers as well.
> The converter was originally, as the name implies, conceived for HTML
> documents but together with the HTML Annotator it can this way be more
> generally useful in enabling NL parsing of a broader class of documents such
> as articles stored in XML documents.
> An example of how this option might work can be given by disabling the
> "inBody"-flag inside the HTMLConverterVisitor. The example also illustrates
> what offsets to apply to such annotations but otherwise the document
> annotation offsets can be used. Empty tags can still be ignored but tags with
> only attributes and no content should preferably be converted.
> Experiments with disabling the "in body"-constraint reveals that there will
> be an additional need to separate the content metadata tags in the converted
> text view. An NL parser reading the text will in many case read different
> tags as one word or one sentence, which is not desirable. Some text delimiter
> should therefore be inserted between tags were required, which optionally
> could be customizable as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)