[
https://issues.apache.org/jira/browse/LUCENE-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12926689#action_12926689
]
Uwe Schindler edited comment on LUCENE-2731 at 10/31/10 6:05 AM:
-----------------------------------------------------------------
By the way, the whole loading in the TokenFilter is broken! XML files should
never be load by a Reader, always by InputStream. The charset detection is part
of the XML spec and the default is UTF-8 if not overridden in the XML file. The
HyphenationCompoundWordTokenFilter supplies ISO-8859-1 as fixed for the reader,
so it is never possible to load other xml files with different charsets. This
is another issue, I will open for 3.x and trunk (as it needs API change).
was (Author: thetaphi):
By the way, the whole loading in the TokenFilter is broken! XML files
should never be load by a Reaer, always by InputStream. The charset detection
is part of the XML spec and the default is UTF-8 if not overridden in the XML
file. The HyphenationCompoundWordTokenFilter supplies ISO-8859-1 as fixed for
the reader, so it is never possible to load other xml files with different
charsets. This is another issue, I will open for 3.x and trunk (as it needs API
change).
> HyphenationCompoundWordTokenFilter fails to load DTD in Crimson parser (JDK
> 1.4)
> --------------------------------------------------------------------------------
>
> Key: LUCENE-2731
> URL: https://issues.apache.org/jira/browse/LUCENE-2731
> Project: Lucene - Java
> Issue Type: Bug
> Components: contrib/analyzers
> Reporter: Uwe Schindler
> Assignee: Uwe Schindler
> Fix For: 2.9.4
>
> Attachments: LUCENE-2731.patch
>
>
> HyphenationCompoundWordTokenFilter loads the DTD in its XML parser from
> memory by supplying EntityResolver. In Java 1.4 (affects Lucene 2.9, but also
> later versions if not Apache Xerces is used as XML parser) this does not
> work, because Cromson does not even ask the entity resolver, if no base URI
> is known. As the hyphenation file is loaded from Reader/InputStream no base
> URI is known. Crimson needs at least a non-null systemId to proceed.
> This patch (Lucene 2.9 only) fakes this by supplying a fake systemId to the
> InputSource.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
