[
https://issues.apache.org/jira/browse/TIKA-2219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15765347#comment-15765347
]
Tim Allison edited comment on TIKA-2219 at 12/20/16 9:51 PM:
-------------------------------------------------------------
Looks like they aren't twiddling with the confidence scores any more (see:
[repo|http://bugs.icu-project.org/trac/browser/icu4j/tags/release-58-1/main/classes/core/src/com/ibm/icu/text/CharsetDetector.java]):
{noformat}
193 for (int i = 0; i < ALL_CS_RECOGNIZERS.size(); i++) {
194 CSRecognizerInfo rcinfo = ALL_CS_RECOGNIZERS.get(i);
195 boolean active = (fEnabledRecognizers != null) ?
fEnabledRecognizers[i] : rcinfo.isDefaultEnabled;
196 if (active) {
197 CharsetMatch m = rcinfo.recognizer.match(this);
198 if (m != null) {
199 matches.add(m);
200 }
201 }
202 }
{noformat}
was (Author: talli...@mitre.org):
Looks like they aren't twiddling with the confidence scores any more
[repo|http://bugs.icu-project.org/trac/browser/icu4j/tags/release-58-1/main/classes/core/src/com/ibm/icu/text/CharsetDetector.java]
{noformat}
193 for (int i = 0; i < ALL_CS_RECOGNIZERS.size(); i++) {
194 CSRecognizerInfo rcinfo = ALL_CS_RECOGNIZERS.get(i);
195 boolean active = (fEnabledRecognizers != null) ?
fEnabledRecognizers[i] : rcinfo.isDefaultEnabled;
196 if (active) {
197 CharsetMatch m = rcinfo.recognizer.match(this);
198 if (m != null) {
199 matches.add(m);
200 }
201 }
202 }
{noformat}
> CharsetDetector no longer detects windows-1252 charset
> ------------------------------------------------------
>
> Key: TIKA-2219
> URL: https://issues.apache.org/jira/browse/TIKA-2219
> Project: Tika
> Issue Type: Bug
> Components: parser
> Affects Versions: 1.14
> Environment: Any.
> Reporter: Pascal Essiembre
> Priority: Minor
> Fix For: 2.0, 1.15
>
>
> Starting with Tika 1.14, windows-1252 is no longer detected, as ISO-8859-1 is
> always detected instead. While not tested, this likely affects other
> windows-125* encodings as well.
> I tracked it down to a change in the
> {{CharsetRecog_sbcs.CharsetRecog_8859_1#getName()}} method. Now it always
> returns "ISO-8859-1" whereas before it was: {{return haveC1Bytes ?
> "windows-1252" : "ISO-8859-1";}}
> Now that condition has been moved to the {{match(CharsetDetector det)}}
> method so that the returned CharsetMatch has the proper name. The problem
> with that is {{CharsetDetector#detectAll()}} method overwrites the correct
> match with a new one that will return the value of {{#getName()}} from the
> {{CharsetRecognizer}} instead (which is always "ISO-8859-1" in this case).
> There might be legitimate reasons why the {{CharsetMatch}} instances in
> {{detectAll()}} method are replaced with new ones, but changing this code in
> that method appears to work for me:
> // Remove this:
> // CharsetMatch m = new CharsetMatch(this, csr,
> confidence);
> // matches.add(m);
> // Add this instead:
> matches.add(charsetMatch);
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
