Aleksandr Dubinsky created TIKA-1406:
----------------------------------------
Summary: Problems in TXT encoding detection
Key: TIKA-1406
URL: https://issues.apache.org/jira/browse/TIKA-1406
Project: Tika
Issue Type: Bug
Components: parser
Affects Versions: 1.6, 1.7
Reporter: Aleksandr Dubinsky
The detection of TXT file encoding often makes mistakes. Two things can improve
the detection in many cases:
- Increase the lookahead from 16k bytes to a larger number, such as 128k or
larger.
- Improve on the brain-dead heuristic that if a file doesn't have a \r then it
must be ISO 8859-1(5) instead of Windows-1252. (For one, it mis-detects files
that don't have any newlines.)
A remaining problem that doesn't have an immediate solution, is the frequent
mis-detection of Windows-1252 as Shift-JIS. A flag to forbit Shift-JIS is
desirable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
