Claudenw commented on code in PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#discussion_r3852102620
##########
src/changes/changes.xml:
##########
@@ -68,6 +68,9 @@ in order to be properly linked in site reports.
</release>
-->
<release version="1.0.0-SNAPSHOT" date="xxxx-yy-zz" description="Current
SNAPSHOT - release to be done">
+ <action issue="RAT-532" type="add" dev="pottlinger">
+ Update to Tika 4.0.0: new charset detection logic in Tika returns
different values compared to 3.x before, such as windows-1252 instead of
ISO-8859-1.
Review Comment:
I think this is an error. We have some windows-1252 files but most are
ISO-8859-1
I think for our purpose we can label windows-1252 as ISO-8859-1. I need to
check the list that is returned from the new Tika and see if it includes
ISO-8859-1 as one of the encodings. I think we should select ISO over windows
when we have the option. This PR needs investigation and work.
##########
apache-rat-core/src/main/java/org/apache/rat/analysis/TikaProcessor.java:
##########
@@ -166,20 +177,30 @@ public static String process(final Document document)
throws RatDocumentAnalysis
* @throws UnsupportedCharsetException on unsupported charset.
*/
private static Charset detectCharset(final InputStream stream, final
DocumentName documentName) throws IOException, UnsupportedCharsetException {
- final int bytesForCharsetDetection = 256;
- CharsetDetector encodingDetector = new
CharsetDetector(bytesForCharsetDetection);
- encodingDetector.setText(stream);
- CharsetMatch charsetMatch = encodingDetector.detect();
- if (charsetMatch != null) {
- try {
- return Charset.forName(charsetMatch.getName());
- } catch (UnsupportedCharsetException e) {
- DefaultLog.getInstance().warn(String.format("Unsupported
character set '%s' in file '%s'",
- charsetMatch.getName(), documentName));
- throw e;
+ stream.mark(BYTES_FOR_CHARSET_DETECTION);
+ try {
+ byte[] sample = stream.readNBytes(BYTES_FOR_CHARSET_DETECTION);
+ if (sample.length == 0) {
+ DefaultLog.getInstance().debug(String.format("No contents in
file '%s'", documentName));
+ return null;
+ }
+
+ Metadata metadata = new Metadata();
+ ParseContext parseContext = new ParseContext();
+
+ try (TikaInputStream tis = TikaInputStream.get(sample, metadata)) {
+ List<EncodingResult> results = ENCODING_DETECTOR.detect(tis,
metadata, parseContext);
+
+ if (results.isEmpty()) {
+ DefaultLog.getInstance().debug(String.format("No encoding
found for file '%s'", documentName));
+ return null;
+ }
Review Comment:
This code does not do the same thing. the debug should be a warning.
And what happend to unsupported character sets?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
