ottlinger commented on code in PR #714:
URL: https://github.com/apache/creadur-rat/pull/714#discussion_r3860617969
##########
src/changes/changes.xml:
##########
@@ -68,6 +68,9 @@ in order to be properly linked in site reports.
</release>
-->
<release version="1.0.0-SNAPSHOT" date="xxxx-yy-zz" description="Current
SNAPSHOT - release to be done">
+ <action issue="RAT-532" type="add" dev="pottlinger">
+ Update to Tika 4.0.0: new charset detection logic in Tika returns
different values compared to 3.x before, such as windows-1252 instead of
ISO-8859-1.
Review Comment:
Out of curiosity I added some logging locally in TikaProcessor to just list
the encodings:
```
[INFO] Running org.apache.rat.analysis.AnalyserFactoryTest
INFO: >>> /jira/RAT147/windows-newlines.txt.bin has encodings:
[[email protected][DECLARATIVE]]
INFO: >>>
/Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/Text.txt
has encodings: [[email protected][STATISTICAL]]
INFO: >>>
/Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/sub/Empty.txt
has encodings: [[email protected][STATISTICAL]]
INFO: >>> /jira/RAT147/unix-newlines.txt.bin has encodings:
[[email protected][DECLARATIVE]]
INFO: >>>
/Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/Text.txt
has encodings: [[email protected][STATISTICAL]]
INFO: >>>
/Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/sub/Empty.txt
has encodings: [[email protected][STATISTICAL]]
INFO: >>>
/Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/Text.txt
has encodings: [[email protected][STATISTICAL]]
INFO: >>>
/Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/LICENSE
has encodings: [[email protected][STATISTICAL]]
INFO: >>>
/Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/Text.txt
has encodings: [[email protected][STATISTICAL]]
INFO: >>>
/Users/me/creadur-rat/apache-rat-core/target/test-classes/examples/exampleData/sub/Empty.txt
has encodings: [[email protected][STATISTICAL]]
....
[INFO] Running org.apache.rat.report.xml.XmlReportFactoryTest
INFO: >>> /ILoggerFactory.java has encodings:
[[email protected][STATISTICAL]]
INFO: >>> /LICENSE has encodings: [[email protected][STATISTICAL]]
INFO: >>> /NOTICE has encodings: [[email protected][STATISTICAL]]
INFO: >>> /Source.java has encodings: [[email protected][STATISTICAL]]
INFO: >>> /Text.txt has encodings: [[email protected][STATISTICAL]]
INFO: >>> /TextHttps.txt has encodings: [[email protected][STATISTICAL]]
INFO: >>> /Xml.xml has encodings: [[email protected][STATISTICAL]]
INFO: >>> /buildr.rb has encodings: [[email protected][STATISTICAL]]
INFO: >>> /generated.txt has encodings: [[email protected][STATISTICAL]]
INFO: >>> /sub/Empty.txt has encodings: [[email protected][STATISTICAL]]
INFO: >>> /tri.txt has encodings: [[email protected][STATISTICAL]]
```
It seems that ISO is not recognized anymore with Tika 4.x.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
