[
https://issues.apache.org/jira/browse/TIKA-4731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18083831#comment-18083831
]
ASF GitHub Bot commented on TIKA-4731:
--------------------------------------
tballison commented on code in PR #2839:
URL: https://github.com/apache/tika/pull/2839#discussion_r3310819996
##########
tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-pkg-module/src/main/java/org/apache/tika/parser/pkg/ZipParser.java:
##########
@@ -565,25 +557,12 @@ private String detectEntryName(ZipArchiveEntry entry,
Metadata parentMetadata,
return new String(entry.getRawName(), config.getEntryEncoding());
}
- // If charset detection is enabled, try to detect and decode
+ // If charset detection is enabled, try to detect and decode.
+ // Mojibuster handles short inputs natively (zip filenames are often
+ // 9-30 bytes); no byte-extension trick needed.
if (config.isDetectCharsetsInEntryNames()) {
byte[] entryName = entry.getRawName();
- // Extend short entry names before detection: statistical detectors
- // (e.g. UniversalEncodingDetector, Icu4j) need enough material to
- // make a confident call. Cyclically repeat the bytes so the
- // detector still sees the same byte distribution.
- byte[] extendedEntryName = entryName;
- if (entryName != null && 0 < entryName.length
- && entryName.length < MIN_BYTES_FOR_DETECTING_CHARSET) {
- int len = entryName.length
- * (MIN_BYTES_FOR_DETECTING_CHARSET / entryName.length);
- extendedEntryName = new byte[len];
- for (int i = 0; i < len; i++) {
- extendedEntryName[i] = entryName[i % entryName.length];
- }
- }
-
- try (TikaInputStream detectStream =
TikaInputStream.get(extendedEntryName)) {
+ try (TikaInputStream detectStream =
TikaInputStream.get(entryName)) {
List<EncodingResult> encResults =
Review Comment:
we won't ship with this until the sjis test passes. good catch. i should
improve documentation, but good enough for now.
> Ongoing improvements to the junk detector
> -----------------------------------------
>
> Key: TIKA-4731
> URL: https://issues.apache.org/jira/browse/TIKA-4731
> Project: Tika
> Issue Type: Task
> Reporter: Tim Allison
> Priority: Minor
>
> With [https://github.com/apache/tika/pull/2818,] I think we have a decent
> shape for the junk detector.
> There are several areas for improvement, but I think it is ready to go.
> This ticket tracks follow on work, including:
> * Smaller model
> * Handling pathological code block changes
> * Handling candidates with different character counts
> * Other items to be discovered in our commoncrawl/govdocs1 corpus?
> We have some coverage for the middle two item, but need to address those more
> directly.
> This work is not a blocker on the 4.0.0-beta-1 release.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
