This is an automated email from the ASF dual-hosted git repository. tballison pushed a commit to branch TIKA-4868-performance-improvements in repository https://gitbox.apache.org/repos/asf/tika.git
commit a1e91c362aa797e4adb9ac0896616951cb9afd91 Author: tallison <[email protected]> AuthorDate: Wed Sep 2 10:31:09 2026 -0400 TIKA-4868: tersify comments; CHANGES entries trimmed --- CHANGES.txt | 102 +++++++++------------ .../org/apache/tika/detect/DefaultDetector.java | 5 +- .../java/org/apache/tika/detect/MagicDetector.java | 5 +- .../main/java/org/apache/tika/mime/Patterns.java | 7 +- .../org/apache/tika/parser/CompositeParser.java | 8 +- .../apache/tika/sax/ToMarkdownContentHandler.java | 6 +- .../tika/parser/microsoft/WordExtractor.java | 8 +- .../org/apache/tika/parser/csv/CSVSniffer.java | 15 +-- .../apache/tika/pipes/core/ContentBytesConfig.java | 7 +- .../serdes/ParseContextSerializer.java | 5 +- 10 files changed, 66 insertions(+), 102 deletions(-) diff --git a/CHANGES.txt b/CHANGES.txt index f0a5cf99d3..54b847e3b5 100644 --- a/CHANGES.txt +++ b/CHANGES.txt @@ -1,80 +1,66 @@ Release 4.1.0 - unreleased + * TikaInputStream.hasReliableLength() distinguishes measured lengths + from declared Content-Length hints, and one-shot streams now carry a + declared length without spooling; detection sizes its magic read only + from measured lengths, so a lying declared length can no longer + truncate it (TIKA-4868). + + * ParseContext entries holding per-parse runtime state (ParseRecord, + ParseTimeout, the parser-map cache) are skipped during serialization + instead of failing as unregistered components (TIKA-4868). + * Mojibuster's adaptive probe strips incrementally instead of - re-stripping the whole buffer on every read (quadratic in read count - on tag-heavy pages; a 293KB page's probe drops 4.5ms -> 0.8ms). - Byte-identical: the resumable scanner holds back at a chunk-final - "<!" so chunked == single-shot at every boundary (exhaustive - boundary tests), and detection output was unchanged across a - 3,034-file differential. JunkDetector's Unicode block lookup also - uses a precomputed BMP table (exhaustive equivalence test) - (TIKA-4868). + re-stripping the whole buffer on every read (quadratic on tag-heavy + pages); JunkDetector's Unicode block lookup uses a precomputed BMP + table. Output unchanged (TIKA-4868). - * Markdown rendering is another ~8x faster on large documents: - a custom Text-node renderer emits unescaped spans in bulk instead of - commonmark's per-character escape-check-and-append (9MB render - 42ms -> 4ms; byte-identical output on a 1,900-document differential - across the test corpus and real-world HTML) (TIKA-4868). + * Markdown rendering is another ~8x faster on large documents: a custom + Text-node renderer emits unescaped spans in bulk instead of + commonmark's per-character escape-check-and-append. Byte-identical + output, guarded by a fast-vs-stock differential test (TIKA-4868). * ZipParser no longer re-decompresses an entry on every rewind when the entry uses a legacy compression method (implode, shrink, bzip2, ...): - those decode per-bit, so the rewind-by-reopen trade that works for - STORED/DEFLATED entries re-decoded the whole entry for each of the - several rewinds the embedded pipeline performs. Such entries now - replay from the budgeted cache. An imploded 606KB zip drops from - 610ms to 102ms in tika-server (TIKA-4868). + such entries replay from the budgeted cache instead of re-opening. + An imploded 606KB zip drops from 610ms to 102ms (TIKA-4868). * More detection/dispatch savings: the message/rfc822 priority-45 magic - is gated behind a ':' scan of the first 30 bytes (every match provably - has one, so results are unchanged and non-mail text skips its ~40 - clauses); CompositeParser caches the built type->parser map in the - ParseContext, so embedded documents reuse the container's map instead - of rebuilding it per document. Plain-text detection drops from 334us - (4.1.0-dev ran it twice) to 52us per document; eml requests ~-45%, - ppt ~-35% in tika-server. MagicDetector also precomputes a 256-entry - first-byte table per pattern (mask and case fold become one array - load per scanned position). Adds a RESOURCE_TIMING log on - org.apache.tika.pipes.timing.resource, silenced by default in the - shipped log4j2 configs (TIKA-4868). - - * WordExtractor (.doc) cleans each character run in one pass instead of - four chained replace/replaceAll copies, and tests paragraph blankness - without a regex replaceAll; ToMarkdownContentHandler collapses line - breaks in one pass with no copy for clean runs. ~39% off parsing a - text-heavy 2MB .doc (TIKA-4868). + is gated behind a lossless ':' scan of the first 30 bytes; + CompositeParser caches the built type->parser map in the ParseContext + so embedded documents reuse the container's map; MagicDetector + precomputes a per-pattern first-byte table. Adds a RESOURCE_TIMING + log, silenced by default in the shipped log4j2 configs (TIKA-4868). + + * WordExtractor (.doc) cleans each character run and tests paragraph + blankness in single passes; ToMarkdownContentHandler collapses line + breaks copy-free for clean runs. ~39% off a text-heavy 2MB .doc + (TIKA-4868). * tika-server's raw-output endpoints (/tika, /tika/text, ...) carry the extracted content as raw UTF-8 bytes from the pipes worker to the HTTP - response instead of a Smile-encoded string: one encode in the worker - replaces a string transcode on both sides of the IPC plus a re-encode - at the HTTP layer (9MB text: 115ms -> 75ms end-to-end). Opt-in via - the new content-bytes-config parse-context component, which moves - CONTENT_ONLY passback content out of tk:content into - EmitData.getContentBytes() (TIKA-4868). + response instead of a Smile-encoded string (9MB text: 115ms -> 75ms). + Opt-in via the new content-bytes-config parse-context component, which + moves CONTENT_ONLY passback content out of tk:content into + EmitData.getContentBytes(); results routed to a regular Emitter get + the content restored to the metadata (TIKA-4868). * Detection hot-path cleanups: MagicMatch resolves its detector via - double-checked locking instead of a synchronized method per eval; - glob patterns are compiled once at registration instead of per - lookup; MimeTypes.forName reads a ConcurrentHashMap (fixing an - unsynchronized-read race) and indexes the normalized key so - parameterized aliases stop re-locking per call; resource names + double-checked locking; glob patterns are compiled once at + registration; MimeTypes.forName reads a ConcurrentHashMap (fixing an + unsynchronized-read race) and indexes normalized keys; resource names containing spaces skip the URI-parse-by-exception; the magic-header - buffer is sized by the stream's known length instead of a fixed - 64KB per detection; and the Adobe Illustrator ranged regex is - gated behind a literal scan (detection results unchanged, - ~28us off every non-AI detection) (TIKA-4868). + buffer is sized by the stream's measured length instead of a fixed + 64KB; the Adobe Illustrator ranged regex is gated behind a literal + scan. Detection results unchanged (TIKA-4868). * Magic detection is ~35% faster on unmatched (e.g. plain-text) input: - MagicDetector range scans find first-byte candidates with a tight - scan before running the full masked/case-folded compare, instead of - paying it at every offset in the range (TIKA-4868). + range scans find first-byte candidates before running the full + masked/case-folded compare (TIKA-4868). * CSVSniffer reads its detection window once into a shared buffer and - runs every delimiter hypothesis against it, instead of re-reading the - stream through a pushback/mark-reset stack per delimiter; windows with - no delimiter and no quote skip the scan outright. Identical results - on a 1,125-file differential; ~28% off small-text parse time in - tika-server (TIKA-4868). + runs every delimiter hypothesis against it; windows with no delimiter + and no quote skip the scan outright. Results unchanged (TIKA-4868). * Pipes workers no longer stall between pre-parse and parse waiting for the client to acknowledge the intermediate-result frame; the diff --git a/tika-core/src/main/java/org/apache/tika/detect/DefaultDetector.java b/tika-core/src/main/java/org/apache/tika/detect/DefaultDetector.java index bda7f5e6f9..200c3eec9a 100644 --- a/tika-core/src/main/java/org/apache/tika/detect/DefaultDetector.java +++ b/tika-core/src/main/java/org/apache/tika/detect/DefaultDetector.java @@ -135,10 +135,7 @@ public class DefaultDetector extends CompositeDetector { @Override public MediaType detect(TikaInputStream tis, Metadata metadata, ParseContext parseContext) throws IOException { - // 0. An override short-circuits detection entirely, as in CompositeDetector. - // Without this check the magic scan below runs even when a prior detect pass - // recorded its result as CONTENT_TYPE_PARSER_OVERRIDE -- doubling detection - // cost on every pipes parse. + // An override short-circuits detection entirely, as in CompositeDetector. MediaType override = detectOverrides(metadata); if (override != null) { return override; diff --git a/tika-core/src/main/java/org/apache/tika/detect/MagicDetector.java b/tika-core/src/main/java/org/apache/tika/detect/MagicDetector.java index 2af36ee54f..e5476204a4 100644 --- a/tika-core/src/main/java/org/apache/tika/detect/MagicDetector.java +++ b/tika-core/src/main/java/org/apache/tika/detect/MagicDetector.java @@ -507,9 +507,8 @@ public class MagicDetector implements Detector { } } } else { - // Range scans (e.g. "\nHeader:" over 0:1024) dominate detection time, so - // find first-byte candidates with a tight scan and run the full compare - // only there, instead of paying the masked/case machinery at every offset. + // Range scans (e.g. "\nHeader:" over 0:1024) dominate detection cost: + // scan for first-byte candidates, run the full compare only there. if (length == 0) { // degenerate empty pattern: preserves the old loop's outcome return startOffset <= endOffset && startOffset <= buffer.length; diff --git a/tika-core/src/main/java/org/apache/tika/mime/Patterns.java b/tika-core/src/main/java/org/apache/tika/mime/Patterns.java index f5f8c234e0..46fe77e0d5 100644 --- a/tika-core/src/main/java/org/apache/tika/mime/Patterns.java +++ b/tika-core/src/main/java/org/apache/tika/mime/Patterns.java @@ -52,10 +52,9 @@ class Patterns implements Serializable { new TreeMap<>(new LengthComparator()); /** - * Compiled forms of {@link #globs}' keys. Matching recompiled every glob regex - * per lookup before; for names that miss the name/extension indexes that was - * a Pattern.compile per glob per call. Transient with a lazy rebuild so a - * serialized form from a build without this field still deserializes. + * Compiled forms of {@link #globs}' keys, so a lookup does not + * Pattern.compile per glob. Transient with a lazy rebuild so a serialized + * form from a build without this field still deserializes. */ private transient Map<String, Pattern> compiledGlobs = new HashMap<>(); private int minExtensionLength = Integer.MAX_VALUE; diff --git a/tika-core/src/main/java/org/apache/tika/parser/CompositeParser.java b/tika-core/src/main/java/org/apache/tika/parser/CompositeParser.java index 7d6a46e0b3..c4de97085a 100644 --- a/tika-core/src/main/java/org/apache/tika/parser/CompositeParser.java +++ b/tika-core/src/main/java/org/apache/tika/parser/CompositeParser.java @@ -249,11 +249,9 @@ public class CompositeParser implements Parser { /** * Cache of built parser maps, stored in the ParseContext so every embedded - * document in a parse reuses the container's map instead of rebuilding it - * (a full walk of every parser's supported types). Keyed by parser instance - * because nested composites share one context. The map is built once per - * (parser, context) and lives as long as the context: a context entry that - * would change a parser's supported types is not picked up mid-parse, nor on + * document in a parse reuses the container's map. Keyed by parser instance + * because nested composites share one context. Lives as long as the context: + * a change to a parser's supported types is not picked up mid-parse, nor on * later parses that reuse the same ParseContext instance. */ private static final class ParserMapCache implements TransientParseState { diff --git a/tika-core/src/main/java/org/apache/tika/sax/ToMarkdownContentHandler.java b/tika-core/src/main/java/org/apache/tika/sax/ToMarkdownContentHandler.java index 6a958b1a7d..8be96fd00a 100644 --- a/tika-core/src/main/java/org/apache/tika/sax/ToMarkdownContentHandler.java +++ b/tika-core/src/main/java/org/apache/tika/sax/ToMarkdownContentHandler.java @@ -420,10 +420,8 @@ public class ToMarkdownContentHandler extends DefaultHandler { if (finished) { return; } - // MarkdownRenderer emits mostly one char at a time; Writer.write(int) is - // synchronized and allocating in every stock Writer, which made rendering - // ~4x slower than the render logic itself on multi-MB documents. The - // unsynchronized buffer turns those calls into array stores. + // The renderer writes mostly one char at a time; stock Writers take a + // synchronized, allocating write(int) for each. The buffer makes those array stores. RenderBuffer buffered = new RenderBuffer(writer); renderer.render(document, buffered); buffered.flush(); diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/WordExtractor.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/WordExtractor.java index 18efa9bcd2..346de85dcc 100644 --- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/WordExtractor.java +++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-microsoft-module/src/main/java/org/apache/tika/parser/microsoft/WordExtractor.java @@ -106,8 +106,7 @@ public class WordExtractor extends AbstractPOIFSExtractor { return count; } - // matches the old regex [\r\n\s]+ ( \s == [ \t\n\x0B\f\r] ), without the - // full replaceAll copy it used to make per paragraph + // blank == matches [\r\n\s]+ ( \s == [ \t\n\x0B\f\r] ) private static boolean isBlankParagraph(String text) { for (int i = 0; i < text.length(); i++) { char c = text.charAt(i); @@ -492,11 +491,6 @@ public class WordExtractor extends AbstractPOIFSExtractor { // Strip the table cell end marker text = text.substring(0, text.length() - 1); } - // One pass over the run replaces what used to be four chained - // replace/replaceAll copies (this runs once per character run): - // \r and every other control char become line breaks, POI's char 30 - // (non-breaking hyphen) and char 31 (non-required hyphen) become - // their Unicode equivalents. text = cleanControlCharacters(text); xhtml.characters(text); } diff --git a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-text-module/src/main/java/org/apache/tika/parser/csv/CSVSniffer.java b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-text-module/src/main/java/org/apache/tika/parser/csv/CSVSniffer.java index 0ef0cd53cf..f5e7c2a405 100644 --- a/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-text-module/src/main/java/org/apache/tika/parser/csv/CSVSniffer.java +++ b/tika-parsers/tika-parsers-standard/tika-parsers-standard-modules/tika-parser-text-module/src/main/java/org/apache/tika/parser/csv/CSVSniffer.java @@ -57,10 +57,8 @@ class CSVSniffer { if (!reader.markSupported()) { reader = new BufferedReader(reader); } - // Every snifflet examines the same window, so read it once into a buffer - // instead of once per delimiter through a mark/reset + pushback stack. - // Grown on demand: markLimit is user-configurable and most inputs are - // far smaller, so an eager char[markLimit] would be waste per parse. + // Every snifflet examines the same window, so read it once. Grown on + // demand: markLimit is user-configurable and usually far exceeds the input. char[] buf = new char[Math.min(markLimit, 8192)]; reader.mark(markLimit); int len = 0; @@ -172,8 +170,7 @@ class CSVSniffer { //hardcode this for now private final char quoteCharacter = '"'; - // The shared window read once by sniff(Reader); pos is the cursor, so - // "unread" is a decrement and the mark-limit check is a bounds check. + // the shared window from sniff(Reader); pos is the cursor private final char[] buf; private final int len; private int pos = 0; @@ -184,8 +181,7 @@ class CSVSniffer { boolean rowZeroEmpty = false; int encapsulated = 0; //number of cells that are encapsulated in dquotes (for now) boolean parseException = false; - // Cell content is never analyzed (see the unquoted() TODO that was here); - // only whether the current cell is non-empty matters. + // only cell non-emptiness matters; content is never analyzed private int unquotedLen = 0; public Snifflet(char delimiter, char[] buf, int len) { @@ -322,8 +318,7 @@ class CSVSniffer { } private int read() throws IOException { - // pos tracks chars consumed exactly as charsRead did (unread decrements - // both), so the original off-by-one mark-limit semantics are preserved. + // markLimit - 1, not markLimit: HitMarkLimit must win over EOF at the boundary if (pos >= markLimit - 1) { throw new HitMarkLimitException(); } diff --git a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/ContentBytesConfig.java b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/ContentBytesConfig.java index a8f8d2c5e3..2e8b3a2b42 100644 --- a/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/ContentBytesConfig.java +++ b/tika-pipes/tika-pipes-core/src/main/java/org/apache/tika/pipes/core/ContentBytesConfig.java @@ -25,10 +25,9 @@ import org.apache.tika.annotation.TikaComponent; * {@code TIKA_CONTENT} into raw UTF-8 bytes on the {@code EmitData}, which travel * as binary over the IPC instead of a Smile-encoded string. A caller who sets this * must read {@code EmitData.getContentBytes()}; the metadata no longer carries the - * content. tika-server sets it for the raw-output endpoints. Results routed to a - * regular Emitter get the content restored into the metadata first - * ({@code EmitDataImpl#restoreContentFromBytes()}), so the move only sticks for - * consumers on the passback path. + * content. tika-server sets it for the raw-output endpoints. Results routed to + * a regular Emitter get the content restored first (see + * {@code EmitDataImpl#restoreContentFromBytes}). */ @TikaComponent(name = "content-bytes-config") public class ContentBytesConfig implements Serializable { diff --git a/tika-serialization/src/main/java/org/apache/tika/serialization/serdes/ParseContextSerializer.java b/tika-serialization/src/main/java/org/apache/tika/serialization/serdes/ParseContextSerializer.java index 658dcaaf51..bff48304cb 100644 --- a/tika-serialization/src/main/java/org/apache/tika/serialization/serdes/ParseContextSerializer.java +++ b/tika-serialization/src/main/java/org/apache/tika/serialization/serdes/ParseContextSerializer.java @@ -80,9 +80,8 @@ public class ParseContextSerializer extends JsonSerializer<ParseContext> { continue; } - // Per-parse runtime state (ParseRecord, ParseTimeout, parser-map cache) - // is never configuration; a context that has been through a parse must - // still serialize. + // per-parse runtime state (ParseRecord, ParseTimeout, ...) is never + // configuration; a context that has been through a parse must still serialize if (value instanceof TransientParseState) { continue; }
