abhinav-phi opened a new pull request, #2118: URL: https://github.com/apache/stormcrawler/pull/2118
## Upgrade to Apache Tika 4.0.0 — Closes #2063 This PR upgrades StormCrawler from Tika 3.3.2 to the first stable release of the Tika 4.x line ([announcement](https://lists.apache.org/thread/fo8ozc8hkkzmrbfmvojtocml4fbm2p3q)), tracking the 4.0.0 milestone. Tika 4.0 ships a number of breaking changes; this PR migrates the codebase, the bundled configuration and the documentation accordingly. No behaviour change is intended for users of the `JSoupParserBolt` and the Tika `ParserBolt` beyond the Tika upgrade itself. ### Dependency changes - `tika.version` `3.3.2` → `4.0.0` in the root POM. Tika 4.x requires Java 17+, which StormCrawler already targets (the project builds on JDK 17-25, CI runs JDK 25), so no compiler settings were needed. - In Tika 4.0, `tika-parsers-standard-package` changed from a **jar** to a **POM** that aggregates the individual `tika-parser-*-module` artifacts. `external/tika` now consumes it with `<type>pom</type>`, which pulls in all standard parser modules transitively — same effective set of parsers as before, including the OCR module. - Added explicit `tika-core` and `tika-serialization` dependencies to `external/tika`: `tika-serialization` provides the new `TikaLoader` used to read the JSON configuration (see below). The existing exclusions (`asm`, `slf4j-log4j12`, and the BSD-licensed `jai-imageio-core`, still a compile dependency of the image parser module) are preserved. - `THIRD-PARTY.txt` regenerated with `license:aggregate-add-third-party` (Tika modules now 4.0.0; new Tika artifacts: `tika-serialization`, `tika-encoding-detector-html`, `tika-encoding-detector-mojibuster`, `tika-ml-core`, `tika-ml-junkdetect`, `tika-parser-datauri-commons`; removals: `tika-parsers-standard-package` jar, `vorbis-java-tika`, POI `poi-ooxml-full`, PDFBox `jempbox`/`xmpbox` 3.0.8, `org.tukaani:xz` — replaced by Tika 4's internal dependencies). ### Code migration **`core/JSoupParserBolt.java`** (only core module using Tika, via `tika-core`): - `TikaConfig` was **removed** in Tika 4. The mime-type detector, previously obtained from `TikaConfig.getDefaultConfig().getDetector()`, is now built with `new DefaultDetector()`, which is exactly what the removed default configuration wired (bytecode of Tika 4's `Tika()` facade confirms it constructs `new DefaultDetector()`). - `Detector.detect()` now takes `(TikaInputStream, Metadata, ParseContext)`; `guessMimeType()` wraps the content bytes with `TikaInputStream.get(content)` and passes an empty `ParseContext`. - `Metadata` no longer implements `HttpHeaders`, so the `Metadata.CONTENT_TYPE` / `Metadata.CONTENT_LENGTH` constants are now taken from `org.apache.http.HttpHeaders`. Code referencing `TikaCoreProperties` constants is unaffected (the `RESOURCE_NAME_KEY` usage stays). **`external/tika/ParserBolt.java`**: - XML configurations are no longer supported in Tika 4; configuration moved to JSON. `instantiateTika()` now loads the configuration through `TikaLoader.load(path, classLoader)` (from the new `tika-serialization` module) and wires the result into the `Tika` facade via `new Tika(loader.loadDetectors(), loader.loadAutoDetectParser())` — the equivalent of the removed `new TikaConfig(url, classLoader)` + `new Tika(tikaConfig)` pair. On failure the bolt still falls back to the default configuration. - `TikaLoader` can only read configuration files from the filesystem. When the configuration is bundled inside a jar (the common case for topologies built as fat jars), `urlToPath()` copies it to a temporary file first. - The parse itself now uses `TikaInputStream.get(content)` in a try-with-resources block, replacing the manual `ByteArrayInputStream` + `finally` close (Tika 4's `Parser.parse()` requires a `TikaInputStream`, and `TikaInputStream` holds `TemporaryResources` that must be released). - Default value of `parser.tika.config.file` is now `tika-config.json`. ### Configuration migration - `external/tika/src/main/resources/tika-config.xml` → `tika-config.json`. The old XML only (a) excluded `TesseractOCRParser` from `DefaultParser` and (b) set `service-loader` warning handlers, which are the Tika 4 defaults. Using Tika's own `XmlToJsonConfigConverter` semantics, this becomes: ```json { "parsers": [ { "default-parser": { "exclude": ["tesseract-ocr-parser"] } } ] } ``` (OCR stays disabled by default in StormCrawler — Tesseract is an optional native dependency.) - Archetype `crawler-conf.yaml` files (core, OpenSearch, Solr) updated to `parse.tika.config.file: "tika-config.json"`. ### Documentation - `external/tika/README.md` and `docs/configuration.adoc` updated for the JSON configuration and the new default file name. ### Testing - `mvn verify` with `CI_ENV=true`: - `stormcrawler-core`: **415 tests, 0 failures** (includes `JSoupParserBoltTest` mime-type detection). - `stormcrawler-tika`: **ParserBoltTest green** — parses a recursive embedded `.docx` (TIKA-2096 scenario) through the new `TikaLoader`/JSON-config path, and exercises the mime-type whitelist. - `stormcrawler-urlfrontier`, `stormcrawler-langid`, archetypes, docs: green. - `-Prat -DskipTests verify -Dskip.format.code=false` (CI's license + format job): the new `tika-config.json` is approved by RAT; google-java-format validation passes on all touched Java files. - Two test failures encountered locally are pre-existing Windows-environment issues, reproduced identically on a pristine checkout of `main` and unrelated to this change: `HttpRobotRulesParserRedirectTest` (WireMock cannot bind port 8089 on Windows) and `WARCHdfsBoltTest` (Hadoop requires `winutils.exe` on Windows); Testcontainers-based tests (OpenSearch/Solr/SQL) require Docker, unavailable locally. CI runs on Ubuntu with Docker where these all pass. ### Notes for reviewers - The Tika 4 metadata key renames (`tk:` prefix) only affect keys **emitted by parsers**, which this bolt copies with a `parse.` prefix into StormCrawler metadata. Users with ParseFilters/indexing rules matching specific `parse.*` Tika keys may be affected by the renames; the upstream [migration guide](https://tika.apache.org/4.0.0/index.html) documents an opt-in legacy-key filter (`metadata-migration-3x-4x.json`) for that case. - HTML remains the domain of `JSoupParserBolt`; the "default content handler is now Markdown" change does not affect StormCrawler, which always supplies its own `BodyContentHandler`/`LinkContentHandler`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
