rahil-c opened a new issue, #19676: URL: https://github.com/apache/hudi/issues/19676
Only `tika-core` ships in the utilities bundles, and `tika-core` registers no parser at all, not even for `text/plain`. Running the bundle as documented therefore produces `parse_status=EMPTY` for every file, empty `extracted_text`, no chunks, and a null vector on every row, with the job exiting 0 and logging nothing. `EMPTY` is also the normal outcome for an image, so nothing distinguishes "this file has no text" from "this deployment cannot extract text at all". Measured on a 2 GB corpus of 616 text and 616 binary files: | classpath | parse_status | chunks | | --- | --- | --- | | bundle as shipped | EMPTY = 1232 | 0 | | plus `tika-parsers-standard-package` | SUCCESS 554, TRUNCATED 62, EMPTY 616 | 115,873 | The 616 EMPTY in the second row are exactly the binary files, which is correct. In the first row every text file was empty too. The documented remedy, supplying the parsers at runtime with `spark-submit --packages`, does not work reliably. Ivy failed to resolve three of Tika's transitive dependencies. Passing the jars with `--jars` instead gets further and then throws `NoSuchMethodError: GzipUtils.getUncompressedFileName`, because Spark ships commons-compress 1.23.0 and `--jars` does not take precedence over Spark's own jars. That last failure is reached in practice rather than in theory. Tika detects format by magic number, and arbitrary binary content occasionally matches a compressed signature. A 1,232-file corpus never hit it; a 64,487-file corpus did. At document-lake scale it is a certainty. The same applies to metadata, which is easy to miss: a JPEG through `TikaDocumentParser` yields 25 metadata fields when the parser modules are present, and nothing at all without them. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
