Davis-Zhang-Onehouse opened a new issue, #20697: URL: https://github.com/apache/hudi/issues/20697
Part of #20683 (RFC-118, #20684). PDF text extraction (PDFBox) into token-bounded chunks, the preprocessor registry and iterator, and balancing of prep work across Spark partitions (hash and assignment strategies). Known limitation of this stage: extraction, normalization and packing keep whole-page strings, so retained text grows with page size rather than being capped by the chunk window. It is fixed later in this chain, before the chain completes. PR: (linked once opened) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
