Davis-Zhang-Onehouse opened a new issue, #20697:
URL: https://github.com/apache/hudi/issues/20697

   Part of #20683 (RFC-118, #20684).
   
   PDF text extraction (PDFBox) into token-bounded chunks, the preprocessor 
registry and iterator, and balancing of prep work across Spark partitions (hash 
and assignment strategies).
   
   Known limitation of this stage: extraction, normalization and packing keep 
whole-page strings, so retained text grows with page size rather than being 
capped by the chunk window. It is fixed later in this chain, before the chain 
completes.
   
   PR: (linked once opened)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to