Davis-Zhang-Onehouse opened a new issue, #20695: URL: https://github.com/apache/hudi/issues/20695
Part of #20683 (RFC-118, #20684). The text preprocessing framework of the embedding stream: the file preprocessor interface, chunks and chunk ids, prep counters, failures and side outputs, BLOB byte sources, text normalization, chunk packing under a token budget, and tokenizers (word count and an OpenAI BPE tokenizer on jtokkit). PR: (linked once opened) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
