Davis-Zhang-Onehouse opened a new pull request, #20696: URL: https://github.com/apache/hudi/pull/20696
### Describe the issue this Pull Request addresses issue: #20695 Part of #20683, RFC-118 (#20684). Stacked on #20694: review only the last commit. Until the PRs below it merge, this PR also shows their commits. Adds the text preprocessing framework that the RFC-118 embedding stream uses to turn files into token-bounded chunks. ### Summary and Changelog - Preprocessor contract: `FilePreprocessor`, `Chunk`, `ChunkIdScheme` / `SequentialChunkIdScheme`, `SourceKind`, `FileMeta`. - Prep accounting: `PrepCounters`, `PrepFailures`, `PrepSideOutput`, `ErrorText`, and the preprocessing exceptions (`PreprocessingException`, `SourceReadException`, `PrepPlatformException`, `PreprocessorConfigurationException`, `EmbeddingPrepFailedException`). - BLOB byte access for preprocessors: `BlobBytesSource`, `BlobBytesSupplier`, `BlobReferenceException`, `BlobTruncatedException`, `SourceFailures`, `StorageReferenceRules`. - Text: `TextNormalizer`, `ChunkPacker` / `PackedChunk` / `PackFailures` with `TokenBudget` and `TokenUnit`, and tokenizers (`Tokenizer`, `TokenizerFactory`, `WordCountTokenizer`, `OpenAiBpeTokenizer` on jtokkit). - 13 test files. All files are new; no existing Hudi file changes. No code was copied from other projects. ### Impact New classes for later stages; no change to existing behaviour. ### Risk Level low: new code only, not yet wired into a running path. Verified on JDK 11 / Spark 3.5 / Scala 2.12 with 117 tests, module compilation, checkstyle and RAT. ### Documentation Update none in this PR; RFC-118 describes preprocessing, and a website page follows once the chain lands. ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
