Davis-Zhang-Onehouse opened a new pull request, #20696:
URL: https://github.com/apache/hudi/pull/20696

   ### Describe the issue this Pull Request addresses
   
   issue: #20695
   Part of #20683, RFC-118 (#20684).
   Stacked on #20694: review only the last commit. Until the PRs below it 
merge, this PR also shows their commits.
   
   Adds the text preprocessing framework that the RFC-118 embedding stream uses 
to turn files into token-bounded chunks.
   
   ### Summary and Changelog
   
   - Preprocessor contract: `FilePreprocessor`, `Chunk`, `ChunkIdScheme` / 
`SequentialChunkIdScheme`, `SourceKind`, `FileMeta`.
   - Prep accounting: `PrepCounters`, `PrepFailures`, `PrepSideOutput`, 
`ErrorText`, and the preprocessing exceptions (`PreprocessingException`, 
`SourceReadException`, `PrepPlatformException`, 
`PreprocessorConfigurationException`, `EmbeddingPrepFailedException`).
   - BLOB byte access for preprocessors: `BlobBytesSource`, 
`BlobBytesSupplier`, `BlobReferenceException`, `BlobTruncatedException`, 
`SourceFailures`, `StorageReferenceRules`.
   - Text: `TextNormalizer`, `ChunkPacker` / `PackedChunk` / `PackFailures` 
with `TokenBudget` and `TokenUnit`, and tokenizers (`Tokenizer`, 
`TokenizerFactory`, `WordCountTokenizer`, `OpenAiBpeTokenizer` on jtokkit).
   - 13 test files.
   
   All files are new; no existing Hudi file changes. No code was copied from 
other projects.
   
   ### Impact
   
   New classes for later stages; no change to existing behaviour.
   
   ### Risk Level
   
   low: new code only, not yet wired into a running path. Verified on JDK 11 / 
Spark 3.5 / Scala 2.12 with 117 tests, module compilation, checkstyle and RAT.
   
   ### Documentation Update
   
   none in this PR; RFC-118 describes preprocessing, and a website page follows 
once the chain lands.
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [PR] feat(ingest): RFC-118: ... Davis-Zhang-Onehouse (via GitHub) via commits

Reply via email to