JiriOndrusek opened a new pull request, #26190: URL: https://github.com/apache/camel/pull/26190
fixes https://issues.apache.org/jira/browse/CAMEL-24621 # What this brings New component camel-langchain4j-ingest: the ingestion half of a RAG setup. Documents flowing through a Camel route are split into segments, embedded and written into a LangChain4j EmbeddingStore — turning any Camel consumer (a directory, S3, Kafka, …) into a feed for a knowledge base, with optional deduplication by document id. The feature originated in Camel Quarkus, where it shipped as the camel-quarkus-langchain4j-ingest extension (CQ 3.39.0). This PR ports the engine to Apache Camel so that the Quarkus extension can delegate to it and other runtimes (Camel Main, Spring Boot, JBang) get the same capability, with Kamelets to follow. Support level: Preview. Follow-ups planned separately: a set of Kamelets in camel-kamelets wrapping this endpoint (file source + tika/docling parsing sinks), and the Camel Quarkus delegation. # Technical details A producer-only endpoint, langchain4j-ingest:pipelineName, backed by LangChain4j's own EmbeddingStoreIngestor (fully explicit builder — no ServiceLoader defaults can leak in): ``` exchange │ ├─ resolve document id │ CamelLangChain4jIngestDocumentId exchange property (set pre-parse by the route, │ else <documentIdHeader> header so a parsed document cannot │ else FAIL forge its own identity) │ ├─ idempotentRepository configured? │ yes: add(exchange,id)? ──false──> reply SKIPPED, done (eager claim, before │ the body is read) ├─ blank text ────────────────────────> reply EMPTY (claim released) ├─ text > maxDocumentSize ────────────> FAIL (claim released) │ ├─ split ── default recursive splitter (maxSegmentSize/maxOverlapSize) │ or a DocumentSplitter bean (documentSplitter=#bean:...) ├─ embed ── batched (embeddingBatchSize, default 32) ──> EmbeddingModel ├─ store ── ONE addAll after all segments are embedded ──> EmbeddingStore │ (a mid-embedding failure writes nothing) │ └─ reply IngestResult(pipeline, documentId, segmentsWritten, INGESTED | EMPTY | SKIPPED) (claim confirmed on INGESTED only) ``` - Every segment carries camel_ingest_pipeline and camel_ingest_document_id metadata, so retrieval can cite its source and a future synchronising engine can find a document's vectors again. - embeddingStore/embeddingModel are autowired from the registry when unambiguous; zero or several candidates fail the start with the option to set. The idempotent repository is referenced by name on purpose and used through the Exchange-aware overloads, matching the idempotent consumer EIP; claim semantics (eager claim, release on EMPTY/failure) are documented including the at-least-once caveat. - TikaTextDecode ships as a helper for the documented parse-route shape: it decodes Tika's pinned-encoding text output without consulting the exchange charset heuristic (CamelCharsetName header outranks the property, so convertBodyTo(String, "UTF-8") would be no substitute). - Tests: 28 unit tests (engine, dedup, bean resolution, validation, real Tika HTML+PDF and a stubbed Docling Serve behind hand-rolled routes) plus an integration test against a real Qdrant container with a real in-process embedding model, asserting semantic retrieval with deliberately no shared vocabulary. - Docs cover the directory and consumer recipes, dedup limits, the size cap (whole-document-in-memory by design) and parser hygiene; catalog, DSL builder factories, bom/coverage wiring included. # Description <!-- - Write a pull request description that is detailed enough to understand what the pull request does, how, and why. --> # Target - [ ] I checked that the commit is targeting the correct branch (Camel 4 uses the `main` branch) # Tracking - [ ] If this is a large change, bug fix, or code improvement, I checked there is a [JIRA issue](https://issues.apache.org/jira/browse/CAMEL) filed for the change (usually before you start working on it). <!-- # *Note*: trivial changes like, typos, minor documentation fixes and other small items do not require a JIRA issue. In this case your pull request should address just this issue, without pulling in other changes. --> # Apache Camel coding standards and style - [ ] I checked that each commit in the pull request has a meaningful subject line and body. <!-- If you're unsure, you can format the pull request title like `[CAMEL-XXX] Fixes bug in camel-file component`, where you replace `CAMEL-XXX` with the appropriate JIRA issue. --> - [ ] I have run `mvn clean install -DskipTests` locally from root folder and I have committed all auto-generated changes. <!-- You can run the aforementioned command in your module so that the build auto-formats your code. This will also be verified as part of the checks and your PR may be rejected if if there are uncommited changes after running `mvn clean install -DskipTests`. You can learn more about the contribution guidelines at https://github.com/apache/camel/blob/main/CONTRIBUTING.md --> # AI-assisted contributions - [ ] If this PR includes AI-generated code, commits have proper co-authorship attribution (e.g., `Co-authored-by` trailers) and the PR description identifies the AI tool used. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
