Jiří Ondrušek created CAMEL-24621:
-------------------------------------
Summary: camel-langchain4j-ingest - document ingestion component:
the missing ingestion half of RAG in Camel
Key: CAMEL-24621
URL: https://issues.apache.org/jira/browse/CAMEL-24621
Project: Camel
Issue Type: New Feature
Reporter: Jiří Ondrušek
Assignee: Jiří Ondrušek
Camel today has strong building blocks for the *retrieval* side of RAG —
camel-langchain4j-chat and camel-langchain4j-agent consume a vector store,
camel-langchain4j-embeddingstore searches one — but nothing that packages
getting documents *into* the store. A user who wants a knowledge base has to
hand-orchestrate splitting, embedding, batching, metadata and deduplication in
every route. [design/langchain4j-evolution.adoc, "Phase 4: The RAG and
AIService
gap"|https://github.com/apache/camel/blob/main/design/langchain4j-evolution.adoc#phase-4-the-rag-and-aiservice-gap]
names this gap explicitly ("there was still no way to do RAG end-to-end").
This ticket proposes closing it with a new component in components/camel-ai.
*What it brings*
One line turns any of Camel's 300+ consumers into a RAG ingestion source:
{code}
from("aws2-s3://product-docs?deleteAfterRead=false")
.to("langchain4j-ingest:products?documentIdHeader=CamelAwsS3Key");
{code}
The producer splits the payload into overlapping segments (recursive splitter,
configurable sizes), embeds them in batches of 32 (staying under embedding
providers' per-request limits), and writes them to any LangChain4j
EmbeddingStore — Qdrant, Milvus, pgvector, Weaviate, Neo4j and the rest —
stamping every segment with camel_ingest_pipeline and camel_ingest_document_id
metadata so retrieval can cite its sources and a future synchronising engine
can find a document's vectors again. Data is written the way LangChain4j itself
writes it (the same compatibility argument that shaped
camel-langchain4j-embeddingstore), so existing retrieval — Camel or plain
LangChain4j — finds it.
On top of the endpoint sit two thin layers, so every audience gets its natural
entry point: a declarative Java API (IngestPipelineDefinition +
IngestPipelineRouteBuilder — "watch this directory, parse with Tika, dedupe,
store there" without writing route internals), and kamelets
(langchain4j-ingest-sink, tika/docling variants, a file source) giving
camel-jbang, camel-k and Pipe users a zero-route-code experience.
*Why users will like it*
* *Nothing new to learn*: it is an ordinary producer endpoint. Store and model
beans autowire when the registry holds exactly one of each; with zero or
several candidates the endpoint fails fast with the fix spelled out in the
message ("name the one to use with embeddingStore=#bean:...").
* *Continuous by nature*: unlike one-shot loaders (e.g. Easy RAG's startup-only
file loading), Camel sources poll and stream — the knowledge base stays current
as S3 buckets, Kafka topics or folders change.
* *Deduplication out of the box*: an optional IdempotentRepository makes
redeliveries and re-listings answer "skipped" instead of duplicating vectors —
first write wins per document id, a blank or failed delivery releases its
claim, and a shared JDBC register gives correct behaviour across replicas.
* *Safe defaults*: a knowledge base reads its source, it never consumes it
(noop, no deletes), half-copied files are waited for, and the document id is
captured before any parser runs, so a crafted document cannot forge its own
identity.
* *Honest result contract*: the reply body is an IngestResult (ingested / empty
/ skipped + segments written), so request-reply callers can react.
*Provenance and proof*
The design is not speculative: it is the field-proven engine of the
camel-quarkus-langchain4j-ingest extension (shipped in Camel Quarkus 3.39.0;
apache/camel-quarkus PRs #9018, #9040, #9078), ported to plain Camel with
runtime-neutral naming. Once released, the Quarkus extension will delegate to
this component, so both runtimes share a single implementation and Camel
Quarkus keeps only its build-time developer experience — one engine, maintained
once. A working PoC exists with a full unit suite plus an integration test that
ingests real files through the declarative API into a real Qdrant container
with a real all-MiniLM-L6-v2 model and verifies that semantic questions sharing
no vocabulary with the documents retrieve the right ones by meaning.
*Scope and follow-ups*
Proposed as Preview. Deliberately append-only in this increment — an edited
document re-ingests alongside its old segments; replace/delete arrive with a
later synchronising engine, for which the document-identity metadata laid down
here is the foundation. Follow-ups tracked separately: contributing the
kamelets to apache/camel-kamelets, a langChain4jRecursiveTokenizer for the
Tokenizer SPI, and the Camel Quarkus delegation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
