gnodet-bot commented on code in PR #26471:
URL: https://github.com/apache/camel/pull/26471#discussion_r4015184619
##########
components/camel-ai/camel-langchain4j-ingest/src/main/docs/langchain4j-ingest-component.adoc:
##########
@@ -96,8 +96,30 @@ replacement.
After ingestion the message body is replaced with the `IngestResult`, so a
request-reply
caller receives the outcome of its call: the pipeline, the document id, the
number of segments
-written and the outcome — `ingested`, `empty` (blank document, nothing
written) or `skipped`
-(duplicate id, see below).
+written and the outcome — `ingested`, `empty` (blank document, nothing
written), `skipped`
+(duplicate id, see below) or `filtered` (rejected by a filter option, see
below).
+
+=== Filtering deliveries
+
+Not everything a consumer delivers belongs in a knowledge base. `includeId`
and `excludeId`
+take comma-separated Ant-style patterns matched against the document id —
exclusion wins —
+and act before the body is read and before the dedup claim, so a rejected
delivery costs
+nothing and never occupies its id. A pattern and an id must agree on a leading
`/`:
+`docs/**` never matches `/docs/guide.md`, so ids derived from absolute paths
need patterns
+starting with `/`. `minDocumentSize` filters documents too short to carry
+retrievable content, and `documentFilter` names a `Predicate` bean evaluated
with the body
+available; both answer `filtered` and release the dedup claim like a blank
document does, so
+a later, acceptable delivery under the same id still ingests. A duplicate id
is answered
+`skipped` before the content filters run, and the predicate runs before the
blank-document
+check — it must tolerate an absent body, and rejecting a blank delivery
answers `filtered`
Review Comment:
The doc says "the predicate runs before the blank-document check — it must
tolerate an absent body, and rejecting a blank delivery answers `filtered`
rather than `empty`". That is correct — `ingestUnlessFiltered()` calls
`filter.matches()` before `service.ingest()`, which does the blank check.
But this contract has no test. The existing
`blankBodyAnswersEmptyNotFiltered` only covers the `minDocumentSize` path
(`direct:min`, which has no `documentFilter`). There is nothing asserting that
when a `documentFilter` rejects a blank body, the outcome is `FILTERED` (not
`EMPTY`), and nothing asserting that a predicate that NPEs on a null/blank body
propagates correctly.
Given the doc explicitly calls this out as a supported and meaningful edge
case, it deserves a test case in `LangChain4jIngestFilterTest`. Something like:
```java
@Test
void blankBodyRejectedByPredicateAnswersFiltered() {
// The predicate runs before the blank-document check:
// a predicate that rejects a blank body answers FILTERED, not EMPTY
IngestResult result = ingest("direct:predicate", "doc-blank-p", " ");
assertThat(result.outcome()).isEqualTo(IngestResult.Outcome.FILTERED);
}
```
(where `direct:predicate` uses `documentFilter=#bean:confidential` — the
`confidential` predicate returns false when the body does not contain
`CONFIDENTIAL`, but a whitespace-only body also doesn't contain `CONFIDENTIAL`,
so it would return false and the outcome should be `FILTERED` — verify this is
the intended semantic.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
