[
https://issues.apache.org/jira/browse/CAMEL-24357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on CAMEL-24357 started by Andrea Cosentino.
------------------------------------------------
> camel-aws2-s3-vectors: consumer returns no results (topK forced to 0),
> ignores the delay option, and can lose vectors on failure
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: CAMEL-24357
> URL: https://issues.apache.org/jira/browse/CAMEL-24357
> Project: Camel
> Issue Type: Bug
> Components: camel-aws
> Reporter: Andrea Cosentino
> Assignee: Andrea Cosentino
> Priority: Major
> Fix For: 4.22.0
>
>
> The aws2-s3-vectors consumer has several defects, all stemming from consumer
> options that shadow the base scheduled-poll options.
> h3. 1. Consumer always sends topK(0) - returns nothing (primary)
> {{AWS2S3VectorsConsumer.poll()}} builds the query with:
> {code:java}
> .topK(Math.min(getMaxMessagesPerPoll(), getConfiguration().getTopK()))
> {code}
> {{getMaxMessagesPerPoll()}} is the base {{ScheduledBatchPollingConsumer}}
> field, which defaults to 0 and is never wired:
> {{AWS2S3VectorsEndpoint.createConsumer()}} only calls
> {{configureConsumer(consumer)}} and never {{setMaxMessagesPerPoll(..)}}
> (unlike {{AWS2S3Endpoint}}). The {{maxMessagesPerPoll}} URI option is bound
> to a *shadow* field on the Configuration that runtime code never reads. So
> the expression is {{Math.min(0, topK) = 0}} and every poll sends {{topK(0)}},
> which AWS S3 Vectors rejects (topK must be >= 1). The consumer delivers zero
> messages out of the box, and setting {{maxMessagesPerPoll}} on the URI does
> not help.
> h3. 2. delay option is ignored
> {{AWS2S3VectorsConfiguration}} redeclares {{delay}} (default 500). The
> generated configurer therefore maps {{?delay=}} to
> {{configuration.setDelay(..)}}, a field that is never read; the real poll
> interval comes from the inherited {{ScheduledPollEndpoint.delay}}. A user
> setting {{?delay=60000}} is silently ignored and the endpoint keeps polling
> at the default interval.
> h3. 3. Vectors are marked processed before routing (event loss on failure)
> {{poll()}} adds the vector id to {{processedVectorIds}} at enqueue time,
> before the exchange is routed. If routing later fails, the vector stays in
> the index but is in the de-dup set, so the fixed similarity query skips it
> forever (until restart). The de-dup set is also cleared only on stop, so it
> grows unbounded.
> h3. Fix
> - Wire {{setMaxMessagesPerPoll}} from the configuration in {{createConsumer}}
> and treat {{maxMessagesPerPoll <= 0}} as "no cap" when computing topK, so a
> real topK (>= 1) is always sent.
> - Remove the shadow {{delay}} field from the Configuration so {{?delay}}
> binds to the inherited scheduled-poll option.
> - Only track {{processedVectorIds}} when {{deleteAfterRead=false}} (deletion
> already prevents re-delivery) and drop the id again on failure so a failed
> exchange is retried on a later poll.
> Affects main (4.22.0) and 4.18.x (the module was added in 4.17.0; it does not
> exist on 4.14.x).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
