Andrea Cosentino created CAMEL-24357:
----------------------------------------
Summary: camel-aws2-s3-vectors: consumer returns no results (topK
forced to 0), ignores the delay option, and can lose vectors on failure
Key: CAMEL-24357
URL: https://issues.apache.org/jira/browse/CAMEL-24357
Project: Camel
Issue Type: Bug
Components: camel-aws
Reporter: Andrea Cosentino
Fix For: 4.22.0
The aws2-s3-vectors consumer has several defects, all stemming from consumer
options that shadow the base scheduled-poll options.
h3. 1. Consumer always sends topK(0) - returns nothing (primary)
{{AWS2S3VectorsConsumer.poll()}} builds the query with:
{code:java}
.topK(Math.min(getMaxMessagesPerPoll(), getConfiguration().getTopK()))
{code}
{{getMaxMessagesPerPoll()}} is the base {{ScheduledBatchPollingConsumer}}
field, which defaults to 0 and is never wired:
{{AWS2S3VectorsEndpoint.createConsumer()}} only calls
{{configureConsumer(consumer)}} and never {{setMaxMessagesPerPoll(..)}} (unlike
{{AWS2S3Endpoint}}). The {{maxMessagesPerPoll}} URI option is bound to a
*shadow* field on the Configuration that runtime code never reads. So the
expression is {{Math.min(0, topK) = 0}} and every poll sends {{topK(0)}}, which
AWS S3 Vectors rejects (topK must be >= 1). The consumer delivers zero messages
out of the box, and setting {{maxMessagesPerPoll}} on the URI does not help.
h3. 2. delay option is ignored
{{AWS2S3VectorsConfiguration}} redeclares {{delay}} (default 500). The
generated configurer therefore maps {{?delay=}} to
{{configuration.setDelay(..)}}, a field that is never read; the real poll
interval comes from the inherited {{ScheduledPollEndpoint.delay}}. A user
setting {{?delay=60000}} is silently ignored and the endpoint keeps polling at
the default interval.
h3. 3. Vectors are marked processed before routing (event loss on failure)
{{poll()}} adds the vector id to {{processedVectorIds}} at enqueue time, before
the exchange is routed. If routing later fails, the vector stays in the index
but is in the de-dup set, so the fixed similarity query skips it forever (until
restart). The de-dup set is also cleared only on stop, so it grows unbounded.
h3. Fix
- Wire {{setMaxMessagesPerPoll}} from the configuration in {{createConsumer}}
and treat {{maxMessagesPerPoll <= 0}} as "no cap" when computing topK, so a
real topK (>= 1) is always sent.
- Remove the shadow {{delay}} field from the Configuration so {{?delay}} binds
to the inherited scheduled-poll option.
- Only track {{processedVectorIds}} when {{deleteAfterRead=false}} (deletion
already prevents re-delivery) and drop the id again on failure so a failed
exchange is retried on a later poll.
Affects main (4.22.0) and 4.18.x (the module was added in 4.17.0; it does not
exist on 4.14.x).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
