Claus Ibsen created CAMEL-26092:
-----------------------------------
Summary: camel-aws2-s3-vectors - the headers of an earlier step
win over the endpoint, so a route from the consumer overwrites the consumed
vectors
Key: CAMEL-26092
URL: https://issues.apache.org/jira/browse/CAMEL-26092
Project: Camel
Issue Type: Bug
Components: camel-aws2
Affects Versions: 4.18.0, 4.22.0, 4.23.0
Reporter: Claus Ibsen
Fix For: 4.24.0
Found while adding the catalog body types of aws2-s3-vectors (CAMEL-24961).
h4. 1. The headers of an earlier step win over the endpoint
The producer reads the bucket and the index from the
{{CamelAwsS3VectorsVectorBucketName}} and {{CamelAwsS3VectorsVectorIndexName}}
headers first, and from its own options only when they are not set
({{AWS2S3VectorsProducer}}, main ed3d719b379c, lines 418-438). The key of
putVectors comes from {{CamelAwsS3VectorsVectorId}} (line 140), and getVectors
and deleteVectors read their keys from it before the body (lines 222 and 249).
The consumer sets all three headers ({{AWS2S3VectorsConsumer}}, lines 123-125),
createVectorBucket sets the bucket and putVectors sets the key (line 177).
Checked on 4.24.0-SNAPSHOT with {{camel run}} against the AWS emulator of
{{camel infra run aws}}, through a proxy that logs the requests:
* a route from
{{aws2-s3-vectors:bt-bucket?vectorIndexName=bt-index&consumerQueryVector=...}}
to
{{aws2-s3-vectors:bt-other?vectorIndexName=bt-other-index&operation=putVectors}}
wrote its vector into bt-bucket/bt-index under the consumed key, so each
consumed vector was overwritten;
* a createVectorBucket on {{aws2-s3-vectors:bt-other}} after a
createVectorBucket on bt-bucket created bt-bucket again ("The vector bucket
bt-bucket already exists");
* a getVectors after a putVectors in the same route ignored its body (the key
"doc-list") and got the key of the putVectors.
aws2-s3 had the same problem with the bucket header and has an explicit
{{CamelAwsS3OverrideBucketName}} header for it.
*Proposed change*: let the endpoint options win, or read only headers meant as
input (an override header), and do not reuse the headers that the consumer and
the producer set as outputs.
h4. 2. The vectors come back without their data, metadata or distance
getVectors, queryVectors and the consumer never set returnData, returnMetadata
or returnDistance on the request (lines 257-261 and 200-205,
{{AWS2S3VectorsConsumer}} lines 85-89), so each vector holds its key only.
Checked on 4.24.0-SNAPSHOT with {{camel run}} against the AWS emulator of
{{camel infra run aws}}, through a proxy that logs the requests: getVectors
gives {{[GetOutputVector(Key=doc-list)]}}. So the
{{CamelAwsS3VectorsSimilarityScore}} header is never set, and the
similarityThreshold option, the {{CamelAwsS3VectorsSimilarityThreshold}} and
{{CamelAwsS3VectorsMetadataFilter}} headers and the consumerMetadataFilter
option do nothing (the consumer only logs the filter, lines 92-96).
h4. 3. A null in a vector or a metadata value that is not a String fails with
an unclear error
A vector {{[0.1, null, 0.3]}} fails with {{NullPointerException: Cannot invoke
"Object.getClass()" because "item" is null}} (line 454), and a
{{CamelAwsS3VectorsVectorMetadata}} value such as the number 2026 fails with
{{ClassCastException: class java.lang.Integer cannot be cast to class
java.lang.String}} (line 504), although S3 Vectors metadata takes numbers and
booleans. Checked on 4.24.0-SNAPSHOT with {{camel run}} against the AWS
emulator of {{camel infra run aws}}, through a proxy that logs the requests.
h4. Also read in the code, not run
* The dataType option offers float16 and the distanceMetric option dot-product
(AWS2S3VectorsConfiguration, lines 70-72), which the DataType and
DistanceMetric enums of the AWS SDK do not have.
* getVectors and deleteVectors pass a List body as the keys without checking
its items (line 482): a List of numbers is sent as {{"keys":[1,2]}}.
* listVectorBuckets and listVectorIndexes give the first page only.
_Claude Code on behalf of davsclaus_
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
