github-actions[bot] commented on code in PR #69363:
URL: https://github.com/apache/doris/pull/69363#discussion_r4332858990
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProvider.java:
##########
@@ -573,10 +578,22 @@ private ParsedMessage parseMessage(Message message) {
if (isCommitted(key, message.messageId(),
message.receiptHandle())) {
continue;
}
+ String path = "s3://" + bucket + "/" + key;
Review Comment:
[P2] Preserve leading slashes in the HEAD object key
For a source `s3://bucket/**`, a notification for the existing key `/a.csv`
passes the matcher, but this path becomes `s3://bucket//a.csv` and
`S3ObjStorage.headObject` reparses it with `ObjectStorageUri`, which strips the
key's leading slash. HEAD therefore checks `a.csv`: if it is absent, the new
missing-object catch makes `deleteSkippedMessages` delete a valid notification
without ingesting `/a.csv`; if both exist, the scan receives the other object's
size/mtime. The previous exact-status BE scan preserves the leading slash. Use
a metadata lookup that preserves the exact decoded key and cover `/a.csv`
separately from `a.csv`.
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProvider.java:
##########
@@ -573,10 +578,22 @@ private ParsedMessage parseMessage(Message message) {
if (isCommitted(key, message.messageId(),
message.receiptHandle())) {
continue;
}
+ String path = "s3://" + bucket + "/" + key;
+ RemoteObject object;
+ try {
+ // Notifications can outlive an overwrite; refresh both scan
size and cache identity.
+ object = objectStorage.headObject(path);
Review Comment:
[P2] Renew receipt visibility during metadata refresh
This HEAD runs serially for every record before `receiveBatch` checks its
deadline or publishes a batch that can be renewed. With default
`max_interval=10`, the receipt lease is 120 seconds, while a supported receive
of two 80-record messages at one second per successful HEAD takes 160 seconds.
If INSERT is still running at the next metadata tick, renewal of those expired
receipts marks the batch expired and pauses/cancels the task; resume can repeat
the same cycle. The file limit also applies only after the whole receive. Keep
current and previously accepted receipts renewed while refreshing metadata. The
[SQS
API](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/APIReference/API_ChangeMessageVisibility.html)
documents non-inflight/invalid-receipt renewal errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
