github-actions[bot] commented on code in PR #69363:
URL: https://github.com/apache/doris/pull/69363#discussion_r4334147490
##########
fe/fe-core/src/main/java/org/apache/doris/job/offset/s3/S3EventSourceOffsetProvider.java:
##########
@@ -573,10 +578,22 @@ private ParsedMessage parseMessage(Message message) {
if (isCommitted(key, message.messageId(),
message.receiptHandle())) {
continue;
}
+ String path = "s3://" + bucket + "/" + key;
+ RemoteObject object;
+ try {
+ // Notifications can outlive an overwrite; refresh both scan
size and cache identity.
+ object = objectStorage.headObject(path);
+ } catch (FileNotFoundException e) {
+ log.info("Skip missing S3 EVENT object: jobId={},
messageId={}, path={}",
+ jobId, message.messageId(), path);
+ continue;
+ } catch (IOException e) {
+ throw new IllegalStateException("Failed to read S3 EVENT
object metadata: " + path, e);
+ }
files.add(key);
// The object must remain unchanged until ingestion completes;
notifications identify keys, not versions.
- fileStatuses.add(new TBrokerFileStatus("s3://" + bucket + "/" +
key, false,
- record.s3.object.size, true));
+ fileStatuses.add(new TBrokerFileStatus(path, false,
object.getSize(), true)
+ .setModificationTime(object.getModificationTime()));
Review Comment:
[P2] Keep S3 generations distinct in the parsed-footer caches
S3 replacements can share a `Last-Modified` second ([HEAD
response](https://docs.aws.amazon.com/AmazonS3/latest/API/API_HeadObject.html),
[HTTP-date
precision](https://www.rfc-editor.org/rfc/rfc9110.html#section-8.8.2.2)). With
block file caching disabled, read a small Parquet A to warm its footer cache,
replace it with an equal-length B in that second, then start/resume a streaming
job while B stays unchanged. Setting this field enables V2 `ParquetFileContext`
caching, which previously bypassed streaming files with mtime=0; its
path/mtime/size key now finds A's footer. A predicate matching B but excluded
by A's statistics can therefore drop B's rows. With
`session.enable_file_scanner_v2=false`, legacy ORC also regresses for
different-length replacements: `FileMetaCache::get_key` switches from path+size
to path+mtime, discarding the distinguishing size. These regressions occur
independently of the preexisting block-cache issue; the LIST assignment has the
same consequence when its
timestamps collide. Use an object version identity or bypass these
mutable-file caches when generations cannot be distinguished, retain size in
legacy keys, and cover same-timestamp replacements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
