danny0405 commented on issue #20338: URL: https://github.com/apache/hudi/issues/20338#issuecomment-6183425212
Thanks for sharing the production measurements. Batching checks by parent directory looks promising for this workload. After tracing the current code, I have a few suggestions for the PR: 1. **Preserve both missing-file modes, not just reverse commit order.** In [WriteProfiles.getFilesFromMetadata](https://github.com/apache/hudi/blob/6bbf798f88d468d1ca3ce774aae991145b0cd0a2/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/sink/partitioner/profile/WriteProfiles.java#L123-L152), streaming can select an older surviving candidate when the newest is missing. With `ignoreMissingFiles=false`, however, a missing candidate encountered during resolution returns `null`, which [triggers the batch reader's full-scan fallback](https://github.com/apache/hudi/blob/6bbf798f88d468d1ca3ce774aae991145b0cd0a2/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/IncrementalInputSplits.java#L212-L219). Conversely, a missing older version must not trigger fallback after a newer version has already resolved that key. A pass-local existence lookup followed by the existing reverse-order resolution would help preserve this behavior. Please also cover MOR, whose map is keyed by full path rather than COW file ID, and distinguish a missing directory from a listing failure such as an authorization error or timeout. 2. **Benchmark sparse updates into large directories before fixing the threshold at two candidates.** Two candidates in a partition containing many retained file versions could make a full listing substantially more expensive than two existence checks. Also, one filesystem listing is not necessarily one storage request: [GCS listings are paginated](https://cloud.google.com/storage/docs/json_api/v1/objects/list). Could the benchmark include actual backend request/page counts, total directory entries, and peak JobManager memory, plus a sparse-update/non-partitioned-table case? That would help decide whether a configurable threshold or a way to retain per-file checks is needed. 3. **Track duplicate discovery as a separate correctness issue.** The [continuous enumerator](https://github.com/apache/hudi/blob/6bbf798f88d468d1ca3ce774aae991145b0cd0a2/hudi-flink-datasource/hudi-flink/src/main/java/org/apache/hudi/source/enumerator/HoodieContinuousSplitEnumerator.java#L93-L130) reads the position in the async discovery callable and advances it in the result callback. This is consistent with the reported stale-position window, although I have not reproduced it. Faster discovery reduces the likelihood but does not guarantee the callback runs before the next invocation. A deterministic test that delays result processing while another discovery becomes due would be useful; any guard should cover discovery through result application. For the batching tests, the key cases seem to be newest-present/older-missing, newest-missing/older-present under both missing-file modes, an entirely removed directory, and MOR base/log candidates. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
