spuru9 commented on PR #29008: URL: https://github.com/apache/flink/pull/29008#issuecomment-6694744107
A few more things from testing this in minikube (FILE and RocksDB) and a closer read. None of them block on their own, but most get more likely now that on-demand fetch is the normal path for old jobs: - If a view evicts job X while a `/jobs/overview` scan runs, the scan can pick up X's not-yet-removed meta entry and mark X as scan-cached before the delete finishes. X then has no local data, drops out of the job list, and later scans skip it. I could reproduce it by holding the overview delete in a wrapper storage. - With the default of one viewed job, two people on two old jobs evict each other on every dashboard poll, so each poll downloads the whole archive again. - `updateOverview()` now also runs on HTTP threads after an eviction, and nothing coordinates it with the scan or the fetch task. On RocksDB I saw the list still show a job that had just been evicted. - `fetchTasks.put` runs after `submit`, so a quick task can hit its `finally` remove before the put. The finished future then stays in the map and `isFetchInFlight` pins that job for good. - Jobs that were only viewed never get into `cachedArchivesPerRefreshDirectory`, so with `clean-expired-jobs` they aren't cleaned up when their remote archive goes away. I deleted the archive of a viewed job and it was still listed and served, while an in-window job was cleaned up correctly. - `cached-retained-jobs` is applied per refresh directory, like `retained-jobs`, but its description doesn't say so. - Tests: removing the scan's `removeAll(mostRecentlyViewedJobs.keySet())` still passes all of `HistoryServerArchiveFetcherTest`, and removing the `isFetchInFlight` check still passes `testLazyModeViewingAnotherJobDoesNotCancelInFlightFetch`. Only `testArchiveWithFetchInFlightIsNotEvictedByViewingAnotherJob` catches the second one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
