prashantwason commented on PR #19790: URL: https://github.com/apache/hudi/pull/19790#issuecomment-5998287828
Thanks @danny0405, @rahil-c and @voonhous for the reviews and the repro. Addressed in 6d0213c: - **Engine registry cache** (@danny0405 P1, @rahil-c): `getMetricRegistry` now registers on every lookup via `compute` instead of inside `computeIfAbsent`, so entries cached in `DISTRIBUTED_REGISTRY_MAP` under a previous SparkContext are recovered. Since engine operations serialize the whole `DISTRIBUTED_REGISTRY_MAP` into their tasks, a single stale entry would fail every `map`/`flatMap`/... even if nobody looks it up again, so those operations now also recover stale entries before shipping the map (`getDistributedRegistries()`). - **Test exercises the stale path** (@danny0405 P2): the tests now remove the accumulator from `AccumulatorContext`, assert `!isRegistered()`, and then assert the result is a different, registered instance that carries the counters and is the cached entry in `REGISTRY_MAP`. Added `testEngineContextRecoversStaleRegistry`, which reproduces your scenario through the engine context, including an `engineContext.map(...)` call. The SparkContext restart is gone. - **Concurrency**: `register()` is synchronized and a stale instance remembers its registered replacement, so concurrent callers and the other cache resolve to the same accumulator instead of registering a second one. - **#19063** (@voonhous): the description now says "Part of #19063 (gap 2)" and explains it is the same stale singleton seen from the other side. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
