LuciferYang commented on PR #58489:
URL: https://github.com/apache/spark/pull/58489#issuecomment-5548580533
**2. No exceptionHandler: startup errors are not retried, and a mid-run
informer death is silent**
`resource-managers/kubernetes/core/src/main/scala/org/apache/spark/scheduler/cluster/k8s/InformerManager.scala:50`
`InformerManager` sets no `exceptionHandler` and never consumes `stopped()`.
In fabric8 7.x the
ExceptionHandler decides retry-vs-stop, and the default chooses stop for any
error before the
first successful sync (it already logs internally — what's missing is the
retry decision), and
for non-GONE WatcherExceptions after startup too. So with the current code,
a mid-run informer
death is silent: events stop, the lister keeps replacing snapshots from a
cache that no longer
updates, and dead executors wait for RPC timeouts to be noticed. And if
start becomes async per
my startup comment without a handler, any startup error turns into a silent
background death
that the hasSynced check would then skip past forever; strictly worse than
crashing.
Please set `exceptionHandler((b, t) => { logError(...); true })` in
`initInformer` (it can only
be set before start) to log and force a retry, or consume the exceptional
completion of
`stopped()` and fail the driver.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
