Samra Pusina created SPARK-61365:
------------------------------------
Summary: Share one loader that skips failed ServiceLoader
providers and gives up after repeated failures
Key: SPARK-61365
URL: https://issues.apache.org/jira/browse/SPARK-61365
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 4.0.5
Reporter: Samra Pusina
AppHistoryServerPlugin.skipUnloadable skips providers that fail to load from a
ServiceLoader iterator and gives up after MAX_LOAD_FAILURES consecutive
failures. (see [https://github.com/apache/spark/pull/58965)] Three other places
have the same loop, all catching Throwable with no failure limit, so they can
loop forever when the iterator does not move past a failed entry (for example
when getResources throws an IOException):
- HadoopDelegationTokenManager.loadProviders
- SparkSession.loadExtensions (classic)
- ConnectionProviderBase.loadProviders
Extract a generic helper (e.g. Utils.skipUnloadable[T]) and use it in all four
places. Details to handle: SparkSession applies ext(extensions) inside the try,
so it needs a callback; the sites log at different levels; and the catch should
be NonFatal || LinkageError (and unwrap a ServiceConfigurationError caused by a
VirtualMachineError).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]