Samra Pusina created SPARK-61365:
------------------------------------

             Summary: Share one loader that skips failed ServiceLoader 
providers and gives up after repeated failures
                 Key: SPARK-61365
                 URL: https://issues.apache.org/jira/browse/SPARK-61365
             Project: Spark
          Issue Type: Improvement
          Components: Spark Core
    Affects Versions: 4.0.5
            Reporter: Samra Pusina


AppHistoryServerPlugin.skipUnloadable skips providers that fail to load from a 
ServiceLoader iterator and gives up after MAX_LOAD_FAILURES consecutive 
failures. (see [https://github.com/apache/spark/pull/58965)] Three other places 
have the same loop, all catching Throwable with no failure limit, so they can 
loop forever when the iterator does not move past a failed entry (for example 
when getResources throws an IOException):

- HadoopDelegationTokenManager.loadProviders

- SparkSession.loadExtensions (classic)

- ConnectionProviderBase.loadProviders

 

Extract a generic helper (e.g. Utils.skipUnloadable[T]) and use it in all four 
places. Details to handle: SparkSession applies ext(extensions) inside the try, 
so it needs a callback; the sites log at different levels; and the catch should 
be NonFatal || LinkageError (and unwrap a ServiceConfigurationError caused by a 
VirtualMachineError).

 

 



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to