[
https://issues.apache.org/jira/browse/SPARK-60925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60925:
-----------------------------------
Labels: pull-request-available (was: )
> Executors tab is empty in the Spark UI when spark-core runs on a
> non-filesystem classpath (OSGi/shaded), because REST API package scanning
> fails
> ------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60925
> URL: https://issues.apache.org/jira/browse/SPARK-60925
> Project: Spark
> Issue Type: Bug
> Components: Bug
> Affects Versions: 4.2.0, 4.1.1, 5.0.0
> Reporter: Anoopam Sharma
> Priority: Major
> Labels: pull-request-available
>
> Description:
> h3. Problem
> The Spark status REST API registers its JAX-RS resources via Jersey *package
> scanning*:
>
> ApiRootResource.scala (context "/api"):
> holder.setInitParameter(ServerProperties.PROVIDER_PACKAGES,
> "org.apache.spark.status.api.v1")
>
> PrometheusResource.scala (context "/metrics"):
> holder.setInitParameter(ServerProperties.PROVIDER_PACKAGES,
> "org.apache.spark.status.api.v1")
>
> Jersey's PackageNamesScanner enumerates the package by walking the classpath
> as
> file/jar URLs. When spark-core is loaded from a classpath that is not
> filesystem-enumerable, the scan finds no resources and the REST endpoints
> (e.g. /api/v1/applications, the Executors/Environment UI tabs backed by them)
> return 404 / fail to initialize. Observed failure:
>
> org.glassfish.jersey.server.internal.scanning.ResourceFinderException:
> The URI scheme bundleresource of the URI ... is not supported.
> Package scanning deployment is not supported for such URIs.
>
> This reproduces under OSGi (resources exposed as bundleresource:// URLs) and
> can
> also affect shaded/relocated or JPMS-modular deployments.
>
> h3. User-visible symptom
> The Executors tab in the Spark Web UI does not populate. The tab is rendered
> by
> executorspage.js, which fetches /api/v1/applications/<appId>/allexecutors.
> Because
> the /api/v1 Jersey servlet fails to register its resources (package scanning
> throws
> ResourceFinderException on bundleresource:// URLs), that REST call returns no
> data
> and the tab appears empty. The Environment tab and other /api/v1-backed views
> are
> affected the same way.
>
> h3. Root cause
> Package scanning depends on an enumerable filesystem classpath, which is an
> environment assumption that does not hold for all valid deployments.
>
> h3. Fix
> Register the REST resources explicitly by class name
> (ServerProperties.PROVIDER_CLASSNAMES) instead of scanning the package:
>
> /api -> ApiRootResource, JacksonMessageWriter
> /metrics -> PrometheusResource, JacksonMessageWriter
>
> These are the only root resource classes and the only @Provider in the
> org.apache.spark.status.api.v1 package; every other resource
> (OneApplicationResource, StagesResource, ApplicationListResource, ...) is
> reached
> through JAX-RS sub-resource locators (methods returning Class[...]), which do
> not
> require registration/scanning. Explicit registration is
> environment-independent
> and is the approach Jersey recommends over package scanning.
>
> h3. Tests
> All HistoryServerSuite tests (run via LevelDBBackendHistoryServerSuite /
> RocksDBBackendHistoryServerSuite) pass; these exercise the live /api/v1 REST
> endpoints. No functional change for the normal (filesystem classpath)
> deployment.
>
>
> Affected files:
> - core/src/main/scala/org/apache/spark/status/api/v1/ApiRootResource.scala
> - core/src/main/scala/org/apache/spark/status/api/v1/PrometheusResource.scala
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
