My hunch is that this behavior was introduced by a patch to start shading
Jetty in Spark 1.3: https://issues.apache.org/jira/browse/SPARK-3996.
Note that Spark's *MetricsSystem* class is marked as *private[spark]* and
thus isn't intended to be interacted with directly by users. It's not
super likely that this API would break, but it's excluded from our MiMa
checks and thus is liable to change in incompatible ways across releases.
If you add these Jetty classes as a compile-only dependency but don't add
them to the runtime classpath, do you get runtime errors? If the metrics
system is usable at runtime and we only have errors when attempting to
compile user code against non-public APIs, then I'm not sure that this is a
high-priority issue to fix since. If the metrics system doesn't work at
runtime, on the other hand, then that's definitely a bug that should be
fixed.
If you'd like to continue debugging this issue, I think we should move this
discussion over to JIRA so it's easier to track and reference.
Hope this helps,
Josh
On Thu, Apr 2, 2015 at 7:34 AM, Jacek Lewandowski <
jacek.lewandow...@datastax.com> wrote:
> A very simple example which works well with Spark 1.2, and fail to compile
> with Spark 1.3:
>
> build.sbt:
>
> name := "untitled"
> version := "1.0"
> scalaVersion := "2.10.4"
> libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0"
>
> Test.scala:
>
> package org.apache.spark.metrics
> import org.apache.spark.SparkEnv
> class Test {
> SparkEnv.get.metricsSystem.report()
> }
>
> Produces:
>
> Error:scalac: bad symbolic reference. A signature in MetricsSystem.class
> refers to term eclipse
> in package org which is not available.
> It may be completely missing from the current classpath, or the version on
> the classpath might be incompatible with the version used when compiling
> MetricsSystem.class.
>
> Error:scalac: bad symbolic reference. A signature in MetricsSystem.class
> refers to term jetty
> in value org.eclipse which is not available.
> It may be completely missing from the current classpath, or the version on
> the classpath might be incompatible with the version used when compiling
> MetricsSystem.class.
>
> This looks like something wrong with shading jetty.
> MetricsSystem references MetricsServlet which references some classes from
> Jetty, in the original package instead of shaded one. I'm not sure, but
> adding the following dependencies solves the problem:
>
> libraryDependencies += "org.eclipse.jetty" % "jetty-server" %
> "8.1.14.v20131031"
> libraryDependencies += "org.eclipse.jetty" % "jetty-servlet" %
> "8.1.14.v20131031"
>
> Is it intended or is it a bug?
>
>
> Thanks !
>
>
> Jacek
>
>
>
