This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 534b59d3c76d [SPARK-57485][BUILD] Exclude package-private Scala types
from the generated Javadoc
534b59d3c76d is described below
commit 534b59d3c76db77a87bd83ca2c8da09668a62dd7
Author: Wenchen Fan <[email protected]>
AuthorDate: Wed Jun 17 10:19:57 2026 -0700
[SPARK-57485][BUILD] Exclude package-private Scala types from the generated
Javadoc
### What changes were proposed in this pull request?
Spark publishes both a Scaladoc and a Javadoc API site. The Javadoc is
generated from Scala sources by genjavadoc, and it currently exposes a large
number of internal types that the Scaladoc correctly hides.
The root cause: a top-level `private[x]` Scala type (e.g. `private[spark]
trait SupportsDelegationToken`) compiles to a JVM-`public` symbol. genjavadoc
emits a `public` Java stub for it even with
`-P:genjavadoc:strictVisibility=true`, and the Javadoc `-public` option can't
filter it because the stub genuinely is public. Scaladoc, by contrast, honors
the access qualifier and drops these types.
This PR adds a filter to `JavaUnidoc / unidoc / unidocAllSources`
(alongside the existing `ignoreUndocumentedPackages`) that drops a generated
stub `<module>/target/java/<pkg>/<Name>.java` **iff every top-level Scala
declaration of `<Name>` in that package is `private[...]`**. A public class
with a `private[...]` companion object (e.g. `SparkConf` — public `class`,
`private[spark] object`) is kept, since the class itself is public.
### Why are the changes needed?
The published Javadoc lists ~1.3k internal types (e.g.
`BarrierCoordinator`, `ContextCleaner`, `ExecutorAllocationManager`, scheduler
RPC messages, `SupportsDelegationToken`) that are `private[spark]` in source
and are absent from the Scaladoc. This both misleads Java users about the
public API surface and makes the two API docs disagree on which types are
public. Filtering them aligns the Java API doc with the Scala one (format still
differs, coverage now matches) without touching ge [...]
### Does this PR introduce _any_ user-facing change?
No code/runtime change. The only user-facing effect is on the generated
Javadoc site: top-level `private[spark]` (and other qualified-private) Scala
types no longer appear as public Java classes. Genuinely public APIs —
including Java-authored ones (`src/main/java`, e.g. the DataSource V2 connector
interfaces) and Java-friendly wrappers like `org.apache.spark.api.java.JavaRDD`
— are unaffected.
### How was this patch tested?
- Validated the filter selects exactly the package-private stubs against
the already-generated `*/target/java` stubs across `core`, `sql/core`,
`sql/api`, `sql/catalyst`, `mllib`, `streaming`: it drops the `private[spark]`
leaks (`SupportsDelegationToken`, `StructuredStreamingIdAwareSchedulerLogging`,
`InternalAccumulator`, ~1.3k total) while keeping public types and
public-class-with-private-companion cases (`SparkConf`, `SparkContext`,
`TaskContext`, `RDD`).
- Confirmed the build definition compiles via `build/sbt reload`.
- A full `build/sbt unidoc` run is the end-to-end integration check;
relying on CI's docs build for that.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Isaac)
This pull request and its description were written by Isaac.
Closes #56538 from cloud-fan/genjavadoc-exclude-package-private.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
(cherry picked from commit 5680d1536bdf2d5225086365173dac6f9edf8d31)
Signed-off-by: Wenchen Fan <[email protected]>
---
project/SparkBuild.scala | 58 ++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 56 insertions(+), 2 deletions(-)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index 920534b0bed0..729364a31e54 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1663,6 +1663,59 @@ object Unidoc {
.map(_.filterNot(_.data.getCanonicalPath.contains("connect-shims")))
}
+ // genjavadoc emits top-level package-private Scala types (`private` or
`private[x]`, e.g.
+ // `private[spark] trait Foo` or a bare `private class Bar`) as *public*
Java stubs even with
+ // `-P:genjavadoc:strictVisibility=true`, because a top-level
package-private type compiles to a
+ // JVM-public symbol, and the Javadoc `-public` option cannot drop a stub
that really is public.
+ // ScalaDoc honors the qualifier and hides such types, so we drop their
stubs here to keep the
+ // published Java API doc aligned with the Scala one. A stub
`<module>/target/java/<pkg>/<Name>.java`
+ // is dropped iff EVERY top-level Scala declaration of `<Name>` in that
package is `private` or
+ // `private[...]`; a public class with a private companion object (e.g.
`SparkConf`) is kept, since
+ // the class itself is public. The private regex tolerates other modifiers
around the access
+ // qualifier (e.g. `final private[x] class`).
+ private val publicTopTypeRe =
+
"""(?m)^(?:@\w+(?:\([^\n)]*\))?\s+)*(?:(?:final|sealed|abstract|implicit|case)\s+)*(?:class|trait|object)\s+(\w+)""".r
+ private val privateTopTypeRe =
+
"""(?m)^(?:@\w+(?:\([^\n)]*\))?\s+)*(?:(?:final|sealed|abstract|implicit|case)\s+)*private(?:\[[^\]]+\])?\s+(?:(?:final|sealed|abstract|implicit|case)\s+)*(?:class|trait|object)\s+(\w+)""".r
+
+ private def dropPackagePrivateJavaStubs(sources: Seq[Seq[File]]):
Seq[Seq[File]] = {
+ val cache = scala.collection.mutable.Map.empty[String, (Set[String],
Set[String])]
+ def scanPkg(dir: String): (Set[String], Set[String]) =
cache.getOrElseUpdate(dir, {
+ val d = new File(dir)
+ val scalaFiles =
+ if (d.isDirectory) d.listFiles.filter(_.getName.endsWith(".scala"))
else Array.empty[File]
+ val text = scalaFiles
+ .map(f => new String(java.nio.file.Files.readAllBytes(f.toPath),
"UTF-8"))
+ .mkString("\n")
+ (publicTopTypeRe.findAllMatchIn(text).map(_.group(1)).toSet,
+ privateTopTypeRe.findAllMatchIn(text).map(_.group(1)).toSet)
+ })
+ val marker = "/target/java/"
+ sources.map(_.filterNot { f =>
+ val path = f.getCanonicalPath.replace('\\', '/')
+ val idx = path.indexOf(marker)
+ if (idx < 0 || !path.endsWith(".java")) {
+ false
+ } else {
+ val rel = path.substring(idx + marker.length) // <pkg>/<Name>.java
+ val slash = rel.lastIndexOf('/')
+ if (slash < 0) {
+ false
+ } else {
+ val moduleRoot = path.substring(0, idx)
+ val pkgPath = rel.substring(0, slash)
+ val name = f.getName.stripSuffix(".java")
+ val (pub, priv) = Seq("src/main/scala", "src/main/scala-2.13")
+ .map(d => scanPkg(s"$moduleRoot/$d/$pkgPath"))
+ .foldLeft((Set.empty[String], Set.empty[String])) {
+ case ((p, q), (a, b)) => (p ++ a, q ++ b)
+ }
+ priv.contains(name) && !pub.contains(name)
+ }
+ }
+ })
+ }
+
val unidocSourceBase = settingKey[String]("Base URL of source links in
Scaladoc.")
lazy val settings = BaseUnidocPlugin.projectSettings ++
@@ -1687,8 +1740,9 @@ object Unidoc {
// Skip class names containing $ and some internal packages in Javadocs
(JavaUnidoc / unidoc / unidocAllSources) := {
- ignoreUndocumentedPackages((JavaUnidoc / unidoc /
unidocAllSources).value)
- .map(_.filterNot(_.getCanonicalPath.contains("org/apache/hadoop")))
+ dropPackagePrivateJavaStubs(
+ ignoreUndocumentedPackages((JavaUnidoc / unidoc /
unidocAllSources).value)
+ .map(_.filterNot(_.getCanonicalPath.contains("org/apache/hadoop"))))
},
(JavaUnidoc / unidoc / javacOptions) := {
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
