This is an automated email from the ASF dual-hosted git repository.
cloud-fan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 280bdae54039 [SPARK-57496][SQL][BUILD] Keep the Types Framework ops
and UDF worker packages out of the published API
280bdae54039 is described below
commit 280bdae54039fea3ccf3c285d5bb80ce7b786956
Author: Wenchen Fan <[email protected]>
AuthorDate: Wed Jun 17 21:29:55 2026 -0700
[SPARK-57496][SQL][BUILD] Keep the Types Framework ops and UDF worker
packages out of the published API
### What changes were proposed in this pull request?
Two related changes that keep internal packages out of the published 4.2.0
API surface:
1. Move the client-side Types Framework ops — `TypeApiOps`,
`TimeTypeApiOps`, `TimestampNanosTypeApiOps` (and the
`TimestampNTZNanosTypeApiOps` / `TimestampLTZNanosTypeApiOps` impls) — from
`org.apache.spark.sql.types.ops` to `org.apache.spark.sql.catalyst.types.ops`,
co-located with the server-side `TypeOps` family. Consumer imports are updated;
same-package consumers drop the now-redundant import.
2. Exclude `org.apache.spark.udf.worker` from the generated API docs in
`project/SparkBuild.scala`'s `ignoreUndocumentedPackages`.
### Why are the changes needed?
The `*ApiOps` types are internal plumbing of the Types Framework (the
client-side counterpart to catalyst's `TypeOps`), but they lived inside the
public `org.apache.spark.sql.types` package, so they leaked into the published
PySpark/Scala API of the unreleased 4.2.0 line.
`org.apache.spark.sql.catalyst.*` is already excluded from both the generated
docs (`ignoreUndocumentedPackages`) and MiMa (`MimaExcludes`), so relocating
them there makes them internal with no new build/MiMa entries [...]
`org.apache.spark.udf.worker` is UDF-worker infrastructure (mostly
protobuf-generated `*OrBuilder` Java plus worker internals) that surfaced as
public API. Its modules aren't MiMa-checked, and the generated Java can't carry
a Scala visibility qualifier, so excluding the package from the docs is the
appropriate fix.
### Does this PR introduce _any_ user-facing change?
No. Relative to released Spark there is no change; the affected types are
new in the unreleased 4.2.0 line and were never intended to be public. This
only removes them from the generated API docs (and, for the ops, the
binary-compatibility surface) before release. There is no behavior change.
### How was this patch tested?
No new tests — this is a package relocation plus a build-config change with
no logic change. The relocated classes are exercised by existing suites (e.g.
`TimestampNanosTypeOpsSuite`) and the cast / `Row` / `HiveResult` paths; CI
compiles all affected modules and runs scalastyle, which enforces the
import-ordering updates made here.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
Closes #56551 from cloud-fan/SPARK-57496.
Authored-by: Wenchen Fan <[email protected]>
Signed-off-by: Wenchen Fan <[email protected]>
---
project/SparkBuild.scala | 1 +
sql/api/src/main/scala/org/apache/spark/sql/Row.scala | 2 +-
.../main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala | 2 +-
.../org/apache/spark/sql/{ => catalyst}/types/ops/TimeTypeApiOps.scala | 2 +-
.../spark/sql/{ => catalyst}/types/ops/TimestampNanosTypeApiOps.scala | 2 +-
.../org/apache/spark/sql/{ => catalyst}/types/ops/TypeApiOps.scala | 2 +-
sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala | 2 +-
.../scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala | 2 +-
.../scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeOps.scala | 1 -
.../org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOps.scala | 1 -
.../main/scala/org/apache/spark/sql/catalyst/types/ops/TypeOps.scala | 1 -
.../spark/sql/catalyst/types/ops/TimestampNanosTypeOpsSuite.scala | 1 -
sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala | 2 +-
.../scala/org/apache/spark/sql/execution/python/EvaluatePython.scala | 2 +-
.../spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala | 2 +-
15 files changed, 11 insertions(+), 14 deletions(-)
diff --git a/project/SparkBuild.scala b/project/SparkBuild.scala
index a4f92eb4ef07..2e458fc42a53 100644
--- a/project/SparkBuild.scala
+++ b/project/SparkBuild.scala
@@ -1657,6 +1657,7 @@ object Unidoc {
.map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/kafka010")))
.map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/types/variant")))
.map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/ui/flamegraph")))
+
.map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/udf/worker")))
.map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/util/collection")))
.map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/util/io")))
.map(_.filterNot(_.getCanonicalPath.contains("org/apache/spark/util/kvstore")))
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/Row.scala
b/sql/api/src/main/scala/org/apache/spark/sql/Row.scala
index 137a8863a604..78842a2df514 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/Row.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/Row.scala
@@ -32,12 +32,12 @@ import org.json4s.jackson.JsonMethods.{compact, pretty,
render}
import org.apache.spark.SparkIllegalArgumentException
import org.apache.spark.annotation.{Stable, Unstable}
import org.apache.spark.sql.catalyst.expressions.GenericRow
+import org.apache.spark.sql.catalyst.types.ops.TypeApiOps
import org.apache.spark.sql.catalyst.util.{DateFormatter, SparkDateTimeUtils,
TimestampFormatter, UDTUtils}
import org.apache.spark.sql.errors.DataTypeErrors
import org.apache.spark.sql.errors.DataTypeErrors.{toSQLType, toSQLValue}
import org.apache.spark.sql.internal.SqlApiConf
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.ops.TypeApiOps
import org.apache.spark.unsafe.types.CalendarInterval
import org.apache.spark.util.ArrayImplicits._
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
index 69b5cc53fa76..5fce0d1491ba 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/encoders/RowEncoder.scala
@@ -22,10 +22,10 @@ import scala.reflect.classTag
import org.apache.spark.sql.{AnalysisException, Row}
import org.apache.spark.sql.catalyst.encoders.AgnosticEncoders.{BinaryEncoder,
BoxedBooleanEncoder, BoxedByteEncoder, BoxedDoubleEncoder, BoxedFloatEncoder,
BoxedIntEncoder, BoxedLongEncoder, BoxedShortEncoder, CalendarIntervalEncoder,
CharEncoder, DateEncoder, DayTimeIntervalEncoder, EncoderField,
GeographyEncoder, GeometryEncoder, InstantEncoder, IterableEncoder,
JavaDecimalEncoder, LocalDateEncoder, LocalDateTimeEncoder, MapEncoder,
NullEncoder, RowEncoder => AgnosticRowEncoder, Strin [...]
+import org.apache.spark.sql.catalyst.types.ops.TypeApiOps
import org.apache.spark.sql.errors.DataTypeErrorsBase
import org.apache.spark.sql.internal.SqlApiConf
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.ops.TypeApiOps
import org.apache.spark.util.ArrayImplicits._
/**
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimeTypeApiOps.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeApiOps.scala
similarity index 98%
rename from
sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimeTypeApiOps.scala
rename to
sql/api/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeApiOps.scala
index 7b5596109290..090aa66b74bb 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimeTypeApiOps.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeApiOps.scala
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.spark.sql.types.ops
+package org.apache.spark.sql.catalyst.types.ops
import java.time.LocalTime
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeApiOps.scala
similarity index 99%
rename from
sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
rename to
sql/api/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeApiOps.scala
index 011e5504e96e..650575d006f1 100644
---
a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TimestampNanosTypeApiOps.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeApiOps.scala
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.spark.sql.types.ops
+package org.apache.spark.sql.catalyst.types.ops
import java.time.{Instant, LocalDateTime, ZoneId, ZoneOffset}
diff --git
a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TypeApiOps.scala
similarity index 99%
rename from
sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
rename to
sql/api/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TypeApiOps.scala
index c76551aacd84..728c6ae40cd4 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/ops/TypeApiOps.scala
+++
b/sql/api/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TypeApiOps.scala
@@ -15,7 +15,7 @@
* limitations under the License.
*/
-package org.apache.spark.sql.types.ops
+package org.apache.spark.sql.catalyst.types.ops
import java.time.ZoneId
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala
b/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala
index b0980a911fdd..b84336cc0f54 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/util/ArrowUtils.scala
@@ -27,9 +27,9 @@ import org.apache.arrow.vector.types.{DateUnit,
FloatingPointPrecision, Interval
import org.apache.arrow.vector.types.pojo.{ArrowType, Field, FieldType, Schema}
import org.apache.spark.SparkException
+import org.apache.spark.sql.catalyst.types.ops.TypeApiOps
import org.apache.spark.sql.errors.ExecutionErrors
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.ops.TypeApiOps
import org.apache.spark.util.ArrayImplicits._
private[sql] object ArrowUtils {
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
index b8d389259544..0fec0bd3e00e 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/ToStringBase.scala
@@ -22,12 +22,12 @@ import java.time.ZoneOffset
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions.codegen._
import org.apache.spark.sql.catalyst.expressions.codegen.Block._
+import org.apache.spark.sql.catalyst.types.ops.TypeApiOps
import org.apache.spark.sql.catalyst.util.{ArrayData, CharVarcharCodegenUtils,
DateFormatter, FractionTimeFormatter, IntervalStringStyles, IntervalUtils,
MapData, TimestampFormatter}
import org.apache.spark.sql.catalyst.util.IntervalStringStyles.ANSI_STYLE
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.BinaryOutputStyle
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.ops.TypeApiOps
import org.apache.spark.unsafe.UTF8StringBuilder
import org.apache.spark.unsafe.types.{CalendarInterval, UTF8String}
import org.apache.spark.util.ArrayImplicits._
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeOps.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeOps.scala
index f4c4b6c5a543..d1700aad05cf 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeOps.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimeTypeOps.scala
@@ -30,7 +30,6 @@ import org.apache.spark.sql.errors.QueryCompilationErrors
import org.apache.spark.sql.execution.arrow.{ArrowFieldWriter, TimeWriter}
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{ObjectType, TimeType}
-import org.apache.spark.sql.types.ops.TimeTypeApiOps
/**
* Server-side (catalyst) operations for TimeType.
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOps.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOps.scala
index ca3976528b25..48628619cdb4 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOps.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOps.scala
@@ -27,7 +27,6 @@ import org.apache.spark.sql.catalyst.types.{PhysicalDataType,
PhysicalTimestampL
import org.apache.spark.sql.catalyst.util.DateTimeUtils
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{ObjectType, TimestampLTZNanosType,
TimestampNTZNanosType}
-import org.apache.spark.sql.types.ops.{TimestampLTZNanosTypeApiOps,
TimestampNTZNanosTypeApiOps}
import org.apache.spark.unsafe.types.TimestampNanosVal
/**
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TypeOps.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TypeOps.scala
index b09feb0fb1cf..ab58336b8ef2 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TypeOps.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/types/ops/TypeOps.scala
@@ -27,7 +27,6 @@ import org.apache.spark.sql.catalyst.expressions.{Expression,
Literal, MutableVa
import org.apache.spark.sql.catalyst.types.PhysicalDataType
import org.apache.spark.sql.execution.arrow.ArrowFieldWriter
import org.apache.spark.sql.types.{DataType, TimestampLTZNanosType,
TimestampNTZNanosType, TimeType}
-import org.apache.spark.sql.types.ops.TypeApiOps
/**
* Server-side (catalyst) type operations for the Types Framework.
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOpsSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOpsSuite.scala
index 2cff609c870e..e8bcc8471953 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOpsSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/types/ops/TimestampNanosTypeOpsSuite.scala
@@ -30,7 +30,6 @@ import org.apache.spark.sql.catalyst.types.{PhysicalDataType,
PhysicalTimestampL
import org.apache.spark.sql.catalyst.util.DateTimeUtils
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.types.{DataType, TimestampLTZNanosType,
TimestampNTZNanosType}
-import org.apache.spark.sql.types.ops.{TimestampLTZNanosTypeApiOps, TypeApiOps}
import org.apache.spark.unsafe.types.{TimestampNanosVal, UTF8String}
/**
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
index c481f4eb7c11..e12e35ea01ed 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/HiveResult.scala
@@ -23,6 +23,7 @@ import java.time._
import org.apache.spark.sql.Row
import org.apache.spark.sql.catalyst.SQLConfHelper
import org.apache.spark.sql.catalyst.expressions.ToStringBase
+import org.apache.spark.sql.catalyst.types.ops.TypeApiOps
import org.apache.spark.sql.catalyst.util.{DateFormatter, DateTimeUtils,
FractionTimeFormatter, STUtils, TimeFormatter, TimestampFormatter}
import org.apache.spark.sql.catalyst.util.IntervalStringStyles.HIVE_STYLE
import org.apache.spark.sql.catalyst.util.IntervalUtils.{durationToMicros,
periodToMonths, toDayTimeIntervalString, toYearMonthIntervalString}
@@ -31,7 +32,6 @@ import
org.apache.spark.sql.execution.datasources.v2.{DescribeTableExec, ShowTab
import org.apache.spark.sql.internal.SQLConf
import org.apache.spark.sql.internal.SQLConf.BinaryOutputStyle
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.ops.TypeApiOps
import org.apache.spark.unsafe.types.{CalendarInterval, VariantVal}
import org.apache.spark.util.ArrayImplicits._
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala
index 874c480b4c9f..cb1acd2f541b 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala
@@ -29,9 +29,9 @@ import org.apache.spark.api.python.SerDeUtil
import org.apache.spark.rdd.RDD
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.expressions._
+import org.apache.spark.sql.catalyst.types.ops.TypeApiOps
import org.apache.spark.sql.catalyst.util.{ArrayBasedMapData, ArrayData,
GenericArrayData, MapData, STUtils}
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.ops.TypeApiOps
import org.apache.spark.unsafe.types.{BinaryView, UTF8String, VariantVal}
object EvaluatePython {
diff --git
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
index 8757fad2933c..28941c033cc3 100644
---
a/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
+++
b/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala
@@ -33,11 +33,11 @@ import
org.apache.hive.service.rpc.thrift.{TCLIServiceConstants, TColumnDesc, TP
import org.apache.spark.internal.{Logging, LogKeys}
import org.apache.spark.internal.LogKeys._
import org.apache.spark.sql.{DataFrame, Row, SparkSession}
+import org.apache.spark.sql.catalyst.types.ops.TypeApiOps
import org.apache.spark.sql.catalyst.util.CharVarcharUtils
import org.apache.spark.sql.catalyst.util.DateTimeConstants.MILLIS_PER_SECOND
import org.apache.spark.sql.internal.{SQLConf, VariableSubstitution}
import org.apache.spark.sql.types._
-import org.apache.spark.sql.types.ops.TypeApiOps
import org.apache.spark.util.{Utils => SparkUtils}
private[hive] class SparkExecuteStatementOperation(
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
