This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 75d80c7795ca [SPARK-51079] Added constructor for ArrowSerializer for
backward compatibility
75d80c7795ca is described below
commit 75d80c7795ca71d24229010ab04ae740473126aa
Author: Haiyang Sun <[email protected]>
AuthorDate: Mon Apr 7 10:16:34 2025 +0900
[SPARK-51079] Added constructor for ArrowSerializer for backward
compatibility
### What changes were proposed in this pull request?
Previously, [PR](https://github.com/apache/spark/pull/49790) extends
ArrowSerializer class to support large variable types in pandas UDF. This PR
makes it more backward-compatible by adding a new constructor for cases where
the flag for large variable is irrelevant (default to false).
### Why are the changes needed?
To achieve versionless in spark connect, we need to be careful about
changing a client API in Scala. For example, the change of an interface like
this could affect Scala UDF referring to this class (as it is part of the spark
connect client package) but running with an older spark connect client.
### How was this patch tested?
N/A. It's a new constructor.
Closes #50513 from haiyangsun-db/SPARK-51079.
Authored-by: Haiyang Sun <[email protected]>
Signed-off-by: Hyukjin Kwon <[email protected]>
---
.../org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala | 6 ++++++
1 file changed, 6 insertions(+)
diff --git
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
index a74fce3a8aba..d79fb25ec1a0 100644
---
a/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
+++
b/sql/connect/common/src/main/scala/org/apache/spark/sql/connect/client/arrow/ArrowSerializer.scala
@@ -52,6 +52,12 @@ class ArrowSerializer[T](
private[this] val allocator: BufferAllocator,
private[this] val timeZoneId: String,
private[this] val largeVarTypes: Boolean) {
+
+ // SPARK-51079: keep the old constructor for backward-compatibility.
+ def this(enc: AgnosticEncoder[T], allocator: BufferAllocator, timeZoneId:
String) = {
+ this(enc, allocator, timeZoneId, false)
+ }
+
private val (root, serializer) =
ArrowSerializer.serializerFor(enc, allocator, timeZoneId, largeVarTypes)
private val vectors = root.getFieldVectors.asScala
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
