This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a commit to branch branch-4.0
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.0 by this push:
new 64d5c19529e2 [SPARK-49872][CORE] Remove jackson JSON string length
limitation
64d5c19529e2 is described below
commit 64d5c19529e279d46b48dc18bd5ec9a6e391ed0c
Author: Wenchen Fan <wenc...@databricks.com>
AuthorDate: Tue Aug 19 08:05:03 2025 +0800
[SPARK-49872][CORE] Remove jackson JSON string length limitation
### What changes were proposed in this pull request?
This is a surgical fix extracted from
https://github.com/apache/spark/pull/49163
The default jackson string limit introduced in jackson 2.15 can be too
small for certain workloads, and this PR removes this limitation to avoid any
regression.
### Why are the changes needed?
fix regression
### Does this PR introduce _any_ user-facing change?
Yes, users won't hit this size limitation anymore.
### How was this patch tested?
https://github.com/apache/spark/pull/49163 tested it. We won't add a test
in this PR as generating a super large JSON will make the CI unstable.
### Was this patch authored or co-authored using generative AI tooling?
no
Closes #52049 from cloud-fan/json.
Lead-authored-by: Wenchen Fan <wenc...@databricks.com>
Co-authored-by: Wenchen Fan <cloud0...@gmail.com>
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
(cherry picked from commit 076618a46cabfb6358f2c1a696edf38db9ae7a6f)
Signed-off-by: Wenchen Fan <wenc...@databricks.com>
---
core/src/main/scala/org/apache/spark/util/JsonProtocol.scala | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
b/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
index 7b99714d7676..f1808c02c06e 100644
--- a/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
+++ b/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala
@@ -22,7 +22,7 @@ import java.util.{Properties, UUID}
import scala.collection.Map
import scala.jdk.CollectionConverters._
-import com.fasterxml.jackson.core.JsonGenerator
+import com.fasterxml.jackson.core.{JsonGenerator, StreamReadConstraints}
import com.fasterxml.jackson.databind.JsonNode
import org.json4s.jackson.JsonMethods.compact
@@ -66,6 +66,11 @@ private[spark] class JsonProtocolOptions(conf: SparkConf) {
private[spark] object JsonProtocol extends JsonUtils {
// TODO: Remove this file and put JSON serialization into each individual
class.
+ // SPARK-49872: Remove jackson JSON string length limitation.
+ mapper.getFactory.setStreamReadConstraints(
+ StreamReadConstraints.builder().maxStringLength(Int.MaxValue).build()
+ )
+
private[util]
val defaultOptions: JsonProtocolOptions = new JsonProtocolOptions(new
SparkConf(false))
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
