[
https://issues.apache.org/jira/browse/SPARK-19796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15893584#comment-15893584
]
Mridul Muralidharan commented on SPARK-19796:
---------------------------------------------
I would not prefer (b) - if we are worried that users are depending on a
private property, sending a truncated version of it is to aggravate it ! I
would rather fail-fast with missing value.
Having said that, while we should limit our internal usage of properties, since
this is also used to propagate user specified key value pairs; adding limits or
log messages might not be optimal. Worst case, if we start detecting that the
properties Map is growing really large, we could broadcast it (ugh ?).
> taskScheduler fails serializing long statements received by thrift server
> -------------------------------------------------------------------------
>
> Key: SPARK-19796
> URL: https://issues.apache.org/jira/browse/SPARK-19796
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 2.2.0
> Reporter: Giambattista
> Priority: Blocker
>
> This problem was observed after the changes made for SPARK-17931.
> In my use-case I'm sending very long insert statements to Spark thrift server
> and they are failing at TaskDescription.scala:89 because writeUTF fails if
> requested to write strings longer than 64Kb (see
> https://www.drillio.com/en/2009/java-encoded-string-too-long-64kb-limit/ for
> a description of the issue).
> As suggested by Imran Rashid I tracked down the offending key: it is
> "spark.job.description" and it contains the complete SQL statement.
> The problem can be reproduced by creating a table like:
> create table test (a int) using parquet
> and by sending an insert statement like:
> scala> val r = 1 to 128000
> scala> println("insert into table test values (" + r.mkString("),(") + ")")
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
