[
https://issues.apache.org/jira/browse/MESOS-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269826#comment-15269826
]
Rob Russo commented on MESOS-4210:
----------------------------------
This does affect our actual production use case. In the latest versions of
spark when using the parquet relation they attempt to speed up file discovery
by parallelizing it into multiple tasks. Unfortunately if using fine grained
mode they tend to do this using a few number of tasks (always seemed to be 8)
and the smaller the number of tasks, the larger the result that needs to be
sent back to the driver from each task to list the discovered files. With our
file setup this exceeds the max protobuf length once the matched files grows
into the tens of thousands (which is not that much for us).
For now we are getting around this using a custom built protobuf jar with a
higher limit set and building mesos with that jar supplied but it would be
great if we could at least have a option to adjust this limit as a mesos config
since the limit can be set at runtime.
I realize that protobufs are not designed to handle large messages but given
this very standard use case of spark on mesos it seems that there should be
some easy workaround here.
> Investigate increasing protobuf protocol message size limit.
> ------------------------------------------------------------
>
> Key: MESOS-4210
> URL: https://issues.apache.org/jira/browse/MESOS-4210
> Project: Mesos
> Issue Type: Bug
> Reporter: Artem Harutyunyan
>
> {noformat}
> [libprotobuf ERROR google/protobuf/io/coded_stream.cc:171] A protocol message
> was rejected because it was too big (more than 67108864 bytes). To increase
> the limit (or to disable these warnings), see
> CodedInputStream::SetTotalBytesLimit() in google/protobuf/io/coded_stream.h.
> F20151217 16:33:44.832834 4076 construct.cpp:48] Check failed: parsed
> Unexpected failure while parsing protobuf
> Check failure stack trace: ***
> @ 0x2b9bab353b68 (unknown)
> @ 0x2b9bab353ac4 (unknown)
> @ 0x2b9bab3534ba (unknown)
> @ 0x2b9bab356274 (unknown)
> @ 0x2b9bab339d09 (unknown)
> @ 0x2b9bab338917 (unknown)
> @ 0x2b9bab33f404 (unknown)
> @ 0x2b9b68350e18 (unknown)
> {noformat}
> The error is presumably caused by a "user sending a very large command line".
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
