[
https://issues.apache.org/jira/browse/TEZ-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337298#comment-14337298
]
Hitesh Shah commented on TEZ-2144:
----------------------------------
[~rohini] I think the user payload is already being compressed in
DagTypeConverters:;convertToDAGPlan(EntityDescriptor descriptor).
It seems like beyond a certain size we might need to fall back to using
LocalResources instead of sending over RPC.
However, a question on the payload. To confirm, is the input split information
in the payload of only the required descriptor i.e. the input of the map stage
and not replicated for to user payloads of all descriptors?
> Compressing user payload
> ------------------------
>
> Key: TEZ-2144
> URL: https://issues.apache.org/jira/browse/TEZ-2144
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
>
> Pig sets the input split information in user payload and when running against
> a table with 10s of 1000s of partitions, DAG submission fails with
> java.io.IOException: Requested data length 305844060 is longer than maximum
> configured RPC length 67108864
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
