[
https://issues.apache.org/jira/browse/PIG-4443?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600374#comment-14600374
]
Rohini Palaniswamy commented on PIG-4443:
-----------------------------------------
bq. Requested data length 160452289 is longer than maximum configured RPC
length 67108864
bq. but I don't see that message because in my case the previous if condition
is false (splitsSerializedSize=1163342 , spillThreshold=33554432)
In your first error you said 160MB (160452289). This time you mentioned 1MB
(1163342). Can you recheck? For 1MB, it should not be failing. Also could you
paste your full stacktrace so that we can see where exactly is the error.
> Write inputsplits in Tez to disk if the size is huge and option to compress
> pig input splits
> --------------------------------------------------------------------------------------------
>
> Key: PIG-4443
> URL: https://issues.apache.org/jira/browse/PIG-4443
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.14.0
> Reporter: Rohini Palaniswamy
> Assignee: Rohini Palaniswamy
> Fix For: 0.15.0
>
> Attachments: PIG-4443-1.patch, PIG-4443-Fix-TEZ-2192-2.patch,
> PIG-4443-Fix-TEZ-2192.patch
>
>
> Pig sets the input split information in user payload and when running against
> a table with 10s of 1000s of partitions, DAG submission fails with
> java.io.IOException: Requested data length 305844060 is longer than maximum
> configured RPC length 67108864
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
