[
https://issues.apache.org/jira/browse/TEZ-2144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14337288#comment-14337288
]
Rohini Palaniswamy commented on TEZ-2144:
-----------------------------------------
There are 3 ways to pass split information in Tez -
https://issues.apache.org/jira/browse/PIG-3564?focusedCommentId=13816848&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13816848
Pig sets it on the DAG submission payload to AM. Calculating splits in AM is
the best. But the probem is with input formats like HCat and HBase, the
delegation token is fetched with getInputSplits().
{code}
tezOp.getLoaderInfo().setInputSplitInfo(MRInputHelpers.generateInputSplitsToMem(conf,
false, 0));
......
......
vertex.setLocationHint(VertexLocationHint.create(tezOp.getLoaderInfo().getInputSplitInfo().getTaskLocationHints()));
vertex.addDataSource(ld.getOperatorKey().toString(),
DataSourceDescriptor.create(InputDescriptor.create(MRInput.class.getName())
.setUserPayload(UserPayload.create(MRRuntimeProtos.MRInputUserPayloadProto.newBuilder()
.setConfigurationBytes(TezUtils.createByteStringFromConf(payloadConf))
.setSplits(tezOp.getLoaderInfo().getInputSplitInfo().getSplitsProto()).build().toByteString().asReadOnlyByteBuffer()))
.setHistoryText(convertToHistoryText("",
payloadConf)),
InputInitializerDescriptor.create(MRInputSplitDistributor.class.getName()),
dag.getCredentials()));
{code}
Pig on Tez HCatLoader jobs with one of the biggest hcat tables with huge number
of partitions hits the RPC limit issue. Running on HDFS data which has even
triple the number of splits(100K+ splits and tasks) does not hit this issue.
HCatBaseInputFormat.java:
{code}
//Call getSplit on the InputFormat, create an
//HCatSplit for each underlying split
//NumSplits is 0 for our purposes
org.apache.hadoop.mapred.InputSplit[] baseSplits =
inputFormat.getSplits(jobConf, 0);
for(org.apache.hadoop.mapred.InputSplit split : baseSplits) {
splits.add(new HCatSplit(
partitionInfo,
split,allCols));
}
{code}
Each hcatSplit duplicates a lot of information - partition schema and table
schema
On the Pig side, trying to workaround the problem by using the alternative of
writing the split files to hdfs for big splits like this. Increasing the RPC
size limit will not help as it is going to 300MB just for a day's worth of
data. Weekly and monthly data scanning will make it run into GBs.
Had a discussion with [~sseth] to see if it make sense for tez to compress the
input split information. It would help a lot in cases like this. Even for HDFS
paths, with the same base path being repeated compression would help. He asked
to create this jira to see if the compression would need to be in Tez or Pig,
check if there will be any gain if each proto is compressed separately or
should the entire payload be compressed
> Compressing user payload
> ------------------------
>
> Key: TEZ-2144
> URL: https://issues.apache.org/jira/browse/TEZ-2144
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Rohini Palaniswamy
>
> Pig sets the input split information in user payload and when running against
> a table with 10s of 1000s of partitions, DAG submission fails with
> java.io.IOException: Requested data length 305844060 is longer than maximum
> configured RPC length 67108864
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
