Ah that makes sense. Thanks again for all the help.
Do you know how the number of splits is calculated?
I also noticed a couple unusual things in our Splits(as seen below). Primarily
getLength() always return 0l, which I’m guessing is possibly causing other
problems as well. Also our getSplits(..,..) function always returns an empty
array of splits. Does that make sense?
Also do you know if there is any developer documentation on custom
StorageHandlers?
public static final class EDXManifestSplit implements InputSplit {
private int splitNum;
private int totalSplits;
public EDXManifestSplit() {}
public EDXManifestSplit(int splitNum, int totalSplits) {
this.splitNum = splitNum;
this.totalSplits = totalSplits;
}
public Manifest finallySplit(JobConf conf) throws IOException {
// load manifest
String configurationInputManifestPath =
conf.get(HadoopProperties.INPUT_MANIFEST_PATH_PROPERTY);
Manifest manifest = new
ManifestLoader().loadEDXManifestFromPath(conf, configurationInputManifestPath,
conf.getBoolean(HadoopProperties.JOB_FLOW_ID_OVERRIDE, false));
return ManifestUtils.sliceManifest(manifest, splitNum, totalSplits);
}
@Override
public void readFields(DataInput in) throws IOException {
splitNum = in.readInt();
totalSplits = in.readInt();
}
// XXX manifest splits can get large so we compress them to save space
// on HDFS
/** {@inheritDoc} */
@Override
public void write(final DataOutput out) throws IOException {
out.writeInt(splitNum);
out.writeInt(totalSplits);
}
@Override
public long getLength() throws IOException {
return 0L;
}
@Override
public String[] getLocations() throws IOException {
return EMPTY_STRING_ARRAY;
}
}
public class EDXManifestInputFormat extends LoggingObject implements
InputFormat<BytesWritable, BytesWritable> {
private static final String[] EMPTY_STRING_ARRAY = new String[0];
private static final InputSplit[] EMPTY_INPUT_SPLIT_ARRAY = new
InputSplit[0];
private static final IonSystem ION = IonSystemBuilder.standard().build();
private volatile NamedKeySource keySource = null;
---
@Override
public InputSplit[] getSplits(JobConf conf, int numSplits) throws
IOException {
List<InputSplit> splits = new ArrayList<>(numSplits);
for (int i = 0; i < numSplits; i++) {
splits.add(new EDXManifestSplit(i, numSplits));
}
return splits.toArray(EMPTY_INPUT_SPLIT_ARRAY);
}
…
}
On 6/24/16, 5:31 PM, "Gopal Vijayaraghavan" <[email protected] on behalf of
[email protected]> wrote:
> While our StorageHandler does utilize a SERDE that correctly returns
>SerDeStats, it seems like the optimizer
is ignoring these values.
AFAIK, the stats impl is assumed to be approximate & aggregate and is
never used for setting up execution.
> Would anyone know how to correctly set these values?
<https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/h
ive/ql/exec/tez/ColumnarSplitSizeEstimator.java#L48>
That's where the sizes are read out for distribution (i.e grouping of
splits etc).
Implementing ColumnarSplit in your split object should do the trick or
wrapping it in an impl.
This is a departure from MapReduce which uses File size instead - with
columnar formats, the total amount of data read out of the file varies as
the selected columns go up/down & similarly how much would be shuffled out.
Cheers,
Gopal
