Hello everyone,
I noticed the following does not work in my pipeline:
@Override
public MergedAttrMapping map(Pair<AttrMapping, Iterable<Integer>>
attrMappingIterablePair) {
MergedAttrMapping out = mapper.map(attrMappingIterablePair.first());
StringBuilder ids = new StringBuilder();
Iterator<Integer> it = attrMappingIterablePair.second().iterator();
while (it.hasNext()) {
Integer I = it.next();
if (i != null && i != 0) {
ids.append(i);
ids.append('|');
}
}
if (ids.length() > 0) {
ids.deleteCharAt(ids.length() -1);
}
out.setIds(ids.toString());
return out;
}
Causing the following exception:
Error: java.util.NoSuchElementException: iterate past last value at
org.apache.hadoop.mapreduce.task.ReduceContextImpl$ValueIterator.next(ReduceContextImpl.java:235)
at
org.apache.crunch.types.avro.AvroPairConverter$AvroWrappedIterable$1.next(AvroPairConverter.java:103)
at
org.apache.crunch.types.PGroupedTableType$HoldLastIterator.next(PGroupedTableType.java:84)
at com.videologygroup.crunch.FteWarehouse$1.map(FteWarehouse.java:268) at
com.videologygroup.crunch.FteWarehouse$1.map(FteWarehouse.java:257) at
org.apache.crunch.MapFn.process(MapFn.java:34) at
org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at
org.apache.crunch.impl.mr.emit.IntermediateEmitter.emit(IntermediateEmitter.java:56)
at org.apache.crunch.MapFn.process(MapFn.java:34) at
org.apache.crunch.impl.mr.run.RTNode.process(RTNode.java:98) at
org.apache.crunch.impl.mr.run.RTNode.processIterable(RTNode.java:113) at
org.apache.crunch.impl.mr.run.CrunchReducer.reduce(CrunchReducer.java:57) at
org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:171) at
org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:627) at
org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389) at
org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) at
java.security.AccessController.doPrivileged(Native Method) at
javax.security.auth.Subject.doAs(Subject.java:415) at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163)
but when I change it to this (change highlighted in yellow):
@Override
public MergedAttrMapping map(Pair<AttrMapping, Iterable<Integer>>
attrMappingIterablePair) {
MergedAttrMapping out = mapper.map(attrMappingIterablePair.first());
StringBuilder dmpAttrs = new StringBuilder();
for (Integer i : attrMappingIterablePair.second()) {
if (i != null && i != 0) {
ids.append(i);
ids.append('|');
}
}
if (ids.length() > 0) {
ids.deleteCharAt(ids.length() -1);
}
out.setIds(ids.toString());
return out;
}
It does.
Is this a bug?
Thanks,
Dave Ortiz
This email is intended only for the use of the individual(s) to whom it is
addressed. If you have received this communication in error, please immediately
notify the sender and delete the original email.
