Thanks for raising an issue. Your assessment seems good. Great detective
work on this one!
Russ Weeks wrote:
I hope this is a false alarm but I put together a simple test case to take
Spark out of the equation... the code still fails with a NoSuchMethodError.
I've raised ACCUMULO-3789.
On Fri, May 8, 2015 at 5:01 PM, Russ Weeks<[email protected]> wrote:
Hey, folks,
I've built some code against Accumulo 1.6.1 (HDP 2.2) and I'm having a bit
of trouble running it against the Accumulo 1.7 jars (a8ef75e)
The error I'm seeing is:
Exception in thread "main" org.apache.spark.SparkException: Job aborted
due to stage failure: Task 0 in stage 5.0 failed 1 times, most recent
failure: Lost task 0.0 in stage 5.0 (TID 4, localhost):
java.lang.NoSuchMethodError:
org.apache.accumulo.core.client.lexicoder.LongLexicoder.decode([B)Ljava/lang/Long;
at
com.phemi.agile.pidx.PartitionedIndexRDDPartitioner.getPartition(PartitionedIndexBuilder.java:164)
at org.apache.spark.util.collection.ExternalSorter.org
$apache$spark$util$collection$ExternalSorter$$getPartition(ExternalSorter.scala:113)
at
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:220)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
The code which triggers the error is pretty straightforward:
@Override
public int getPartition(Object keyObj) {
Key key = (Key)keyObj;
long partId = longLex.decode(key.getRow().copyBytes());
if (partId< 0 || partId>= numPartitions) {
throw new IllegalArgumentException("Key " + key + " invalid
partition: " + partId);
}
return (int)partId;
}
The decode method exists in 1.7's LongLexicoder, of course, but its
implementation has been moved to the abstract parent class AbstractEncoder.
It doesn't seem to me like that should break any method signature
definitions, and yet...
If this isn't a compatibility problem, what am I doing wrong here?
Thanks,
-Russ
