Yeah my fix didn't fix anything; was barking up the wrong tree. The toObject() was the right one I think hotspot just optimized out the intermediate calls. Going to try upgrading to 1.8 before downgrading to u79, will also look at the Phoenix source code with respect to concurrency issues.
> On Feb 15, 2016, at 12:25 PM, Andrew Purtell <[email protected]> wrote: > > You might also consider moving back down to 7u79 > >> On Feb 15, 2016, at 10:35 AM, Jonathan Leech <[email protected]> wrote: >> >> Has anyone else seen this? Happening under load in jdk 1.7.0_80 / phoenix >> 4.5.2 - cloudera labs. Based on the source code, It seems the JVM is calling >> the wrong toObject(), and then dumping. The correct toObject() method is a >> couple parent classes away with some generics and Sun / Oracle must have >> gotten something wrong with that special case under load. Thinking I may be >> able to just patch in a call to the correct method to stop the bleeding. >> Anyone else have other ideas, thoughts, or also want the patch? >> >> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native >> code) >> >> J 7432 C2 >> org.apache.phoenix.schema.types.PDecimal.toObject([BIILorg/apache/phoenix/schema/types/PDataType;Lorg/apache/phoenix/schema/SortOrder;Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Object; >> (376 bytes) @ 0x00007f8332f3de52 [0x00007f8332f3d9c0+0x492] >> >> J 14641 C2 >> org.apache.phoenix.expression.DecimalAddExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z >> (178 bytes) @ 0x00007f83349ecde8 [0x00007f83349eca00+0x3e8] >> >> J 11096 C2 >> org.apache.phoenix.expression.CoerceExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z >> (68 bytes) @ 0x00007f8333ed824c [0x00007f8333ed81c0+0x8c] >> >> J 9742 C2 >> org.apache.phoenix.compile.ExpressionProjector.getValue(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/phoenix/schema/types/PDataType;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Ljava/lang/Object; >> (88 bytes) @ 0x00007f83329f57b8 [0x00007f83329f5760+0x58] >> >> J 8783 C2 >> org.apache.phoenix.compile.UpsertCompiler.upsertSelect(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/schema/TableRef;Lorg/apache/phoenix/compile/RowProjector;Lorg/apache/phoenix/iterate/ResultIterator;[I[I)Lorg/apache/phoenix/execute/MutationState; >> (633 bytes) @ 0x00007f833367c448 [0x00007f833367ba00+0xa48] >> >> J 10636 C2 >> org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/iterate/ResultIterator;Lorg/apache/hadoop/hbase/client/Scan;Ljava/lang/String;)Lorg/apache/phoenix/iterate/PeekingResultIterator; >> (168 bytes) @ 0x00007f8333d63924 [0x00007f8333d63180+0x7a4] >> >> J 8646 C2 >> org.apache.phoenix.iterate.ParallelIterators$1.call()Ljava/lang/Object; (5 >> bytes) @ 0x00007f833358010c [0x00007f833357fe40+0x2cc] >> >> J 9466 C2 org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run()V >> (12 bytes) @ 0x00007f83338e7240 [0x00007f83338e7180+0xc0] >> >> J 7047 C2 >> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V >> (225 bytes) @ 0x00007f8332e4971c [0x00007f8332e49580+0x19c] >> >> J 12402 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (58 bytes) >> @ 0x00007f833420ad2c [0x00007f833420ac80+0xac] >> >> j java.lang.Thread.run()V+26 >> >> v ~StubRoutines::call_stub >> >> V [libjvm.so+0x6020a5] JavaCalls::call_helper(JavaValue*, methodHandle*, >> JavaCallArguments*, Thread*)+0x365 >> >> V [libjvm.so+0x600b08] JavaCalls::call(JavaValue*, methodHandle, >> JavaCallArguments*, Thread*)+0x28 >> >> V [libjvm.so+0x600dd7] JavaCalls::call_virtual(JavaValue*, KlassHandle, >> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197 >> >> V [libjvm.so+0x600ef7] JavaCalls::call_virtual(JavaValue*, Handle, >> KlassHandle, Symbol*, Symbol*, Thread*)+0x47 >> >> V [libjvm.so+0x67f8d5] thread_entry(JavaThread*, Thread*)+0xe5 >> >> V [libjvm.so+0x95bfff] JavaThread::thread_main_inner()+0xdf >> >> V [libjvm.so+0x95c105] JavaThread::run()+0xf5 >> >> V [libjvm.so+0x8238c8] java_start(Thread*)+0x108
