As I was seeing sporadic errors similar to that of PHOENIX-2169 in addition to the core dumps, I decided to refactor the upsert select into separate upsert and select statements. So far this has fixed both the errors and the core dumps. I believe there to be a concurrency issue in upsert select between two salted tables, similar to PHOENIX-2169.
- Jonathan > On Feb 16, 2016, at 10:02 AM, Jonathan Leech <[email protected]> wrote: > > Yeah my fix didn't fix anything; was barking up the wrong tree. The > toObject() was the right one I think hotspot just optimized out the > intermediate calls. Going to try upgrading to 1.8 before downgrading to u79, > will also look at the Phoenix source code with respect to concurrency issues. > > > >> On Feb 15, 2016, at 12:25 PM, Andrew Purtell <[email protected]> >> wrote: >> >> You might also consider moving back down to 7u79 >> >>> On Feb 15, 2016, at 10:35 AM, Jonathan Leech <[email protected]> wrote: >>> >>> Has anyone else seen this? Happening under load in jdk 1.7.0_80 / phoenix >>> 4.5.2 - cloudera labs. Based on the source code, It seems the JVM is >>> calling the wrong toObject(), and then dumping. The correct toObject() >>> method is a couple parent classes away with some generics and Sun / Oracle >>> must have gotten something wrong with that special case under load. >>> Thinking I may be able to just patch in a call to the correct method to >>> stop the bleeding. Anyone else have other ideas, thoughts, or also want the >>> patch? >>> >>> Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native >>> code) >>> >>> J 7432 C2 >>> org.apache.phoenix.schema.types.PDecimal.toObject([BIILorg/apache/phoenix/schema/types/PDataType;Lorg/apache/phoenix/schema/SortOrder;Ljava/lang/Integer;Ljava/lang/Integer;)Ljava/lang/Object; >>> (376 bytes) @ 0x00007f8332f3de52 [0x00007f8332f3d9c0+0x492] >>> >>> J 14641 C2 >>> org.apache.phoenix.expression.DecimalAddExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z >>> (178 bytes) @ 0x00007f83349ecde8 [0x00007f83349eca00+0x3e8] >>> >>> J 11096 C2 >>> org.apache.phoenix.expression.CoerceExpression.evaluate(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Z >>> (68 bytes) @ 0x00007f8333ed824c [0x00007f8333ed81c0+0x8c] >>> >>> J 9742 C2 >>> org.apache.phoenix.compile.ExpressionProjector.getValue(Lorg/apache/phoenix/schema/tuple/Tuple;Lorg/apache/phoenix/schema/types/PDataType;Lorg/apache/hadoop/hbase/io/ImmutableBytesWritable;)Ljava/lang/Object; >>> (88 bytes) @ 0x00007f83329f57b8 [0x00007f83329f5760+0x58] >>> >>> J 8783 C2 >>> org.apache.phoenix.compile.UpsertCompiler.upsertSelect(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/schema/TableRef;Lorg/apache/phoenix/compile/RowProjector;Lorg/apache/phoenix/iterate/ResultIterator;[I[I)Lorg/apache/phoenix/execute/MutationState; >>> (633 bytes) @ 0x00007f833367c448 [0x00007f833367ba00+0xa48] >>> >>> J 10636 C2 >>> org.apache.phoenix.compile.MutatingParallelIteratorFactory.newIterator(Lorg/apache/phoenix/compile/StatementContext;Lorg/apache/phoenix/iterate/ResultIterator;Lorg/apache/hadoop/hbase/client/Scan;Ljava/lang/String;)Lorg/apache/phoenix/iterate/PeekingResultIterator; >>> (168 bytes) @ 0x00007f8333d63924 [0x00007f8333d63180+0x7a4] >>> >>> J 8646 C2 >>> org.apache.phoenix.iterate.ParallelIterators$1.call()Ljava/lang/Object; (5 >>> bytes) @ 0x00007f833358010c [0x00007f833357fe40+0x2cc] >>> >>> J 9466 C2 >>> org.apache.phoenix.job.JobManager$InstrumentedJobFutureTask.run()V (12 >>> bytes) @ 0x00007f83338e7240 [0x00007f83338e7180+0xc0] >>> >>> J 7047 C2 >>> java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V >>> (225 bytes) @ 0x00007f8332e4971c [0x00007f8332e49580+0x19c] >>> >>> J 12402 C2 java.util.concurrent.ThreadPoolExecutor$Worker.run()V (58 bytes) >>> @ 0x00007f833420ad2c [0x00007f833420ac80+0xac] >>> >>> j java.lang.Thread.run()V+26 >>> >>> v ~StubRoutines::call_stub >>> >>> V [libjvm.so+0x6020a5] JavaCalls::call_helper(JavaValue*, methodHandle*, >>> JavaCallArguments*, Thread*)+0x365 >>> >>> V [libjvm.so+0x600b08] JavaCalls::call(JavaValue*, methodHandle, >>> JavaCallArguments*, Thread*)+0x28 >>> >>> V [libjvm.so+0x600dd7] JavaCalls::call_virtual(JavaValue*, KlassHandle, >>> Symbol*, Symbol*, JavaCallArguments*, Thread*)+0x197 >>> >>> V [libjvm.so+0x600ef7] JavaCalls::call_virtual(JavaValue*, Handle, >>> KlassHandle, Symbol*, Symbol*, Thread*)+0x47 >>> >>> V [libjvm.so+0x67f8d5] thread_entry(JavaThread*, Thread*)+0xe5 >>> >>> V [libjvm.so+0x95bfff] JavaThread::thread_main_inner()+0xdf >>> >>> V [libjvm.so+0x95c105] JavaThread::run()+0xf5 >>> >>> V [libjvm.so+0x8238c8] java_start(Thread*)+0x108
