Murtadha Hubail created ASTERIXDB-1474:
------------------------------------------
Summary: Cannot query a dataset with big object after restart
Key: ASTERIXDB-1474
URL: https://issues.apache.org/jira/browse/ASTERIXDB-1474
Project: Apache AsterixDB
Issue Type: Bug
Components: Storage
Reporter: Murtadha Hubail
Priority: Critical
If you try to query a dataset which contains a big object record after an
instance restart (or simply after the dataset gets deactivated and activated
again), you get the following exception
{noformat}
attempt to unlock read lock, not locked by current thread
[IllegalMonitorStateException]
{noformat}
The complete stacktrace is below
{noformat}
java.lang.ArrayIndexOutOfBoundsException: 1948279919
at
org.apache.hyracks.storage.am.lsm.btree.tuples.LSMBTreeTupleReference.isAntimatter(LSMBTreeTupleReference.java:88)
at
org.apache.hyracks.storage.am.lsm.btree.tuples.LSMBTreeTupleReference.resetByTupleOffset(LSMBTreeTupleReference.java:60)
at
org.apache.hyracks.storage.am.lsm.btree.tuples.LSMBTreeTupleReference.resetByTupleIndex(LSMBTreeTupleReference.java:75)
at
org.apache.hyracks.storage.am.btree.frames.OrderedSlotManager.findTupleIndex(OrderedSlotManager.java:44)
at
org.apache.hyracks.storage.am.btree.frames.BTreeNSMLeafFrame.findTupleIndex(BTreeNSMLeafFrame.java:388)
at
org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.getLowKeyIndex(BTreeRangeSearchCursor.java:219)
at
org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.open(BTreeRangeSearchCursor.java:291)
at
org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:794)
at
org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:679)
at
org.apache.hyracks.storage.am.btree.impls.BTree.search(BTree.java:206)
at
org.apache.hyracks.storage.am.btree.impls.BTree.access$6(BTree.java:190)
at
org.apache.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.search(BTree.java:973)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.hasNext(LSMBTreePointSearchCursor.java:74)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.hasNext(LSMBTreeSearchCursor.java:86)
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.writeSearchResults(IndexSearchOperatorNodePushable.java:149)
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:184)
at
org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93)
at
org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushAndReset(AbstractOneInputOneOutputOneFramePushRuntime.java:63)
at
org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushIfNotFailed(AbstractOneInputOneOutputOneFramePushRuntime.java:69)
at
org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
at
org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
at
org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
at
org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.NullPointerException
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
at org.apache.hyracks.control.nc.Task.run(Task.java:263)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.NullPointerException
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
... 5 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.NullPointerException
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:187)
at
org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93)
at
org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushAndReset(AbstractOneInputOneOutputOneFramePushRuntime.java:63)
at
org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushIfNotFailed(AbstractOneInputOneOutputOneFramePushRuntime.java:69)
at
org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
at
org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
at
org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
at
org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.NullPointerException
at
org.apache.hyracks.storage.am.bloomfilter.impls.MurmurHash128Bit.hash3_x64_128(MurmurHash128Bit.java:49)
at
org.apache.hyracks.storage.am.bloomfilter.impls.BloomFilter.contains(BloomFilter.java:93)
at
org.apache.hyracks.storage.am.lsm.common.impls.BloomFilterAwareBTreePointSearchCursor.hasNext(BloomFilterAwareBTreePointSearchCursor.java:39)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.hasNext(LSMBTreePointSearchCursor.java:75)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.hasNext(LSMBTreeSearchCursor.java:86)
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.writeSearchResults(IndexSearchOperatorNodePushable.java:149)
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:184)
... 14 more
org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
at org.apache.hyracks.control.nc.Task.run(Task.java:263)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
... 5 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
at
org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
at
org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
at
org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
at
org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read lock,
not locked by current thread
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
at
org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
at
org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
... 11 more
Jun 07, 2016 4:20:13 AM org.apache.hyracks.control.cc.work.JobCleanupWork run
WARNING: Ignoring duplicate cleanup for JobRun with id: JID:0
Jun 07, 2016 4:20:13 AM org.apache.hyracks.control.nc.Joblet close
WARNING: Freeing leaked 327680 bytes
Jun 07, 2016 4:20:13 AM org.apache.hyracks.control.nc.Joblet close
WARNING: Freeing leaked 327680 bytes
org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
at
org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
at org.apache.hyracks.control.nc.Task.run(Task.java:319)
... 3 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
at org.apache.hyracks.control.nc.Task.run(Task.java:263)
... 3 more
Caused by: java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
... 5 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
at
org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
at
org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
at
org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
at
org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read lock,
not locked by current thread
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
at
org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
at
org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
... 11 more
org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
at
org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
at org.apache.hyracks.control.nc.Task.run(Task.java:319)
... 3 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
at org.apache.hyracks.control.nc.Task.run(Task.java:263)
... 3 more
Caused by: java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
... 5 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
at
org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
at
org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
at
org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
at
org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read lock,
not locked by current thread
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
at
org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
at
org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
... 11 more
Jun 07, 2016 4:20:13 AM org.apache.asterix.api.http.servlet.APIServlet doPost
SEVERE: Job failed on account of:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
at
org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
at org.apache.hyracks.control.nc.Task.run(Task.java:319)
... 3 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
at org.apache.hyracks.control.nc.Task.run(Task.java:263)
... 3 more
Caused by: java.util.concurrent.ExecutionException:
org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
... 5 more
Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
java.lang.IllegalMonitorStateException: attempt to unlock read lock, not locked
by current thread
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
at
org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
at
org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
at
org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
at
org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
at
org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 more
Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read lock,
not locked by current thread
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
at
java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
at
java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
at
org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
at
org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
at
org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
at
org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
... 11 more
{noformat}
To reproduce the issue execute the following:
{noformat}
drop dataverse testdv2 if exists;
create dataverse testdv2;
use dataverse testdv2;
create type testtype as closed {
id: int64,
name: string,
hobbies: {{string}}
}
create dataset testds(testtype) primary key id;
use dataverse testdv2;
load dataset testds
using localfs
(("path"="asterix_nc1://target/data/big-object/big_object_20M.adm"),("format"="adm"))
pre-sorted;
{noformat}
After that, restart the instance and run the following query:
{noformat}
use dataverse testdv2;
for $d in dataset("testds")
where $d.id = 1
return $d
{noformat}
I traced the problem and found out the issue is related to checking the first
metadata page of the index. If the first page in the index contains supplement
blocks, it will be cached in the buffer cache without its supplement blocks
during the metadata page checking. Then, when the same page is requested again
during a search, it will be returned from the buffer cache without its
supplement blocks, causing all types of unexpected behaviors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
