[
https://issues.apache.org/jira/browse/ASTERIXDB-1474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Blow reassigned ASTERIXDB-1474:
---------------------------------------
Assignee: Michael Blow
Thanks for the detailed analysis; I'll take care of it.
> Cannot query a dataset with big object after restart
> ----------------------------------------------------
>
> Key: ASTERIXDB-1474
> URL: https://issues.apache.org/jira/browse/ASTERIXDB-1474
> Project: Apache AsterixDB
> Issue Type: Bug
> Components: Storage
> Reporter: Murtadha Hubail
> Assignee: Michael Blow
> Priority: Critical
>
> If you try to query a dataset which contains a big object record after an
> instance restart (or simply after the dataset gets deactivated and activated
> again), you get the following exception
> {noformat}
> attempt to unlock read lock, not locked by current thread
> [IllegalMonitorStateException]
> {noformat}
> The complete stacktrace is below
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: 1948279919
> at
> org.apache.hyracks.storage.am.lsm.btree.tuples.LSMBTreeTupleReference.isAntimatter(LSMBTreeTupleReference.java:88)
> at
> org.apache.hyracks.storage.am.lsm.btree.tuples.LSMBTreeTupleReference.resetByTupleOffset(LSMBTreeTupleReference.java:60)
> at
> org.apache.hyracks.storage.am.lsm.btree.tuples.LSMBTreeTupleReference.resetByTupleIndex(LSMBTreeTupleReference.java:75)
> at
> org.apache.hyracks.storage.am.btree.frames.OrderedSlotManager.findTupleIndex(OrderedSlotManager.java:44)
> at
> org.apache.hyracks.storage.am.btree.frames.BTreeNSMLeafFrame.findTupleIndex(BTreeNSMLeafFrame.java:388)
> at
> org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.getLowKeyIndex(BTreeRangeSearchCursor.java:219)
> at
> org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.open(BTreeRangeSearchCursor.java:291)
> at
> org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:794)
> at
> org.apache.hyracks.storage.am.btree.impls.BTree.performOp(BTree.java:679)
> at
> org.apache.hyracks.storage.am.btree.impls.BTree.search(BTree.java:206)
> at
> org.apache.hyracks.storage.am.btree.impls.BTree.access$6(BTree.java:190)
> at
> org.apache.hyracks.storage.am.btree.impls.BTree$BTreeAccessor.search(BTree.java:973)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.hasNext(LSMBTreePointSearchCursor.java:74)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.hasNext(LSMBTreeSearchCursor.java:86)
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.writeSearchResults(IndexSearchOperatorNodePushable.java:149)
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:184)
> at
> org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushAndReset(AbstractOneInputOneOutputOneFramePushRuntime.java:63)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushIfNotFailed(AbstractOneInputOneOutputOneFramePushRuntime.java:69)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at
> org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.NullPointerException
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.NullPointerException
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.NullPointerException
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:187)
> at
> org.apache.hyracks.dataflow.common.comm.io.AbstractFrameAppender.write(AbstractFrameAppender.java:93)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushAndReset(AbstractOneInputOneOutputOneFramePushRuntime.java:63)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.flushIfNotFailed(AbstractOneInputOneOutputOneFramePushRuntime.java:69)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:55)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at
> org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.hyracks.storage.am.bloomfilter.impls.MurmurHash128Bit.hash3_x64_128(MurmurHash128Bit.java:49)
> at
> org.apache.hyracks.storage.am.bloomfilter.impls.BloomFilter.contains(BloomFilter.java:93)
> at
> org.apache.hyracks.storage.am.lsm.common.impls.BloomFilterAwareBTreePointSearchCursor.hasNext(BloomFilterAwareBTreePointSearchCursor.java:39)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.hasNext(LSMBTreePointSearchCursor.java:75)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.hasNext(LSMBTreeSearchCursor.java:86)
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.writeSearchResults(IndexSearchOperatorNodePushable.java:149)
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.nextFrame(IndexSearchOperatorNodePushable.java:184)
> ... 14 more
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at
> org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read
> lock, not locked by current thread
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
> at
> org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
> at
> org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
> ... 11 more
> Jun 07, 2016 4:20:13 AM org.apache.hyracks.control.cc.work.JobCleanupWork run
> WARNING: Ignoring duplicate cleanup for JobRun with id: JID:0
> Jun 07, 2016 4:20:13 AM org.apache.hyracks.control.nc.Joblet close
> WARNING: Freeing leaked 327680 bytes
> Jun 07, 2016 4:20:13 AM org.apache.hyracks.control.nc.Joblet close
> WARNING: Freeing leaked 327680 bytes
> org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
> at
> org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
> at org.apache.hyracks.control.nc.Task.run(Task.java:319)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at
> org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read
> lock, not locked by current thread
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
> at
> org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
> at
> org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
> ... 11 more
> org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
> at
> org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
> at org.apache.hyracks.control.nc.Task.run(Task.java:319)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at
> org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read
> lock, not locked by current thread
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
> at
> org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
> at
> org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
> ... 11 more
> Jun 07, 2016 4:20:13 AM org.apache.asterix.api.http.servlet.APIServlet doPost
> SEVERE: Job failed on account of:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> org.apache.hyracks.api.exceptions.HyracksException: Job failed on account of:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.control.cc.job.JobRun.waitForCompletion(JobRun.java:212)
> at
> org.apache.hyracks.control.cc.work.WaitForJobCompletionWork$1.run(WaitForJobCompletionWork.java:48)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.control.common.utils.ExceptionUtils.setNodeIds(ExceptionUtils.java:45)
> at org.apache.hyracks.control.nc.Task.run(Task.java:319)
> ... 3 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:218)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.initialize(SuperActivityOperatorNodePushable.java:83)
> at org.apache.hyracks.control.nc.Task.run(Task.java:263)
> ... 3 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.runInParallel(SuperActivityOperatorNodePushable.java:212)
> ... 5 more
> Caused by: org.apache.hyracks.api.exceptions.HyracksDataException:
> java.lang.IllegalMonitorStateException: attempt to unlock read lock, not
> locked by current thread
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:211)
> at
> org.apache.hyracks.algebricks.runtime.operators.base.AbstractOneInputOneOutputOneFramePushRuntime.close(AbstractOneInputOneOutputOneFramePushRuntime.java:57)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.AssignRuntimeFactory$1.close(AssignRuntimeFactory.java:122)
> at
> org.apache.hyracks.algebricks.runtime.operators.std.EmptyTupleSourceRuntimeFactory$1.close(EmptyTupleSourceRuntimeFactory.java:60)
> at
> org.apache.hyracks.algebricks.runtime.operators.meta.AlgebricksMetaOperatorDescriptor$1.initialize(AlgebricksMetaOperatorDescriptor.java:116)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable.lambda$initialize$0(SuperActivityOperatorNodePushable.java:83)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:205)
> at
> org.apache.hyracks.api.rewriter.runtime.SuperActivityOperatorNodePushable$1.call(SuperActivityOperatorNodePushable.java:202)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ... 3 more
> Caused by: java.lang.IllegalMonitorStateException: attempt to unlock read
> lock, not locked by current thread
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.unmatchedUnlockException(ReentrantReadWriteLock.java:444)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$Sync.tryReleaseShared(ReentrantReadWriteLock.java:428)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.releaseShared(AbstractQueuedSynchronizer.java:1341)
> at
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.unlock(ReentrantReadWriteLock.java:881)
> at
> org.apache.hyracks.storage.common.buffercache.CachedPage.releaseReadLatch(CachedPage.java:121)
> at
> org.apache.hyracks.storage.am.btree.impls.BTreeRangeSearchCursor.close(BTreeRangeSearchCursor.java:91)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreePointSearchCursor.close(LSMBTreePointSearchCursor.java:211)
> at
> org.apache.hyracks.storage.am.lsm.btree.impls.LSMBTreeSearchCursor.close(LSMBTreeSearchCursor.java:97)
> at
> org.apache.hyracks.storage.am.common.dataflow.IndexSearchOperatorNodePushable.close(IndexSearchOperatorNodePushable.java:208)
> ... 11 more
> {noformat}
> To reproduce the issue execute the following:
> {noformat}
> drop dataverse testdv2 if exists;
> create dataverse testdv2;
> use dataverse testdv2;
> create type testtype as closed {
> id: int64,
> name: string,
> hobbies: {{string}}
> }
> create dataset testds(testtype) primary key id;
> use dataverse testdv2;
> load dataset testds
> using localfs
> (("path"="asterix_nc1://target/data/big-object/big_object_20M.adm"),("format"="adm"))
> pre-sorted;
> {noformat}
> After that, restart the instance and run the following query:
> {noformat}
> use dataverse testdv2;
> for $d in dataset("testds")
> where $d.id = 1
> return $d
> {noformat}
> I traced the problem and found out the issue is related to checking the first
> metadata page of the index. If the first page in the index contains
> supplement blocks, it will be cached in the buffer cache without its
> supplement blocks during the metadata page checking. Then, when the same page
> is requested again during a search, it will be returned from the buffer cache
> without its supplement blocks, causing all types of unexpected behaviors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
