[ https://issues.apache.org/jira/browse/IGNITE-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ksenia Rybakova updated IGNITE-3300: ------------------------------------ Priority: Blocker (was: Major) > ArrayIndexOutOfBoundsException: -1 during capitalization benchmark running > -------------------------------------------------------------------------- > > Key: IGNITE-3300 > URL: https://issues.apache.org/jira/browse/IGNITE-3300 > Project: Ignite > Issue Type: Bug > Components: general > Affects Versions: 1.7 > Reporter: Ksenia Rybakova > Priority: Blocker > > Test config: > 1 client, 16 servers > warmup 60 > duration 1h > preload 5M > operations: capitalization benchmark > backups count 1 > After ~30 min after start the following errors occur at the driver: > {noformat} > [08:42:51,367][ERROR][sys-#5%null%][GridTaskWorker] Failed to obtain remote > job result policy for result from ComputeTask.result(..) method (will fail > the whole task): GridJobResultImpl [job=C4V2 > [r=o.a.i.yardstick.cache.load.IgniteCapitalizationBenchmark$ScanQueryBroadcastClosure@41013d64], > sib=GridJobSiblingImpl > [sesId=bfb21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > jobId=51c21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > nodeId=79226bcc-651f-4af7-8d39-fd4019b7708c, isJobDone=false], > jobCtx=GridJobContextImpl > [jobId=51c21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63, timeoutObj=null, > attrs={}], node=TcpDiscoveryNode [id=79226bcc-651f-4af7-8d39-fd4019b7708c, > addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47501, > /10.20.0.216:47501, /127.0.0.1:47501], discPort=47501, order=5, intOrder=5, > lastExchangeTime=1465485122287, loc=false, ver=1.7.0#20160603-sha1:82573436, > isClient=false], ex=class o.a.i.compute.ComputeUserUndeclaredException: > Failed to execute job due to unexpected runtime exception > [jobId=51c21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > ses=GridJobSessionImpl [ses=GridTaskSessionImpl > [taskName=o.a.i.yardstick.cache.load.IgniteCapitalizationBenchmark$ScanQueryBroadcastClosure, > dep=LocalDeployment [super=GridDeployment [ts=1465485111989, depMode=SHARED, > clsLdr=sun.misc.Launcher$AppClassLoader@6da264f1, > clsLdrId=cee28b53551-79226bcc-651f-4af7-8d39-fd4019b7708c, userVer=0, > loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, > undeployed=false, usage=0]], > taskClsName=o.a.i.yardstick.cache.load.IgniteCapitalizationBenchmark$ScanQueryBroadcastClosure, > sesId=bfb21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > startTime=1465486810898, endTime=9223372036854775807, > taskNodeId=a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > clsLdr=sun.misc.Launcher$AppClassLoader@6da264f1, closed=false, cpSpi=null, > failSpi=null, loadSpi=null, usage=1, fullSup=false, > subjId=a8d91d06-89bd-41a5-8b4f-6a02416dfc63, mapFut=IgniteFuture > [orig=GridFutureAdapter [resFlag=0, res=null, startTime=1465486811180, > endTime=0, ignoreInterrupts=false, state=INIT]]], > jobId=51c21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63]], hasRes=true, > isCancelled=false, isOccupied=true] > class org.apache.ignite.IgniteException: Remote job threw user exception > (override or implement ComputeTask.result(..) method if you would like to > have automatic failover for this exception). > at > org.apache.ignite.compute.ComputeTaskAdapter.result(ComputeTaskAdapter.java:101) > at > org.apache.ignite.internal.processors.task.GridTaskWorker$3.apply(GridTaskWorker.java:912) > at > org.apache.ignite.internal.processors.task.GridTaskWorker$3.apply(GridTaskWorker.java:905) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6491) > at > org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:905) > at > org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:801) > at > org.apache.ignite.internal.processors.task.GridTaskProcessor.processJobExecuteResponse(GridTaskProcessor.java:995) > at > org.apache.ignite.internal.processors.task.GridTaskProcessor$JobMessageListener.onMessage(GridTaskProcessor.java:1220) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105) > at > org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: class org.apache.ignite.compute.ComputeUserUndeclaredException: > Failed to execute job due to unexpected runtime exception > [jobId=51c21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > ses=GridJobSessionImpl [ses=GridTaskSessionImpl > [taskName=org.apache.ignite.yardstick.cache.load.IgniteCapitalizationBenchmark$ScanQueryBroadcastClosure, > dep=LocalDeployment [super=GridDeployment [ts=1465485111989, depMode=SHARED, > clsLdr=sun.misc.Launcher$AppClassLoader@6da264f1, > clsLdrId=cee28b53551-79226bcc-651f-4af7-8d39-fd4019b7708c, userVer=0, > loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, > undeployed=false, usage=0]], > taskClsName=org.apache.ignite.yardstick.cache.load.IgniteCapitalizationBenchmark$ScanQueryBroadcastClosure, > sesId=bfb21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > startTime=1465486810898, endTime=9223372036854775807, > taskNodeId=a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > clsLdr=sun.misc.Launcher$AppClassLoader@6da264f1, closed=false, cpSpi=null, > failSpi=null, loadSpi=null, usage=1, fullSup=false, > subjId=a8d91d06-89bd-41a5-8b4f-6a02416dfc63, mapFut=IgniteFuture > [orig=GridFutureAdapter [resFlag=0, res=null, startTime=1465486811180, > endTime=0, ignoreInterrupts=false, state=INIT]]], > jobId=51c21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63]] > at > org.apache.ignite.internal.processors.job.GridJobWorker.handleThrowable(GridJobWorker.java:607) > at > org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:551) > at > org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1161) > at > org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1766) > ... 7 more > Caused by: javax.cache.CacheException: class > org.apache.ignite.IgniteCheckedException: Failed to execute SQL query. > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.queryLocalFields(GridQueryProcessor.java:983) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:689) > at > org.apache.ignite.yardstick.cache.load.IgniteCapitalizationBenchmark$ScanQueryBroadcastClosure.run(IgniteCapitalizationBenchmark.java:246) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4V2.execute(GridClosureProcessor.java:2206) > at > org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:509) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6459) > at > org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:503) > ... 11 more > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to execute > SQL query. > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:847) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:870) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryFields(IgniteH2Indexing.java:738) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor$7.applyx(GridQueryProcessor.java:965) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor$7.applyx(GridQueryProcessor.java:959) > at > org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:1769) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.queryLocalFields(GridQueryProcessor.java:959) > ... 17 more > Caused by: org.h2.jdbc.JdbcSQLException: General error: > "java.lang.ArrayIndexOutOfBoundsException: -1"; SQL statement: > SELECT _key FROM "deposit".Deposit WHERE affKey=? [50000-175] > at > org.h2.message.DbException.getJdbcSQLException(DbException.java:332) > at org.h2.message.DbException.get(DbException.java:161) > at org.h2.message.DbException.convert(DbException.java:284) > at org.h2.command.Command.executeQuery(Command.java:208) > at > org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:106) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:844) > ... 24 more > Caused by: java.lang.ArrayIndexOutOfBoundsException: -1 > at java.util.ArrayList.elementData(ArrayList.java:400) > at java.util.ArrayList.get(ArrayList.java:413) > at > org.apache.ignite.internal.processors.affinity.GridAffinityAssignment.get(GridAffinityAssignment.java:152) > at > org.apache.ignite.internal.processors.affinity.GridAffinityAssignmentCache.nodes(GridAffinityAssignmentCache.java:387) > at > org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.nodes(GridCacheAffinityManager.java:251) > at > org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:287) > at > org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:278) > at > org.apache.ignite.internal.processors.cache.GridCacheAffinityManager.primary(GridCacheAffinityManager.java:302) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing$7$3.apply(IgniteH2Indexing.java:1730) > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase$FilteringIterator.accept(GridH2IndexBase.java:251) > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase$FilteringIterator.accept(GridH2IndexBase.java:200) > at > org.apache.ignite.internal.util.lang.GridFilteredIterator.hasNext(GridFilteredIterator.java:59) > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2Cursor.next(GridH2Cursor.java:59) > at org.h2.index.IndexCursor.next(IndexCursor.java:274) > at org.h2.table.TableFilter.next(TableFilter.java:359) > at org.h2.command.dml.Select.queryFlat(Select.java:527) > at org.h2.command.dml.Select.queryWithoutCache(Select.java:632) > at org.h2.command.dml.Query.query(Query.java:297) > at org.h2.command.dml.Query.query(Query.java:284) > at org.h2.command.dml.Query.query(Query.java:36) > at org.h2.command.CommandContainer.query(CommandContainer.java:91) > at org.h2.command.Command.executeQuery(Command.java:196) > ... 26 more > Finishing main test [ts=1465486971375, date=Thu Jun 09 08:42:51 PDT 2016] > ERROR: Shutting down benchmark driver to unexpected exception. > {noformat} > At the servers: > {noformat} > [08:42:51,383][ERROR][pub-#12%null%][GridJobWorker] Failed to execute job due > to unexpected runtime exception > [jobId=74c21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > ses=GridJobSessionImpl [ses=GridTaskSessionImpl > [taskName=o.a.i.yardstick.cache.load.IgniteCapitalizationBenchmark$ScanQueryBroadcastClosure, > dep=LocalDeployment [super=GridDeployment [ts=1465485109866, depMode=SHARED, > clsLdr=sun.misc.Launcher$AppClassLoader@40914272, > clsLdrId=6c628b53551-aa875cfe-4e8c-4f21-bdb4-aed0d3b56764, userVer=0, > loc=true, sampleClsName=java.lang.String, pendingUndeploy=false, > undeployed=false, usage=0]], > taskClsName=o.a.i.yardstick.cache.load.IgniteCapitalizationBenchmark$ScanQueryBroadcastClosure, > sesId=bfb21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > startTime=1465486810898, endTime=9223372036854775807, > taskNodeId=a8d91d06-89bd-41a5-8b4f-6a02416dfc63, > clsLdr=sun.misc.Launcher$AppClassLoader@40914272, closed=false, cpSpi=null, > failSpi=null, loadSpi=null, usage=1, fullSup=false, > subjId=a8d91d06-89bd-41a5-8b4f-6a02416dfc63, mapFut=IgniteFuture > [orig=GridFutureAdapter [resFlag=0, res=null, startTime=1465486811190, > endTime=0, ignoreInterrupts=false, state=INIT]]], > jobId=74c21563551-a8d91d06-89bd-41a5-8b4f-6a02416dfc63]] > javax.cache.CacheException: class org.apache.ignite.IgniteCheckedException: > Failed to execute SQL query. > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.queryLocalFields(GridQueryProcessor.java:983) > at > org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:689) > at > org.apache.ignite.yardstick.cache.load.IgniteCapitalizationBenchmark$ScanQueryBroadcastClosure.run(IgniteCapitalizationBenchmark.java:246) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$C4V2.execute(GridClosureProcessor.java:2206) > at > org.apache.ignite.internal.processors.job.GridJobWorker$2.call(GridJobWorker.java:509) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6459) > at > org.apache.ignite.internal.processors.job.GridJobWorker.execute0(GridJobWorker.java:503) > at > org.apache.ignite.internal.processors.job.GridJobWorker.body(GridJobWorker.java:456) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > org.apache.ignite.internal.processors.job.GridJobProcessor.processJobExecuteRequest(GridJobProcessor.java:1161) > at > org.apache.ignite.internal.processors.job.GridJobProcessor$JobExecutionListener.onMessage(GridJobProcessor.java:1766) > at > org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1219) > at > org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:847) > at > org.apache.ignite.internal.managers.communication.GridIoManager.access$1700(GridIoManager.java:105) > at > org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:810) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to execute > SQL query. > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:847) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQueryWithTimer(IgniteH2Indexing.java:870) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryFields(IgniteH2Indexing.java:738) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor$7.applyx(GridQueryProcessor.java:965) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor$7.applyx(GridQueryProcessor.java:959) > at > org.apache.ignite.internal.util.lang.IgniteOutClosureX.apply(IgniteOutClosureX.java:36) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.executeQuery(GridQueryProcessor.java:1769) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.queryLocalFields(GridQueryProcessor.java:959) > ... 17 more > Caused by: org.h2.jdbc.JdbcSQLException: General error: "class > org.apache.ignite.IgniteException: Thread got interrupted while trying to > acquire index lock."; SQL statement: > SELECT _key FROM "deposit".Deposit WHERE affKey=? [50000-175] > at > org.h2.message.DbException.getJdbcSQLException(DbException.java:332) > at org.h2.message.DbException.get(DbException.java:161) > at org.h2.message.DbException.convert(DbException.java:284) > at org.h2.command.Command.executeQuery(Command.java:208) > at > org.h2.jdbc.JdbcPreparedStatement.executeQuery(JdbcPreparedStatement.java:106) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.executeSqlQuery(IgniteH2Indexing.java:844) > ... 24 more > Caused by: class org.apache.ignite.IgniteException: Thread got interrupted > while trying to acquire index lock. > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.lock(GridH2Table.java:241) > at org.h2.table.TableFilter.lock(TableFilter.java:145) > at org.h2.command.dml.Select.queryWithoutCache(Select.java:618) > at org.h2.command.dml.Query.query(Query.java:297) > at org.h2.command.dml.Query.query(Query.java:284) > at org.h2.command.dml.Query.query(Query.java:36) > at org.h2.command.CommandContainer.query(CommandContainer.java:91) > at org.h2.command.Command.executeQuery(Command.java:196) > ... 26 more > Caused by: java.lang.InterruptedException > at > java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1244) > at > java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.tryLock(ReentrantReadWriteLock.java:1115) > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.lock(GridH2Table.java:237) > ... 33 more > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)