Re: java.lang.IllegalArgumentException: Can not set final

[smurphy]

Hi Ilya,

I don't think the errors are caused by the binary objects, but by having
custom objects in the EntryProcessor. Checkout the attached
FragmentAssignToScannerEntryProcessor.java. Instead of injecting the
BinaryFragmentConverter into FragmentAssignToScannerEntryProcessor, I moved
all the binary object code from BinaryFragmentConverter directly into
FragmentAssignToScannerEntryProcessor. 
This works fine.

Then I created the static FragmentAssignToScannerEntryProcessor.Tester
class, which has no binary objects, and added it as a member variable of
FragmentAssignToScannerEntryProcessor and that resulted in the same
IllegalArgumentException as before: "failed to read field [name=tester]"

FragmentAssignToScannerEntryProcessor.java

  

stackTrace.txt
  



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: java.lang.IllegalArgumentException: Can not set final

[smurphy]

Thanks for the responses.

Ilya - I did try your suggestion: removing the final modifier, deleting the
constructor and only using getters and setters. 
I even went as far as to make the fields public. 
All these changes resulted in the an IllegalArgumentException.

If it helps, I was able to write a JUnit that replicated the exception and
fixed it along the lines of the following article:

https://docs.oracle.com/javase/tutorial/reflect/member/fieldTrouble.html



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: java.lang.IllegalArgumentException: Can not set final

[smurphy]

Hmm pasting the stack trace into the page didn't work.
Here it is as an attachment..:


stackTrace.txt
  





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

java.lang.IllegalArgumentException: Can not set final

[smurphy]

FragmentAssignToScannerEntryProcessor.java

  
BinaryFragmentConverter.java

  

I am running 2 server nodes with peer class loading enabled and no custom
classes on the server classpaths.
I am using an EntryProcessor with a custom class constructor injected as a
private final member variable (both files are attached).
The custom class cannot be deserialized with the following stack trace:






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: ClassNotFoundException when remotely calling cache.withKeepBinary().get(key)

[smurphy]

Ilya,

Apologies for the slow response..
You are right - this fixed my issue.
Thanks,

Steve



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

ClassNotFoundException when remotely calling cache.withKeepBinary().get(key)

[smurphy]

(Full logs and test class should be attached at the bottom..)

I am running ver. 2.5.0 in client/server mode and am calling
ignite.compute().call(..) on an IgniteCallable.
Server node does not have any of my code on its classpath, but peer class
loading it set to true on both the server and the client node.

I am getting java.lang.ClassNotFoundException when executing the following
line in my IgniteCallable:
BinaryObject binaryObject = _binaryCache.get(id);

I expected a binary object. I debugged into GridCacheAdapter (line 1345):
boolean keepBinary = ctx.keepBinary();

This method is returning false (I would expect this value to be true.?)
If I manually set the value to true, then the test will pass. 

The callable will run successfully if I run it locally, as in not putting it
in the ignite.compute.call(..) method. 
Also, the TestObject is successfully put into the cache (as a binary
object).
It is the get that is deserializing to the custom java object.

PeerClassLoadingTest.java

  

ignite.logs
  




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Upgrade from 2.1.0 to 2.4.0 resulting in error within transaction block

[smurphy]

>From my reading of the javadoc of org.apache.ignite.IgniteCache, only methods
that are inherited from javax.cache.Cache are transactional. So, get(..) and
put(..) are transactional, but query(..) is not..

>From the javadoc...:

Transactions
Cache API supports transactions. You can group and set of cache methods
within a transaction to provide ACID-compliant behavior. See
IgniteTransactions for more information. Methods which can be used inside
transaction (put, get...) throw TransactionException. See
TransactionException for more information.





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Upgrade from 2.1.0 to 2.4.0 resulting in error within transaction block

[smurphy]

Another update..

I changed the code (shown at the beginning of this thread) that deletes
records from:

SqlFieldsQuery sqlQuery = new SqlFieldsQuery("delete from EngineFragment 
where " + criteria()); 
fragmentCache.query(sqlQuery.setArgs(criteria.getArgs())); 

to:

_fragmentCache.remove(id);


This code works successfully within a transaction..




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Upgrade from 2.1.0 to 2.4.0 resulting in error within transaction block

[smurphy]

I neglected to say that I also removed Spring's ChainedTransactionManager and
used SpringTransactionManager in the annotation, which also resulted in the
same stack trace..

@Transactional("igniteTxMgr")
// code that deletes from cache... 


// Here is how the transaction manager is wired up..
@Bean
public PlatformTransactionManager igniteTxMgr(Ignite ignite) {
SpringTransactionManager igniteTxMgr = new
SpringTransactionManager();
igniteTxMgr.setIgniteInstanceName(ignite.name());
igniteTxMgr.setTransactionConcurrency(OPTIMISTIC);
return igniteTxMgr;
}



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Upgrade from 2.1.0 to 2.4.0 resulting in error within transaction block

[smurphy]

Code works in Ignite 2.1.0. Upgrading to 2.4.0 produces the stack trace
below. The delete statement that is causing the error is:

SqlFieldsQuery sqlQuery = new SqlFieldsQuery("delete from EngineFragment
where " + criteria());
fragmentCache.query(sqlQuery.setArgs(criteria.getArgs()));

The code above is called from within a transactional block managed by a
PlatformTransactionManager which is in turn managed by Spring's
ChainedTransactionManager. If the @Transactional annotation is removed from
the surrounding code, then the code works ok...

2018-03-28 15:50:05,748 WARN  [engine 127.0.0.1] progress_monitor_2 unknown
unknown {ProgressMonitorImpl.java:112} - Scan
[ec7af5e8-a773-40fd-9722-f81103de73dc] is unable to process!
javax.cache.CacheException: Failed to process key '247002'
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:618)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxyImpl.query(IgniteCacheProxyImpl.java:557)
at
org.apache.ignite.internal.processors.cache.GatewayProtectedCacheProxy.query(GatewayProtectedCacheProxy.java:382)
at
com.company.core.dao.ignite.IgniteFragmentDao.delete(IgniteFragmentDao.java:143)
at
com.company.core.dao.ignite.IgniteFragmentDao$$FastClassBySpringCGLIB$$c520aa1b.invoke()
at 
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:136)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
at
com.company.core.dao.ignite.IgniteFragmentDao$$EnhancerBySpringCGLIB$$ce60f71c.delete()
at
com.company.core.core.service.impl.InternalScanServiceImpl.purgeScanFromGrid(InternalScanServiceImpl.java:455)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
at com.sun.proxy.$Proxy417.purgeScanFromGrid(Unknown Source)
at 
com.company.core.core.async.tasks.PurgeTask.process(PurgeTask.java:85)
at sun.reflect.GeneratedMethodAccessor197.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
at com.sun.proxy.$Proxy418.process(Unknown Source)
at
com.company.core.core.async.impl.ProgressMonitorImpl._runTasks(ProgressMonitorImpl.java:128)
at
com.company.core.core.async.impl.ProgressMonitorImpl.lambda$null$0(ProgressMonitorImpl.java:98)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at
com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:108)
at
com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:41)
at
com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:77)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at

Re: SpringTransactionManager and ChainedTransactionManager

[smurphy]

In actuality I rewrote the tests and removed the
AbstractTransactionalJUnit4SpringContextTests super class and the
transactions worked. I also rewrote the the test as an integration test on a
tomcat server and confirmed that the transaction managers worked.

I did see the same javadoc comments that you mentioned and found that the
order was not important. Also, from the following article that is references
in Spring's Transaction documentation:

ChainedTransactionManager which is ‘a crude implementation of a transaction
manager just links together a list of other transaction managers to
implement the transaction synchronization. If the business processing is
successful they all commit, and if not they all roll back.”

Full article: 

https://www.javaworld.com/article/2077963/open-source-tools/distributed-transactions-in-spring--with-and-without-xa.html?page=2



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

SpringTransactionManager and ChainedTransactionManager

[smurphy]

Hello,

I am using SpringTransactionManager & HibernateTransactionManager in a
Spring ChainedTransactionManager in order to rollback all database and cache
updates
in the event of there being any failures in either transaction manager.

I am finding that rollbacks in HibernateTransactionManager do cause
rollbacks in SpringTransactionManager but not vice versa - database changes
are not rolled back when Ignite transactions rollback.

Here are the logs from my test:

2018-02-02 15:09:00,509 DEBUG [unknown 192.168.1.10] main unknown unknown
{AbstractPlatformTransactionManager.java:367} - Creating new transaction
with name [com.dna.chain.service.impl.TestServiceImpl.testeroo]:
PROPAGATION_REQUIRED,ISOLATION_DEFAULT; 'chainedTransactionManager'
2018-02-02 15:09:00,514 DEBUG [unknown 192.168.1.10] main unknown unknown
{AbstractPlatformTransactionManager.java:851} - Initiating transaction
rollback
2018-02-02 15:09:00,514 DEBUG [unknown 192.168.1.10] main unknown unknown
{AbstractPlatformTransactionManager.java:1020} - Resuming suspended
transaction after completion of inner transaction

Here are the configured transaction managers:

@Bean
public ChainedTransactionManager chainedTransactionManager(
PlatformTransactionManager hibernateTxMgr, PlatformTransactionManager
igniteTxMgr) {
ChainedTransactionManager transactionManager = new
ChainedTransactionManager(
new PlatformTransactionManager[] { hibernateTxMgr, igniteTxMgr
});
return transactionManager;
 }

@Bean
public PlatformTransactionManager hibernateTxMgr(SessionFactory
sessionFactory) {
HibernateTransactionManager manager = new
HibernateTransactionManager(sessionFactory);
return manager;
}

@Bean
public PlatformTransactionManager igniteTxMgr(IgniteSpringBean ignite)
throws IOException {
SpringTransactionManager igniteTxMgr = new
SpringTransactionManager();
igniteTxMgr.setGridName(ignite.getConfiguration().getGridName());
return igniteTxMgr;
}

And test code:

@Transactional("chainedTransactionManager")
public void testeroo(Consumer consumer, Params params) {
consumer.accept(params);
}

// Inserts params into databases first and then puts params into cache and
throws exception
public final class Scenario2 implements Consumer {

@Override
public void accept(Params params) {

scheduleDao.saveOrUpdate(schedule);
   jobConfigDao.saveOrUpdate(jobConfig);
   cache_1.put(pair.getFirst(), pair.getSecond());
   throw new RuntimeException("inbound is null")
}
   }






--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Grid freezing

[smurphy]

I figured this out at long last...

The root cause of the problem was the Scan object's toString method:

@Override
public String toString() {
return ReflectionToStringBuilder.reflectionToString(this);
}


It used the apache common-lang's RefelectionToStringBuilder and this jar was
not in the ignite libs folder.
Once in there, transactions worked with multiple server nodes under heavy
loads.
The problem was easy to reproduce - just remove this jar from the libs
folder.

The error message that results is the same as shown above and does not point
to the real underlying cause, so that is what caused the confusion:
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.implicitSingleResult(IgniteTxLocalAdapter.java:352)
 

I am guessing that because this ReflectionToStringBuilder.toString() is only
used in the Scan object's toString() method, it is never needed by Ignite
until at some point an exception is thrown and a logger calls the toString
on the Scan object and a java.lang.NoClassDefFoundError is not
anticipated...

I tried invoking the ReflectionToStringBuilder.toString() more directly from
the dequeuer code itself and I did see the expected error in the logs:
java.lang.NoClassDefFoundError:
org/apache/commons/lang/builder/ReflectionToStringBuilder at...



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Grid freezing

[smurphy]

The WARNING message above is not consistently showing up in the logs. 
Nodes are not going down either. 
What I have found is that running 1 server and 2 clients does NOT freeze,
even under heavy loads.
But having 2 ignite servers does freeze up. 
Two servers are unsuccessful either on separate boxes or on the same box
using localhost.

Looking at the thread dump of one server, 7 threads are waiting at
TransactionProxyImpl.commit (line 259) (Excerpt 1 & 2 below)

The thread dump of the other server has 8 thread waiting at
IgniteCacheProxy.getAllOutTx (line 1328) (Excerpt 3)
The thread number of one of these waiting threads is also waiting in the
CacheAffinitySharedManager  (Excerpt 4)
Even though Excerpt 3 and 4 call getAllOutTx, they are the first place in
the the transaction block where an Ignite cache is called.

*Excerpt 1:*

"pub-#75%dna%" prio=5 tid=96 WAITING
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park()
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:176)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:779)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:103)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:461)
at
org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:283)
   Local Variable: java.lang.NullPointerException#2
at
org.apache.ignite.internal.util.future.GridCompoundFuture.markInitialized(GridCompoundFuture.java:269)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare0(GridDhtTxPrepareFuture.java:1468)
   Local Variable: java.lang.NoClassDefFoundError#1
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.mapIfLocked(GridDhtTxPrepareFuture.java:668)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare(GridDhtTxPrepareFuture.java:1034)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareAsyncLocal(GridNearTxLocal.java:3343)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.prepareColocatedTx(IgniteTxHandler.java:221)
   Local Variable:
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler#1
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepareLocal(GridNearOptimisticSerializableTxPrepareFuture.java:576)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepare(GridNearOptimisticSerializableTxPrepareFuture.java:493)
   Local Variable: org.apache.ignite.internal.util.GridLeanMap#24
   Local Variable:
org.apache.ignite.spi.discovery.tcp.internal.TcpDiscoveryNode#1
   Local Variable:
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxMapping#4
   Local Variable:
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxPrepareRequest#1
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepare(GridNearOptimisticSerializableTxPrepareFuture.java:407)
   Local Variable: java.util.AbstractMap$2#1
   Local Variable:
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture$MiniFuture#2
   Local Variable: java.util.AbstractMap$2#2
   Local Variable: java.util.ArrayList$Itr#1
   Local Variable: java.util.ArrayList#7898
   Local Variable: java.util.HashMap#5544
   Local Variable:
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxMapping#1
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticSerializableTxPrepareFuture.prepare0(GridNearOptimisticSerializableTxPrepareFuture.java:314)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepareOnTopology(GridNearOptimisticTxPrepareFutureAdapter.java:137)
   Local Variable:
org.apache.ignite.internal.processors.affinity.AffinityTopologyVersion#33
   Local Variable:
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture#11
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearOptimisticTxPrepareFutureAdapter.prepare(GridNearOptimisticTxPrepareFutureAdapter.java:74)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.prepareNearTxLocal(GridNearTxLocal.java:3110)
   Local Variable:

Re: Grid freezing

[smurphy]

Here is an additional log message that suggests that rolling back the
transaction is not working.
I do not know why the isolation level in this log is marked as
READ_COMMITTED. All transactions are configured to be Optimistic and
Serializable..:


2017-10-26 10:49:12,524 ERROR [dna-scan-engine 172.29.11.197]
progress_monitor_1   {Log4JLogger.java:495} -  Failed to add
cause to the end of cause chain (cause is printed here but will not be
propagated to callee): class o.a.i.i.transactions.IgniteTx
TimeoutCheckedException: Failed to acquire lock within provided timeout for
transaction [timeout=1, tx=GridNearTxLocal
[mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping
[entries=[IgniteTxEntry [key=KeyCacheObjectImpl [part=311, val=311, hasVal
Bytes=true], cacheId=-211228266, txKey=IgniteTxKey [key=KeyCacheObjectImpl
[part=311, val=311, hasValBytes=true], cacheId=-211228266], val=[op=DELETE,
val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null],
entryProcessorsCol=null, ttl=-1, conflictExpireTi
me=-1, conflictVer=null, explicitVer=null, dhtVer=null, filters=[],
filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry
[super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=311, val=311, hasValBytes=true], val=null,
 startVer=1509032890673, ver=GridCacheVersion [topVer=120512802,
order=1509032890673, nodeOrder=4], hash=311, extras=null, flags=0]]],
prepared=0, locked=false, nodeId=790731d7-c002-436f-ae3e-9a3bd7944581,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flag
s=0, partUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion
[topVer=120512802, order=1509032890671, nodeOrder=4]]], explicitLock=false,
dhtVer=null, last=false, nearEntries=0, clientFirst=false,
node=790731d7-c002-436f-ae3e-9a3bd7944581]], nearLocallyMapped=false,
 colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false,
thread=,
mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping
[entries=[IgniteTxEntry [key=KeyCacheObjectImpl [part=311, val=311,
hasValBytes=
true], cacheId=-211228266, txKey=IgniteTxKey [key=KeyCacheObjectImpl
[part=311, val=311, hasValBytes=true], cacheId=-211228266], val=[op=DELETE,
val=null], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP, val=null],
entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1,
 conflictVer=null, explicitVer=null, dhtVer=null, filters=[],
filtersPassed=false, filtersSet=true, entry=GridDhtDetachedCacheEntry
[super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=311, val=311, hasValBytes=true], val=null,
start
Ver=1509032890673, ver=GridCacheVersion [topVer=120512802,
order=1509032890673, nodeOrder=4], hash=311, extras=null, flags=0]]],
prepared=0, locked=false, nodeId=790731d7-c002-436f-ae3e-9a3bd7944581,
locMapped=false, expiryPlc=null, transferExpiryPlc=false, flags=0, p
artUpdateCntr=0, serReadVer=null, xidVer=GridCacheVersion [topVer=120512802,
order=1509032890671, nodeOrder=4]]], explicitLock=false, dhtVer=null,
last=false, nearEntries=0, clientFirst=false,
node=790731d7-c002-436f-ae3e-9a3bd7944581]], super=GridDhtTxLocalAdapter [n
earOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false,
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true,
recovery=false], super=IgniteTxAdapter [xidVer=Gr
idCacheVersion [topVer=120512802, order=1509032890671, nodeOrder=4],
writeVer=null, implicit=true, loc=true, threadId=208,
startTime=1509032942451, nodeId=f55eb4f4-57cb-4a6e-b1db-5f91b8a0f34e,
startVer=GridCacheVersion [topVer=120512802, order=1509032890671, nodeOrder
=4], endVer=null, isolation=READ_COMMITTED, concurrency=OPTIMISTIC,
timeout=1, sysInvalidate=false, sys=false, plc=2,
commitVer=GridCacheVersion [topVer=120512802, order=1509032890671,
nodeOrder=4], finalizing=NONE, invalidParts=null, state=PREPARING,
timedOut=fal
se, topVer=AffinityTopologyVersion [topVer=4, minorTopVer=8],
duration=10024ms, onePhaseCommit=false], size=1
class org.apache.ignite.IgniteCheckedException: Failed to commit
transaction:
GridNearTxLocal[id=f219e595f51--072e-e122--0004,
concurrency=OPTIMISTIC, isolation=READ_COMMITTED, state=ROLLED_BACK,
invalidate=false, rollbackOnly=true, nodeId=f55eb4f4
-57cb-4a6e-b1db-5f91b8a0f34e, duration=10037]
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxFinishFuture.finish(GridNearTxFinishFuture.java:423)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal.rollbackNearTxLocalAsync(GridNearTxLocal.java:3255)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$5.applyx(GridNearTxLocal.java:1624)
at
org.apache.ignite.internal.processors.cache.distributed.near.GridNearTxLocal$5.applyx(GridNearTxLocal.java:1614)
at

Re: Grid freezing

[smurphy]

One thing that I see in the logs looks to me like a NullPointerException on
attempting to commit an optimistic/serializable transaction.
My reading of this excpetion is that an optimistic lock conflict is
correctly detected in prepare0 but that  eventually the code incorrectly
attempts to return this.ret which is null rather than an
IgniteTxOptimisticCheckedException...:



Oct 25 19:25:46 srvr.company apache-ignite[45092]: [19:25:46] (err) Failed
to execute compound future reducer: GridDhtTxPrepareFuture
[futId=2a3ec065f51-75e18206-0fee-4926-99be-bb6fee7b5828, err=null,
replied=1, mapped=1, reads=[], writes=[IgniteTxEntry [key=KeyCacheObjectImpl
[part=115, val=115, hasValBytes=true], cacheId=-211228266, txKey=IgniteTxKey
[key=KeyCacheObjectImpl [part=115, val=115, hasValBytes=true],
cacheId=-211228266], val=[op=UPDATE,
val=com.company.dna.scan.fragment.node.domain.Scan [idHash=1148364470,
hash=525731140, _fragmentIds=HashSet {}, _scannerGroupId=16,
_visibility=EXTERNAL, _scanUUID=d1a7a992-6ff2-4eb0-8398-06e45b15d08b,
_customerId=6779780, _customerMaxReached=false, _priority=1,
_scanStartedMsgSent=false, _empty=true, _idx=115, _canceled=true,
_scanId=45534, _version=2]], prevVal=[op=NOOP, val=null], oldVal=[op=NOOP,
val=null], entryProcessorsCol=null, ttl=-1, conflictExpireTime=-1,
conflictVer=null, explicitVer=null, dhtVer=null, filters=[],
filtersPassed=false, filtersSet=false, entry=GridDhtCacheEntry [rdrs=[],
part=115, super=GridDistributedCacheEntry [super=GridCacheMapEntry
[key=KeyCacheObjectImpl [part=115, val=115, hasValBytes=true],
val=com.company.dna.scan.fragment.node.domain.Scan [idHash=582900940,
hash=-1150571357, _fragmentIds=HashSet {}, _scannerGroupId=16,
_visibility=EXTERNAL, _scanUUID=d1a7a992-6ff2-4eb0-8398-06e45b15d08b,
_customerId=6779780, _customerMaxReached=false, _priority=1,
_scanStartedMsgSent=true, _empty=true, _idx=115, _canceled=false,
_scanId=45534, _version=3], startVer=1508977480914, ver=GridCacheVersion
[topVer=120361566, order=1508977480909, nodeOrder=33], hash=115,
extras=GridCacheMvccEntryExtras [mvcc=GridCacheMvcc
[locs=[GridCacheMvccCandidate [nodeId=47680ca3-827f-4788-af9e-d73dd7b030e0,
ver=GridCacheVersion [topVer=120361566, order=1508977480913, nodeOrder=33],
threadId=111, id=396, topVer=AffinityTopologyVersion [topVer=39,
minorTopVer=0], reentry=null,
otherNodeId=7a2804c4-5fae-4258-b6e2-96e75a219ea7, otherVer=GridCacheVersion
[topVer=120361566, order=1508977480890, nodeOrder=39], mappedD
Oct 25 19:25:46 srvr.company apache-ignite[45092]: htNodes=null,
mappedNearNodes=null, ownerVer=null, serOrder=GridCacheVersion
[topVer=120361566, order=1508977480890, nodeOrder=39],
key=KeyCacheObjectImpl [part=115, val=115, hasValBytes=true],
masks=local=1|owner=1|ready=1|reentry=0|used=0|tx=1|single_implicit=0|dht_local=1|near_local=0|removed=0|read=0,
prevVer=null, nextVer=null]], rmts=null]], flags=2]]], prepared=1,
locked=false, nodeId=null, locMapped=false, expiryPlc=null,
transferExpiryPlc=false, flags=0, partUpdateCntr=0,
serReadVer=GridCacheVersion [topVer=120361566, order=1508977480161,
nodeOrder=33], xidVer=null]], trackable=true, nearMiniId=2, last=true,
retVal=false, ret=null, lockKeys=[], forceKeysFut=null, locksReady=true,
invoke=false, timeoutObj=PrepareTimeoutObject [timeout=9976],
xid=GridCacheVersion [topVer=120361566, order=1508977480913, nodeOrder=33],
innerFuts=[], super=GridCompoundFuture
[rdc=o.a.i.i.processors.cache.distributed.dht.GridDhtTxPrepareFuture$1@38648d98,
initFlag=1, lsnrCalls=0, done=false, cancelled=false, err=null,
futs=[]]]java.lang.NullPointerException
Oct 25 19:25:46 srvr.company apache-ignite[45092]: at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.implicitSingleResult(IgniteTxLocalAdapter.java:352)
Oct 25 19:25:46 srvr.company apache-ignite[45092]: at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.createPrepareResponse(GridDhtTxPrepareFuture.java:875)
Oct 25 19:25:46 srvr.company apache-ignite[45092]: at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:763)
Oct 25 19:25:46 srvr.company apache-ignite[45092]: at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.onDone(GridDhtTxPrepareFuture.java:103)
Oct 25 19:25:46 srvr.company apache-ignite[45092]: at
org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:450)
Oct 25 19:25:46 srvr.company apache-ignite[45092]: at
org.apache.ignite.internal.util.future.GridCompoundFuture.checkComplete(GridCompoundFuture.java:278)
Oct 25 19:25:46 srvr.company apache-ignite[45092]: at
org.apache.ignite.internal.util.future.GridCompoundFuture.markInitialized(GridCompoundFuture.java:269)
Oct 25 19:25:46 srvr.company apache-ignite[45092]: at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxPrepareFuture.prepare0(GridDhtTxPrepareFuture.java:1468)
Oct 25 19:25:46 srvr.company 

Re: Grid freezing

[smurphy]

log_1.txt
  
log_2.txt
  
log_3.txt
  

I don't see any nodes shutting down until 9 minutes after this particular
log..
But please see the attached are server logs for fuller details..





--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Grid freezing

[smurphy]

No - this is what the invocation looks like:

DequeuePortionsCallable job = new
DequeuePortionsCallable(ignitePortionDequeuer);
DequeuedPortionResponse response = _ignite.compute().call(job);




--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Grid freezing

[smurphy]

I added a transaction timeout of 1 millis and a transation size of 100
(transaction is Optimistic and Serializable)...

I see no TransactionTimeoutExceptions, just the following:
CacheException: class org.apache.ignite.IgniteInterruptedException: Got
interrupted while waiting for future to complete..



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Grid freezing

[smurphy]

Thanks Evgenii,

I'll add that catch block and see if it sheds any light on the issue. 

The client's transaction configuration is set to Optimistic and Serializable
and the transaction within the try with resources block is explicitly set to
Optimistic and Serializable, which should preclude deadlock...but there does
seem to be deadlock..

The servers do not have any transaction configuration set explicitly..



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Grid freezing

[smurphy]

IgnitePortionDequeuer.java

  

top.visor
  

Hi Evgenii,

See the attached top command and java file..



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Grid freezing

[smurphy]

threaddump.tdump
  

I am using Ignite v2.1 and my code using Optimistic/Serializable
transactions and is locking up. When it does, there are a lot of `WARNING:
Found long running transaction` and `WARNING: Found long running cache
future` messages in the logs (see below). Eventually the grid freezes and
there are a lot of "IgniteInterruptedException: Got interrupted while
waiting for future to complete." (see below). Also, see attached thread dump
when the grid is finally stopped.

Using visor, I could see the active threads increase until all are consumed.
I boosted the thread pool count to 100 and saw the active threads rise
steadily until all were consumed. At theat point, the grid freezes up.

Using visor ping multiple times, the servers always ping successfully but
the client fails intermittently. 

LONG RUNNING TRANSACTION:

2017-10-19 13:03:06,698 WARN  [ ] grid-timeout-worker-#15%null%  
{Log4JLogger.java:480} - Found long running transaction
[startTime=12:30:33.985, curTime=13:03:06.682, tx=GridNearTxLocal
[mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping
[entries=[IgniteTxEntry [key=KeyCacheObjectImpl [part=126,
val=GridCacheInternalKeyImpl [name=SCAN_IDX, grpName=default-ds-group],
hasValBytes=true], cacheId=1481046058, txKey=IgniteTxKey
[key=KeyCacheObjectImpl [part=126, val=GridCacheInternalKeyImpl
[name=SCAN_IDX, grpName=default-ds-group], hasValBytes=true],
cacheId=1481046058], val=[op=TRANSFORM, val=null], prevVal=[op=NOOP,
val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=[IgniteBiTuple
[val1=GetAndIncrementProcessor [], val2=[Ljava.lang.Object;@49d9dc6c]],
ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null,
dhtVer=null, filters=[], filtersPassed=false, filtersSet=true,
entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry
[super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=126,
val=GridCacheInternalKeyImpl [name=SCAN_IDX, grpName=default-ds-group],
hasValBytes=true], val=null, startVer=1508434155232, ver=GridCacheVersion
[topVer=119473256, order=1508434155232, nodeOrder=2243], hash=-375255214,
extras=null, flags=0]]], prepared=0, locked=false,
nodeId=36b4d422-d011-4f77-919a-b8ffb089614b, locMapped=false,
expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0,
serReadVer=null, xidVer=GridCacheVersion [topVer=119473256,
order=1508434155230, nodeOrder=2243]]], explicitLock=false, dhtVer=null,
last=false, nearEntries=0, clientFirst=false,
node=36b4d422-d011-4f77-919a-b8ffb089614b]], nearLocallyMapped=false,
colocatedLocallyMapped=false, needCheckBackup=null, hasRemoteLocks=false,
thread=,
mappings=IgniteTxMappingsSingleImpl [mapping=GridDistributedTxMapping
[entries=[IgniteTxEntry [key=KeyCacheObjectImpl [part=126,
val=GridCacheInternalKeyImpl [name=SCAN_IDX, grpName=default-ds-group],
hasValBytes=true], cacheId=1481046058, txKey=IgniteTxKey
[key=KeyCacheObjectImpl [part=126, val=GridCacheInternalKeyImpl
[name=SCAN_IDX, grpName=default-ds-group], hasValBytes=true],
cacheId=1481046058], val=[op=TRANSFORM, val=null], prevVal=[op=NOOP,
val=null], oldVal=[op=NOOP, val=null], entryProcessorsCol=[IgniteBiTuple
[val1=GetAndIncrementProcessor [], val2=[Ljava.lang.Object;@49d9dc6c]],
ttl=-1, conflictExpireTime=-1, conflictVer=null, explicitVer=null,
dhtVer=null, filters=[], filtersPassed=false, filtersSet=true,
entry=GridDhtDetachedCacheEntry [super=GridDistributedCacheEntry
[super=GridCacheMapEntry [key=KeyCacheObjectImpl [part=126,
val=GridCacheInternalKeyImpl [name=SCAN_IDX, grpName=default-ds-group],
hasValBytes=true], val=null, startVer=1508434155232, ver=GridCacheVersion
[topVer=119473256, order=1508434155232, nodeOrder=2243], hash=-375255214,
extras=null, flags=0]]], prepared=0, locked=false,
nodeId=36b4d422-d011-4f77-919a-b8ffb089614b, locMapped=false,
expiryPlc=null, transferExpiryPlc=false, flags=0, partUpdateCntr=0,
serReadVer=null, xidVer=GridCacheVersion [topVer=119473256,
order=1508434155230, nodeOrder=2243]]], explicitLock=false, dhtVer=null,
last=false, nearEntries=0, clientFirst=false,
node=36b4d422-d011-4f77-919a-b8ffb089614b]], super=GridDhtTxLocalAdapter
[nearOnOriginatingNode=false, nearNodes=[], dhtNodes=[], explicitLock=false,
super=IgniteTxLocalAdapter [completedBase=null, sndTransformedVals=false,
depEnabled=false, txState=IgniteTxImplicitSingleStateImpl [init=true,
recovery=false], super=IgniteTxAdapter [xidVer=GridCacheVersion
[topVer=119473256, order=1508434155230, nodeOrder=2243], writeVer=null,
implicit=true, loc=true, threadId=109, startTime=1508434233985,
nodeId=cb0332bd-1c18-4695-a75f-0d0ed4637b55, startVer=GridCacheVersion
[topVer=119473256, order=1508434155230, nodeOrder=2243], endVer=null,
isolation=READ_COMMITTED, concurrency=OPTIMISTIC, timeout=0,
sysInvalidate=false, sys=true, plc=2, commitVer=GridCacheVersion
[topVer=119473256, order=1508434155230, nodeOrder=2243], finalizing=NONE,
invalidParts=null, state=PREPARING,