We have a 3 node cluster with persistence enabled, sometimes I see below error
happening.
Ignite Server log
2021-07-19 21:12:13,650 INFO [http-nio-9050-exec-1] o.a.c.h.Http11Processor
[DirectJDKLog.java:175] Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at
DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in method name
[0x160x030x030x010xb40x010x000x010xb00x030x030x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x000x01p0x000x010x000x020x000x030x000x040x000x050x000x060x000x070x000x080x000x090x000x0a0x000x0b0x000x0c0x000x0d0x000x0e0x000x0f0x000x100x000x110x000x120x000x130x000x140x000x150x000x160x000x170x000x180x000x190x000x1a0x000x1b0x00/0x0000x0010x0020x0030x0040x0050x0060x0070x0080x0090x00:0x00;0x00<0x00=0x00>0x00?0x00@0x00g0x00h0x00i0x00j0x00k0x00l0x00m0x00A0x00B0x00C0x00D0x00E0x00F0x000x840x000x850x000x860x000x870x000x880x000x890x000xba0x000xbb0x000xbc0x000xbd0x000xbe0x000xbf0x000xc00x000xc10x000xc20x000xc30x000xc40x000xc50x000x9c0x000x9d0x000x9e0x000x9f0x000xa00x000xa10x000xa20x000xa30x000xa40x000xa50x000xa60x000xa70xc00x010xc00x020xc00x030xc00x040xc00x050xc00x060xc00x070xc00x080xc00x090xc00x0a0xc00x0b0xc00x0c0xc00x0d0xc00x0e0xc00x0f0xc00x100xc00x110xc00x120xc00x130xc00x140xc00x150xc00x160xc00x170xc00x180xc00x190xc0#0xc0$0xc0%0xc0&0xc0'0xc0(0xc0)0xc0*0xc0+0xc0,0xc0-0xc0.0xc0/0xc010xc000xc020xc0s0xc0r0xc0t0xc0u0xc0v0xc0w0xc0x0xc0z0xc0y0xc0{0xc0|0xc0}0xc0~0xc00x7f0xc00x800xc00x810xc00x820xc00x830xc00x840xc00x850xc00x860xc00x870xc00x880xc00x890xc00x8a0xc00x8b0xc00x8c0xc00x8d0xc00x8e0xc00x8f0xc00x900xc00x910xc00x920xc00x930xc00x940xc00x950xc00x960xc00x970xc00x980xc00x990xc00x9a0xc00x9b0x000x960x000x970x000x980x000x990x000x9a0x000x9b0xcc0xa80xcc0xa90xcc0xaa0xcc0xab0xcc0xac0xcc0xad0xcc0xae0x020x000x010x000x160x000x0a0x000x0a0x000x080x000x170x000x190x000x180x000x160x000x0b0x000x040x030x000x01...].
HTTP method names must be tokens
at
org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:418)
at
org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260)
at
org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
at
org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868)
at
org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1590)
at
org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
2021-07-19 21:12:14,114 WARN [grid-nio-worker-tcp-rest-0-#38%blade-cache%]
o.a.i.i.p.r.p.t.GridTcpRestProtocol [JavaLogger.java:295] Client disconnected
abruptly due to network connection loss or because the connection was left open
on application shutdown. [cls=class o.a.i.i.util.nio.GridNioException,
msg=Failed to parse incoming packet (invalid packet start)
[ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=441 cap=8192],
super=AbstractNioClientWorker [idx=0, bytesRcvd=0, bytesSent=0, bytesRcvd0=0,
bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-rest-0,
igniteInstanceName=blade-cache, finished=false, heartbeatTs=1626743534110,
hashCode=507494530, interrupted=false,
runner=grid-nio-worker-tcp-rest-0-#38%blade-cache%]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, closeSocket=true,
outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl
[locAddr=/10.148.213.242:11211, rmtAddr=/10.148.213.242:54330,
createTime=1626743534110, closeTime=0, bytesSent=0, bytesRcvd=441,
bytesSent0=0, bytesRcvd0=441, sndSchedTime=1626743534110,
lastSndTime=1626743534110, lastRcvTime=1626743534110, readsPaused=false,
filterChain=FilterChain[filters=[GridNioCodecFilter [parser=GridTcpRestParser
[marsh=JdkMarshaller [clsFilter=o.a.i.marshaller.MarshallerUtils$1@dafac83],
routerClient=false], directMode=false]], accepted=true, markedForClose=false]],
b=16]]
2021-07-19 21:12:14,117 WARN [grid-nio-worker-tcp-rest-1-#39%blade-cache%]
o.a.i.i.p.r.p.t.GridTcpRestProtocol [JavaLogger.java:295] Client disconnected
abruptly due to network connection loss or because the connection was left open
on application shutdown. [cls=class o.a.i.i.util.nio.GridNioException,
msg=Failed to parse incoming packet (invalid packet start)
[ses=GridSelectorNioSessionImpl [worker=ByteBufferNioClientWorker
[readBuf=java.nio.HeapByteBuffer[pos=0 lim=441 cap=8192],
super=AbstractNioClientWorker [idx=1, bytesRcvd=0, bytesSent=0, bytesRcvd0=0,
bytesSent0=0, select=true, super=GridWorker [name=grid-nio-worker-tcp-rest-1,
igniteInstanceName=blade-cache, finished=false, heartbeatTs=1626743534110,
hashCode=1070416289, interrupted=false,
runner=grid-nio-worker-tcp-rest-1-#39%blade-cache%]]], writeBuf=null,
readBuf=null, inRecovery=null, outRecovery=null, closeSocket=true,
outboundMessagesQueueSizeMetric=null, super=GridNioSessionImpl
[locAddr=/10.148.213.242:11211, rmtAddr=/10.148.213.242:54332,
createTime=1626743534110, closeTime=0, bytesSent=0, bytesRcvd=441,
bytesSent0=0, bytesRcvd0=441, sndSchedTime=1626743534110,
lastSndTime=1626743534110, lastRcvTime=1626743534110, readsPaused=false, filte
Client log
Caused by:
org.apache.ignite.internal.processors.cache.CacheInvalidStateException: Failed
to execute the cache operation (all partition owners have left the grid,
partition data has been lost) [cacheName=USER_IGNITE_CACHE_PRIMARY,
partition=79, key=UserKeyCacheObjectImpl [part=79,
val=429E98A8-F594-4D1B-BE68-23BA6E027E25, hasValBytes=false]]
During this time , I verified the topology and saw all nodes to be up and
running fine. I had to restart all the nodes to recover from this error.
Cache Configuration
public CacheConfiguration<String, User> getPrimaryUserCacheConfiguration() {
final CacheConfiguration<String, User> cacheConfiguration = new
CacheConfiguration<>(CacheIdentifiers.USER_IGNITE_CACHE_PRIMARY.toString());
cacheConfiguration.setName(CacheIdentifiers.USER_IGNITE_CACHE_PRIMARY.toString());
cacheConfiguration.setIndexedTypes(String.class, User.class);
cacheConfiguration.setCacheMode(CacheMode.PARTITIONED);
cacheConfiguration.setStoreKeepBinary(true);
RendezvousAffinityFunction rendezvousAffinityFunction = new
RendezvousAffinityFunction();
rendezvousAffinityFunction.setPartitions(512);
cacheConfiguration.setBackups(1);
cacheConfiguration.setAffinity(rendezvousAffinityFunction);
//cacheConfiguration.setOnheapCacheEnabled(true);
return cacheConfiguration;
}
Could someone please help me understand why this error would occur and how to
recover from this ?
Also I have configured health check to identify this situation but I was not
able to determine with the below check. Am I doing something wrong?
private Collection<Integer> getForLostPartitions(){
Collection<Integer> lostPartitions =
ignite.cache(CacheIdentifiers.USER_IGNITE_CACHE_PRIMARY.toString()).lostPartitions();
return lostPartitions;
}
if(checkForLostPartitions()){
Health.down().withDetail("Lost
Partitions",getForLostPartitions().size()).withDetail("Partitions Lost ",
getForLostPartitions().toArray()).build();
}
Any help would be appreciated.
Thanks
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________
This message is for information purposes only, it is not a recommendation,
advice, offer or solicitation to buy or sell a product or service nor an
official confirmation of any transaction. It is directed at persons who are
professionals and is not intended for retail customer use. Intended for
recipient only. This message is subject to the terms at:
www.barclays.com/emaildisclaimer.
For important disclosures, please see:
www.barclays.com/salesandtradingdisclaimer regarding market commentary from
Barclays Sales and/or Trading, who are active market participants;
https://www.investmentbank.barclays.com/disclosures/barclays-global-markets-disclosures.html
regarding our standard terms for the Investment Bank of Barclays where we
trade with you in principal-to-principal wholesale markets transactions; and in
respect of Barclays Research, including disclosures relating to specific
issuers, please see http://publicresearch.barclays.com.
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________
If you are incorporated or operating in Australia, please see
https://www.home.barclays/disclosures/importantapacdisclosures.html for
important disclosure.
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________
How we use personal information see our privacy notice
https://www.investmentbank.barclays.com/disclosures/personalinformationuse.html
_________________________________________________________________________________________________________________________________________________________________________________________________________________________________
