Thank you Zhenya, it seems that the problem was caused by the fact that centos deleted the whole ignite work directory as it was unaccessed for 10 days. Ignite probably creates the workdir only when starting, but the application did not restart during 2 weeks. It seems that Ignite failed to serialize class because it assumed that the marshaller folder still exists.
The solution is to never use /tmp for ignite workdir (or disable tmpfiles.d). On Wed, 2021-03-31 at 09:01 +0300, Zhenya Stanilovsky wrote: what output of: stat /tmp/ignite-workspace/marshaller/-961185899.classname0 ? If it`s all ok here, possibly some privileges issues ? Hello, we are using CentOS Linux release 7.8.2003 (Core), which deletes files from /tmp no sooner than 10 days after accessed, so this should not be an issue. On Tue, 2021-03-30 at 18:40 +0300, Zhenya Stanilovsky wrote: hi, https://askubuntu.com/questions/20783/how-is-the-tmp-directory-cleaned-up ? Hi, We use a cluster consisting of 2 Ignite nodes. Deployment is done without downtime or cluster reset. When attempted to deploy new version of an application contaning new caches on node 1, Ignite fails to write the class file on node 2. Peer class loading is enabled. The directory does exist and already contains other serialized classes and has proper permissions. Somehow Ignite fails to open a file it has just created, with FileNotFoundException: No such file or directory Do you have any suggestions what could be the cause? Thank you, Matej [ERROR] [29.03.2021 14:34:09.686] [] [68.166.6:47500]-#2] [i.i.MarshallerMappingFileStore]: Failed to write class name to file [platformId=0id=-961185899, clsName=org.profile.AbTest, file=/tmp/ignite-workspace/marshaller/-961185899.classname0] java.io.FileNotFoundException: /tmp/ignite-workspace/marshaller/-961185899.classname0 (No such file or directory) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187) at org.apache.ignite.internal.MarshallerMappingFileStore.writeMapping(MarshallerMappingFileStore.java:97) at org.apache.ignite.internal.MarshallerMappingFileStore.mergeAndWriteMapping(MarshallerMappingFileStore.java:222) at org.apache.ignite.internal.MarshallerContextImpl.onMappingDataReceived(MarshallerContextImpl.java:191) at org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.processIncomingMappings(GridMarshallerMappingProcessor.java:356) at org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.onJoiningNodeDataReceived(GridMarshallerMappingProcessor.java:336) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onExchange(GridDiscoveryManager.java:906) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:2090) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:4816) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:3089) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2795) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7766) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [ERROR] [29.03.2021 14:34:09.687] [] [68.166.6:47500]-#2] [i.i.MarshallerMappingFileStore]: Failed to write class name to file [platformId=0id=-1710898632, clsName=org.profile.GridCard, file=/tmp/ignite-customer2/marshaller/-1710898632.classname0] java.io.FileNotFoundException: /tmp/ignite-customer2/marshaller/-1710898632.classname0 (No such file or directory) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237) at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187) at org.apache.ignite.internal.MarshallerMappingFileStore.writeMapping(MarshallerMappingFileStore.java:97) at org.apache.ignite.internal.MarshallerMappingFileStore.mergeAndWriteMapping(MarshallerMappingFileStore.java:222) at org.apache.ignite.internal.MarshallerContextImpl.onMappingDataReceived(MarshallerContextImpl.java:191) at org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.processIncomingMappings(GridMarshallerMappingProcessor.java:356) at org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.onJoiningNodeDataReceived(GridMarshallerMappingProcessor.java:336) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onExchange(GridDiscoveryManager.java:906) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:2090) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:4816) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:3089) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2795) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7766) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [ INFO] [29.03.2021 14:34:09.687] [] [68.166.6:47500]-#2] [CacheObjectBinaryProcessorImpl]: New BinaryMetadata[typeId=-961185899, typeName=org.profile.AbTest] is received from node 70033db6-99a4-4dcf-b546-1206d3516b17; adding it locally [ INFO] [29.03.2021 14:34:09.687] [] [68.166.6:47500]-#2] [CacheObjectBinaryProcessorImpl]: New BinaryMetadata[typeId=-1710898632, typeName=org.profile.GridCard] is received from node 70033db6-99a4-4dcf-b546-1206d3516b17; adding it locally [ INFO] [29.03.2021 14:34:09.689] [] [68.166.6:47500]-#2] [ o.a.i.s.d.t.TcpDiscoverySpi]: New next node [newNext=TcpDiscoveryNode [id=70033db6-99a4-4dcf-b546-1206d3516b17, consistentId=127.0.0.1,192.168.166.163:47500, addrs=ArrayList [127.0.0.1, 192.168.165.164], sockAddrs=HashSet [/127.0.0.1:47500, prod-app27.dmn/192.168.165.164:47500], discPort=47500, order=0, intOrder=18, lastExchangeTime=1617021249663, loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=false]] [ INFO] [29.03.2021 14:34:09.738] [] [otifier-worker-#41] [.p.c.GridClusterStateProcessor]: Received activate request with BaselineTopology[id=0] [ INFO] [29.03.2021 14:34:09.738] [] [otifier-worker-#41] [.p.c.GridClusterStateProcessor]: Started state transition: true [ INFO] [29.03.2021 14:34:09.738] [] [otifier-worker-#41] [.p.c.GridClusterStateProcessor]: Received state change finish message: true [ INFO] [29.03.2021 14:34:09.738] [] [o-event-worker-#42] [a.i.i.m.d.GridDiscoveryManager]: Added new node to topology: TcpDiscoveryNode [id=70033db6-99a4-4dcf-b546-1206d3516b17, consistentId=127.0.0.1,192.168.165.164:47500, addrs=ArrayList [127.0.0.1, 192.168.165.164], sockAddrs=HashSet [/127.0.0.1:47500, prod-app27.dmn.sk/192.168.165.164:47500], discPort=47500, order=33, intOrder=18, lastExchangeTime=1617021249663, loc=false, ver=2.8.1#20200521-sha1:86422096, isClient=false] [ INFO] [29.03.2021 14:34:09.738] [] [o-event-worker-#42] [a.i.i.m.d.GridDiscoveryManager]: Topology snapshot [ver=33, locNode=344660b0, servers=3, clients=0, state=ACTIVE, CPUs=16, offheap=11.0GB, heap=5.5GB] [ INFO] [29.03.2021 14:34:09.738] [] [o-event-worker-#42] [a.i.i.m.d.GridDiscoveryManager]: ^-- Baseline [id=0, size=3, online=3, offline=0] [ INFO] [29.03.2021 14:34:09.739] [] [xchange-worker-#43] [ o.a.i.i.e.time]: Started exchange init [topVer=AffinityTopologyVersion [topVer=33, minorTopVer=0], crd=false, evt=NODE_JOINED, evtNode=70033db6-99a4-4dcf-b546-1206d3516b17, customEvt=null, allowMerge=false, exchangeFreeSwitch=false]
