Hi, I was going through this post on stackoverflow which is about the same issue. The fact that snapshot works for apache ignite bit not in ultimate edition indicates there is some bug in later. Could you please confirm. We have around 15 caches with 2 backups. I changed backups to zero but still see this issue. Could you please advise further.
https://stackoverflow.com/questions/72041292/is-there-a-fix-for-too-many-open-files-error-in-gridgain-cluster On Mon, Jun 6, 2022 at 9:13 PM Surinder Mehra <redni...@gmail.com> wrote: > Hi, > I was experimenting with the GG ultimate edition to take snapshots and > encountered the below error and cluster stops. Please note that this works > in the ignite free version and we don't see too many files open error. Is > this a bug or we are missing some configuration? > > version: gridgain-8.8.19 > > /bin./snapshot-utility.sh snapshot -type=full > > [21:03:51,693][SEVERE][db-snapshot-executor-stripe-0-#35][] Critical > system error detected. Will be handled accordingly to configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], > failureCtx=FailureContext [type=CRITICAL_ERROR, err=class > o.a.i.i.processors.cache.persistence.StorageException: Failed to initialize > partition file: > /home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name****/part-88.bin] > class > org.apache.ignite.internal.processors.cache.persistence.StorageException: > Failed to initialize partition file: > /home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name****/part-88.bin > at > org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:519) > at > org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:405) > at > org.apache.ignite.internal.processors.cache.persistence.pagemem.PageReadWriteManagerImpl.read(PageReadWriteManagerImpl.java:68) > at > org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:577) > at > org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:911) > at > org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:730) > at > org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:711) > at > org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotCreateFuture.completeSavingAllocatedIndex(SnapshotCreateFuture.java:1304) > at > org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotCreateFuture.completeSnapshotCreation(SnapshotCreateFuture.java:1486) > at > org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotCreateFuture.doFinalStage(SnapshotCreateFuture.java:1171) > at > org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotOperationFuture.completeStagesLocally(SnapshotOperationFuture.java:2352) > at > org.gridgain.grid.internal.processors.cache.database.snapshot.SnapshotOperationFuture$10.run(SnapshotOperationFuture.java:2286) > at > org.apache.ignite.internal.util.StripedExecutor$Stripe.body(StripedExecutor.java:567) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: java.nio.file.FileSystemException: > /home/usr/tools/gridgain-ultimate-8.8.19/work/db/node00-c221fe71-5d29-4cd7-ab0f-9fa8240711b2/cache-name****/part-88.bin: > Too many open files > at > java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) > at > java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) > at > java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) > at > java.base/sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:201) > at > java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:253) > at > java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:311) > at > org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.<init>(AsyncFileIO.java:65) > at > org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:43) > at > org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:491) > ... 14 more > [21:03:51,695][SEVERE][db-snapshot-executor-stripe-0-#35][FailureProcessor] > No deadlocked threads detected. > [21:03:51,767][SEVERE][db-snapshot-executor-stripe-0-#35][FailureProcessor] > Thread dump at 2022/06/06 21:03:51 IST > Thread [name="main", id=1, state=WAITING, blockCnt=4, waitCnt=4169] > Lock [object=java.util.concurrent.CountDownLatch$Sync@5b60e356, > ownerName=null, ownerId=-1] > at java.base@11.0.14.1/jdk.internal.misc.Unsafe.park(Native > Method) > at > java.base@11.0.14.1/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194) > at > java.base@11.0.14.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:885) > at > java.base@11.0.14.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1039) > at > java.base@11.0.14.1/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1345) > at > java.base@11.0.14.1/java.util.concurrent.CountDownLatch.await(CountDownLatch.java:232) > at > app//o.a.i.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:391) > > > Config: > > <bean class="org.apache.ignite.configuration.IgniteConfiguration"> > > <property name="peerClassLoadingEnabled" value="true"/> > <property name="deploymentMode" value="CONTINUOUS"/> > <property name="dataStorageConfiguration"> > <bean > class="org.apache.ignite.configuration.DataStorageConfiguration"> > <property name="defaultDataRegionConfiguration"> > <bean > class="org.apache.ignite.configuration.DataRegionConfiguration"> > <property name="persistenceEnabled" value="true"/> > </bean> > </property> > </bean> > </property> > <property name="pluginConfigurations"> > <bean class="org.gridgain.grid.configuration.GridGainConfiguration"> > <property name="snapshotConfiguration"> > <bean > class="org.gridgain.grid.configuration.SnapshotConfiguration"> > <property name="snapshotsPath" value="/home/ignitesnapshots/"/> > </bean> > </property> > </bean> > </property> > </bean> > </beans> >
