Re: Failed to accept TCP connection. java.net.SocketException: Too many open files
Hi, I think that most of the open file descriptors relate to partition files. Perhaps, in terms of performance, this is not a very good way to open/close files after each read/write. Thanks. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to accept TCP connection. java.net.SocketException: Too many open files
Hello, It looks like there is a new possibility which allows decreasing the number of open file handles. As of v2.2, Apache ignite provides a new concept - Cache Groups [1]. Caches within a single cache group share various internal structures and it allows to mix data of caches in shared partition files. So, this feature should decrease the number of open file handles. [1] https://apacheignite.readme.io/v2.3/docs/cache-groups Thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to accept TCP connection. java.net.SocketException: Too many open files
Hi Slava, As per the problem statement defined by Ashish, Whenever a client finishes off writing, shouldn't the handles be closed and therefore the ever increasing number of handles will not be a problem otherwise if the handles of an earlier client operation are not disposed, the resources are still being consumed ? Is it something which has been taken care of post 2.1 ? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to accept TCP connection. java.net.SocketException: Too many open files
Thanks for your help. I will look into pointers suggested by you. Thanks, -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to accept TCP connection. java.net.SocketException: Too many open files
Hi Ashish, > is there a way I can control the number of open file handle in Ignite ? > Or is there a way to control the bin file size ?* If I am not mistaken, there is no way to control the bin file size or number of open file handles. The one thing that can be changed is a number of partitions for your cache. Please see [1] [2] The default values are the following: 1024 - partitioned cache, 512 - replicated cache. I don't think that big number of open file handles is a good reason for reducing the number of partitions. Anyway, the main rule is that a number of partitions should be bigger than a number of the nodes in a cluster in order to get a uniform number of partitions' assignments across all the cluster nodes. > If I change the cache mode from Replicated to Partitioned. Will it help ? In case of 2 nodes, I think it won't help. [1] https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/cache/affinity/rendezvous/RendezvousAffinityFunction.html [2] https://ignite.apache.org/releases/latest/javadoc/org/apache/ignite/configuration/CacheConfiguration.html Thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to accept TCP connection. java.net.SocketException: Too many open files
Hello, As I mentioned above, Ignite Native Persistence creates a dedicated file on the file system for every partition that this node is either a primary or backup. Well, I guess that you are running two Ignite nodes on the same physical machine. The replicated cache has 512 partitions by default. In that case, number of file handles is approximately equal - 2 (nodes) * 30 (caches) * 512 (partitions) = 30720 Thanks! -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Failed to accept TCP connection. java.net.SocketException: Too many open files
Hi, I am still getting "Too many open files" issue even after increasing the limit to ~32K. Following is my ignite configuration: I am running 2 nodes in cluster configured with REPLICATION mode and FULL_SYNC. I have populated ingnite with around ~30 different caches (Requirement to have close to ~200 caches at any point of time). Each cache has around 1.2 million entries. When each cache is created, it increases the file handle by around 1050. I could see that for each cache 500+ bin files are getting created IgnitePersistentStore directory. My system runs out of file handle after ~30 caches have been created in Ignite. Ignite cluster does not seem to release file handle even after client which is populating cache is killed. Reason for this could be as Ignite cluster will keep file handle open in case it need to read data again from the cache. I have couple of questions : 1. is there a way I can control the number of open file handle in Ignite ? Either by increasing the bin file size or with 2. If I change the cache mode from Replication to partition, will it help (I assume so as each node will be having only partial data ) ? Below is the exception trace I am getting : a.util.OptionalDouble [idHash=1820269203, hash=-668113279, isPresent=true, value=0.03775351299133245]] class org.apache.ignite.IgniteCheckedException: Can't open file: part-280.bin at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.init(FilePageStore.java:397) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStore.read(FilePageStore.java:291) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:288) at org.apache.ignite.internal.processors.cache.persistence.file.FilePageStoreManager.read(FilePageStoreManager.java:273) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:569) at org.apache.ignite.internal.processors.cache.persistence.pagemem.PageMemoryImpl.acquirePage(PageMemoryImpl.java:487) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.getOrAllocatePartitionMetas(GridCacheOffheapManager.java:1014) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.init0(GridCacheOffheapManager.java:881) at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.invoke(GridCacheOffheapManager.java:1231) at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:352) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3179) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.initialValue(GridCacheMapEntry.java:2556) at org.apache.ignite.internal.processors.datastreamer.DataStreamerImpl$IsolatedUpdater.receive(DataStreamerImpl.java:1999) at org.apache.ignite.internal.processors.datastreamer.DataStreamerUpdateJob.call(DataStreamerUpdateJob.java:140) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.localUpdate(DataStreamProcessor.java:383) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.processRequest(DataStreamProcessor.java:302) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor.access$000(DataStreamProcessor.java:59) at org.apache.ignite.internal.processors.datastreamer.DataStreamProcessor$1.onMessage(DataStreamProcessor.java:89) at org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1556) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:1184) at org.apache.ignite.internal.managers.communication.GridIoManager.access$4200(GridIoManager.java:126) at org.apache.ignite.internal.managers.communication.GridIoManager$9.run(GridIoManager.java:1097) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) Caused by: java.io.FileNotFoundException: /opt/Ignite/IgnitePersistentStore/10_63_192_28_10_63_193_37_10_63_194_55_127_0_0_1_47500/cache-1324-LOCKED-2/part-280.bin (Too many open files) at java.io.RandomAccessFile.open0(Native Method) at java.io.RandomAccessFile.open(RandomAccessFile.java:316) at java.io.RandomAccessFile.(RandomAccessFile.java:243) at org.apache.ignite.internal.processors.cache.persistence.file.RandomAccessFileIOFactory.create(RandomAccessFileIOFactory.java:38) at
Re: Failed to accept TCP connection. java.net.SocketException: Too many open files
Hi, Ignite Persistent store requires creating a number of files like a WAL (write-ahead log), page store (which is implemented as file per partition) etc. So, you need to increase the open files limit. In order to do that you can edit limits.conf (nofile - max number of open files) or you can use 'ulimit' command. Please see the details https://apacheignite.readme.io/v2.1/docs/jvm-and-system-tuning#file-descriptors Thanks! 2017-09-05 9:18 GMT+03:00 userx <gagan...@gmail.com>: > Hi all, > > I am not sure about the steps how the following error happened but can > someone help explain what the logs have to say. > > 2017-09-05 04:45:52,731 WARN > [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl > - Failed to save metadata for typeId: -1689269445; exceptio > n was thrown: > /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_ > 63_170_29_127_0_0_1_47500/-1689269445.bin > (Too many open files) > 2017-09-05 04:45:52,864 WARN > [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl > - Failed to save metadata for typeId: -1689269445; exceptio > n was thrown: > /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_ > 63_170_29_127_0_0_1_47500/-1689269445.bin > (Too many open files) > 2017-09-05 04:45:52,943 WARN > [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl > - Failed to save metadata for typeId: 1537855308; exception > was thrown: > /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_ > 63_170_29_127_0_0_1_47500/1537855308.bin > (Too many open files) > 2017-09-05 04:45:52,972 WARN > [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl > - Failed to save metadata for typeId: 1537855308; exception > was thrown: > /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_ > 63_170_29_127_0_0_1_47500/1537855308.bin > (Too many open files) > 2017-09-05 04:45:52,979 WARN > [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl > - Failed to save metadata for typeId: -741311308; exception > was thrown: > /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_ > 63_170_29_127_0_0_1_47500/-741311308.bin > (Too many open files) > 2017-09-05 04:45:52,985 WARN > [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl > - Failed to save metadata for typeId: -741311308; exception > was thrown: > /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_ > 63_170_29_127_0_0_1_47500/-741311308.bin > (Too many open files) > 2017-09-05 04:45:52,991 WARN > [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl > - Failed to save metadata for typeId: 1055444204; exception > was thrown: > /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_ > 63_170_29_127_0_0_1_47500/1055444204.bin > (Too many open files) > 2017-09-05 04:45:52,996 WARN > [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl > - Failed to save metadata for typeId: 1055444204; exception > was thrown: > /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_ > 63_170_29_127_0_0_1_47500/1055444204.bin > (Too many open files) > 2017-09-05 04:45:53,011 WARN > [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl > - Failed to save metadata for typeId: 674065617; exception > was thrown: > /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_ > 63_170_29_127_0_0_1_47500/674065617.bin > (Too many open files) > 2017-09-05 04:45:53,015 WARN > [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apache.ignite.internal.processors.cache.binary. > CacheObjectBinaryProcessorImpl > - Failed to save metadata for typeId: 674065617; exception > was thrown: > /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_ > 63_170_29_127_0_0_1_47500/674065617.bin > (Too many open files) > 2017-09-05 04:45:53,861 ERROR > [tcp-disco-srvr-#2%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} > org.apa
Re: Failed to accept TCP connection. java.net.SocketException: Too many open files
Hi all, Looking at https://apacheignite.readme.io/v2.1/docs/jvm-and-system-tuning it looks like the recommended setting for ulimit is 32768. Pardon my limited knowledge, but can some one please tell me why this large number is required ? Current settings have 1024 as the limit but before I increase the same, I need to understand the logic behind the same. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Failed to accept TCP connection. java.net.SocketException: Too many open files
Hi all, I am not sure about the steps how the following error happened but can someone help explain what the logs have to say. 2017-09-05 04:45:52,731 WARN [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl - Failed to save metadata for typeId: -1689269445; exceptio n was thrown: /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_63_170_29_127_0_0_1_47500/-1689269445.bin (Too many open files) 2017-09-05 04:45:52,864 WARN [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl - Failed to save metadata for typeId: -1689269445; exceptio n was thrown: /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_63_170_29_127_0_0_1_47500/-1689269445.bin (Too many open files) 2017-09-05 04:45:52,943 WARN [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl - Failed to save metadata for typeId: 1537855308; exception was thrown: /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_63_170_29_127_0_0_1_47500/1537855308.bin (Too many open files) 2017-09-05 04:45:52,972 WARN [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl - Failed to save metadata for typeId: 1537855308; exception was thrown: /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_63_170_29_127_0_0_1_47500/1537855308.bin (Too many open files) 2017-09-05 04:45:52,979 WARN [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl - Failed to save metadata for typeId: -741311308; exception was thrown: /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_63_170_29_127_0_0_1_47500/-741311308.bin (Too many open files) 2017-09-05 04:45:52,985 WARN [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl - Failed to save metadata for typeId: -741311308; exception was thrown: /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_63_170_29_127_0_0_1_47500/-741311308.bin (Too many open files) 2017-09-05 04:45:52,991 WARN [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl - Failed to save metadata for typeId: 1055444204; exception was thrown: /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_63_170_29_127_0_0_1_47500/1055444204.bin (Too many open files) 2017-09-05 04:45:52,996 WARN [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl - Failed to save metadata for typeId: 1055444204; exception was thrown: /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_63_170_29_127_0_0_1_47500/1055444204.bin (Too many open files) 2017-09-05 04:45:53,011 WARN [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl - Failed to save metadata for typeId: 674065617; exception was thrown: /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_63_170_29_127_0_0_1_47500/674065617.bin (Too many open files) 2017-09-05 04:45:53,015 WARN [tcp-disco-msg-worker-#3%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.internal.processors.cache.binary.CacheObjectBinaryProcessorImpl - Failed to save metadata for typeId: 674065617; exception was thrown: /temp/Ignite/Work/binary_meta/10_63_142_35_10_63_155_178_10_63_170_29_127_0_0_1_47500/674065617.bin (Too many open files) 2017-09-05 04:45:53,861 ERROR [tcp-disco-srvr-#2%e89cfda3-beb6-4eca-ada2-fb4c3b2eebbc%] {} org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi - Failed to accept TCP connection. java.net.SocketException: Too many open files (Accept faile d) at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409) at java.net.ServerSocket.implAccept(ServerSocket.java:545) at java.net.ServerSocket.accept(ServerSocket.java:513) at org.apache.ignite.spi.discovery.tcp.ServerImpl$TcpServer.body(ServerImpl.java:5592) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/