Hello Jeremy, Sorry for the confusion and verbose message - it has nothing to do with Ignite files or file locking in general - I was just trying to provide some background on what the application is doing and how ignite locks will be used. The only thing (about ignite) I really need to know is how costly creating and destroying reentrant locks is and if destroying them will free up all resources completely and if locks with the same name can be created again later on. For example in hazelcast destroyed distributed locks become unusable but are not removed and can not be recreated so the only way to work with them is preallocate a finite number of locks
Thank you, Alex On Sun, Nov 10, 2024 at 4:52 PM Jeremy McMillan <j...@gridgain.com> wrote: > I don't understand what Ignite does with processing filesystem > directories. This sounds like a question for whoever is supporting your OS. > It is generally a very bad idea to write to files which belong to Ignite. I > suspect you're struggling with OS file locking semantics and concurrent > file access. If this is what you are doing with your Ignite client, you > likely need to operate similar to what you did for Hazelcast, which like > Ignite is downstream of your concurrent file processing and locking. > > > On Sat, Nov 9, 2024 at 8:16 PM Alex Roytman <roytm...@gmail.com> wrote: > >> Hello Everyone, >> >> I am processing files (in a complex way) in a large number of file system >> directories. For the most part, at any given time different directories >> will be touched but the same directory may be written to from different >> threads or nodes (and it may take some time to complete processing) so I >> need to do it under a lock. >> >> I would like to ask if there is a significant cost in creating, using and >> destroying a lock and if resources will be cleanly released. >> In my Hazelcast based implementation I hash all directory paths being >> used to 256 locks because if I create/use/destroy a lock for each >> directory, the locks are never fully deallocated upon destruction and >> fairly quickly eat up all resources. >> >> 256 (configurable) locks seem to provide sufficient concurrency even for >> processing that may take 100 of seconds >> >> Would you expect ignite 3 the same behavior as ignite 2 in this respect >> or it may be different >> >> Would you recommend I should continue using this strategy with Ignite? >> >> Thank you, >> Alex >> >
