Well, as you are finding, this "simple" approach is causing you problems. The only thing I can add is to just be cautious about relying on side-effects or presumptions about these multiple readers. Because they could change at any time. I've never considered such an approach because I have always thought of a File Stream / Reader and the actual file as a 1:1 relationship. Anything else (except for 100% read only) sounds dodgy. You mention you sometimes going into append mode. But this would invalidate all the other read locks.
I agree that caches are not magic - they only work when on average, access tends to cause clumps of common data. If you were to pull apart the guts of database i wonder if you would find multiple file readers. My guess is no. On Dec 26, 1:07 am, Eitan <[email protected]> wrote: > Because it's a simple feature that I need and I don't want an overkill > solution that I do not control its thread count, its memory > consumption and cpu usage. > It is just a simple component inside my highly performance server that > needs to keep a single list of redo log in the disk if it takes to > much memory which will be very light and won't impact the server > performance. This occurs when the backup server it replicates to is > temporarily down and the redo log starts to accumulate. There are many > consideration which I can't really explain, but I do not wish to use > any embedded or remote third party overkill product for this very > simplified capabilities, like I've described above, unlimited single > list of elements that can only be added at the end of the list, > removed from the first and iterated over the list in read only mode. > > Any other generic product addresses a much more complex scenario and > probably will perform worse because of that. > > I already implemented it and it works fine, but I am not sure about > the multi reader over a file regarding the visibility issue and I was > hoping that someone knows what kind of guarantee is there for the > visibility of multiple RandomAccessFile inside the same process over > the same file. (only one in rw mode and the rest in r mode) > > Thanks > > On Dec 25, 12:41 pm, Viktor Klang <[email protected]> wrote: > > > > > Why a file mate? > > Why not Redis, Cassandra, MongoDB, SQLite whatnot? > > > On Fri, Dec 25, 2009 at 10:20 AM, Eitan <[email protected]> wrote: > > > What about what I said above, using multiple FileChannel/ > > > RandomAccessFile to read from the same file (not writing, there's a > > > single writer multi reader mutual exclusion protection over these > > > FileChannel/RandomAccessFile access), all I am worried about is the > > > visibility that updates the writer will do will be visible to the > > > readers once done, FileChannel guarantee this and I suspect it must > > > rely on this kind of guarantee from the RandomAccessFile that creates > > > it though it is not documented, But according to the simple analysis I > > > of the RandomAccessFile getChannel method I did, it seems that it does > > > need to guarantee it in order for FileChannel to guarantee it. > > > > Since my implementation here is just an "unlimited" single list of > > > objects that starts in the memory and once there's no room continue in > > > the disk, I went for this solution and not using an over kill embedded > > > data base. I only need to support appending objects at the end of the > > > list, removing from the start and iterating over the list in read only > > > mode. (Either from the start to the end or from the end to the start). > > > > Regarding the caching you suggested, I created caching mechanism and > > > there are a few layers of abstraction over this file, but, I am > > > talking here about the lowest level of the abstraction which access > > > the file and that might occur concurrently and I want to remove the > > > contention there as well. (for instance 10 threads currently iterating > > > over the list all at a different location) besides the whole issue > > > here is to move the data from the memory to the disk because I am out > > > of memory space, so caching the 10g file is not an option, I only > > > cache the end and the start of the list since they are accesses most > > > frequently. > > > The time spending ratio here is not the problem, there can be tens of > > > threads concurrently iterating over the list and I want them to be > > > able to iterate concurrently and not wait for each other. According to > > > amdal's law even if the time spent on reading is just 10%, I can only > > > accelerate this process by at most 10 so 100 threads iterating will > > > work no faster than 10. > > > > Basically if I get the visibility guarantee then my multi > > > RandomAccessFile in read mode over the file is good enough solution, > > > let the disk and the OS do the caching of accessing the same file > > > sections and keep my program more concurrent. > > > > Thanks > > > > On Dec 24, 11:45 pm, Christian Catchpole <[email protected]> > > > wrote: > > > > Is not the simplest solution to allow each thread to get a reference > > > > to the *single* RandomAccessFile instance and simply synchronize on it > > > > while reading (not forgetting to reset the offset into the file). > > > > It's not going to be super performant, but how is it ever going to be > > > > with multiple threads thrashing on one file? If anything it might > > > > reduce thrashing as you are controlling the points of file > > > > contention. What is the ratio of time spent reading the file to time > > > > spent operating on the data? > > > > > If all these threads will in-fact thrash on the file it might only be > > > > solved with: > > > > > 1. Better scheduling than "free for all" threads. > > > > 2. a large OS level disk cache > > > > 3. an in memory disk cache > > > > 4. some kind of abstract layer over the file which does caching and > > > > returns you "model objects" rather than being seen as a byte stream. > > > > > On Dec 25, 1:38 am, Eitan <[email protected]> wrote: > > > > > > Looking at FileChannel it has the following part in its documentation: > > > > > > The view of a file provided by an instance of this class is guaranteed > > > > > to be consistent with other views of the same file provided by other > > > > > instances in the same program. The view provided by an instance of > > > > > this > > > > > class may or may not, however, be consistent with the views seen by > > > > > other > > > > > concurrently-running programs due to caching performed by the > > > > > underlying > > > > > operating system and delays induced by network-filesystem protocols. > > > > > This > > > > > is true regardless of the language in which these other programs are > > > > > written, and whether they are running on the same machine or on some > > > > > other > > > > > machine. The exact nature of any such inconsistencies are system- > > > > > dependent > > > > > and are therefore unspecified. > > > > > > This looks exactly like the guarantee I need, now I've looked into > > > > > RandomAccessFile.getChannel() implementation and it holds a single > > > > > member of FileChannel which is initialized on demand and the next > > > > > calls will return the same instance. > > > > > > So the only way I see to create multiple FileChannel instances over > > > > > the same file is to open multiple RandomAccessFile instances over that > > > > > file and use getChannel() on each of these instances. Since > > > > > FileChannel does provide this guarantee I tend to believe it relies > > > > > upon a guarantee that RandomAccessFile provide, otherwise it would > > > > > need some mechanism to identify that different FileChannel instances > > > > > over different RandomAccessFile instances are actually over the same > > > > > file in order to provide the specified guarantee. > > > > > > What do you think? > > > > > > On Dec 21, 9:33 am, Ben Schulz <[email protected]> wrote: > > > > > > > My guess is that Java IO takes advantage of any OS optimizations > > > there > > > > > > are, on another OS there may be a per-file-pointer cache or > > > something. > > > > > > That would obviously invalidate your assumptions. I suggest you look > > > > > > into using asynchronous I/O which will get rid of any > > > multiple-reader- > > > > > > bottlenecks. This should also rule out third parties tampering with > > > > > > the file (since you already hold the write lock). > > > > > > > With kind regards > > > > > > Ben > > > > > > > On 21 Dez., 07:50, Eitan <[email protected]> wrote: > > > > > > > > I am trying to avoid locking the access to file for multiple > > > readers > > > > > > > can be multi concurrent readers and it will be a bottleneck. > > > > > > > I've created a multi threaded test for it and it seems to work > > > fine, > > > > > > > but I am not still not confident about it. I was hoping to find > > > some > > > > > > > documentation about it which specifies the behavior. > > > > > > > > On Dec 20, 11:40 pm, Christian Catchpole <[email protected]> > > > > > > > wrote: > > > > > > > > > Can you even get a second lock on a file that way? Sounds > > > dangerous. > > > > > > > > I would just create a some kind of service that each Thread can > > > access > > > > > > > > that will synchronise disk access down to one RandomAccessFile. > > > > > > > > > On Dec 21, 2:48 am, Eitan <[email protected]> wrote: > > > > > > > > > > I want to use a file in a single process with single writer or > > > multi > > > > > > > > > readers mutual exclusion, for that purpose I use number of > > > > > > > > > RandomAccessFile instances over the same file which are open > > > > > > > > > in > > > read > > > > > > > > > mode and are used concurrently. I also have a single > > > RandomAccessFile > > > > > > > > > open in "rw" mode which updates the file (not when it is being > > > > > > > > > actively read). > > > > > > > > > I do not close the Random Access File handles at any point. > > > > > > > > > > I wanted to know if the data which is being written to the > > > > > > > > > file > > > with > > > > > > > > > the RandomAccessFile in rw mode will always be visible to the > > > readers > > > > > > > > > after the the write is complete? > > > > > > > > > (I am not using rws mode for performance issues, only for > > > > > > > > > these > > > > > > > > > readers which are in the same process). > > > > > > > > > > Thanks > > > > > > > > > Eitan > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > "The Java Posse" group. > > > To post to this group, send email to [email protected]. > > > To unsubscribe from this group, send email to > > > [email protected]<javaposse%2bunsubscr...@googlegroups > > > .com> > > > . > > > For more options, visit this group at > > >http://groups.google.com/group/javaposse?hl=en. > > > -- > > Viktor Klang > > | "A complex system that works is invariably > > | found to have evolved from a simple system > > | that worked." - John Gall > > > Blog: klangism.blogspot.com > > Twttr: twitter.com/viktorklang > > Code: github.com/viktorklang -- You received this message because you are subscribed to the Google Groups "The Java Posse" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
