Why a file mate? Why not Redis, Cassandra, MongoDB, SQLite whatnot? On Fri, Dec 25, 2009 at 10:20 AM, Eitan <[email protected]> wrote:
> What about what I said above, using multiple FileChannel/ > RandomAccessFile to read from the same file (not writing, there's a > single writer multi reader mutual exclusion protection over these > FileChannel/RandomAccessFile access), all I am worried about is the > visibility that updates the writer will do will be visible to the > readers once done, FileChannel guarantee this and I suspect it must > rely on this kind of guarantee from the RandomAccessFile that creates > it though it is not documented, But according to the simple analysis I > of the RandomAccessFile getChannel method I did, it seems that it does > need to guarantee it in order for FileChannel to guarantee it. > > Since my implementation here is just an "unlimited" single list of > objects that starts in the memory and once there's no room continue in > the disk, I went for this solution and not using an over kill embedded > data base. I only need to support appending objects at the end of the > list, removing from the start and iterating over the list in read only > mode. (Either from the start to the end or from the end to the start). > > Regarding the caching you suggested, I created caching mechanism and > there are a few layers of abstraction over this file, but, I am > talking here about the lowest level of the abstraction which access > the file and that might occur concurrently and I want to remove the > contention there as well. (for instance 10 threads currently iterating > over the list all at a different location) besides the whole issue > here is to move the data from the memory to the disk because I am out > of memory space, so caching the 10g file is not an option, I only > cache the end and the start of the list since they are accesses most > frequently. > The time spending ratio here is not the problem, there can be tens of > threads concurrently iterating over the list and I want them to be > able to iterate concurrently and not wait for each other. According to > amdal's law even if the time spent on reading is just 10%, I can only > accelerate this process by at most 10 so 100 threads iterating will > work no faster than 10. > > Basically if I get the visibility guarantee then my multi > RandomAccessFile in read mode over the file is good enough solution, > let the disk and the OS do the caching of accessing the same file > sections and keep my program more concurrent. > > Thanks > > On Dec 24, 11:45 pm, Christian Catchpole <[email protected]> > wrote: > > Is not the simplest solution to allow each thread to get a reference > > to the *single* RandomAccessFile instance and simply synchronize on it > > while reading (not forgetting to reset the offset into the file). > > It's not going to be super performant, but how is it ever going to be > > with multiple threads thrashing on one file? If anything it might > > reduce thrashing as you are controlling the points of file > > contention. What is the ratio of time spent reading the file to time > > spent operating on the data? > > > > If all these threads will in-fact thrash on the file it might only be > > solved with: > > > > 1. Better scheduling than "free for all" threads. > > 2. a large OS level disk cache > > 3. an in memory disk cache > > 4. some kind of abstract layer over the file which does caching and > > returns you "model objects" rather than being seen as a byte stream. > > > > On Dec 25, 1:38 am, Eitan <[email protected]> wrote: > > > > > Looking at FileChannel it has the following part in its documentation: > > > > > The view of a file provided by an instance of this class is guaranteed > > > to be consistent with other views of the same file provided by other > > > instances in the same program. The view provided by an instance of > > > this > > > class may or may not, however, be consistent with the views seen by > > > other > > > concurrently-running programs due to caching performed by the > > > underlying > > > operating system and delays induced by network-filesystem protocols. > > > This > > > is true regardless of the language in which these other programs are > > > written, and whether they are running on the same machine or on some > > > other > > > machine. The exact nature of any such inconsistencies are system- > > > dependent > > > and are therefore unspecified. > > > > > This looks exactly like the guarantee I need, now I've looked into > > > RandomAccessFile.getChannel() implementation and it holds a single > > > member of FileChannel which is initialized on demand and the next > > > calls will return the same instance. > > > > > So the only way I see to create multiple FileChannel instances over > > > the same file is to open multiple RandomAccessFile instances over that > > > file and use getChannel() on each of these instances. Since > > > FileChannel does provide this guarantee I tend to believe it relies > > > upon a guarantee that RandomAccessFile provide, otherwise it would > > > need some mechanism to identify that different FileChannel instances > > > over different RandomAccessFile instances are actually over the same > > > file in order to provide the specified guarantee. > > > > > What do you think? > > > > > On Dec 21, 9:33 am, Ben Schulz <[email protected]> wrote: > > > > > > My guess is that Java IO takes advantage of any OS optimizations > there > > > > are, on another OS there may be a per-file-pointer cache or > something. > > > > That would obviously invalidate your assumptions. I suggest you look > > > > into using asynchronous I/O which will get rid of any > multiple-reader- > > > > bottlenecks. This should also rule out third parties tampering with > > > > the file (since you already hold the write lock). > > > > > > With kind regards > > > > Ben > > > > > > On 21 Dez., 07:50, Eitan <[email protected]> wrote: > > > > > > > I am trying to avoid locking the access to file for multiple > readers > > > > > can be multi concurrent readers and it will be a bottleneck. > > > > > I've created a multi threaded test for it and it seems to work > fine, > > > > > but I am not still not confident about it. I was hoping to find > some > > > > > documentation about it which specifies the behavior. > > > > > > > On Dec 20, 11:40 pm, Christian Catchpole <[email protected]> > > > > > wrote: > > > > > > > > Can you even get a second lock on a file that way? Sounds > dangerous. > > > > > > I would just create a some kind of service that each Thread can > access > > > > > > that will synchronise disk access down to one RandomAccessFile. > > > > > > > > On Dec 21, 2:48 am, Eitan <[email protected]> wrote: > > > > > > > > > I want to use a file in a single process with single writer or > multi > > > > > > > readers mutual exclusion, for that purpose I use number of > > > > > > > RandomAccessFile instances over the same file which are open in > read > > > > > > > mode and are used concurrently. I also have a single > RandomAccessFile > > > > > > > open in "rw" mode which updates the file (not when it is being > > > > > > > actively read). > > > > > > > I do not close the Random Access File handles at any point. > > > > > > > > > I wanted to know if the data which is being written to the file > with > > > > > > > the RandomAccessFile in rw mode will always be visible to the > readers > > > > > > > after the the write is complete? > > > > > > > (I am not using rws mode for performance issues, only for these > > > > > > > readers which are in the same process). > > > > > > > > > Thanks > > > > > > > Eitan > > -- > > You received this message because you are subscribed to the Google Groups > "The Java Posse" group. > To post to this group, send email to [email protected]. > To unsubscribe from this group, send email to > [email protected]<javaposse%[email protected]> > . > For more options, visit this group at > http://groups.google.com/group/javaposse?hl=en. > > > -- Viktor Klang | "A complex system that works is invariably | found to have evolved from a simple system | that worked." - John Gall Blog: klangism.blogspot.com Twttr: twitter.com/viktorklang Code: github.com/viktorklang -- You received this message because you are subscribed to the Google Groups "The Java Posse" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/javaposse?hl=en.
