BTW: you're right.. the journal-sync tool was not actually syncing on NIO Mapped and NIO are performing similarly. (Mapped being slight faster)
I am also changing AIO to not use O_DIRECT in case of sync=false... (I don't know why someone would use AIO with sync=false, but in any case.. i will make the change) On Fri, Feb 3, 2017 at 3:58 AM, nigro_franz <[email protected]> wrote: > @clebertsuconic But even if is a personal request, IMHO would be really great > to have a separate (open source of course!) project that aims to provide a > journal as a first class citizen, maybe with pluggable features like > different binary encoders/decoders (SBE, ProtoBuf, FlatBuffers..), > replication/compaction/paging and using different stores (redis, SQL, > cassandra, elasticsearch, Kafka)...the possibilities could be infinite, > wdyt? > >> when you do sync=false, is that actually writing to the >> disk, or it's just putting it in memory until you call force or close >> the disk? > > AFAIK it is really writing to the disk, but asynchronously and using the OS > configuration for it (on linux: numa balancing, swappiness, > dirty_expire_centisecs, dirty_ratio, dirty_writeback_centisecs and many > more). > >> I'm not arguing how safe is to use either, but on my crash tests the >> messages weren't lost if I waited some time before killing the server. > > Cool! Effectively It's guaranteed to be safe with process failure, indeed > AFAIK pretty almost any OS that provides mmap ensures that every writes > performed on mapped memory will be persisted on physical storage even if the > owner process will crash! The only way to force a mmap file to lose data is > with power failures/OS hard-crashes (that sadly Murphy's law said that will > happen for sure eheh). > >> IAIO with no-sync probably has a higher level of guarantees of being >>> actually on the disk in case of crashes.. but if the user don't need >>> syncs, that's a rock & roll possibility. > > Yes, i think the same, it has a good balance between reliability (on process > crashes, but on the cloud it could be enough :P) and speed but AIO is > unbeatable from this POV: if the write is reported as performed it is no way > on the disk, even on OS failures! > Another 2 points of difference will be memory and cpu usage: the writes > you're seeing are flushed by the OS on the disk in background, using the > virtual memory directly as a buffer (but on lAIO there is no need to use > it!); indeed the CPU usage will be, for high writes rate, a little higher > than the lAIO version, due to this async flushes, right? > >> Take a look on my branch, mapped at my fork > > Thanks!For sure!!! > > > > -- > View this message in context: > http://activemq.2283324.n4.nabble.com/DISCUSS-Memory-Mapped-Journal-Type-tp4721494p4721560.html > Sent from the ActiveMQ - Dev mailing list archive at Nabble.com. -- Clebert Suconic
