On Saturday, January 24, 2015 at 1:29:28 PM UTC+11, Nikolaus Rath wrote: > > > > > Some applications (notably PostgreSQL) do not update inode dates when > they > > update files, specifically to reduce IO load. ie. the data is changed, > but > > the modification dates (and quite probably size) are not. > > Are you sure that's correct?
No. It's something I vaguely recall from years ago, and it may not relate to unix, and it may only relate to flushing vs update dates, which would be irrelevant in the s3sql context. > > So the -c option becomes important, at least in this case. > > > > I also (sometimes) change the file modification dates on photos to the > > original photo date after trivial edits: eg changing EXIF data. In this > > case the date and size remain the same. > > That might cause problems indeed. But seriously, I'd simply not do that > instead of trying to patch S3QL to support a bigger cache. > While it's not a sufficient reason to patch, the 100% cache does have other advantages (like allowing the file system to work when the network is down/disconnected). > The checksum is calculated before compression. Ah, excellent! I did wonder about that, but mistakenly believed the cache was compressed. > > > > Normal Copy: > > > > No. It's > > 1. Calculate checksum > 2. If different: > a. compress > b. encrypt > c. upload > > > rsync -c copy with complete cache: > > > > No, that wrong as well. The cache is stored uncompressed. > So it becomes: 1. compare local data 2. if different: a. checksum b. compress c. encrypt d. upload which is of negligible value IFF file size and date are trustworthy (which still concerns me). I must admit I see the 'rsync -c' option as a much stronger guarantee than a copy based on file dates & sizes, and this could be irrational. But, if 100% cache can be achieved at nearly zero performance impact *and* it supports rsync -c as well as an offline mode, this seems to me like a big gain at minimal cost. -- You received this message because you are subscribed to the Google Groups "s3ql" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
