On 2025-01-13 Mon 04:25 GMT-08, Oswald Buddenhagen <oswald.buddenha...@gmx.de> wrote:
> On Mon, Jan 13, 2025 at 01:33:54AM -0800, Hong Xu wrote: >>Does isync perform any checks (like some checksum) to prevent data loss >>in an event of RAM error? >> > no, and the idea doesn't even make much sense. there would have to be a > checksum to compare to, and that couldn't be calculated before the data > was actually loaded into ram. I did a bit of research into IMAP, and I agree that there's no way to fully fix this unless there are data integrity support at the IMAP level. > > the whole idea of software-based ram-checking would make sense only for > data that lives in ram for a long time, where corruption due to storage > deterioration would be much more likely than corruption during transfer, > or the added time due to computing the checksums at storage and > retrieval. > > isync stores message payloads only for a few seconds in its network > buffers. if you are concerned about integrity at that time scale, get > hardware that can do it for you automatically. it would be total > insanity to half-assedly harden each application. Even though I don't think isync can fully fix the issue here, I'd still like to clarify the concern for isync users, which is real and shouldn't be dismissed. I'd also like to share what I do to mitigate. === Concern === With proper backup, storage corruption is not a concern here, because storage corruptions can hardly go undetected due to the builtin error checks in hard drives, even on consumer-grade hardware. When a corruption occurs, data can be read without error only at a super-low probability. See [1] and its reference. However, consumer-grade hardware is prone to RAM bit flips, at a higher rate than what most people probably think. The most recent research I knew is discussed on Wikipedia [2], which shows the error rate was on the magnitude of 1 bit error per gigabyte of RAM per 1.8 hours. And they truly affect data integrity, which is a frequent topic discussed in the NAS community for one example, because data can go corrupted in RAM in the period of being copied from one location to another (e.g., [3]). === My Solution === I'd also like to share what I do to mitigate silent data corruption when it's impractical to use an ECC RAM: - Regularly back up email folders. This can become a life-saver if a RAM becomes faulty, in which case many file corruptions and system crashes becomes noticeable. I can then restore the backups and let them sync back to the server with isync. - Save all important data, like attachment, to somewhere else with data integrity checks, such as a local git repository, or a subversion repository if there are large binaries. In this way, I don't worry about silent data corruptions when copying them around in the future, and occasional bit flip in emails becomes non-concerning. [1] https://serverfault.com/a/77721/158338 [2] https://en.wikipedia.org/wiki/ECC_memory#Research [3] https://forums.truenas.com/t/ecc-memory-for-home-servers-or-not/7099 -- Hong _______________________________________________ isync-devel mailing list isync-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/isync-devel
