Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Sat, 07 Oct 2017 19:11:42 +0200, J. Roeleveld wrote: > > > mdbox? Is this a single file per mail folder? > > > > It's multiple mails per file and multiple files per mailbox. > > > > > The main reason I switched to maildir several decades ago was > > > precisely the issues (by design) mbox has. > > > A single corrupted email WILL kill the entire folder. > > > > https://wiki2.dovecot.org/MailboxFormat/dbox > > Interesting, but I still consider multiple emails inside a single file > a recepy for disaster. It increases the risk, but by how much? The wiki page doesn't give any indication of how many mails are stored in each file. Is is 5 mails per file or 5 files per mailbox? > The following is another cause for concern: > "This also means that you must not lose the dbox index files, they > can't be regenerated without data loss. " I was concerned about that but further reading indicates that these files hold only metadata. You may lose information on which mails have been read or flagged but the mails are still there. It also adds complication at the MDA level as procmail and friends can't simply save the mail in a directory. On the other hand, Dovecot is mature and well used software to these concerns have probably already been addressed somewhere. -- Neil Bothwick Blessed be the pessimist for he hath made backups. pgpaEot3gjrEz.pgp Description: OpenPGP digital signature
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Sat, Oct 7, 2017 at 12:12 PM, Neil Bothwick wrote: > On Sat, 7 Oct 2017 07:06:24 -0400, Rich Freeman wrote: > >> btrfs isn't horrible, but it basically hasn't been optimized at all. >> The developers are mainly focused on getting it to not destroy your >> data, with mixed success. An obvious example of this is that if you >> read a file from a pair of mirrors, the filesystem decides which drive >> in the pair to use based on whether the PID doing the read is even or >> odd. >> >> Fundamentally I haven't seen any arguments as to why btrfs should be >> any worse than zfs. It just hasn't been implemented completely. But, >> if you want a filesystem today and not in 10 years you need to take >> that into account. > > I switched from ZFS to btrfs a few years ago when it appeared that ZFS > wasn't really going anywhere while btrfs was under active development. It > looks like I backed the wrong horse and should investigate switching back. > Well, they're both FOSS, and honestly I feel like btrfs has more potential, but zfs is much more usable today. Btrfs has features which make it a lot more flexible in smaller installs (like being able to remove disks, and treating snapshots as full citizens). However, zfs generally can get the job done and is far less likely to eat your data in the process. I was also a btrfs hold-out for a long time, and I look forward to using it again some day, but it hasn't matured like I originally hoped. -- Rich
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Saturday, October 7, 2017 6:13:57 PM CEST Neil Bothwick wrote: > On Sat, 07 Oct 2017 14:59:39 +0200, J. Roeleveld wrote: > > > Although, I will also be switching to dovecot's mdbox format when I > > > set up my next server, so the issue of lots of small files won't be > > > nearly as big. > > > > mdbox? Is this a single file per mail folder? > > It's multiple mails per file and multiple files per mailbox. > > > The main reason I switched to maildir several decades ago was precisely > > the issues (by design) mbox has. > > A single corrupted email WILL kill the entire folder. > > https://wiki2.dovecot.org/MailboxFormat/dbox Interesting, but I still consider multiple emails inside a single file a recepy for disaster. The following is another cause for concern: "This also means that you must not lose the dbox index files, they can't be regenerated without data loss. " -- Joost
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Sat, 07 Oct 2017 14:59:39 +0200, J. Roeleveld wrote: > > Although, I will also be switching to dovecot's mdbox format when I > > set up my next server, so the issue of lots of small files won't be > > nearly as big. > > mdbox? Is this a single file per mail folder? It's multiple mails per file and multiple files per mailbox. > The main reason I switched to maildir several decades ago was precisely > the issues (by design) mbox has. > A single corrupted email WILL kill the entire folder. https://wiki2.dovecot.org/MailboxFormat/dbox -- Neil Bothwick For every action, there is an equal and opposite malfunction. pgpQ2FiTZtFGj.pgp Description: OpenPGP digital signature
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Sat, 7 Oct 2017 07:06:24 -0400, Rich Freeman wrote: > btrfs isn't horrible, but it basically hasn't been optimized at all. > The developers are mainly focused on getting it to not destroy your > data, with mixed success. An obvious example of this is that if you > read a file from a pair of mirrors, the filesystem decides which drive > in the pair to use based on whether the PID doing the read is even or > odd. > > Fundamentally I haven't seen any arguments as to why btrfs should be > any worse than zfs. It just hasn't been implemented completely. But, > if you want a filesystem today and not in 10 years you need to take > that into account. I switched from ZFS to btrfs a few years ago when it appeared that ZFS wasn't really going anywhere while btrfs was under active development. It looks like I backed the wrong horse and should investigate switching back. -- Neil Bothwick C&W music backward: get yer dog, wife, job, truck, kids, and sobriety back. pgpIkfcfhzYDe.pgp Description: OpenPGP digital signature
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Saturday, October 7, 2017 11:28:08 AM CEST Tanstaafl wrote: > On 10/6/2017, 2:12:00 PM, J. Roeleveld wrote: > > I had a large partition with reiserfs. > > Running fsck always failed due to running out of memory. > > > > Partition was quite a bit larger than 2TB (around 6TB) and contained > > a huge (millions) amount of files, > but having an fsck become > > impossible with 16GB memory available was rather annoying. > > Ah, yes, I had a similar problem occasionally when a user would decide > to delete (or move to a different folder) a bunch (as many as tens of > thousands) of messages at once... Thunderbird would go non responsive, > and the server was brought to its knees. I'd have to kill their server > processes, and then the user would end up with a bunch of duplicate > messages in their maildirs. > > Very annoying. Actually, I used to do this a lot using a webmail client (when I was still able to run squirrelmail without having to change back to an old PHP version) and never actually had any issues with this. Neither with reiserfs or ext4. I would put that down to either hardware or issues with the chosen IMAP- server. For reference: I have been using Cyrus for a very long time. -- Joost
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Saturday, October 7, 2017 11:18:33 AM CEST Tanstaafl wrote: > On 10/6/2017, 8:53:27 AM, Philip Webb wrote: > > 171005 christos kotsis wrote: > >> I just noticed that ReiserFS has significant performance > >> over ext3, 4 when dealing with small files. > > > > I've long relied on ReiserFS for everything except /boot > > & have never had any problems with my files or drives. > > I have many small files + a few big PDFs -- perhaps c 20 MB ea -- > > & the big ones simply stay where I put them, so no changes to handle. > > I used ReiserFS for many - 8+ - years on our old mail server, selected > for its performance with large numbers of small (maildir) files, and > never had a problem. Same here, apart from that one partition where the fsck never worked. > But during the last rebuild when virtualizing everything, sometime > around 2012, I switched to XFS, and believe I saw a performance gain, > and no more long fsck's during the rare reboots... and again, no problems. My last rebuild was earlier this year, my mail had already been migrated to ext4 without issues. (Did not notice any performance issues) > Personally, I can't wait until btrfs is fully ready/stable, and have > been considering FreeBSD (or FreeNAS) just for ZFS, for the reliability > factor, but have wondered about performance for mail servers. > > Anyone have any experience with comparing performance with either btrfs > or ZFS against either ReiserFS or XFS for a maildir based mail server? My mailserver (Cyrus) uses ext4 for the mailboxes. This is on a partition which is accessed via iSCSI. Which is a zvol on a ZFS pool. Eg: disks <-> ZFS <-> zvol <-> iSCSI <-> ext4 I am not noticing any significant performance issues, the ones I am can be resolved by adding a dedicated SLOG en L2ARC, but this will only help the systems hanging in the rack as those are connected with a 20Gbe link. Rest of the systems won't get more than 1Gbe. I have several large mailboxes: - postgresql-hackers = 195,000 items - gentoo-user = 240,000 items - Xen-devel = 366,000 items The others are below 100,000. I use these as archives and regularly search through these before reverting to Google or asking on the relevant mailing lists. > Although, I will also be switching to dovecot's mdbox format when I set > up my next server, so the issue of lots of small files won't be nearly > as big. mdbox? Is this a single file per mail folder? The main reason I switched to maildir several decades ago was precisely the issues (by design) mbox has. A single corrupted email WILL kill the entire folder. -- Joost
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Sat, Oct 7, 2017 at 6:28 AM, Neil Bothwick wrote: > On Sat, 7 Oct 2017 05:18:33 -0400, Tanstaafl wrote: > >> Anyone have any experience with comparing performance with either btrfs >> or ZFS against either ReiserFS or XFS for a maildir based mail server? > > I tried btrfs on a mail server and it was unbearably slow. Disabling > copy-on-write made a big difference, but it still went a lot faster when > I switched to ext4. > > I haven't used XFS in years, maybe it's time to revisit it. > I haven't used xfs in a while, but here is my sense of things, for a basic configuration (filesystem running on one drive or a mirrored pair): xfs > ext4 > zfs >>> btrfs At least, that is in terms of most conventional measures of performance (reading and writing files on a typical filesystem). If you want to measure performance in terms of how long your system is down after a controller error then both zfs and btrfs will have an advantage. I mention it because I think that integrity shouldn't take a back seat to performance 99% of the time. It has performance benefits of its own, but you only see them every couple of years when something fails. btrfs isn't horrible, but it basically hasn't been optimized at all. The developers are mainly focused on getting it to not destroy your data, with mixed success. An obvious example of this is that if you read a file from a pair of mirrors, the filesystem decides which drive in the pair to use based on whether the PID doing the read is even or odd. Fundamentally I haven't seen any arguments as to why btrfs should be any worse than zfs. It just hasn't been implemented completely. But, if you want a filesystem today and not in 10 years you need to take that into account. Now, ZFS has a bunch of tricks available to improve things like SSD read caches and write logs. But, you could argue that other filesystems support separate journal devices and there is bcache so I think if you want to look at those features you need to compare apples to apples. ZFS certainly integrates it all nicely, but then it has other "features" like not being able to remove a drive from a storage pool, or revert to a snapshot without deleting all the subsequent snapshots. In general though I think zfs will always suffer a bit in performance because it is copy-on-write. If you want to change 1 block in the middle of a file, ext4 and xfs can just write over that 1 block, while zfs and btrfs are going to write that block someplace else and do a metadata dance to map it over the original block. I just don't see how that will ever be faster. Of course, if you have a hardware failure in the middle of an operation zfs and btrfs basically guarantee that the writes behave as if they were atomic, while you only get that benefit with ext4/xfs if you do full journaling with a significant performance hit, and if you're using mdadm underneath then you lose that guarantee. Both zfs and btrfs avoid the raid write hole (though to be fair you don't want to go anywhere near parity raid on btrfs anytime soon). I'm not saying that there isn't a place for performance-above-all. For an ephemeral worker node you already have 47 backups running and if the node fails you restart it, so if it needs to write some data to disk performance is probably the only concern. Ditto for any data that has no long-term value/etc. However, for most general-purpose filesystems I think integrity should be the #1 concern, because you won't notice that 20us access time difference but you probably will notice hour spent restoring from backups, assuming you even have backups. -- Rich
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Sat, 7 Oct 2017 05:18:33 -0400, Tanstaafl wrote: > Anyone have any experience with comparing performance with either btrfs > or ZFS against either ReiserFS or XFS for a maildir based mail server? I tried btrfs on a mail server and it was unbearably slow. Disabling copy-on-write made a big difference, but it still went a lot faster when I switched to ext4. I haven't used XFS in years, maybe it's time to revisit it. -- Neil Bothwick Walk softly and carry a fully charged phazer. pgpRvvjXncZeO.pgp Description: OpenPGP digital signature
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On 10/6/2017, 2:12:00 PM, J. Roeleveld wrote: > I had a large partition with reiserfs. > Running fsck always failed due to running out of memory. > > Partition was quite a bit larger than 2TB (around 6TB) and contained > a huge (millions) amount of files, > but having an fsck become > impossible with 16GB memory available was rather annoying. Ah, yes, I had a similar problem occasionally when a user would decide to delete (or move to a different folder) a bunch (as many as tens of thousands) of messages at once... Thunderbird would go non responsive, and the server was brought to its knees. I'd have to kill their server processes, and then the user would end up with a bunch of duplicate messages in their maildirs. Very annoying.
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On 10/6/2017, 8:53:27 AM, Philip Webb wrote: > 171005 christos kotsis wrote: >> I just noticed that ReiserFS has significant performance >> over ext3, 4 when dealing with small files. > I've long relied on ReiserFS for everything except /boot > & have never had any problems with my files or drives. > I have many small files + a few big PDFs -- perhaps c 20 MB ea -- > & the big ones simply stay where I put them, so no changes to handle. I used ReiserFS for many - 8+ - years on our old mail server, selected for its performance with large numbers of small (maildir) files, and never had a problem. But during the last rebuild when virtualizing everything, sometime around 2012, I switched to XFS, and believe I saw a performance gain, and no more long fsck's during the rare reboots... and again, no problems. Personally, I can't wait until btrfs is fully ready/stable, and have been considering FreeBSD (or FreeNAS) just for ZFS, for the reliability factor, but have wondered about performance for mail servers. Anyone have any experience with comparing performance with either btrfs or ZFS against either ReiserFS or XFS for a maildir based mail server? Although, I will also be switching to dovecot's mdbox format when I set up my next server, so the issue of lots of small files won't be nearly as big.
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
Mick wrote: > On Friday, 6 October 2017 19:12:00 BST J. Roeleveld wrote: > >> I had a large partition with reiserfs. >> Running fsck always failed due to running out of memory. >> >> Partition was quite a bit larger than 2TB (around 6TB) and contained a huge >> (millions) amount of files, but having an fsck become impossible with 16GB >> memory available was rather annoying. >> >> -- >> Joost > It should also be noted that reiserfs may be getting some bitrot over the > years. I haven't used it for a while now, but when I converted an old PC > from > reiserfs to ext4 it acquired a new lease of life in terms of performance. > For > every day personal use I tend to prefer it, although I also use btrfs on a > trial basis. Same here. When I built my current rig several years ago, I switched to ext4 from reiserfs myself. At the time, there was very little development and its future was uncertain since the guy that came up with it was shall we say, no longer available. I haven't looked into it but is it even maintained like it should be or just sitting out there with the occasional patch up by whoever has a few minutes? If I were building a new rig today, I don't think I'd even consider reiserfs. Dale :-) :-)
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Friday, 6 October 2017 19:12:00 BST J. Roeleveld wrote: > I had a large partition with reiserfs. > Running fsck always failed due to running out of memory. > > Partition was quite a bit larger than 2TB (around 6TB) and contained a huge > (millions) amount of files, but having an fsck become impossible with 16GB > memory available was rather annoying. > > -- > Joost It should also be noted that reiserfs may be getting some bitrot over the years. I haven't used it for a while now, but when I converted an old PC from reiserfs to ext4 it acquired a new lease of life in terms of performance. For every day personal use I tend to prefer it, although I also use btrfs on a trial basis. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On 2017-10-06 20:12, J. Roeleveld wrote: On 5 October 2017 22:45:50 GMT+02:00, christos kotsis wrote: I just noticed that ReiserFS has significant performance over ext3, 4 when dealing with small files. On 5 Oct 2017 11:32 pm, "christos kotsis" wrote: If the big data are used often,and I/O performance is desirable, then I would go for two partitions. One would be either ext3 or ext4, with huge block size, while the second could be one of two with small block size(minimum 1024). On 5 Oct 2017 10:46 pm, wrote: Hi, Installing gentoo on new laptop and it has 2TB disk. I want to use 1.8TB for data where will be big files and also huge amount of small files, thus I want to ask which FS is best for this. Until now I've used reiserfs on cca 0.5TB partition, but I don't know if it's also good choice for that big partition. I've thought about zfs, but I don't need snapshots and such stuff mostly scaling is requirement. Thanks Pat Freehosting PIPNI - http://www.pipni.cz/ I had a large partition with reiserfs. Running fsck always failed due to running out of memory. Partition was quite a bit larger than 2TB (around 6TB) and contained a huge (millions) amount of files, but having an fsck become impossible with 16GB memory available was rather annoying. -- Joost Hi, Thanks to all, it will be dev env, so sources and big DB. I'll try reiserfs first (I have long time experiences with this one and have 32GB RAM :-D) and if this fails then xfs. Thanks again. Pat Freehosting PIPNI - http://www.pipni.cz/
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On 5 October 2017 22:45:50 GMT+02:00, christos kotsis wrote: >I just noticed that ReiserFS has significant performance over ext3, 4 >when >dealing with small files. > >On 5 Oct 2017 11:32 pm, "christos kotsis" >wrote: > >If the big data are used often,and I/O performance is desirable, then I >would go for two partitions. >One would be either ext3 or ext4, with huge block size, while the >second >could be one of two with small block size(minimum 1024). > > >On 5 Oct 2017 10:46 pm, wrote: > >Hi, > >Installing gentoo on new laptop and it has 2TB disk. I want to use >1.8TB >for data where will be big files and also huge amount of small files, >thus >I want to ask which FS is best for this. Until now I've used reiserfs >on >cca 0.5TB partition, but I don't know if it's also good choice for that >big >partition. > >I've thought about zfs, but I don't need snapshots and such stuff >mostly >scaling is requirement. > >Thanks > >Pat > > >Freehosting PIPNI - http://www.pipni.cz/ I had a large partition with reiserfs. Running fsck always failed due to running out of memory. Partition was quite a bit larger than 2TB (around 6TB) and contained a huge (millions) amount of files, but having an fsck become impossible with 16GB memory available was rather annoying. -- Joost -- Sent from my Android device with K-9 Mail. Please excuse my brevity.
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
On Fri, Oct 6, 2017 at 5:53 AM, Philip Webb wrote: > 171005 christos kotsis wrote: >> I just noticed that ReiserFS has significant performance >> over ext3, 4 when dealing with small files. > > I've long relied on ReiserFS for everything except /boot > & have never had any problems with my files or drives. > I have many small files + a few big PDFs -- perhaps c 20 MB ea -- > & the big ones simply stay where I put them, so no changes to handle. > Unless your needs are fairly specialized (in which case you probably wouldn't be looking for advice on this list), I'd probably stick with the more mainstream filesystems. I doubt reiserfs will eat your data, but it has been generally falling out of use. IMO if your goal isn't to experiment with alternate filesystems, there are really only a couple of mainstream choices out there for a general-purpose workstation filesystem: 1. Ext4: This should just be your default if you don't want to care about your filesystem. It is ubiquitous for a reason. It won't eat your data, and everybody knows what to expect from it. If your filesystem is fairly small and being used for a root, or otherwise has a lot of small files, then make sure to override the inode defaults. Other than that it just works. 2. Xfs: If you absolutely have to mess with a filesystem (especially for multimedia) this isn't a bad alternative. You won't be able to shrink it, but for the most part it behaves a lot like ext4. Zfs is starting to cross over into experimental territory, IMO, though it generally is fairly stable. I care about data integrity, so it is what I tend to run (well, aside from one btrfs filesystem I haven't switched over). I had a SATA port misbehave and spread silent corruption all over a disk, and zfs got me through it without anything but some warning alerts/etc and a need to rebuild after I moved the drive to another controller (and marked a big X over the port). If I were using mdadm I'd have had to rebuild from backups at a cost of hours of downtime (a fairly large array), and might have lost recently-written data entirely as might have been in use for longer before detecting the error, leaving me a dilemma of figuring out which backup versions were good, with the answer being something older. Even if I didn't have redundancy zfs (or btrfs) would have complained loudly about the issue. I do use snapshots because they're cheap, but rolling back is pretty rare. Unless you have a very specialized need I wouldn't go messing with block sizes or anything like that in any of these cases. -- Rich
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
171005 christos kotsis wrote: > I just noticed that ReiserFS has significant performance > over ext3, 4 when dealing with small files. I've long relied on ReiserFS for everything except /boot & have never had any problems with my files or drives. I have many small files + a few big PDFs -- perhaps c 20 MB ea -- & the big ones simply stay where I put them, so no changes to handle. -- ,, SUPPORT ___//___, Philip Webb ELECTRIC /] [] [] [] [] []| Cities Centre, University of Toronto TRANSIT`-O--O---' purslowatchassdotutorontodotca
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
I just noticed that ReiserFS has significant performance over ext3, 4 when dealing with small files. On 5 Oct 2017 11:32 pm, "christos kotsis" wrote: If the big data are used often,and I/O performance is desirable, then I would go for two partitions. One would be either ext3 or ext4, with huge block size, while the second could be one of two with small block size(minimum 1024). On 5 Oct 2017 10:46 pm, wrote: Hi, Installing gentoo on new laptop and it has 2TB disk. I want to use 1.8TB for data where will be big files and also huge amount of small files, thus I want to ask which FS is best for this. Until now I've used reiserfs on cca 0.5TB partition, but I don't know if it's also good choice for that big partition. I've thought about zfs, but I don't need snapshots and such stuff mostly scaling is requirement. Thanks Pat Freehosting PIPNI - http://www.pipni.cz/
Re: [gentoo-user] {OT?} which fs on 1.8TB partition
If the big data are used often,and I/O performance is desirable, then I would go for two partitions. One would be either ext3 or ext4, with huge block size, while the second could be one of two with small block size(minimum 1024). On 5 Oct 2017 10:46 pm, wrote: Hi, Installing gentoo on new laptop and it has 2TB disk. I want to use 1.8TB for data where will be big files and also huge amount of small files, thus I want to ask which FS is best for this. Until now I've used reiserfs on cca 0.5TB partition, but I don't know if it's also good choice for that big partition. I've thought about zfs, but I don't need snapshots and such stuff mostly scaling is requirement. Thanks Pat Freehosting PIPNI - http://www.pipni.cz/
[gentoo-user] {OT?} which fs on 1.8TB partition
Hi, Installing gentoo on new laptop and it has 2TB disk. I want to use 1.8TB for data where will be big files and also huge amount of small files, thus I want to ask which FS is best for this. Until now I've used reiserfs on cca 0.5TB partition, but I don't know if it's also good choice for that big partition. I've thought about zfs, but I don't need snapshots and such stuff mostly scaling is requirement. Thanks Pat Freehosting PIPNI - http://www.pipni.cz/