Re: [gentoo-user] ceph on gentoo?
On Tue, Dec 30, 2014 at 06:43:26PM +0100, J. Roeleveld wrote: On Tuesday, December 30, 2014 08:11:15 AM Bruce Hill, Jr. wrote: Can you offer any technical suggestions as for what to check? Do you leave the messages on the mailserver? In that case, ensure your POP3-client keeps a list of message-ids (UIDL) and only downloads messages that haven't been downloaded before. I don't know if there is an equivalent for mutt as I don't use that. -- Joost Thanks for this reply. That one original message didn't get removed from the mailserver, and I hadn't scrolled down quite far enough to see it. Once it was removed, it seems to have stopped repeating; and a host of other messages to the list that hadn't arrived came through (especially all the replies in this thread which weren't previously seen in mutt). -- Bruce
Re: [gentoo-user] ceph on gentoo?
On December 27, 2014 at 10:19 AM Andrew Savchenko birc...@gentoo.org wrote: Please stop insults and offensive language. I just sent replies to the list, this is verifiable by mail headers. My apologies to you sir. If you have mail problems, check your MTA or whatever you are using to receive e-mail from this list. As you can see, other people don't have this problems. On my workstation mail is POP3 using mutt and mail-mta/msmtp is the MTA. Just my guess: greylisting is broken (or had a temporary lag) on mail server you are using. There is no greylisting/blacklisting being done. I checked mail at the web interface for the hosting company, and there was no repeat of messages here; only in Mutt. Now there is another account doing the same thing. Can you offer any technical suggestions as for what to check? Best regards, Andrew Savchenko Kindest regards, Bruce
Re: [gentoo-user] ceph on gentoo?
On Tuesday, December 30, 2014 08:11:15 AM Bruce Hill, Jr. wrote: On December 27, 2014 at 10:19 AM Andrew Savchenko birc...@gentoo.org wrote: If you have mail problems, check your MTA or whatever you are using to receive e-mail from this list. As you can see, other people don't have this problems. On my workstation mail is POP3 using mutt and mail-mta/msmtp is the MTA. Just my guess: greylisting is broken (or had a temporary lag) on mail server you are using. There is no greylisting/blacklisting being done. I checked mail at the web interface for the hosting company, and there was no repeat of messages here; only in Mutt. Now there is another account doing the same thing. Can you offer any technical suggestions as for what to check? Do you leave the messages on the mailserver? In that case, ensure your POP3-client keeps a list of message-ids (UIDL) and only downloads messages that haven't been downloaded before. Here is what the man page for fetchmail says about it: *** -U | --uidl (Keyword: uidl) Force UIDL use (effective only with POP3). Force client-side tracking of 'newness' of messages (UIDL stands for unique ID listing and is described in RFC1939). Use with 'keep' to use a mailbox as a baby news drop for a group of users. The fact that seen messages are skipped is logged, unless error logging is done through syslog while running in daemon mode. Note that fetchmail may automatically enable this option depending on upstream server capa- bilities. Note also that this option may be removed and forced enabled in a future fetchmail version. See also: --idfile. *** I don't know if there is an equivalent for mutt as I don't use that. -- Joost
Re: [gentoo-user] ceph on gentoo?
On Fri, 26 Dec 2014 20:55:07 -0500, Rich Freeman wrote: If you'd like to receive a few thousand emails from santacl...@north.pole care of the list of your choice just let me know. Well done Rich, you've just posted Santa's address in plain text where all the spam address harvesters will find it. You won't be getting anything from him next year! Mind you, at his age, those offers of viagra may be useful... -- Neil Bothwick COBOL: (n.) an old computer language, designed to be read and not run. Unfortunately, it is often run anyway. pgpAcOc5X1IjK.pgp Description: OpenPGP digital signature
Re: [gentoo-user] ceph on gentoo?
Hi, On Fri, 26 Dec 2014 00:38:58 -0600 Bruce Hill wrote: To whoever controls this list... I just arrived home to find my mailbox spammed with hundreds of messages from this luser Andrew Savchenko birc...@gentoo.org Please stop insults and offensive language. I just sent replies to the list, this is verifiable by mail headers. If you have mail problems, check your MTA or whatever you are using to receive e-mail from this list. As you can see, other people don't have this problems. What is the explanation for this please? Just my guess: greylisting is broken (or had a temporary lag) on mail server you are using. Best regards, Andrew Savchenko pgpFkYZ2ZzDfy.pgp Description: PGP signature
Re: [gentoo-user] ceph on gentoo?
Thomas Mueller wrote: from Bruce Hill: To whoever controls this list... I just arrived home to find my mailbox spammed with hundreds of messages from this luser Andrew Savchenko birc...@gentoo.org What is the explanation for this please? I didn't get these spams. Are you sure they are from Andrew Savchenko? Check the headers: spammers are known to fake their email address. Tom I didn't get any here either. Unless Gmail filtered it which should be disabled. Dale :-) :-)
Re: [gentoo-user] ceph on gentoo?
Am 26.12.2014 um 09:11 schrieb Dale: I didn't get any here either. Unless Gmail filtered it which should be disabled. me = 3rd one not getting them. Without gmail (but other antispam-measures ...). S
Re: [gentoo-user] ceph on gentoo?
On Dec 26, 2014, at 10:15, Stefan G. Weichinger li...@xunil.at wrote: Am 26.12.2014 um 09:11 schrieb Dale: I didn't get any here either. Unless Gmail filtered it which should be disabled. me = 3rd one not getting them. Without gmail (but other antispam-measures ...). +1
Re: [gentoo-user] ceph on gentoo?
On Tue, Dec 23, 2014 at 06:20:05PM +0300, Andrew Savchenko wrote: Hi, On Tue, 23 Dec 2014 15:22:26 +0100 Stefan G. Weichinger wrote: Anyone here running ceph / http://ceph.com/ on gentoo? As server(s) or client or ... ? I am learning about this right now and currently on my way to a first small test cluster. Very interesting possibilities ! We used it about a year ago for our infrastructure (backup and live sync of HA systems), obviously both servers and clients were used, both on Gentoo. We stopped this because of numerous kernel panics, not to mention that it was quite slow even after tuning. So we switch to another solution for data sync and backups: clsync. (It was developed from scratch for our needs, this is not a filesystem, but may be considered as more powerful alternative to lsyncd.) Though this was a year ago or so. Your mileage may vary and it is likely that during this year stability was improved. Ceph is very promising by both design and capabilities. Best regards, Andrew Savchenko Andrew, Can you answer why my email client has HUNDREDS of the same reply from you in this thread? I've never seen this behavior in my life. Thanks
Re: [gentoo-user] ceph on gentoo?
On Fri, Dec 26, 2014 at 8:47 PM, Bruce Hill da...@happypenguincomputers.com wrote: Can you answer why my email client has HUNDREDS of the same reply from you in this thread? I've never seen this behavior in my life. Can you take this off the list? If you want somebody from Gentoo to confirm that the list had nothing to do with this I suggest filing a bug or contacting infra@g.o. There are many things that could cause the behavior you see, and most of them have nothing to do with Andrew. If you'd like to receive a few thousand emails from santacl...@north.pole care of the list of your choice just let me know. -- Rich
Re: [gentoo-user] ceph on gentoo?
To whoever controls this list... I just arrived home to find my mailbox spammed with hundreds of messages from this luser Andrew Savchenko birc...@gentoo.org What is the explanation for this please?
Re: [gentoo-user] ceph on gentoo?
from Bruce Hill: To whoever controls this list... I just arrived home to find my mailbox spammed with hundreds of messages from this luser Andrew Savchenko birc...@gentoo.org What is the explanation for this please? I didn't get these spams. Are you sure they are from Andrew Savchenko? Check the headers: spammers are known to fake their email address. Tom
Re: [gentoo-user] ceph on gentoo?
Am 24.12.2014 um 02:02 schrieb Andrew Savchenko: Ad slow: what kind of hardware did you use and how many nodes/osds? We used 3 servers, where each server was both node and osd (that's our hardware limitation). Each machine had hardware alike 2x Xeon E5450, 16 GB and 2 Gbps network connectivity (via bonding of two 1 Gbps interfaces). We went through a lot of software and kernel tuning, this helped to solve many issues, but not all of them: ceph nodes still got kernel panics once in a while. This was unacceptable and we moved for other approaches to our issues. Hmm, that dampens my enthusiasm ;-) I watched a presentation on youtube yesterday where they recommended one SSD as journal per ~4 harddisks ... and 4-8 hard disks per OSD node maximum (if I remember correctly). Plus ~1 GHz / 1 core of CPU per OSD ... as a rule of thumb. And 500 MB RAM per OSD ... that were the recommendations in http://youtu.be/C3lxGuAWEWU - Did you have the journal separated on SSDs? I think that would make quite a difference both in performance and cost ;) Do you remember the kernel version and ceph version? How many disks / OSDs? Sorry for being so curious .. Thanks, Stefan
Re: [gentoo-user] ceph on gentoo?
On Wed, 24 Dec 2014 10:58:35 +0100 Stefan G. Weichinger wrote: Did you have the journal separated on SSDs? We don't have SSDs at all. I think that would make quite a difference both in performance and cost ;) Do you remember the kernel version and ceph version? Not exactly :/ It was something rather new at that time like 3.12.x. How many disks / OSDs? 3 OSDs with raid6 attached to each one. Sorry for being so curious .. Not a problem :) Best regards, Andrew Savchenko pgpp8nppQ12M9.pgp Description: PGP signature
[gentoo-user] ceph on gentoo?
Anyone here running ceph / http://ceph.com/ on gentoo? As server(s) or client or ... ? I am learning about this right now and currently on my way to a first small test cluster. Very interesting possibilities ! Stefan
Re: [gentoo-user] ceph on gentoo?
Hi, On Tue, 23 Dec 2014 15:22:26 +0100 Stefan G. Weichinger wrote: Anyone here running ceph / http://ceph.com/ on gentoo? As server(s) or client or ... ? I am learning about this right now and currently on my way to a first small test cluster. Very interesting possibilities ! We used it about a year ago for our infrastructure (backup and live sync of HA systems), obviously both servers and clients were used, both on Gentoo. We stopped this because of numerous kernel panics, not to mention that it was quite slow even after tuning. So we switch to another solution for data sync and backups: clsync. (It was developed from scratch for our needs, this is not a filesystem, but may be considered as more powerful alternative to lsyncd.) Though this was a year ago or so. Your mileage may vary and it is likely that during this year stability was improved. Ceph is very promising by both design and capabilities. Best regards, Andrew Savchenko pgpVVw_WacZ9H.pgp Description: PGP signature
Re: [gentoo-user] ceph on gentoo?
On 2014-12-23 15:22, Stefan G. Weichinger wrote: Anyone here running ceph / http://ceph.com/ on gentoo? As server(s) or client or ... ? I am learning about this right now and currently on my way to a first small test cluster. Very interesting possibilities ! Stefan I tried the filesystem with kernel 3.7 a year ago (to export distfiles to several machines). Since it's kernel based a bug caused my system to reboot and sadly it was a database. However the project mentioned that the filesystem wasn't production ready that time. Never tried the object storage though.
Re: [gentoo-user] ceph on gentoo?
Am 23.12.2014 um 16:25 schrieb Tomas Mozes: I tried the filesystem with kernel 3.7 a year ago (to export distfiles to several machines). Since it's kernel based a bug caused my system to reboot and sadly it was a database. However the project mentioned that the filesystem wasn't production ready that time. Never tried the object storage though. cephfs still is mentioned as kind of beta in most of the talks I saw on youtube. I am going to try the object store ... and I am interested in using it with qemu/kvm. S
Re: [gentoo-user] ceph on gentoo?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Am 23.12.2014 um 16:20 schrieb Andrew Savchenko: Hi, On Tue, 23 Dec 2014 15:22:26 +0100 Stefan G. Weichinger wrote: Anyone here running ceph / http://ceph.com/ on gentoo? As server(s) or client or ... ? I am learning about this right now and currently on my way to a first small test cluster. Very interesting possibilities ! We used it about a year ago for our infrastructure (backup and live sync of HA systems), obviously both servers and clients were used, both on Gentoo. We stopped this because of numerous kernel panics, not to mention that it was quite slow even after tuning. So we switch to another solution for data sync and backups: clsync. (It was developed from scratch for our needs, this is not a filesystem, but may be considered as more powerful alternative to lsyncd.) Though this was a year ago or so. Your mileage may vary and it is likely that during this year stability was improved. Ceph is very promising by both design and capabilities. I agree! I expect that there were many changes over the time of a year ... they went from v0.72 (5th stable release) in Nov 2013 to v0.80 in May 2014 (6th stable release) ... and v0.87 in Oct 2014 (7th ...) We get 0.80.7 in ~amd64 now ... I will see. Ad slow: what kind of hardware did you use and how many nodes/osds? Thanks, Stefan -BEGIN PGP SIGNATURE- Version: GnuPG v2 iQIcBAEBAgAGBQJUmYv5AAoJEClcuD1V0PzmjTEP/2xdLh/rw9SEYzjeJwShrZOn su0tl9bqgGLC+FdKjSJ5XYCkV7VEGe4UTG1SRbPIyF246y88LoRxKZNpMcrm367o rw7dGI81uwYQPOe/ajDPXHeJTZjvfNslQugzxvHL9OxRhUcNrnw1kN3ymL3WlqCS REYcVsvEh3JL8y2edPZLMVGT5FV1P6U6UgehGYYfwT0jNJQQINEq31jf/fB/3k4n /jrnB45eKJRxwNXDm0HhtwmKWXOKWF9d2B9qvHKkYCtPnZt/5pPbDh1CX6OlU1Gm SuaVZZzCTSA88umZKq7rBKmrs09v458OlvvdcsRb7EVQ/bF26KKM0RT2xzDXvx1A KuEveiDcSulijFpxL+rk4GTNpyrc9/oz3SBKYK97VPIv+MS+IPsnAnXiVy3EYg1I UGOdy3UjIVCaZn3FVnvgbgJq6hxmsYpFB+3YED6Ei0f80efHEn+L4oeXIgaF411n 57dOsjpnm3WnfHqht7BsU+qIDfD3haPgh0RBAVFk1KPBzwqvU0fJIFbEIPUN379E iIZlZsX9BQocAsyGzTku3G2AZScRoSjYNXfT///vAYnm2BHb274mjBkzsjp3NExN r8GZx/Rjb1qBnhIJsPWKe1sMXf4zeZ+bq4d9a8pzEycrFE2YYFJggOObLJFpF505 W9ahxI/qsoWQ/gX6+HkU =Xl3X -END PGP SIGNATURE-
Re: [gentoo-user] ceph on gentoo?
Am 23.12.2014 um 16:28 schrieb Stefan G. Weichinger: Am 23.12.2014 um 16:25 schrieb Tomas Mozes: I tried the filesystem with kernel 3.7 a year ago (to export distfiles to several machines). Since it's kernel based a bug caused my system to reboot and sadly it was a database. However the project mentioned that the filesystem wasn't production ready that time. Never tried the object storage though. cephfs still is mentioned as kind of beta in most of the talks I saw on youtube. I am going to try the object store ... and I am interested in using it with qemu/kvm. got my first two demo nodes up and in-sync ... what a success ;-) As so often with new technology one has to learn and understand things first ... the next nodes should be way easier to set up. I already set up a block device on the store and mounted it on my desktop machine ... it works! Performance aside ... right now the cluster runs on 2 VMs. I should also file a bug already: /usr/bin/rbd looks for /sbin/udevadm while it is located in /usr/bin solved it with ln -s for now ... but, you know, it should configure this at build-time. Now for some qemu/libvirt-testing ... S
Re: [gentoo-user] ceph on gentoo?
On Tue, Dec 23, 2014 at 3:27 PM, Stefan G. Weichinger li...@xunil.at wrote: got my first two demo nodes up and in-sync ... what a success ;-) I started to look into ceph, and my biggest issue is that they don't protect against silent corruption. They do checksum data during transit, but not at rest. That means that you could end up with 3 different copies of a file and no way to know which one is the right one. Simply storing the data on btrfs isn't enough - that will protect against files changing on the disk itself, but you could STILL end up with 3 different copies of a file on different nodes and no way to know which one is right, if the error happens at a higher level than the btrfs filesystem/disk. -- Rich
Re: [gentoo-user] ceph on gentoo?
Am 23.12.2014 um 21:40 schrieb Rich Freeman: On Tue, Dec 23, 2014 at 3:27 PM, Stefan G. Weichinger li...@xunil.at wrote: got my first two demo nodes up and in-sync ... what a success ;-) I started to look into ceph, and my biggest issue is that they don't protect against silent corruption. They do checksum data during transit, but not at rest. That means that you could end up with 3 different copies of a file and no way to know which one is the right one. Simply storing the data on btrfs isn't enough - that will protect against files changing on the disk itself, but you could STILL end up with 3 different copies of a file on different nodes and no way to know which one is right, if the error happens at a higher level than the btrfs filesystem/disk. but ... oh my. *sigh* I assume the devs there have a clever answer to this as well? At least for the future ... now that btrfs is declared stable at least for the more trivial setups (read: not RAID5/6) by Chris Mason himself ... btrfs should be usable for ceph-OSDs soon. In the other direction: what protects against these errors you mention? S
Re: [gentoo-user] ceph on gentoo?
On Tue, Dec 23, 2014 at 3:54 PM, Stefan G. Weichinger li...@xunil.at wrote: Am 23.12.2014 um 21:40 schrieb Rich Freeman: On Tue, Dec 23, 2014 at 3:27 PM, Stefan G. Weichinger li...@xunil.at wrote: got my first two demo nodes up and in-sync ... what a success ;-) I started to look into ceph, and my biggest issue is that they don't protect against silent corruption. They do checksum data during transit, but not at rest. That means that you could end up with 3 different copies of a file and no way to know which one is the right one. Simply storing the data on btrfs isn't enough - that will protect against files changing on the disk itself, but you could STILL end up with 3 different copies of a file on different nodes and no way to know which one is right, if the error happens at a higher level than the btrfs filesystem/disk. but ... oh my. *sigh* I assume the devs there have a clever answer to this as well? At least for the future ... now that btrfs is declared stable at least for the more trivial setups (read: not RAID5/6) by Chris Mason himself ... btrfs should be usable for ceph-OSDs soon. Proclamations of stability do not stable make. :) I'm using btrfs now, but I've had my share of headaches (especially with the 3.15/16 kernels). I think the rate of change/regressions is a good long-term sign, but I'd stick to longterm kernels if you use it (which is different advice than I used to give). In the other direction: what protects against these errors you mention? If I had a solution I'd be using it. I don't use ceph. Btrfs protects against them just fine for a single system. The problem with ceph is cross-node consistency. -- Rich
Re: [gentoo-user] ceph on gentoo?
On 23/12/14 23:28, Stefan G. Weichinger wrote: Am 23.12.2014 um 16:25 schrieb Tomas Mozes: I tried the filesystem with kernel 3.7 a year ago (to export distfiles to several machines). Since it's kernel based a bug caused my system to reboot and sadly it was a database. However the project mentioned that the filesystem wasn't production ready that time. Never tried the object storage though. cephfs still is mentioned as kind of beta in most of the talks I saw on youtube. I am going to try the object store ... and I am interested in using it with qemu/kvm. S Tried that (qemu/kvm mostly gentoo VM's up to 64G) ... gave up as it was too slow/unstable with the hardware I had. You need a 10G network and a much larger number of hosts than 3 to do serious I/O. I do think its something that is only practical with a datacentre sized installation. Using it for VM images was very slow and unstable - I did use btrfs under it and not xfs (the recommended for production?) - moved to pure btrfs and its is SO much better :) Rebuilding it every few weeks and having to keep backups of a couple of terrabytes of disposable data because rebuilding was so slow was the last straw ... What I would like (and what I was looking at ceph to do) was a distributable (across a relatively slow WAN) synced file system that placed only data in use close to the host using it - will have to get back to it one day.
Re: [gentoo-user] ceph on gentoo?
Hi, On Tue, 23 Dec 2014 16:36:25 +0100 Stefan G. Weichinger wrote: Am 23.12.2014 um 16:20 schrieb Andrew Savchenko: [...] We used it about a year ago for our infrastructure (backup and live sync of HA systems), obviously both servers and clients were used, both on Gentoo. We stopped this because of numerous kernel panics, not to mention that it was quite slow even after tuning. So we switch to another solution for data sync and backups: clsync. (It was developed from scratch for our needs, this is not a filesystem, but may be considered as more powerful alternative to lsyncd.) Though this was a year ago or so. Your mileage may vary and it is likely that during this year stability was improved. Ceph is very promising by both design and capabilities. I agree! I expect that there were many changes over the time of a year ... they went from v0.72 (5th stable release) in Nov 2013 to v0.80 in May 2014 (6th stable release) ... and v0.87 in Oct 2014 (7th ...) We get 0.80.7 in ~amd64 now ... I will see. Ad slow: what kind of hardware did you use and how many nodes/osds? We used 3 servers, where each server was both node and osd (that's our hardware limitation). Each machine had hardware alike 2x Xeon E5450, 16 GB and 2 Gbps network connectivity (via bonding of two 1 Gbps interfaces). We went through a lot of software and kernel tuning, this helped to solve many issues, but not all of them: ceph nodes still got kernel panics once in a while. This was unacceptable and we moved for other approaches to our issues. Best regards, Andrew Savchenko pgplP4mXOrMjz.pgp Description: PGP signature
