Re: [ceph-users] Asked for emperor, got firefly. (You can't take the sky from me?)
Its just a text file, you can change it/create it on all your nodes before
you run ceph-deploy.
W dniu 02.09.2014 o 21:37 J David pisze:
On Tue, Sep 2, 2014 at 2:50 PM, Konrad Gutkowski
wrote:
You need to set higher priority for ceph repo, check "ceph-deploy with
--release (--stable) for dumpling?" thread.
Right, this is the same issue as that. It looks like the 0.80.1
packages are coming from Ubuntu; this is the first time we have used
Ubuntu nodes instead of Debian and Debian doesn't have ceph packages
in its default repos, so we haven't seen this before.
But ceph-deploy installs the ceph repo and the packages in one step.
("ceph-deploy install hostname")
So at what point should one change the repo priority to get the desired
result?
Also, is this a ceph-deploy bug? Does it need reporting, or is there
one already? Searching found only bug 8533, which sounds like the
same issue for yum-based repositories; that bug says it is resolved,
but it's not clear whether that applies to .deb repositories as well,
nor is it clear what version one has to be at to get the fix.
And finally, is it safe to *only* upgrade ceph-deploy? Will newer
versions work with our 0.72.2 ceph cluster?
Thanks!
--
Konrad Gutkowski
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Asked for emperor, got firefly. (You can't take the sky from me?)
Hi, You need to set higher priority for ceph repo, check "ceph-deploy with --release (--stable) for dumpling?" thread. W dniu 02.09.2014 o 19:18 J David pisze: On Tue, Sep 2, 2014 at 1:00 PM, Alfredo Deza wrote: correct, if you don't specify what release you want/need, ceph-deploy will use the latest stable release (firefly as of this writing) So, ceph-deploy set up emperor repositories in /etc/apt/sources.list.d/ceph.list and then didn't use them? If that is what happened, then why 0.80.1 and not 0.80.5? Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Konrad Gutkowski ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] script for commissioning a node with multiple osds, added to cluster as a whole
Hi, You could use ceph-deploy. There wont be any difference in the total amount of data being moved. W dniu 29.08.2014 o 18:53 Chad Seys pisze: Hi All, Does anyone have a script or sequence of commands to prepare all drives on a single computer for use by ceph, and then start up all OSDs on the computer at one time? I feel this would be faster and less network traffic than adding one drive at a time, which is what the current script does. Thanks! Chad. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Konrad Gutkowski ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] ceph-deploy with --release (--stable) for dumpling?
Ceph-deploy should set priority for ceph repository, which it doesn't, this usually installs the best available version from any repository. I don't know if this is intentional, but you can change this yourself - google "apt repository priority" and change it on all your nodes. W dniu 26.08.2014 o 02:52 Nigel Williams pisze: ceph-deploy --release dumpling or previously ceph-deploy --stable dumpling now results in Firefly (0.80.1) being installed, is this intentional? I'm adding another host with more OSDs and guessing it is preferable to deploy the same version. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Konrad Gutkowski ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] flashcache from fb and dm-cache??
It does improve performance somewhat, just don't expect miracles. At this point using any of the caching solutions we give you similar results. I did only limited tests with small clusters (2-3 osd nodes). W dniu 30.07.2014 o 16:40 German Anders pisze: Also, does someone try flashcache from facebook on ceph? cons? pros? any perf improvement? and dm-cache? German Anders -- Konrad Gutkowski___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Bad Write-Performance on Ceph/Possible bottlenecks?
Hi, I wouldn't put those SSD's in raid, just use them separately as journals for half of your's HDD's. This should make your write performance somewhat better. W dniu 04.07.2014 o 11:13 Marco Allevato pisze: Hello Ceph-Community, I’m writing here because we have a bad write-performance on our Ceph-Cluster of about As an overview the technical details of our Cluster: 3 x monitoring-Servers; each with 2 x 1 Gbit/s NIC configured as Bond (Link Aggregation-Mode) 5 x datastore-Servers; each with 10 x 4 TB HDDs serving as OSDs, as Journal we use a 15 GB LVM on an 256 GB SSD-Raid1; 2 x 10 Gbit/s NIC configured as Bond (Link Aggregation->Mode) ceph.conf [global] auth_service_required = cephx filestore_xattr_use_omap = true auth_client_required = cephx auth_cluster_required = cephx mon_host = 172.30.30.8,172.30.30.9 mon_initial_members = monitoring1, monitoring2, monitoring3 fsid = 5f22ab94-8d96-48c2-88d3-cff7bad443a9 public network = 172.30.30.0/24 [mon.monitoring1] host = monitoring1 addr = 172.30.30.8:6789 [mon.monitoring2] host = monitoring2 addr = 172.30.30.9:6789 [mon.monitoring3] host = monitoring3 addr = 172.30.30.10:6789 [filestore] filestore max sync interval = 10 [osd] osd recovery max active = 1 osd journal size = 15360 osd op threads = 40 osd disk threads = 40 [osd.0] host = datastore1 [osd.1] host = datastore1 [osd.2] host = datastore1 [osd.3] host = datastore1 [osd.4] host = datastore1 [osd.5] host = datastore1 [osd.6] host = datastore1 [osd.7] host = datastore1 [osd.8] host = datastore1 [osd.9] host = datastore1 [osd.10] host = datastore2 [osd.11] host = datastore2 [osd.11] host = datastore2 [osd.12] host = datastore2 [osd.13] host = datastore2 [osd.14] host = datastore2 [osd.15] host = datastore2 [osd.16] host = datastore2 [osd.17] host = datastore2 [osd.18] host = datastore2 [osd.19] host = datastore2 [osd.20] host = datastore3 [osd.21] host = datastore3 [osd.22] host = datastore3 [osd.23] host = datastore3 [osd.24] host = datastore3 [osd.25] host = datastore3 [osd.26] host = datastore3 [osd.27] host = datastore3 [osd.28] host = datastore3 [osd.29] host = datastore3 [osd.30] host = datastore4 [osd.31] host = datastore4 [osd.32] host = datastore4 [osd.33] host = datastore4 [osd.34] host = datastore4 [osd.35] host = datastore4 [osd.36] host = datastore4 [osd.37] host = datastore4 [osd.38] host = datastore4 [osd.39] host = datastore4 [osd.0] host = datastore5 [osd.40] host = datastore5 [osd.41] host = datastore5 [osd.42] host = datastore5 [osd.43] host = datastore5 [osd.44] host = datastore5 [osd.45] host = datastore5 [osd.46] host = datastore5 [osd.47] host = datastore5 [osd.48] host = datastore5 We have 3 pools: -> 2 x 1000 pgs with 2 Replicas distributing the data equally to two racks (Used for datastore 1-4) -> 1 x 100 pgs without replication; data only stored on datastore 5. This Pool is used to compare the performance on local disks without networking Here are the performance values, which I get using fio-Bench on a 32GB rbd: On 1000 pgs-Pool with distribution fio --bs=1M --rw=randwrite --ioengine=libaio --direct=1 --iodepth=32 --runtime=60 --name=/dev/rbd/pool1/bench1 fio-2.0.13 Starting 1 process Jobs: 1 (f=1): [w] [100.0% done] [0K/312.0M/0K /s] [0 /312 /0 iops] [eta 00m:00s] /dev/rbd/pool1/bench1: (groupid=0, jobs=1): err= 0: pid=21675: Fri Jul 4 11:03:52 2014 write: io=21071MB, bw=358989KB/s, iops=350 , runt= 60104msec slat (usec): min=127 , max=8040 , avg=511.49, stdev=216.27 clat (msec): min=5 , max=4018 , avg=90.74, stdev=215.83 lat (msec): min=6 , max=4018 , avg=91.25, stdev=215.83 clat percentiles (msec): | 1.00th=[8], 5.00th=[9], 10.00th=[ 11], 20.00th=[ 15], | 30.00th=[ 21], 40.00th=[ 30], 50.00th=[ 45], 60.00th=[ 63], | 70.00th=[ 83], 80.00th=[ 105], 90.00th=[ 129], 95.00th=[ 190], | 99.00th=[ 1254], 99.50th=[ 1680], 99.90th=[ 2409], 99.95th=[ 2638], | 99.99th=[ 3556] bw (KB/s) : min=68210, max=479232, per=100.00%, avg=368399.55, stdev=84457.12 lat (msec) : 10=9.50%, 20=20.02%, 50=23.56%, 100=24.56%, 250=18.09% lat (msec) : 500=1.39%, 750=0.81%, 1000=0.65%, 2000=1.13%, >=2000=0.29% cpu : usr=11.17%, sys=7.46%, ctx=17772, majf=0, minf=24 IO depths: 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=99.9%, >=
Re: [ceph-users] OSD Data not evenly distributed
Hi, Increasing PG number for pools that hold data might help if you didn't do that already. Check out this thread: http://lists.ceph.com/pipermail/ceph-users-ceph.com/2014-January/027094.html You might find some tips there (although it was pre firefly). W dniu 28.06.2014 o 14:44 Jianing Yang pisze: Hi, all My cluster has been running for about 4 month now. I have about 108 osds and all are 600G SAS Disk. Their disk usage is between 70% and 85%. It seems that ceph cannot distribute data evenly by default settings. Is there any configuration that helps distribute data more evenly? Thanks very much ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Konrad Gutkowski ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Issues related to Ceph (firefly)
Hi, W dniu 03.06.2014 o 13:47 pisze: Von: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] Im Auftrag von Sherry Shahbazi Gesendet: Dienstag, 3. Juni 2014 13:35 An: ceph-users@lists.ceph.com Betreff: [ceph-users] Issues related to Ceph (firefly) Hi guys, There are couple of issues that I faced: 1) Ceph automatically changes /etc/apt/sources.list.d/ceph.list! no matter what did I set (emperor) it would change it to firefly. 2) On one of my hosts, /etc/ceph will not be created, so I have to create /etc/ceph manually and push ceph.conf to it! 3) PGs stuck inactive, and it seems to be take forever to create them! 4) "ceph osd dump | grep size" shows size=3! while I set min_size and max_size to 2!!! I also set "osd pool default size = 2" in ceph.conf, but that also did not help! ceph osd pool set size 2 This might also solve (2) Any ideas regarding any of these issue is highly appreciated. Thanks, Sherry [Zeller, Jan (ID)] Hi, I can confirm this. Have (still) same problem (especially regarding pt. 3) with Ubuntu 14.04 and with firefly in general using 3 x OSDs and 1 x MON. The only version which I was able to install was 0.72.2 --- Jan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Konrad Gutkowski ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] creative ideas for ssd garbage collection on OSD
W dniu 10.03.2014 o 07:54 Stefan Priebe - Profihost AG pisze: Am 07.03.2014 16:56, schrieb Konrad Gutkowski: Hi, If those are journal drives you could have n+1 ssd's and swap them at some intervals, could introduce more problems. If it required data to be synchronized one could operate it with degraded raid1 to swap disks, would introduce unnecessary wear though... Just a thought. No they're running as OSD disks. It depends on your operational constraints then, don't know how quickly ssd's can complete GC, but if you can take the whole cluster offline for a short while if should be simple and clean way to solve this. But I guess you already thought about it. My thinking is.. shouldn't TRIM suffice? (internet tells me it should) W dniu 07.03.2014 o 15:22 Stefan Priebe - Profihost AG pisze: Hello list, a lot of SSDs do their garbage collection only if the SSD is idle. But in a ceph cluster the ssd gets never idle. Does anybody have creative ideas how to solve this? Greets, Stefan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Regards, Konrad Gutkowski ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] creative ideas for ssd garbage collection on OSD
Hi, If those are journal drives you could have n+1 ssd's and swap them at some intervals, could introduce more problems. If it required data to be synchronized one could operate it with degraded raid1 to swap disks, would introduce unnecessary wear though... Just a thought. W dniu 07.03.2014 o 15:22 Stefan Priebe - Profihost AG pisze: Hello list, a lot of SSDs do their garbage collection only if the SSD is idle. But in a ceph cluster the ssd gets never idle. Does anybody have creative ideas how to solve this? Greets, Stefan ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Regards, Konrad Gutkowski ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] pushing rbd write performance
Hi all, I've been testing a small cluster with 4 nodes and ssd storage, it seems i cant manage to push it beyond ~250 MB/s qemu - ~300MB/s krbd with quite erratic bandwidth behavior. 4 nodes, 3 ssd's each, cut to 160GB partition to get uniform i/o distribution. Raw disk performance is around <300MB/s + up to 40k iops (various intel dries). Network: 10g eth, jumbo frames 9k, txqueue 20k Backing fs is btrfs, 10G journal Rbd was set up with 8m object size, no striping. Both krbd i qemu was ran on one of the nodes, used fio for benchmarking with various settings. Any ideas? relevant config part (some values may not make much sense since i tried many ways to push it harder): [client] rbd_cache = true rbd_cache_writethrough_until_flush = false rbd_cache_size = 2 GiB rbd_cache_max_dirty = 2 GiB rbd_cache_target_dirty = 256 MiB rbd_cache_max_dirty_age = 1.0 [osd] max open files = 112400 osd op threads = 12 osd disk threads = 1 journal dio = true journal aio = true journal max write bytes = 1 GiB journal max write entries = 5 journal queue max bytes = 1 GiB journal queue max ops = 5 filestore op threads = 6 filestore queue max ops = 4096 filestore queue max bytes = 16 MiB filestore queue committing max ops = 4096 filestore queue committing max bytes = 16 MiB filestore min sync interval = 15 filestore max sync interval = 15 filestore fd cache size = 10240 filestore journal parallel = true -- Regards, Konrad Gutkowski ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] RBD+KVM problems with sequential read
Hi,W dniu 07.02.2014 o 08:14 Ирек Фасихов pisze:[...]Why might such a low speed sequential read? Do ideas on this issue? Iirc you need to set your readahead for the device higher (inside the vm) to compensate for network rtt.blockdev --setra x /dev/vdaThanks.-- С уважением, Фасихов Ирек НургаязовичМоб.: +79229045757 Regards,Konrad Gutkowski___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
