Re: [ceph-users] Compile from source with Kinetic support
Hello, Ceph is still not compiling when if I add Kinetic support option. Could you have a look on the log and tell me what's missing ? -- Best regards, Julien On 12/02/2014 09:53 AM, Julien Lutran wrote: It's ok for KeyValueDB.cc now, but I have another problem with src/os/KineticStore.h : http://pastebin.com/nhSDQs70 On 12/01/2014 05:58 PM, Haomai Wang wrote: Sorry, it's a typo /WITH_KINETIC/HAVE_KINETIC/ :-) On Tue, Dec 2, 2014 at 12:51 AM, Julien Lutran julien.lut...@ovh.net mailto:julien.lut...@ovh.net wrote: Sorry, It didn't change anything : root@host:~/sources/ceph# head -12 src/os/KeyValueDB.cc // -*- mode:C++; tab-width:8; c-basic-offset:2; indent-tabs-mode:t -*- // vim: ts=8 sw=2 smarttab #include KeyValueDB.h #include LevelDBStore.h #ifdef HAVE_LIBROCKSDB #include RocksDBStore.h #endif #ifdef WITH_KINETIC #include KineticStore.h #endif root@host:~/sources/ceph# make [...] CXX os/libos_la-KeyValueDB.lo os/KeyValueDB.cc: In static member function 'static KeyValueDB* KeyValueDB::create(CephContext*, const string, const string)': os/KeyValueDB.cc:21:16: error: expected type-specifier before 'KineticStore' return new KineticStore(cct); ^ os/KeyValueDB.cc:21:16: error: expected ';' before 'KineticStore' os/KeyValueDB.cc:21:32: error: 'KineticStore' was not declared in this scope return new KineticStore(cct); ^ os/KeyValueDB.cc: In static member function 'static int KeyValueDB::test_init(const string, const string)': os/KeyValueDB.cc:39:12: error: 'KineticStore' has not been declared return KineticStore::_test_init(g_ceph_context); ^ make[3]: *** [os/libos_la-KeyValueDB.lo] Error 1 On 12/01/2014 03:22 PM, Haomai Wang wrote: #ifdef WITH_KINETIC #include KineticStore.h #endif -- Best Regards, Wheat ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] 'rbd list' stuck
Hi Why command 'rbd list'executed on monitor stuck,any prompt should be appreciated! Backtree: [810bfdee] futex_wait_queue_me+0xde/0x140 [810c0969] futex_wait+0x179/0x280 [810c297e] do_futex+0xfe/0x5e0 [810c2ee0] SyS_futex+0x80/0x180 [815f2119] system_call_fastpath+0x16/0x1b [] 0x Best Regards! YangBin ZTE Information Security Notice: The information contained in this mail (and any attachment transmitted herewith) is privileged and confidential and is intended for the exclusive use of the addressee(s). If you are not an intended recipient, any disclosure, reproduction, distribution or other dissemination or use of the information contained is strictly prohibited. If you have received this mail in error, please delete it and notify us immediately. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Help with Integrating Ceph with various Cloud Storage
Hello Manoj My answers to your queries. # For testing purpose you can install Ceph on virtual machines ( multiple instances of virtual box for multiple mon, osd ). Its good to practice Ceph with multiple MON and OSD. # For real data storage , please use physical servers , virtual servers are good to go only for testing and learning purpose. # S3,azure blob,swift are object storage which are different from each other , similarly Ceph also supports object storage and you can access Ceph object storage with S3 and Swift protocols. You cannot use Ceph storage with amazon,azure but you can use ceph with Swift and s3 API. # I am not much familiar with cloud storage , but based on the experience of openstack , libvirt should be installed from cloudstack servers only , Ceph servers should only do storage for you. Karan Singh Systems Specialist , Storage Platforms CSC - IT Center for Science, Keilaranta 14, P. O. Box 405, FIN-02101 Espoo, Finland mobile: +358 503 812758 tel. +358 9 4572001 fax +358 9 4572302 http://www.csc.fi/ On 17 Dec 2014, at 09:57, Manoj Singh respond2ma...@gmail.com wrote: Hi All, I am new to Ceph. Due to physical machines shortage I have installed Ceph cluster with single OSD and MON in a single Virtual Machine. I have few queries as below: 1. Whether having the Ceph setup on a VM is fine or it require to be on Physical server. 2. Since Amazon S3, Azure Blob Storage, Swift are Object based Storage, what is the feasibility of attaching these Cloud Storage to Ceph and to be able to allocate disc space from the same while creating new VM from local CloudStack or OpenStack 3. When I am integrating CloudStack with Ceph whether libvert should be installed on the CloudStack management server or on Ceph server. From diagram given in Ceph documentation it's bit confusing. Thank you in advance. your help shall be really appreciated. Best Regards, Manoj Kumar ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com smime.p7s Description: S/MIME cryptographic signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Is cache tiering production ready?
Hi, Since firefly, ceph can support cache tiering. Cache tiering: support for creating ‘cache pools’ that store hot, recently accessed objects with automatic demotion of colder data to a base tier. Typically the cache pool is backed by faster storage devices like SSDs. I'm testing cache tiering, and everything goes well. But rados -h shows that: CACHE POOLS: (for testing/development only) cache-flush obj-name flush cache pool object (blocking) cache-try-flush obj-name flush cache pool object (non-blocking) cache-evict obj-name evict cache pool object cache-flush-evict-allflush+evict all objects cache-try-flush-evict-alltry-flush+evict all objects rados -v ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) Is cache tiering production ready? Thanks a lot! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] File System stripping data
On 12/17/2014 11:21 AM, John Spray wrote: On Wed, Dec 17, 2014 at 2:07 AM, Kevin Shiah agan...@gmail.com wrote: setfattr -n ceph.dir.layout.stripe_count -v 2 dir And return: setfattr: dir: Operation not supported Works for me on master. What ceph version are you using? I just tried something similar on Giant (0.87) and I saw this in the logs: parse_layout_vxattr name layout.pool value 'cephfs_svo' invalid data pool 3 reply request -22 I resolves the pool to a ID, but then it's unable to set it? John ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Wido den Hollander 42on B.V. Ceph trainer and consultant Phone: +31 (0)20 700 9902 Skype: contact42on ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] File System stripping data
On 12/17/2014 12:35 PM, John Spray wrote: On Wed, Dec 17, 2014 at 10:25 AM, Wido den Hollander w...@42on.com wrote: I just tried something similar on Giant (0.87) and I saw this in the logs: parse_layout_vxattr name layout.pool value 'cephfs_svo' invalid data pool 3 reply request -22 I resolves the pool to a ID, but then it's unable to set it? Was the 'cephfs_svo' pool already added as a data pool with ceph mds add_data_pool? Ah, indeed. Working fine right now. Same goes for any other layout settings. There are paths where if a pool was added very recently, MDSs/clients might not know about the pool yet and can generate errors like this. John -- Wido den Hollander 42on B.V. Ceph trainer and consultant Phone: +31 (0)20 700 9902 Skype: contact42on ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Erasure coded PGs incomplete
Hello Loic, Thanks for you help, I’ve take a look to my crush map and I replace step chooseleaf indep 0 type osd” by step choose indep 0 type osd” and all PGs was created successfully. At. Italo Santos http://italosantos.com.br/ On Tuesday, December 16, 2014 at 8:39 PM, Loic Dachary wrote: Hi, The 2147483647 means that CRUSH did not find enough OSD for a given PG. If you check the crush rule associated with the erasure coded pool, you will most probably find why. Cheers On 16/12/2014 23:32, Italo Santos wrote: Hello, I'm trying to create an erasure pool following http://docs.ceph.com/docs/master/rados/operations/erasure-code/, but when I try create a pool with a specifc erasure-code-profile (myprofile) the PGs became on incomplete state. Anyone can help me? Below the profile I created: root@ceph0001:~# ceph osd erasure-code-profile get myprofile directory=/usr/lib/ceph/erasure-code k=6 m=2 plugin=jerasure technique=reed_sol_van The status of cluster: root@ceph0001:~# ceph health HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean health detail: root@ceph0001:~# ceph health detail HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean pg 2.9 is stuck inactive since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck inactive since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck inactive since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck inactive since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck inactive since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck inactive since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck inactive since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck inactive since forever, current state incomplete, last acting [9,15,2147483647,4,2,2147483647,2147483647,2147483647] pg 2.1 is stuck inactive since forever, current state incomplete, last acting [2,4,2147483647,13,2147483647,10,2147483647,2147483647] pg 2.0 is stuck inactive since forever, current state incomplete, last acting [14,1,2147483647,4,10,2147483647,2147483647,2147483647] pg 2.3 is stuck inactive since forever, current state incomplete, last acting [14,11,6,2147483647,2147483647,2147483647,2,2147483647] pg 2.2 is stuck inactive since forever, current state incomplete, last acting [13,5,11,2147483647,2147483647,3,2147483647,2147483647] pg 2.9 is stuck unclean since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck unclean since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck unclean since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck unclean since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck unclean since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck unclean since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck unclean since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck unclean since forever, current state incomplete, last acting [9,15,2147483647,4,2,2147483647,2147483647,2147483647] pg 2.1 is stuck unclean since forever, current state incomplete, last acting [2,4,2147483647,13,2147483647,10,2147483647,2147483647] pg 2.0 is stuck unclean since forever, current state incomplete, last acting [14,1,2147483647,4,10,2147483647,2147483647,2147483647] pg 2.3 is stuck unclean since forever, current state incomplete, last acting [14,11,6,2147483647,2147483647,2147483647,2,2147483647] pg 2.2 is stuck unclean since forever, current state incomplete, last acting [13,5,11,2147483647,2147483647,3,2147483647,2147483647] pg 2.9 is incomplete, acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] (reducing pool ecpool min_size from 6 may help; search ceph.com/docs (http://ceph.com/docs) for 'incomplete') pg 2.8 is incomplete, acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] (reducing pool ecpool min_size from 6 may help; search ceph.com/docs (http://ceph.com/docs) for
[ceph-users] cephfs not mounting on boot
Both fuse and kernel module fail to mount, The mons mds are on two other nodes, so they are available when this node is booting. They can be mounted manually after boot. my fstab: idmin /mnt/cephfs fuse.ceph defaults,nonempty,_netdev 0 0 vnb.proxmox.softlog,vng.proxmox.softlog,vnt.proxmox.softlog:/ /mnt/test ceph_netdev,defaults,namemin,secretfile=/etc/pve/priv/admin.secret 0 0 -- Lindsay signature.asc Description: This is a digitally signed message part. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs kernel module reports error on mount
Hmm, from a quick google it appears you are not the only one who has seen this symptom with mount.ceph. Our mtab code appears to have diverged a bit from the upstream util-linux repo, so it seems entirely possible we have a bug in ours somewhere. I've opened http://tracker.ceph.com/issues/10351 to track it. Cheers, John On Wed, Dec 17, 2014 at 1:31 PM, Lindsay Mathieson lindsay.mathie...@gmail.com wrote: mount reports: mount: error writing /etc/mtab: Invalid argument fstab entry is: vnb.proxmox.softlog,vng.proxmox.softlog,vnt.proxmox.softlog:/ /mnt/test ceph _netdev,defaults,name=admin,secretfile=/etc/pve/priv/admin.secret 0 0 However the mounts is successful and a mtab entry is made. debian wheezy, ceph 0.87 -- Lindsay ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs not mounting on boot
Can you tell us more about how they fail? Error messages on console, anything in syslog? In the absence of other clues, you might want to try checking that the network is coming up before ceph tries to mount. John On Wed, Dec 17, 2014 at 1:34 PM, Lindsay Mathieson lindsay.mathie...@gmail.com wrote: Both fuse and kernel module fail to mount, The mons mds are on two other nodes, so they are available when this node is booting. They can be mounted manually after boot. my fstab: id=admin /mnt/cephfs fuse.ceph defaults,nonempty,_netdev 0 0 vnb.proxmox.softlog,vng.proxmox.softlog,vnt.proxmox.softlog:/ /mnt/test ceph _netdev,defaults,name=admin,secretfile=/etc/pve/priv/admin.secret 0 0 -- Lindsay ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] cephfs not mounting on boot
On Wed, 17 Dec 2014 02:02:52 PM John Spray wrote: Can you tell us more about how they fail? Error messages on console, anything in syslog? Not quite sure what to look for, but I did a quick scan on ceph through dmesg syslog, nothing stood out In the absence of other clues, you might want to try checking that the network is coming up before ceph tries to mount. Now I think on it, that might just be it - I seem to recall a similar problem with cifs mounts, despite having the _netdev option. I had to issue a mount in /etc/network/if-up.d/ I'll test than and get back to you -- Lindsay signature.asc Description: This is a digitally signed message part. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is cache tiering production ready?
Cache tiering is a stable, functioning system. Those particular commands are for testing and development purposes, not something you should run (although they ought to be safe). -Greg On Wed, Dec 17, 2014 at 1:44 AM Yujian Peng pengyujian5201...@126.com wrote: Hi, Since firefly, ceph can support cache tiering. Cache tiering: support for creating ‘cache pools’ that store hot, recently accessed objects with automatic demotion of colder data to a base tier. Typically the cache pool is backed by faster storage devices like SSDs. I'm testing cache tiering, and everything goes well. But rados -h shows that: CACHE POOLS: (for testing/development only) cache-flush obj-name flush cache pool object (blocking) cache-try-flush obj-name flush cache pool object (non-blocking) cache-evict obj-name evict cache pool object cache-flush-evict-allflush+evict all objects cache-try-flush-evict-alltry-flush+evict all objects rados -v ceph version 0.80.7 (6c0127fcb58008793d3c8b62d925bc91963672a3) Is cache tiering production ready? Thanks a lot! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Ceph rbd mapped but files all have 0 byte size
Dear All, We have set up ceph and used it for about one year already. Here is a summary of the setting. We used 3 servers to run the ceph. cs02, cs03, cs04 Here is how we set up the ceph: 1. We created several OSDs on three of these servers. using command like: ceph-deploy osd create cs02:/dev/sdc …. cs03:/dev/… cs04:/dev/…. 2. And have created MDS on cs02: ceph-deploy mds create ilab-cs02 3. After that, we have created a RADOS block device on cs02 by rbd create rbd-research --size 1024 4. Then mapped rbd-research sudo tbd map rbd-resrearch —pool rbd 5. Then make file system sudo mkfs.ext4 /dev/rbd/rbd/rbd-research 6. Then mkdir and mount the rbd by adding this line to /etc/fstab /dev/rbd/rbd/rbd-research /mnt/retinadata ext4 defaults,users 0 2 7. Then mount mount /mnt/retinadata It worked reliably until recently we had a power off of our servers accidentally. After power recovered. cs03, cs04 were automatically boot up, while cs02 were not automatically boot up. There is a message shown on cs02 telling something like “not able to mount /mnt/retinadata, device not found, press S to ignore and continue booting, press M to manual configure”. We selected S and booted up the system. Then we found that, /mnt/retinadata was not mount and the rbd image at /dev/rbd/rbd/rbd1 was not there. We map the rbd image once again by. sudo tbd map rbd-research —pool rbd Then we were able to mount /mnt/retinadata But the result we have now is : 1. All the file system structures are there. 2. All the files are of 0 byte size. Could anybody help on this issue? Thank you very much in advance. Some more information. We tried to reboot cs02 again. And we see a full screen of error message like: [44038.215233] libceph: connect 192.168.1.31:6789 socket error on write [44038.215308] libceph: mon1 192.1.168.31:6789 error -101 libceph: connect 192.168.1.41:6812 error -101 libceph: osd22 192.168.1.41:6812 socket error on write Best Regards, Cyan Cheng ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Help with SSDs
Hey there, Is there a good work around if our SSDs are not handling D_SYNC very well? We invested a ton of money into Samsung 840 EVOS and they are not playing well with D_SYNC. Would really appreciate the help! Thank you, Bryson ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Incorrect description in document at chapter 'Crush Operation'?
Hi,all
I found content below at
http://ceph.com/docs/master/rados/operations/crush-map :
step choose firstn {num} type {bucket-type}
Description: Selects the number of buckets of the given type. The
number is usually the number of replicas in the pool (i.e., pool size).
If {num} == 0, choose pool-num-replicas buckets (all available).
If {num} 0 pool-num-replicas, choose that many buckets.
If {num} 0, it means pool-num-replicas - {num}.
However, corresponding source code is that (V0.87, mapper.c ,
crush_do_rule(...)) :
if (numrep = 0) {
numrep += result_max;
if (numrep = 0)
continue;
}
I think the correct description is 'If {num} 0, it means pool-num-replicas
+ {num}'.
--
TongLei | XOR Media
Storage Application Department of XOR Media China
Tel. 86-21-61213323 Fax. 86-21-52065237
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Ceph rbd mapped but files all have 0 byte size
On Tue, Dec 16, 2014 at 6:19 AM, Cyan Cheng cheng.1...@gmail.com wrote: Dear All, We have set up ceph and used it for about one year already. Here is a summary of the setting. We used 3 servers to run the ceph. cs02, cs03, cs04 Here is how we set up the ceph: 1. We created several OSDs on three of these servers. using command like: ceph-deploy osd create cs02:/dev/sdc …. cs03:/dev/… cs04:/dev/…. 2. And have created MDS on cs02: ceph-deploy mds create ilab-cs02 3. After that, we have created a RADOS block device on cs02 by rbd create rbd-research --size 1024 4. Then mapped rbd-research sudo tbd map rbd-resrearch —pool rbd I assume that's rbd map. Mapping images on the same physical box that's also running OSDs works in general but isn't a very good idea. 5. Then make file system sudo mkfs.ext4 /dev/rbd/rbd/rbd-research 6. Then mkdir and mount the rbd by adding this line to /etc/fstab /dev/rbd/rbd/rbd-research /mnt/retinadata ext4 defaults,users 0 2 7. Then mount mount /mnt/retinadata It worked reliably until recently we had a power off of our servers accidentally. After power recovered. cs03, cs04 were automatically boot up, while cs02 were not automatically boot up. There is a message shown on cs02 telling something like “not able to mount /mnt/retinadata, device not found, press S to ignore and continue booting, press M to manual configure”. We selected S and booted up the system. Then we found that, /mnt/retinadata was not mount and the rbd image at /dev/rbd/rbd/rbd1 was not there. We map the rbd image once again by. sudo tbd map rbd-research —pool rbd Then we were able to mount /mnt/retinadata But the result we have now is : 1. All the file system structures are there. 2. All the files are of 0 byte size. All files or just those that you (your system) were working with at the time of the power reset? Could anybody help on this issue? Thank you very much in advance. Some more information. We tried to reboot cs02 again. And we see a full screen of error message like: [44038.215233] libceph: connect 192.168.1.31:6789 socket error on write [44038.215308] libceph: mon1 192.1.168.31:6789 error -101 libceph: connect 192.168.1.41:6812 error -101 libceph: osd22 192.168.1.41:6812 socket error on write That's Network is unreachable so probably something's wrong with your network. Thanks, Ilya ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Certificate has expired
Hi, We have some problems with ceph-deploy install node This is the error I get when I run the installation: [mon01][INFO ] Running command: sudo rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc [mon01][INFO ] Running command: sudo rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc [mon01][WARNIN] curl: (60) Peer's Certificate has expired. Can you fix this problem? We can run the install with other parameters? Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Certificate has expired
Strange, when I visit https://ceph.com, I get a certificate that doesn't expire until 10 February 2015. Perhaps check the clock on your node isn't in the future? John On Wed, Dec 17, 2014 at 4:16 PM, Emilio emilio.mor...@adam.es wrote: Hi, We have some problems with ceph-deploy install node This is the error I get when I run the installation: [mon01][INFO ] Running command: sudo rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc [mon01][INFO ] Running command: sudo rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc [mon01][WARNIN] curl: (60) Peer's Certificate has expired. Can you fix this problem? We can run the install with other parameters? Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Erasure coded PGs incomplete
Hi, Thanks for the update : good news are much appreciated :-) Would you have time to review the documentation at https://github.com/ceph/ceph/pull/3194/files ? It was partly motivated by the problem you had. Cheers On 17/12/2014 14:03, Italo Santos wrote: Hello Loic, Thanks for you help, I’ve take a look to my crush map and I replace step chooseleaf indep 0 type osd” by step choose indep 0 type osd” and all PGs was created successfully. At. *Italo Santos* http://italosantos.com.br/ On Tuesday, December 16, 2014 at 8:39 PM, Loic Dachary wrote: Hi, The 2147483647 means that CRUSH did not find enough OSD for a given PG. If you check the crush rule associated with the erasure coded pool, you will most probably find why. Cheers On 16/12/2014 23:32, Italo Santos wrote: Hello, I'm trying to create an erasure pool following http://docs.ceph.com/docs/master/rados/operations/erasure-code/, but when I try create a pool with a specifc erasure-code-profile (myprofile) the PGs became on incomplete state. Anyone can help me? Below the profile I created: root@ceph0001:~# ceph osd erasure-code-profile get myprofile directory=/usr/lib/ceph/erasure-code k=6 m=2 plugin=jerasure technique=reed_sol_van The status of cluster: root@ceph0001:~# ceph health HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean health detail: root@ceph0001:~# ceph health detail HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean pg 2.9 is stuck inactive since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck inactive since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck inactive since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck inactive since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck inactive since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck inactive since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck inactive since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck inactive since forever, current state incomplete, last acting [9,15,2147483647,4,2,2147483647,2147483647,2147483647] pg 2.1 is stuck inactive since forever, current state incomplete, last acting [2,4,2147483647,13,2147483647,10,2147483647,2147483647] pg 2.0 is stuck inactive since forever, current state incomplete, last acting [14,1,2147483647,4,10,2147483647,2147483647,2147483647] pg 2.3 is stuck inactive since forever, current state incomplete, last acting [14,11,6,2147483647,2147483647,2147483647,2,2147483647] pg 2.2 is stuck inactive since forever, current state incomplete, last acting [13,5,11,2147483647,2147483647,3,2147483647,2147483647] pg 2.9 is stuck unclean since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck unclean since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck unclean since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck unclean since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck unclean since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck unclean since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck unclean since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck unclean since forever, current state incomplete, last acting [9,15,2147483647,4,2,2147483647,2147483647,2147483647] pg 2.1 is stuck unclean since forever, current state incomplete, last acting [2,4,2147483647,13,2147483647,10,2147483647,2147483647] pg 2.0 is stuck unclean since forever, current state incomplete, last acting [14,1,2147483647,4,10,2147483647,2147483647,2147483647] pg 2.3 is stuck unclean since forever, current state incomplete, last acting [14,11,6,2147483647,2147483647,2147483647,2,2147483647] pg 2.2 is stuck unclean since forever, current state incomplete, last acting [13,5,11,2147483647,2147483647,3,2147483647,2147483647] pg 2.9 is incomplete, acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] (reducing pool ecpool min_size from 6 may help; search ceph.com/docs http://ceph.com/docs for 'incomplete') pg 2.8 is
Re: [ceph-users] Certificate has expired
Yes, sorry this server was in the past! Thx! On 17/12/14 17:40, John Spray wrote: Strange, when I visit https://ceph.com, I get a certificate that doesn't expire until 10 February 2015. Perhaps check the clock on your node isn't in the future? John On Wed, Dec 17, 2014 at 4:16 PM, Emilio emilio.mor...@adam.es wrote: Hi, We have some problems with ceph-deploy install node This is the error I get when I run the installation: [mon01][INFO ] Running command: sudo rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc [mon01][INFO ] Running command: sudo rpm --import https://ceph.com/git/?p=ceph.git;a=blob_plain;f=keys/release.asc [mon01][WARNIN] curl: (60) Peer's Certificate has expired. Can you fix this problem? We can run the install with other parameters? Thanks! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- adamvozip.es Emilio Moreno Fernández Departamento de Sistemas www.adam.es La experiencia nos avala. 25 años de historia. Tel. 902 902 685 Carrer Artesans, 7 - Parc Tecnològic del Vallès 08290 Cerdanyola del Vallès - Barcelona www.adam.es http://www.adam.es Advertencia legal: Este mensaje y, en su caso, los ficheros anexos son confidenciales, especialmente en lo que respecta a los datos personales, y se dirigen exclusivamente al destinatario referenciado. Si usted no lo es y lo ha recibido por error o tiene conocimiento del mismo por cualquier motivo, le rogamos que nos lo comunique por este medio y proceda a destruirlo o borrarlo, y que en todo caso se abstenga de utilizar, reproducir, alterar, archivar o comunicar a terceros el presente mensaje y ficheros anexos, todo ello bajo pena de incurrir en responsabilidades legales. El emisor no garantiza la integridad, rapidez o seguridad del presente correo, ni se responsabiliza de posiblesperjuicios derivados de la captura, incorporaciones de virus o cualesquiera otras manipulaciones efectuadas por terceros. ecotech No imprimas si no es necesario. Protejamos el Medio Ambiente. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Erasure coded PGs incomplete
Hello, I’ve take a look to this documentation (which help a lot) and if I understand right, when I set a profile like: === ceph osd erasure-code-profile set isilon k=8 m=2 ruleset-failure-domain=host === And create a pool following the recommendations on doc, I’ll need (100*16)/2 = 800 PGs, I’ll need the sufficient number of hosts to support create total PGs? Regards. Italo Santos http://italosantos.com.br/ On Wednesday, December 17, 2014 at 2:42 PM, Loic Dachary wrote: Hi, Thanks for the update : good news are much appreciated :-) Would you have time to review the documentation at https://github.com/ceph/ceph/pull/3194/files ? It was partly motivated by the problem you had. Cheers On 17/12/2014 14:03, Italo Santos wrote: Hello Loic, Thanks for you help, I’ve take a look to my crush map and I replace step chooseleaf indep 0 type osd” by step choose indep 0 type osd” and all PGs was created successfully. At. *Italo Santos* http://italosantos.com.br/ On Tuesday, December 16, 2014 at 8:39 PM, Loic Dachary wrote: Hi, The 2147483647 means that CRUSH did not find enough OSD for a given PG. If you check the crush rule associated with the erasure coded pool, you will most probably find why. Cheers On 16/12/2014 23:32, Italo Santos wrote: Hello, I'm trying to create an erasure pool following http://docs.ceph.com/docs/master/rados/operations/erasure-code/, but when I try create a pool with a specifc erasure-code-profile (myprofile) the PGs became on incomplete state. Anyone can help me? Below the profile I created: root@ceph0001:~# ceph osd erasure-code-profile get myprofile directory=/usr/lib/ceph/erasure-code k=6 m=2 plugin=jerasure technique=reed_sol_van The status of cluster: root@ceph0001:~# ceph health HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean health detail: root@ceph0001:~# ceph health detail HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean pg 2.9 is stuck inactive since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck inactive since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck inactive since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck inactive since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck inactive since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck inactive since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck inactive since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck inactive since forever, current state incomplete, last acting [9,15,2147483647,4,2,2147483647,2147483647,2147483647] pg 2.1 is stuck inactive since forever, current state incomplete, last acting [2,4,2147483647,13,2147483647,10,2147483647,2147483647] pg 2.0 is stuck inactive since forever, current state incomplete, last acting [14,1,2147483647,4,10,2147483647,2147483647,2147483647] pg 2.3 is stuck inactive since forever, current state incomplete, last acting [14,11,6,2147483647,2147483647,2147483647,2,2147483647] pg 2.2 is stuck inactive since forever, current state incomplete, last acting [13,5,11,2147483647,2147483647,3,2147483647,2147483647] pg 2.9 is stuck unclean since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck unclean since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck unclean since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck unclean since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck unclean since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck unclean since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck unclean since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck unclean since forever, current state incomplete, last acting [9,15,2147483647,4,2,2147483647,2147483647,2147483647]
Re: [ceph-users] Erasure coded PGs incomplete
On 17/12/2014 18:18, Italo Santos wrote: Hello, I’ve take a look to this documentation (which help a lot) and if I understand right, when I set a profile like: === ceph osd erasure-code-profile set isilon k=8 m=2 ruleset-failure-domain=host === And create a pool following the recommendations on doc, I’ll need (100*16)/2 = 800 PGs, I’ll need the sufficient number of hosts to support create total PGs? You will need k+m = 10 host per OSD. If you only have 10 hosts that should be ok and the 800 PGs will use these 10 OSD in various orders. It also means that you will end up having 800 PG per OSD which is a bit too mche. If you have 20 OSDs that will be better : each PG will get 10 OSD out of 20 and each OSD will have 400 PGs. Ideally you want the number of PG per OSD to be in the range (approximately) [20,300]. Cheers Regards. *Italo Santos* http://italosantos.com.br/ On Wednesday, December 17, 2014 at 2:42 PM, Loic Dachary wrote: Hi, Thanks for the update : good news are much appreciated :-) Would you have time to review the documentation at https://github.com/ceph/ceph/pull/3194/files ? It was partly motivated by the problem you had. Cheers On 17/12/2014 14:03, Italo Santos wrote: Hello Loic, Thanks for you help, I’ve take a look to my crush map and I replace step chooseleaf indep 0 type osd” by step choose indep 0 type osd” and all PGs was created successfully. At. *Italo Santos* http://italosantos.com.br/ On Tuesday, December 16, 2014 at 8:39 PM, Loic Dachary wrote: Hi, The 2147483647 means that CRUSH did not find enough OSD for a given PG. If you check the crush rule associated with the erasure coded pool, you will most probably find why. Cheers On 16/12/2014 23:32, Italo Santos wrote: Hello, I'm trying to create an erasure pool following http://docs.ceph.com/docs/master/rados/operations/erasure-code/, but when I try create a pool with a specifc erasure-code-profile (myprofile) the PGs became on incomplete state. Anyone can help me? Below the profile I created: root@ceph0001:~# ceph osd erasure-code-profile get myprofile directory=/usr/lib/ceph/erasure-code k=6 m=2 plugin=jerasure technique=reed_sol_van The status of cluster: root@ceph0001:~# ceph health HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean health detail: root@ceph0001:~# ceph health detail HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean pg 2.9 is stuck inactive since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck inactive since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck inactive since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck inactive since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck inactive since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck inactive since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck inactive since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck inactive since forever, current state incomplete, last acting [9,15,2147483647,4,2,2147483647,2147483647,2147483647] pg 2.1 is stuck inactive since forever, current state incomplete, last acting [2,4,2147483647,13,2147483647,10,2147483647,2147483647] pg 2.0 is stuck inactive since forever, current state incomplete, last acting [14,1,2147483647,4,10,2147483647,2147483647,2147483647] pg 2.3 is stuck inactive since forever, current state incomplete, last acting [14,11,6,2147483647,2147483647,2147483647,2,2147483647] pg 2.2 is stuck inactive since forever, current state incomplete, last acting [13,5,11,2147483647,2147483647,3,2147483647,2147483647] pg 2.9 is stuck unclean since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck unclean since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck unclean since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck unclean since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck unclean since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck unclean since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck unclean since forever,
Re: [ceph-users] Erasure coded PGs incomplete
Loic, So, if want have a failure domain by host, I’ll need set up a erasure profile which k+m = total number of hosts I have, right? Regards. Italo Santos http://italosantos.com.br/ On Wednesday, December 17, 2014 at 3:24 PM, Loic Dachary wrote: On 17/12/2014 18:18, Italo Santos wrote: Hello, I’ve take a look to this documentation (which help a lot) and if I understand right, when I set a profile like: === ceph osd erasure-code-profile set isilon k=8 m=2 ruleset-failure-domain=host === And create a pool following the recommendations on doc, I’ll need (100*16)/2 = 800 PGs, I’ll need the sufficient number of hosts to support create total PGs? You will need k+m = 10 host per OSD. If you only have 10 hosts that should be ok and the 800 PGs will use these 10 OSD in various orders. It also means that you will end up having 800 PG per OSD which is a bit too mche. If you have 20 OSDs that will be better : each PG will get 10 OSD out of 20 and each OSD will have 400 PGs. Ideally you want the number of PG per OSD to be in the range (approximately) [20,300]. Cheers Regards. *Italo Santos* http://italosantos.com.br/ On Wednesday, December 17, 2014 at 2:42 PM, Loic Dachary wrote: Hi, Thanks for the update : good news are much appreciated :-) Would you have time to review the documentation at https://github.com/ceph/ceph/pull/3194/files ? It was partly motivated by the problem you had. Cheers On 17/12/2014 14:03, Italo Santos wrote: Hello Loic, Thanks for you help, I’ve take a look to my crush map and I replace step chooseleaf indep 0 type osd” by step choose indep 0 type osd” and all PGs was created successfully. At. *Italo Santos* http://italosantos.com.br/ On Tuesday, December 16, 2014 at 8:39 PM, Loic Dachary wrote: Hi, The 2147483647 means that CRUSH did not find enough OSD for a given PG. If you check the crush rule associated with the erasure coded pool, you will most probably find why. Cheers On 16/12/2014 23:32, Italo Santos wrote: Hello, I'm trying to create an erasure pool following http://docs.ceph.com/docs/master/rados/operations/erasure-code/, but when I try create a pool with a specifc erasure-code-profile (myprofile) the PGs became on incomplete state. Anyone can help me? Below the profile I created: root@ceph0001:~# ceph osd erasure-code-profile get myprofile directory=/usr/lib/ceph/erasure-code k=6 m=2 plugin=jerasure technique=reed_sol_van The status of cluster: root@ceph0001:~# ceph health HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean health detail: root@ceph0001:~# ceph health detail HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean pg 2.9 is stuck inactive since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck inactive since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck inactive since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck inactive since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck inactive since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck inactive since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck inactive since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck inactive since forever, current state incomplete, last acting [9,15,2147483647,4,2,2147483647,2147483647,2147483647] pg 2.1 is stuck inactive since forever, current state incomplete, last acting [2,4,2147483647,13,2147483647,10,2147483647,2147483647] pg 2.0 is stuck inactive since forever, current state incomplete, last acting [14,1,2147483647,4,10,2147483647,2147483647,2147483647] pg 2.3 is stuck inactive since forever, current state incomplete, last acting [14,11,6,2147483647,2147483647,2147483647,2,2147483647] pg 2.2 is stuck inactive since forever, current state incomplete, last acting [13,5,11,2147483647,2147483647,3,2147483647,2147483647] pg 2.9 is stuck unclean since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck unclean
Re: [ceph-users] Erasure coded PGs incomplete
On 17/12/2014 19:22, Italo Santos wrote: Loic, So, if want have a failure domain by host, I’ll need set up a erasure profile which k+m = total number of hosts I have, right? Yes, k+m has to be = number of hosts. Regards. *Italo Santos* http://italosantos.com.br/ On Wednesday, December 17, 2014 at 3:24 PM, Loic Dachary wrote: On 17/12/2014 18:18, Italo Santos wrote: Hello, I’ve take a look to this documentation (which help a lot) and if I understand right, when I set a profile like: === ceph osd erasure-code-profile set isilon k=8 m=2 ruleset-failure-domain=host === And create a pool following the recommendations on doc, I’ll need (100*16)/2 = 800 PGs, I’ll need the sufficient number of hosts to support create total PGs? You will need k+m = 10 host per OSD. If you only have 10 hosts that should be ok and the 800 PGs will use these 10 OSD in various orders. It also means that you will end up having 800 PG per OSD which is a bit too mche. If you have 20 OSDs that will be better : each PG will get 10 OSD out of 20 and each OSD will have 400 PGs. Ideally you want the number of PG per OSD to be in the range (approximately) [20,300]. Cheers Regards. *Italo Santos* http://italosantos.com.br/ On Wednesday, December 17, 2014 at 2:42 PM, Loic Dachary wrote: Hi, Thanks for the update : good news are much appreciated :-) Would you have time to review the documentation at https://github.com/ceph/ceph/pull/3194/files ? It was partly motivated by the problem you had. Cheers On 17/12/2014 14:03, Italo Santos wrote: Hello Loic, Thanks for you help, I’ve take a look to my crush map and I replace step chooseleaf indep 0 type osd” by step choose indep 0 type osd” and all PGs was created successfully. At. *Italo Santos* http://italosantos.com.br/ On Tuesday, December 16, 2014 at 8:39 PM, Loic Dachary wrote: Hi, The 2147483647 means that CRUSH did not find enough OSD for a given PG. If you check the crush rule associated with the erasure coded pool, you will most probably find why. Cheers On 16/12/2014 23:32, Italo Santos wrote: Hello, I'm trying to create an erasure pool following http://docs.ceph.com/docs/master/rados/operations/erasure-code/, but when I try create a pool with a specifc erasure-code-profile (myprofile) the PGs became on incomplete state. Anyone can help me? Below the profile I created: root@ceph0001:~# ceph osd erasure-code-profile get myprofile directory=/usr/lib/ceph/erasure-code k=6 m=2 plugin=jerasure technique=reed_sol_van The status of cluster: root@ceph0001:~# ceph health HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean health detail: root@ceph0001:~# ceph health detail HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean pg 2.9 is stuck inactive since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck inactive since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck inactive since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck inactive since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck inactive since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck inactive since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck inactive since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck inactive since forever, current state incomplete, last acting [9,15,2147483647,4,2,2147483647,2147483647,2147483647] pg 2.1 is stuck inactive since forever, current state incomplete, last acting [2,4,2147483647,13,2147483647,10,2147483647,2147483647] pg 2.0 is stuck inactive since forever, current state incomplete, last acting [14,1,2147483647,4,10,2147483647,2147483647,2147483647] pg 2.3 is stuck inactive since forever, current state incomplete, last acting [14,11,6,2147483647,2147483647,2147483647,2,2147483647] pg 2.2 is stuck inactive since forever, current state incomplete, last acting [13,5,11,2147483647,2147483647,3,2147483647,2147483647] pg 2.9 is stuck unclean since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck unclean since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck unclean since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck unclean since forever, current state incomplete, last acting
Re: [ceph-users] Erasure coded PGs incomplete
Understood. Thanks for your help, the cluster is healthy now :D Also, using for example k=6,m=1 and failure domain by host I’ll be able lose all OSD on the same host, but if a lose 2 disks on different hosts I can lose data right? So, it is possible been a failure domain which allow me to lose an OSD or a host? Regards. Italo Santos http://italosantos.com.br/ On Wednesday, December 17, 2014 at 4:27 PM, Loic Dachary wrote: On 17/12/2014 19:22, Italo Santos wrote: Loic, So, if want have a failure domain by host, I’ll need set up a erasure profile which k+m = total number of hosts I have, right? Yes, k+m has to be = number of hosts. Regards. *Italo Santos* http://italosantos.com.br/ On Wednesday, December 17, 2014 at 3:24 PM, Loic Dachary wrote: On 17/12/2014 18:18, Italo Santos wrote: Hello, I’ve take a look to this documentation (which help a lot) and if I understand right, when I set a profile like: === ceph osd erasure-code-profile set isilon k=8 m=2 ruleset-failure-domain=host === And create a pool following the recommendations on doc, I’ll need (100*16)/2 = 800 PGs, I’ll need the sufficient number of hosts to support create total PGs? You will need k+m = 10 host per OSD. If you only have 10 hosts that should be ok and the 800 PGs will use these 10 OSD in various orders. It also means that you will end up having 800 PG per OSD which is a bit too mche. If you have 20 OSDs that will be better : each PG will get 10 OSD out of 20 and each OSD will have 400 PGs. Ideally you want the number of PG per OSD to be in the range (approximately) [20,300]. Cheers Regards. *Italo Santos* http://italosantos.com.br/ On Wednesday, December 17, 2014 at 2:42 PM, Loic Dachary wrote: Hi, Thanks for the update : good news are much appreciated :-) Would you have time to review the documentation at https://github.com/ceph/ceph/pull/3194/files ? It was partly motivated by the problem you had. Cheers On 17/12/2014 14:03, Italo Santos wrote: Hello Loic, Thanks for you help, I’ve take a look to my crush map and I replace step chooseleaf indep 0 type osd” by step choose indep 0 type osd” and all PGs was created successfully. At. *Italo Santos* http://italosantos.com.br/ On Tuesday, December 16, 2014 at 8:39 PM, Loic Dachary wrote: Hi, The 2147483647 means that CRUSH did not find enough OSD for a given PG. If you check the crush rule associated with the erasure coded pool, you will most probably find why. Cheers On 16/12/2014 23:32, Italo Santos wrote: Hello, I'm trying to create an erasure pool following http://docs.ceph.com/docs/master/rados/operations/erasure-code/, but when I try create a pool with a specifc erasure-code-profile (myprofile) the PGs became on incomplete state. Anyone can help me? Below the profile I created: root@ceph0001:~# ceph osd erasure-code-profile get myprofile directory=/usr/lib/ceph/erasure-code k=6 m=2 plugin=jerasure technique=reed_sol_van The status of cluster: root@ceph0001:~# ceph health HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean health detail: root@ceph0001:~# ceph health detail HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean pg 2.9 is stuck inactive since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck inactive since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck inactive since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck inactive since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck inactive since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck inactive since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck inactive since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck inactive since forever, current state
[ceph-users] Frozen Erasure-coded-pool
I am trying to setup a small VM ceph cluster to excersise before creating a real cluster. Currently there are two osd's on the same host. I wanted to create an erasure coded pool with k=1 and m=1 (yes I know it's stupid, but it is a test case). On top of it there is a cache tier (writeback) and I used the pool to make a rados blockdevice with it. But as I wanted to format it with ext4 the system suddenly hangs. At the moment I do not understand why. I discovered that after the creation of the 'cold-storage' the active primaries are setup correctly (about one half of the pgs to osd.0 and the other half to osd.1). But the second osd in the active group is always nonsense (MAXINT, a placeholder for 'not there'?). To my suprise the state is 'active+clean' - how can this be, shouldn't it be 'active+degraded'? These are the commands I used (from my recollection) :# ceph osd erasure-code-profile get ec_1_1 directory=/usr/lib/x86_64-linux-gnu/ceph/erasure-code k=1 m=1 plugin=jerasure ruleset-failure-domain=osd technique=reed_sol_van :# ceph osd pool create liverpool 300 300 erasure ec_1_1 :# ceph osd pool create cache 100 100 replicated :# ceph osd tier add liverpool cache :# ceph osd tier cache-mode writeback :# ceph osd tier set-overlay liverpool cache :# rbd --pool liverpool create --size 1500 testdisk :# rbd --pool liverpool map testdisk :# mkfs.ext4 /dev/rbd/liverpool/testdisk Now the mkfs freezes and I can see this thru ceph -w: 2014-12-17 19:08:56.466846 mon.0 [INF] pgmap v2062: 400 pgs: 400 active+clean; 140 bytes data, 88220 kB used, 2418 MB / 2504 MB avail; 47 B/s rd, 0 op/s 2014-12-17 19:11:20.697190 mon.0 [INF] pgmap v2064: 400 pgs: 307 stale+active+clean, 93 active+clean; 140 bytes data, 106 MB used, 2397 MB / 2504 MB avail 2014-12-17 19:11:20.388468 osd.1 [WRN] 6 slow requests, 6 included below; oldest blocked for 124.270960 secs 2014-12-17 19:11:20.388556 osd.1 [WRN] slow request 124.270960 seconds old, received at 2014-12-17 19:09:16.116251: osd_op(client.6155.1:508 rb.0.1807.2ae8944a.0005 [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~24576] 24.e6ca00e6 ondisk+write e590) v4 currently waiting for subops from 0 [repeated a few times] 2014-12-17 19:11:21.911696 mon.0 [INF] osdmap e592: 2 osds: 1 up, 2 in 2014-12-17 19:11:22.053272 mon.0 [INF] pgmap v2065: 400 pgs: 307 stale+active+clean, 93 active+clean; 140 bytes data, 106 MB used, 2397 MB / 2504 MB avail 2014-12-17 19:11:24.826008 mon.0 [INF] osd.0 10.0.0.141:6800/7919 boot 2014-12-17 19:11:24.827218 mon.0 [INF] osdmap e593: 2 osds: 2 up, 2 in 2014-12-17 19:11:24.935173 mon.0 [INF] pgmap v2066: 400 pgs: 307 stale+active+clean, 93 active+clean; 140 bytes data, 106 MB used, 2397 MB / 2504 MB avail 2014-12-17 19:11:26.072303 mon.0 [INF] osdmap e594: 2 osds: 2 up, 2 in 2014-12-17 19:11:26.220102 mon.0 [INF] pgmap v2067: 400 pgs: 307 stale+active+clean, 93 active+clean; 140 bytes data, 106 MB used, 2397 MB / 2504 MB avail 2014-12-17 19:11:30.702281 mon.0 [INF] pgmap v2068: 400 pgs: 307 stale+active+clean, 93 active+clean; 16308 kB data, 138 MB used, 2366 MB / 2504 MB avail; 1471 kB/s wr, 7 op/s; 2184 kB/s, 0 objects/s recovering 2014-12-17 19:11:32.050330 mon.0 [INF] pgmap v2069: 400 pgs: 400 active+clean; 33924 kB data, 167 MB used, 2337 MB / 2504 MB avail; 4543 kB/s wr, 46 op/s; 3565 kB/s, 1 objects/s recovering 2014-12-17 19:13:30.569447 mon.0 [INF] pgmap v2070: 400 pgs: 400 active+clean; 33924 kB data, 143 MB used, 2361 MB / 2504 MB avail How is this explained? What have I done wrong? Greetings! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Frozen Erasure-coded-pool
Hi Max, On 17/12/2014 20:57, Max Power wrote: I am trying to setup a small VM ceph cluster to excersise before creating a real cluster. Currently there are two osd's on the same host. I wanted to create an erasure coded pool with k=1 and m=1 (yes I know it's stupid, but it is a test case). This is going to fail in an undefined way and it should fail early instead of allowing it to proceed. I've created http://tracker.ceph.com/issues/10358, thanks for catching this :-) I'm not sure what to make of the rest of what you observed, it probably is tainted by the fact that the erasure coded pool cannot proceed. Cheers On top of it there is a cache tier (writeback) and I used the pool to make a rados blockdevice with it. But as I wanted to format it with ext4 the system suddenly hangs. At the moment I do not understand why. I discovered that after the creation of the 'cold-storage' the active primaries are setup correctly (about one half of the pgs to osd.0 and the other half to osd.1). But the second osd in the active group is always nonsense (MAXINT, a placeholder for 'not there'?). To my suprise the state is 'active+clean' - how can this be, shouldn't it be 'active+degraded'? These are the commands I used (from my recollection) :# ceph osd erasure-code-profile get ec_1_1 directory=/usr/lib/x86_64-linux-gnu/ceph/erasure-code k=1 m=1 plugin=jerasure ruleset-failure-domain=osd technique=reed_sol_van :# ceph osd pool create liverpool 300 300 erasure ec_1_1 :# ceph osd pool create cache 100 100 replicated :# ceph osd tier add liverpool cache :# ceph osd tier cache-mode writeback :# ceph osd tier set-overlay liverpool cache :# rbd --pool liverpool create --size 1500 testdisk :# rbd --pool liverpool map testdisk :# mkfs.ext4 /dev/rbd/liverpool/testdisk Now the mkfs freezes and I can see this thru ceph -w: 2014-12-17 19:08:56.466846 mon.0 [INF] pgmap v2062: 400 pgs: 400 active+clean; 140 bytes data, 88220 kB used, 2418 MB / 2504 MB avail; 47 B/s rd, 0 op/s 2014-12-17 19:11:20.697190 mon.0 [INF] pgmap v2064: 400 pgs: 307 stale+active+clean, 93 active+clean; 140 bytes data, 106 MB used, 2397 MB / 2504 MB avail 2014-12-17 19:11:20.388468 osd.1 [WRN] 6 slow requests, 6 included below; oldest blocked for 124.270960 secs 2014-12-17 19:11:20.388556 osd.1 [WRN] slow request 124.270960 seconds old, received at 2014-12-17 19:09:16.116251: osd_op(client.6155.1:508 rb.0.1807.2ae8944a.0005 [set-alloc-hint object_size 4194304 write_size 4194304,write 4091904~24576] 24.e6ca00e6 ondisk+write e590) v4 currently waiting for subops from 0 [repeated a few times] 2014-12-17 19:11:21.911696 mon.0 [INF] osdmap e592: 2 osds: 1 up, 2 in 2014-12-17 19:11:22.053272 mon.0 [INF] pgmap v2065: 400 pgs: 307 stale+active+clean, 93 active+clean; 140 bytes data, 106 MB used, 2397 MB / 2504 MB avail 2014-12-17 19:11:24.826008 mon.0 [INF] osd.0 10.0.0.141:6800/7919 boot 2014-12-17 19:11:24.827218 mon.0 [INF] osdmap e593: 2 osds: 2 up, 2 in 2014-12-17 19:11:24.935173 mon.0 [INF] pgmap v2066: 400 pgs: 307 stale+active+clean, 93 active+clean; 140 bytes data, 106 MB used, 2397 MB / 2504 MB avail 2014-12-17 19:11:26.072303 mon.0 [INF] osdmap e594: 2 osds: 2 up, 2 in 2014-12-17 19:11:26.220102 mon.0 [INF] pgmap v2067: 400 pgs: 307 stale+active+clean, 93 active+clean; 140 bytes data, 106 MB used, 2397 MB / 2504 MB avail 2014-12-17 19:11:30.702281 mon.0 [INF] pgmap v2068: 400 pgs: 307 stale+active+clean, 93 active+clean; 16308 kB data, 138 MB used, 2366 MB / 2504 MB avail; 1471 kB/s wr, 7 op/s; 2184 kB/s, 0 objects/s recovering 2014-12-17 19:11:32.050330 mon.0 [INF] pgmap v2069: 400 pgs: 400 active+clean; 33924 kB data, 167 MB used, 2337 MB / 2504 MB avail; 4543 kB/s wr, 46 op/s; 3565 kB/s, 1 objects/s recovering 2014-12-17 19:13:30.569447 mon.0 [INF] pgmap v2070: 400 pgs: 400 active+clean; 33924 kB data, 143 MB used, 2361 MB / 2504 MB avail How is this explained? What have I done wrong? Greetings! ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Loïc Dachary, Artisan Logiciel Libre signature.asc Description: OpenPGP digital signature ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Erasure coded PGs incomplete
On 17/12/2014 19:46, Italo Santos wrote: Understood. Thanks for your help, the cluster is healthy now :D Also, using for example k=6,m=1 and failure domain by host I’ll be able lose all OSD on the same host, but if a lose 2 disks on different hosts I can lose data right? So, it is possible been a failure domain which allow me to lose an OSD or a host? That's actually a good way to put it :-) Regards. *Italo Santos* http://italosantos.com.br/ On Wednesday, December 17, 2014 at 4:27 PM, Loic Dachary wrote: On 17/12/2014 19:22, Italo Santos wrote: Loic, So, if want have a failure domain by host, I’ll need set up a erasure profile which k+m = total number of hosts I have, right? Yes, k+m has to be = number of hosts. Regards. *Italo Santos* http://italosantos.com.br/ On Wednesday, December 17, 2014 at 3:24 PM, Loic Dachary wrote: On 17/12/2014 18:18, Italo Santos wrote: Hello, I’ve take a look to this documentation (which help a lot) and if I understand right, when I set a profile like: === ceph osd erasure-code-profile set isilon k=8 m=2 ruleset-failure-domain=host === And create a pool following the recommendations on doc, I’ll need (100*16)/2 = 800 PGs, I’ll need the sufficient number of hosts to support create total PGs? You will need k+m = 10 host per OSD. If you only have 10 hosts that should be ok and the 800 PGs will use these 10 OSD in various orders. It also means that you will end up having 800 PG per OSD which is a bit too mche. If you have 20 OSDs that will be better : each PG will get 10 OSD out of 20 and each OSD will have 400 PGs. Ideally you want the number of PG per OSD to be in the range (approximately) [20,300]. Cheers Regards. *Italo Santos* http://italosantos.com.br/ On Wednesday, December 17, 2014 at 2:42 PM, Loic Dachary wrote: Hi, Thanks for the update : good news are much appreciated :-) Would you have time to review the documentation at https://github.com/ceph/ceph/pull/3194/files ? It was partly motivated by the problem you had. Cheers On 17/12/2014 14:03, Italo Santos wrote: Hello Loic, Thanks for you help, I’ve take a look to my crush map and I replace step chooseleaf indep 0 type osd” by step choose indep 0 type osd” and all PGs was created successfully. At. *Italo Santos* http://italosantos.com.br/ On Tuesday, December 16, 2014 at 8:39 PM, Loic Dachary wrote: Hi, The 2147483647 means that CRUSH did not find enough OSD for a given PG. If you check the crush rule associated with the erasure coded pool, you will most probably find why. Cheers On 16/12/2014 23:32, Italo Santos wrote: Hello, I'm trying to create an erasure pool following http://docs.ceph.com/docs/master/rados/operations/erasure-code/, but when I try create a pool with a specifc erasure-code-profile (myprofile) the PGs became on incomplete state. Anyone can help me? Below the profile I created: root@ceph0001:~# ceph osd erasure-code-profile get myprofile directory=/usr/lib/ceph/erasure-code k=6 m=2 plugin=jerasure technique=reed_sol_van The status of cluster: root@ceph0001:~# ceph health HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean health detail: root@ceph0001:~# ceph health detail HEALTH_WARN 12 pgs incomplete; 12 pgs stuck inactive; 12 pgs stuck unclean pg 2.9 is stuck inactive since forever, current state incomplete, last acting [4,10,15,2147483647,3,2147483647,2147483647,2147483647] pg 2.8 is stuck inactive since forever, current state incomplete, last acting [0,2147483647,4,2147483647,10,2147483647,15,2147483647] pg 2.b is stuck inactive since forever, current state incomplete, last acting [8,3,14,2147483647,5,2147483647,2147483647,2147483647] pg 2.a is stuck inactive since forever, current state incomplete, last acting [11,7,2,2147483647,2147483647,2147483647,15,2147483647] pg 2.5 is stuck inactive since forever, current state incomplete, last acting [12,8,5,1,2147483647,2147483647,2147483647,2147483647] pg 2.4 is stuck inactive since forever, current state incomplete, last acting [5,2147483647,13,1,2147483647,2147483647,8,2147483647] pg 2.7 is stuck inactive since forever, current state incomplete, last acting [12,2,10,7,2147483647,2147483647,2147483647,2147483647] pg 2.6 is stuck inactive since forever, current state incomplete, last acting [9,15,2147483647,4,2,2147483647,2147483647,2147483647] pg 2.1 is stuck inactive since forever, current state incomplete, last acting [2,4,2147483647,13,2147483647,10,2147483647,2147483647] pg 2.0 is stuck inactive since forever, current state incomplete, last acting [14,1,2147483647,4,10,2147483647,2147483647,2147483647] pg 2.3 is stuck inactive since forever, current state incomplete, last acting [14,11,6,2147483647,2147483647,2147483647,2,2147483647] pg 2.2 is stuck inactive since forever, current state incomplete, last acting
[ceph-users] Double-mounting of RBD
I have a somewhat interesting scenario. I have an RBD of 17TB formatted using XFS. I would like it accessible from two different hosts, one mapped/mounted read-only, and one mapped/mounted as read-write. Both are shared using Samba 4.x. One Samba server gives read-only access to the world for the data. The other gives read-write access to a very limited set of users who occasionally need to add data. However, when testing this, when changes are made to the read-write Samba server the changes don't seem to be seen by the read-only Samba server. Is there some file system caching going on that will eventually be flushed? Am I living dangerously doing what I have set up? I thought I would avoid most/all potential file system corruption by making sure there is only one read-write access method. Thanks for any answers. Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Double-mounting of RBD
On Wed, Dec 17, 2014 at 2:31 PM, McNamara, Bradley bradley.mcnam...@seattle.gov wrote: I have a somewhat interesting scenario. I have an RBD of 17TB formatted using XFS. I would like it accessible from two different hosts, one mapped/mounted read-only, and one mapped/mounted as read-write. Both are shared using Samba 4.x. One Samba server gives read-only access to the world for the data. The other gives read-write access to a very limited set of users who occasionally need to add data. However, when testing this, when changes are made to the read-write Samba server the changes don’t seem to be seen by the read-only Samba server. Is there some file system caching going on that will eventually be flushed? Am I living dangerously doing what I have set up? I thought I would avoid most/all potential file system corruption by making sure there is only one read-write access method. Thanks for any answers. Well, you'll avoid corruption by only having one writer, but the other reader is still caching data in-memory that will prevent it from seeing the writes on the disk. Plus I have no idea if mounting xfs read-only actually prevents it from making any writes to the disk; I think some FSes will do stuff like defragment internal data structures in that mode, maybe? -Greg ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Double-mounting of RBD
On 12/17/2014 03:49 PM, Gregory Farnum wrote: On Wed, Dec 17, 2014 at 2:31 PM, McNamara, Bradley bradley.mcnam...@seattle.gov wrote: I have a somewhat interesting scenario. I have an RBD of 17TB formatted using XFS. I would like it accessible from two different hosts, one mapped/mounted read-only, and one mapped/mounted as read-write. Both are shared using Samba 4.x. One Samba server gives read-only access to the world for the data. The other gives read-write access to a very limited set of users who occasionally need to add data. However, when testing this, when changes are made to the read-write Samba server the changes don’t seem to be seen by the read-only Samba server. Is there some file system caching going on that will eventually be flushed? Am I living dangerously doing what I have set up? I thought I would avoid most/all potential file system corruption by making sure there is only one read-write access method. Thanks for any answers. Well, you'll avoid corruption by only having one writer, but the other reader is still caching data in-memory that will prevent it from seeing the writes on the disk. Plus I have no idea if mounting xfs read-only actually prevents it from making any writes to the disk; I think some FSes will do stuff like defragment internal data structures in that mode, maybe? -Greg FSes mounted read-only still do tend to do things like journal replay, but since the block device is mapped read-only that won't be a problem in this case. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] File System stripping data
Hi John, I am using 0.56.1. Could it be because data striping is not supported in this version? Kevin On Wed Dec 17 2014 at 4:00:15 AM PST Wido den Hollander w...@42on.com wrote: On 12/17/2014 12:35 PM, John Spray wrote: On Wed, Dec 17, 2014 at 10:25 AM, Wido den Hollander w...@42on.com wrote: I just tried something similar on Giant (0.87) and I saw this in the logs: parse_layout_vxattr name layout.pool value 'cephfs_svo' invalid data pool 3 reply request -22 I resolves the pool to a ID, but then it's unable to set it? Was the 'cephfs_svo' pool already added as a data pool with ceph mds add_data_pool? Ah, indeed. Working fine right now. Same goes for any other layout settings. There are paths where if a pool was added very recently, MDSs/clients might not know about the pool yet and can generate errors like this. John -- Wido den Hollander 42on B.V. Ceph trainer and consultant Phone: +31 (0)20 700 9902 Skype: contact42on ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Help with SSDs
Hello, On Tue, 16 Dec 2014 08:58:23 -0700 Bryson McCutcheon wrote: Hey there, Is there a good work around if our SSDs are not handling D_SYNC very well? We invested a ton of money into Samsung 840 EVOS and they are not playing well with D_SYNC. Would really appreciate the help! Baring any reply from the developers, I'm pretty sure the answer is no, as D_SYNC is a vital part of the journaling process. You _may_ get away with a hardware cache RAID controller (some models/makers will use that cache in JBOD mode, too). And while I very much can sympathize with you (I have dozens of new Toshiba DTxx drives sitting on a pile that perform quite nicely for a week before they go into half to 10% speed mode for at least 10 hours) and don't want to sound snarky, penultimately you get what you pay for. Journals on consumer SSDs are a bad idea. Aside form the performance issues you're experiencing, Samsung doesn't really give a TBW value for them, the estimation based on their claims is about 100TBW for the 120GB model. While not bad, it's something I'd be weary about with regards to long term usage. http://ssdendurancetest.com/ssd-endurance-test-report/Samsung-840-EVO-120 The only reliable game in town at this point in time are Intel DC S3700 models, the 200GB model for example has a TBW of 1.8PB and will keep its speed w/o the need for TRIM or massive underprovisioning. Christian -- Christian BalzerNetwork/Systems Engineer ch...@gol.com Global OnLine Japan/Fusion Communications http://www.gol.com/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Double-mounting of RBD
On Wednesday, December 17, 2014, Josh Durgin josh.dur...@inktank.com wrote: On 12/17/2014 03:49 PM, Gregory Farnum wrote: On Wed, Dec 17, 2014 at 2:31 PM, McNamara, Bradley bradley.mcnam...@seattle.gov wrote: I have a somewhat interesting scenario. I have an RBD of 17TB formatted using XFS. I would like it accessible from two different hosts, one mapped/mounted read-only, and one mapped/mounted as read-write. Both are shared using Samba 4.x. One Samba server gives read-only access to the world for the data. The other gives read-write access to a very limited set of users who occasionally need to add data. However, when testing this, when changes are made to the read-write Samba server the changes don’t seem to be seen by the read-only Samba server. Is there some file system caching going on that will eventually be flushed? Am I living dangerously doing what I have set up? I thought I would avoid most/all potential file system corruption by making sure there is only one read-write access method. Thanks for any answers. Well, you'll avoid corruption by only having one writer, but the other reader is still caching data in-memory that will prevent it from seeing the writes on the disk. Plus I have no idea if mounting xfs read-only actually prevents it from making any writes to the disk; I think some FSes will do stuff like defragment internal data structures in that mode, maybe? -Greg FSes mounted read-only still do tend to do things like journal replay, but since the block device is mapped read-only that won't be a problem in this case. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com Someone commented that the OS with the readonly mount will still do something potentially damaging to the filesystem at mount time. Something along the lines of replaying the xfs journal and the read write OS being unaware of it. Dig through the ceph mailing list archives. Jake ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Help with SSDs
On 12/17/2014 02:58 AM, Bryson McCutcheon wrote: Is there a good work around if our SSDs are not handling D_SYNC very well? We invested a ton of money into Samsung 840 EVOS and they are not playing well with D_SYNC. Would really appreciate the help! Just in case it's linked with the recent performance issue: are they upgraded/refreshed with the latest firmware? (http://www.samsung.com/au/consumer/pc-peripherals/solid-state-drive/ssd-840-evo/MZ-7TE1T0BW) I have EVOs too, what to you mean by not playing well with D_SYNC? Is there something I can test on my side to compare results with you, as I have mine flashed? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Reproducable Data Corruption with cephfs kernel driver
I'be been experimenting with CephFS for funning KVM images (proxmox). cephfs fuse version - 0.87 cephfs kernel module - kernel version 3.10 Part of my testing involves running a Windows 7 VM up and running CrystalDiskMark to check the I/O in the VM. Its surprisingly good with both the fuse and the kernel driver, seq reads writes are actually faster than the underlying disk, so I presume the FS is aggressively caching. With the fuse driver I have no problems. With the kernel driver, the benchmark runs fine, but when I reboot the VM the drive is corrupted and unreadable, every time. Rolling back to a snapshot fixes the disk. This does not happen unless I run the benchmark, which I presume is writing a lot of data. No problems with the same test for Ceph rbd, or NFS. -- Lindsay ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Help with SSDs
Hi Mikaël, I have EVOs too, what to you mean by not playing well with D_SYNC? Is there something I can test on my side to compare results with you, as I have mine flashed? http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ described how test the ssd-performance for an journal ssd (your ssd will be overwritten!!). Udo ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Help with SSDs
what to you mean by not playing well with D_SYNC? Hi, check this blog: http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ - Mail original - De: Mikaël Cluseau mclus...@isi.nc À: Bryson McCutcheon brysonmccutch...@gmail.com, ceph-users ceph-users@lists.ceph.com Envoyé: Jeudi 18 Décembre 2014 03:16:07 Objet: Re: [ceph-users] Help with SSDs On 12/17/2014 02:58 AM, Bryson McCutcheon wrote: Is there a good work around if our SSDs are not handling D_SYNC very well? We invested a ton of money into Samsung 840 EVOS and they are not playing well with D_SYNC. Would really appreciate the help! Just in case it's linked with the recent performance issue: are they upgraded/refreshed with the latest firmware? (http://www.samsung.com/au/consumer/pc-peripherals/solid-state-drive/ssd-840-evo/MZ-7TE1T0BW) I have EVOs too, what to you mean by not playing well with D_SYNC? Is there something I can test on my side to compare results with you, as I have mine flashed? ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Help with SSDs
Looking at the blog, I notice he disabled the write cache before the tests: doing this on my m550 resulted in *improved* dsync results (300 IOPS - 700 IOPS) still not great obviously, but ... interesting. So do experiment with the settings to see if you can get the 840's working better for you. While you can't do much about the endurance lifetime being a bit low, you could possibly improve performance using a journal *file* that is located on the 840's (you'll need to symlink it - disclaimer - have not tried this myself, but will experiment if you are interested). Slightly different open() options are used in this case and these cheaper consumer SSD seem to work better with them. Regards Mark On 18/12/14 18:08, Alexandre DERUMIER wrote: what to you mean by not playing well with D_SYNC? Hi, check this blog: http://www.sebastien-han.fr/blog/2014/10/10/ceph-how-to-test-if-your-ssd-is-suitable-as-a-journal-device/ ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] 'rbd list' stuck
The cluster state must be wrong,but how to recovery?
root@node3 ceph-cluster]# ceph -w
cluster 1365f2dd-b86c-436c-a64f-3318a937f3c2
health HEALTH_WARN 64 pgs incomplete; 64 pgs stale; 64 pgs stuck
inactive; 64 pgs stuck stale; 64 pgs stuck unclean; 8 requests are blocked
32 sec
monmap e1: 1 mons at {node3=10.74.120.79:6789/0}, election epoch 1,
quorum 0 node3
osdmap e38: 3 osds: 3 up, 3 in
pgmap v80: 64 pgs, 1 pools, 0 bytes data, 0 objects
100 MB used, 164 GB / 164 GB avail
64 stale+incomplete
2014-12-18 09:51:44.156184 mon.0 [INF] pgmap v80: 64 pgs: 64
stale+incomplete; 0 bytes data, 100 MB used, 164 GB / 164 GB avail
Hi
Why command 'rbd list'executed on monitor stuck,any prompt should be
appreciated!
Backtree:
[810bfdee] futex_wait_queue_me+0xde/0x140
[810c0969] futex_wait+0x179/0x280
[810c297e] do_futex+0xfe/0x5e0
[810c2ee0] SyS_futex+0x80/0x180
[815f2119] system_call_fastpath+0x16/0x1b
[] 0x
Best Regards!
YangBin
ZTE Information Security Notice: The information contained in this mail (and
any attachment transmitted herewith) is privileged and confidential and is
intended for the exclusive use of the addressee(s). If you are not an intended
recipient, any disclosure, reproduction, distribution or other dissemination or
use of the information contained is strictly prohibited. If you have received
this mail in error, please delete it and notify us immediately.
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
