Hi Samuel,
Here is another assert, but this time with debug filestore = 20.
Does this reveal anything?
2016-09-03 16:12:44.122451 7fec728c9700 20 list_by_hash_bitwise prefix 08F3
2016-09-03 16:12:44.123046 7fec728c9700 20 list_by_hash_bitwise prefix 08F30042
2016-09-03 16:12:44.123068 7fec728c9700 20 list_by_hash_bitwise prefix 08FB
2016-09-03 16:12:44.123669 7fec728c9700 20 list_by_hash_bitwise prefix 08FB00D8
2016-09-03 16:12:44.123687 7fec728c9700 20 list_by_hash_bitwise prefix 08F708EF
2016-09-03 16:12:44.123738 7fec728c9700 20 filestore(/var/lib/ceph/osd/ceph-4)
objects: 0x7fec728c6e60
2016-09-03 16:12:44.123753 7fec728c9700 10 osd.4 pg_epoch: 5023 pg[7.80( v
1096'91073 (727'87762,1096'91073] local-les=5023 n=31613 ec=32 les/c/f
5023/5023/0 5022/5022/4987) [9,4] r=1 lpr=5022 pi=4984-502
1/17 luod=0'0 crt=1096'91073 lcod 0'0 active] be_scan_list scanning 25 objects
deeply
2016-09-03 16:12:44.123803 7fec728c9700 10 filestore(/var/lib/ceph/osd/ceph-4)
stat 7.80_head/#7:0100377b:::1000019e202.00000000:head# = 0 (size 11644)
2016-09-03 16:12:44.123810 7fec728c9700 15 filestore(/var/lib/ceph/osd/ceph-4)
getattrs 7.80_head/#7:0100377b:::1000019e202.00000000:head#
2016-09-03 16:12:44.123865 7fec728c9700 20 filestore(/var/lib/ceph/osd/ceph-4)
fgetattrs 132 getting '_'
2016-09-03 16:12:44.123876 7fec728c9700 20 filestore(/var/lib/ceph/osd/ceph-4)
fgetattrs 132 getting '_parent'
2016-09-03 16:12:44.123880 7fec728c9700 20 filestore(/var/lib/ceph/osd/ceph-4)
fgetattrs 132 getting 'snapset'
2016-09-03 16:12:44.123884 7fec728c9700 20 filestore(/var/lib/ceph/osd/ceph-4)
fgetattrs 132 getting '_layout'
2016-09-03 16:12:44.123889 7fec728c9700 10 filestore(/var/lib/ceph/osd/ceph-4)
getattrs no xattr exists in object_map r = 0
2016-09-03 16:12:44.123890 7fec728c9700 10 filestore(/var/lib/ceph/osd/ceph-4)
getattrs 7.80_head/#7:0100377b:::1000019e202.00000000:head# = 0
2016-09-03 16:12:44.123894 7fec728c9700 10 osd.4 pg_epoch: 5023 pg[7.80( v
1096'91073 (727'87762,1096'91073] local-les=5023 n=31613 ec=32 les/c/f
5023/5023/0 5022/5022/4987) [9,4] r=1 lpr=5022 pi=4984-502
1/17 luod=0'0 crt=1096'91073 lcod 0'0 active] be_deep_scrub
7:0100377b:::1000019e202.00000000:head seed 4294967295
2016-09-03 16:12:44.123904 7fec728c9700 15 filestore(/var/lib/ceph/osd/ceph-4)
read 7.80_head/#7:0100377b:::1000019e202.00000000:head# 0~524288
2016-09-03 16:12:44.124020 7fec728c9700 10 filestore(/var/lib/ceph/osd/ceph-4)
FileStore::read 7.80_head/#7:0100377b:::1000019e202.00000000:head#
0~11644/524288
2016-09-03 16:12:44.124033 7fec728c9700 15 filestore(/var/lib/ceph/osd/ceph-4)
read 7.80_head/#7:0100377b:::1000019e202.00000000:head# 11644~524288
2016-09-03 16:12:44.129766 7fec6e0c0700 -1 *** Caught signal (Aborted) **
in thread 7fec6e0c0700 thread_name:tp_osd_recov
ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
1: (()+0x8ebb02) [0x560bbe037b02]
2: (()+0x10330) [0x7fec9b31d330]
3: (gsignal()+0x37) [0x7fec9937fc37]
4: (abort()+0x148) [0x7fec99383028]
5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
const*)+0x265) [0x560bbe12ef85]
6: (ReplicatedPG::scan_range(int, int, PG::BackfillInterval*,
ThreadPool::TPHandle&)+0xad2) [0x560bbdc11482]
7: (ReplicatedPG::update_range(PG::BackfillInterval*,
ThreadPool::TPHandle&)+0x614) [0x560bbdc11ac4]
8: (ReplicatedPG::recover_backfill(int, ThreadPool::TPHandle&, bool*)+0x337)
[0x560bbdc31c87]
9: (ReplicatedPG::start_recovery_ops(int, ThreadPool::TPHandle&, int*)+0x8a0)
[0x560bbdc63160]
10: (OSD::do_recovery(PG*, ThreadPool::TPHandle&)+0x355) [0x560bbdaf3555]
11: (OSD::RecoveryWQ::_process(PG*, ThreadPool::TPHandle&)+0xd)
[0x560bbdb3c0dd]
12: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa6e) [0x560bbe12018e]
13: (ThreadPool::WorkThread::entry()+0x10) [0x560bbe121070]
14: (()+0x8184) [0x7fec9b315184]
15: (clone()+0x6d) [0x7fec9944337d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
--- begin dump of recent events ---
-80> 2016-09-03 16:12:44.102928 7fec728c9700 20 list_by_hash_bitwise prefix
08B702C7
-79> 2016-09-03 16:12:44.102953 7fec728c9700 20 list_by_hash_bitwise prefix
08BF
-78> 2016-09-03 16:12:44.103614 7fec728c9700 20 list_by_hash_bitwise prefix
08BF0464
-77> 2016-09-03 16:12:44.103675 7fec728c9700 20 list_by_hash_bitwise prefix
087
-76> 2016-09-03 16:12:44.103753 7fec728c9700 20 list_by_hash_bitwise prefix
0870
-75> 2016-09-03 16:12:44.104343 7fec728c9700 20 list_by_hash_bitwise prefix
0870000B
-74> 2016-09-03 16:12:44.104363 7fec728c9700 20 list_by_hash_bitwise prefix
0878
-73> 2016-09-03 16:12:44.105032 7fec728c9700 20 list_by_hash_bitwise prefix
0878005D
-72> 2016-09-03 16:12:44.105054 7fec728c9700 20 list_by_hash_bitwise prefix
0874
-71> 2016-09-03 16:12:44.105693 7fec728c9700 20 list_by_hash_bitwise prefix
087400A0
-70> 2016-09-03 16:12:44.105714 7fec728c9700 20 list_by_hash_bitwise prefix
087C
-69> 2016-09-03 16:12:44.106376 7fec728c9700 20 list_by_hash_bitwise prefix
087C0487
-68> 2016-09-03 16:12:44.106398 7fec728c9700 20 list_by_hash_bitwise prefix
0872
-67> 2016-09-03 16:12:44.106998 7fec728c9700 20 list_by_hash_bitwise prefix
0872082C
-66> 2016-09-03 16:12:44.107020 7fec728c9700 20 list_by_hash_bitwise prefix
087A
-65> 2016-09-03 16:12:44.107578 7fec728c9700 20 list_by_hash_bitwise prefix
087A00CC
-64> 2016-09-03 16:12:44.107598 7fec728c9700 20 list_by_hash_bitwise prefix
0876
-63> 2016-09-03 16:12:44.108302 7fec728c9700 20 list_by_hash_bitwise prefix
0876048C
-62> 2016-09-03 16:12:44.108333 7fec728c9700 20 list_by_hash_bitwise prefix
087E
-61> 2016-09-03 16:12:44.108869 7fec728c9700 20 list_by_hash_bitwise prefix
087E0C81
-60> 2016-09-03 16:12:44.108888 7fec728c9700 20 list_by_hash_bitwise prefix
0871
-59> 2016-09-03 16:12:44.109529 7fec728c9700 20 list_by_hash_bitwise prefix
0871005D
-58> 2016-09-03 16:12:44.109550 7fec728c9700 20 list_by_hash_bitwise prefix
0879
-57> 2016-09-03 16:12:44.110204 7fec728c9700 20 list_by_hash_bitwise prefix
087900EF
-56> 2016-09-03 16:12:44.110228 7fec728c9700 20 list_by_hash_bitwise prefix
0875
-55> 2016-09-03 16:12:44.110853 7fec728c9700 20 list_by_hash_bitwise prefix
08750802
-54> 2016-09-03 16:12:44.110874 7fec728c9700 20 list_by_hash_bitwise prefix
087D
-53> 2016-09-03 16:12:44.111517 7fec728c9700 20 list_by_hash_bitwise prefix
087D0201
-52> 2016-09-03 16:12:44.111540 7fec728c9700 20 list_by_hash_bitwise prefix
0873
-51> 2016-09-03 16:12:44.112135 7fec728c9700 20 list_by_hash_bitwise prefix
0873087A
-50> 2016-09-03 16:12:44.112161 7fec728c9700 20 list_by_hash_bitwise prefix
087B
-49> 2016-09-03 16:12:44.112827 7fec728c9700 20 list_by_hash_bitwise prefix
087B0025
-48> 2016-09-03 16:12:44.112839 7fec728c9700 20 list_by_hash_bitwise prefix
0877
-47> 2016-09-03 16:12:44.113485 7fec728c9700 20 list_by_hash_bitwise prefix
08770C10
-46> 2016-09-03 16:12:44.113497 7fec728c9700 20 list_by_hash_bitwise prefix
087F
-45> 2016-09-03 16:12:44.114123 7fec728c9700 20 list_by_hash_bitwise prefix
087F0056
-44> 2016-09-03 16:12:44.114138 7fec728c9700 20 list_by_hash_bitwise prefix
08F
-43> 2016-09-03 16:12:44.114784 7fec728c9700 20 list_by_hash_bitwise prefix
08F0
-42> 2016-09-03 16:12:44.115485 7fec728c9700 20 list_by_hash_bitwise prefix
08F00034
-41> 2016-09-03 16:12:44.115518 7fec728c9700 20 list_by_hash_bitwise prefix
08F8
-40> 2016-09-03 16:12:44.116135 7fec728c9700 20 list_by_hash_bitwise prefix
08F80845
-39> 2016-09-03 16:12:44.116153 7fec728c9700 20 list_by_hash_bitwise prefix
08F4
-38> 2016-09-03 16:12:44.116722 7fec728c9700 20 list_by_hash_bitwise prefix
08F400A4
-37> 2016-09-03 16:12:44.116741 7fec728c9700 20 list_by_hash_bitwise prefix
08FC
-36> 2016-09-03 16:12:44.117359 7fec728c9700 20 list_by_hash_bitwise prefix
08FC00B7
-35> 2016-09-03 16:12:44.117378 7fec728c9700 20 list_by_hash_bitwise prefix
08F2
-34> 2016-09-03 16:12:44.117968 7fec728c9700 20 list_by_hash_bitwise prefix
08F200FC
-33> 2016-09-03 16:12:44.117986 7fec728c9700 20 list_by_hash_bitwise prefix
08FA
-32> 2016-09-03 16:12:44.118561 7fec728c9700 20 list_by_hash_bitwise prefix
08FA0026
-31> 2016-09-03 16:12:44.118578 7fec728c9700 20 list_by_hash_bitwise prefix
08F6
-30> 2016-09-03 16:12:44.119209 7fec728c9700 20 list_by_hash_bitwise prefix
08F600C2
-29> 2016-09-03 16:12:44.119228 7fec728c9700 20 list_by_hash_bitwise prefix
08FE
-28> 2016-09-03 16:12:44.119863 7fec728c9700 20 list_by_hash_bitwise prefix
08FE00F8
-27> 2016-09-03 16:12:44.119881 7fec728c9700 20 list_by_hash_bitwise prefix
08F1
-26> 2016-09-03 16:12:44.120531 7fec728c9700 20 list_by_hash_bitwise prefix
08F100D8
-25> 2016-09-03 16:12:44.120549 7fec728c9700 20 list_by_hash_bitwise prefix
08F9
-24> 2016-09-03 16:12:44.121194 7fec728c9700 20 list_by_hash_bitwise prefix
08F9095D
-23> 2016-09-03 16:12:44.121211 7fec728c9700 20 list_by_hash_bitwise prefix
08F5
-22> 2016-09-03 16:12:44.121801 7fec728c9700 20 list_by_hash_bitwise prefix
08F50CEC
-21> 2016-09-03 16:12:44.121817 7fec728c9700 20 list_by_hash_bitwise prefix
08FD
-20> 2016-09-03 16:12:44.122434 7fec728c9700 20 list_by_hash_bitwise prefix
08FD00EF
-19> 2016-09-03 16:12:44.122451 7fec728c9700 20 list_by_hash_bitwise prefix
08F3
-18> 2016-09-03 16:12:44.123046 7fec728c9700 20 list_by_hash_bitwise prefix
08F30042
-17> 2016-09-03 16:12:44.123068 7fec728c9700 20 list_by_hash_bitwise prefix
08FB
-16> 2016-09-03 16:12:44.123669 7fec728c9700 20 list_by_hash_bitwise prefix
08FB00D8
-15> 2016-09-03 16:12:44.123687 7fec728c9700 20 list_by_hash_bitwise prefix
08F708EF
-14> 2016-09-03 16:12:44.123738 7fec728c9700 20
filestore(/var/lib/ceph/osd/ceph-4) objects: 0x7fec728c6e60
-13> 2016-09-03 16:12:44.123753 7fec728c9700 10 osd.4 pg_epoch: 5023
pg[7.80( v 1096'91073 (727'87762,1096'91073] local-les=5023 n=31613 ec=32
les/c/f 5023/5023/0 5022/5022/4987) [9,4] r=1 lpr=5022 pi=4984-5021/17 luod=0'0
crt=1096'91073 lcod 0'0 active] be_scan_list scanning 25 objects deeply
-12> 2016-09-03 16:12:44.123803 7fec728c9700 10
filestore(/var/lib/ceph/osd/ceph-4) stat
7.80_head/#7:0100377b:::1000019e202.00000000:head# = 0 (size 11644)
-11> 2016-09-03 16:12:44.123810 7fec728c9700 15
filestore(/var/lib/ceph/osd/ceph-4) getattrs
7.80_head/#7:0100377b:::1000019e202.00000000:head#
-10> 2016-09-03 16:12:44.123865 7fec728c9700 20
filestore(/var/lib/ceph/osd/ceph-4) fgetattrs 132 getting '_'
-9> 2016-09-03 16:12:44.123876 7fec728c9700 20
filestore(/var/lib/ceph/osd/ceph-4) fgetattrs 132 getting '_parent'
-8> 2016-09-03 16:12:44.123880 7fec728c9700 20
filestore(/var/lib/ceph/osd/ceph-4) fgetattrs 132 getting 'snapset'
-7> 2016-09-03 16:12:44.123884 7fec728c9700 20
filestore(/var/lib/ceph/osd/ceph-4) fgetattrs 132 getting '_layout'
-6> 2016-09-03 16:12:44.123889 7fec728c9700 10
filestore(/var/lib/ceph/osd/ceph-4) getattrs no xattr exists in object_map r = 0
-5> 2016-09-03 16:12:44.123890 7fec728c9700 10
filestore(/var/lib/ceph/osd/ceph-4) getattrs
7.80_head/#7:0100377b:::1000019e202.00000000:head# = 0
-29> 2016-09-03 16:12:44.119228 7fec728c9700 20 list_by_hash_bitwise prefix
08FE
7: (ReplicatedPG::update_range(PG::BackfillInterval*,
ThreadPool::TPHandle&)+0x614) [0x560bbdc11ac4]
8: (ReplicatedPG::recover_backfill(int, ThreadPool::TPHandle&, bool*)+0x337)
[0x560bbdc31c87]
9: (ReplicatedPG::start_recovery_ops(int, ThreadPool::TPHandle&, int*)+0x8a0)
[0x560bbdc63160]
10: (OSD::do_recovery(PG*, ThreadPool::TPHandle&)+0x355) [0x560bbdaf3555]
11: (OSD::RecoveryWQ::_process(PG*, ThreadPool::TPHandle&)+0xd)
[0x560bbdb3c0dd]
12: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa6e) [0x560bbe12018e]
13: (ThreadPool::WorkThread::entry()+0x10) [0x560bbe121070]
14: (()+0x8184) [0x7fec9b315184]
15: (clone()+0x6d) [0x7fec9944337d]
NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to
interpret this.
> On Sep 2, 2016, at 12:25, Samuel Just <sj...@redhat.com> wrote:
>
> Probably an EIO. You can reproduce with debug filestore = 20 to confirm.
> -Sam
>
> On Fri, Sep 2, 2016 at 10:18 AM, Reed Dier <reed.d...@focusvq.com> wrote:
>> OSD has randomly stopped for some reason. Lots of recovery processes
>> currently running on the ceph cluster. OSD log with assert below:
>>
>> -14> 2016-09-02 11:32:38.672460 7fcf65514700 5 -- op tracker -- seq: 1147,
>> time: 2016-09-02 11:32:38.672460, event: queued_for_pg, op:
>> osd_sub_op_reply(unknown.0.0:0 7.d1 MIN [scrub-reserve] ack, result = 0)
>> -13> 2016-09-02 11:32:38.672533 7fcf70d40700 5 -- op tracker -- seq:
>> 1147, time: 2016-09-02 11:32:38.672533, event: reached_pg, op:
>> osd_sub_op_reply(unknown.0.0:0 7.d1 MIN [scrub-reserve] ack, result = 0)
>> -12> 2016-09-02 11:32:38.672548 7fcf70d40700 5 -- op tracker -- seq:
>> 1147, time: 2016-09-02 11:32:38.672548, event: started, op:
>> osd_sub_op_reply(unknown.0.0:0 7.d1 MIN [scrub-reserve] ack, result = 0)
>> -11> 2016-09-02 11:32:38.672548 7fcf7cd58700 1 -- [].28:6800/27735 <==
>> mon.0 [].249:6789/0 60 ==== pg_stats_ack(0 pgs tid 45) v1 ==== 4+0+0 (0 0 0)
>> 0x55a4443b1400 con 0x55a4434a4e80
>> -10> 2016-09-02 11:32:38.672559 7fcf70d40700 1 -- [].28:6801/27735 -->
>> [].31:6801/2070838 -- osd_sub_op(unknown.0.0:0 7.d1 MIN [scrub-unreserve] v
>> 0'0 snapset=0=[]:[]) v12 -- ?+0 0x55a443aec100 con 0x55a443be0600
>> -9> 2016-09-02 11:32:38.672571 7fcf70d40700 5 -- op tracker -- seq:
>> 1147, time: 2016-09-02 11:32:38.672571, event: done, op:
>> osd_sub_op_reply(unknown.0.0:0 7.d1 MIN [scrub-reserve] ack, result = 0)
>> -8> 2016-09-02 11:32:38.681929 7fcf7b555700 1 -- [].28:6801/27735 <==
>> osd.2 [].26:6801/9468 148 ==== MBackfillReserve GRANT pgid: 15.11,
>> query_epoch: 4235 v3 ==== 30+0+0 (3067148394 0 0) 0x55a4441f65a0 con
>> 0x55a4434ab200
>> -7> 2016-09-02 11:32:38.682009 7fcf7b555700 5 -- op tracker -- seq:
>> 1148, time: 2016-09-02 11:32:38.682008, event: done, op: MBackfillReserve
>> GRANT pgid: 15.11, query_epoch: 4235
>> -6> 2016-09-02 11:32:38.682068 7fcf73545700 5 osd.4 pg_epoch: 4235
>> pg[15.11( v 895'400028 (859'397021,895'400028] local-les=4234 n=166739
>> ec=732 les/c/f 4234/4003/0 4232/4233/4233) [2,4]/[4] r=0 lpr=4233
>> pi=4002-4232/47 (log bound mismatch
>> , actual=[859'396822,895'400028]) bft=2 crt=895'400028 lcod 0'0 mlcod 0'0
>> active+undersized+degraded+remapped+wait_backfill] exit
>> Started/Primary/Active/WaitRemoteBackfillReserved 221.748180 6 0.000056
>> -5> 2016-09-02 11:32:38.682109 7fcf73545700 5 osd.4 pg_epoch: 4235
>> pg[15.11( v 895'400028 (859'397021,895'400028] local-les=4234 n=166739
>> ec=732 les/c/f 4234/4003/0 4232/4233/4233) [2,4]/[4] r=0 lpr=4233
>> pi=4002-4232/47 (log bound mismatch
>> , actual=[859'396822,895'400028]) bft=2 crt=895'400028 lcod 0'0 mlcod 0'0
>> active+undersized+degraded+remapped+wait_backfill] enter
>> Started/Primary/Active/Backfilling
>> -4> 2016-09-02 11:32:38.682584 7fcf7b555700 1 -- [].28:6801/27735 <==
>> osd.6 [].30:6801/44406 171 ==== osd pg remove(epoch 4235; pg6.19; ) v2 ====
>> 30+0+0 (522063165 0 0) 0x55a44392f680 con 0x55a443bae100
>> -3> 2016-09-02 11:32:38.682600 7fcf7b555700 5 -- op tracker -- seq:
>> 1149, time: 2016-09-02 11:32:38.682600, event: started, op: osd pg
>> remove(epoch 4235; pg6.19; )
>> -2> 2016-09-02 11:32:38.682616 7fcf7b555700 5 osd.4 4235
>> queue_pg_for_deletion: 6.19
>> -1> 2016-09-02 11:32:38.685425 7fcf7b555700 5 -- op tracker -- seq:
>> 1149, time: 2016-09-02 11:32:38.685421, event: done, op: osd pg remove(epoch
>> 4235; pg6.19; )
>> 0> 2016-09-02 11:32:38.690487 7fcf6c537700 -1 osd/ReplicatedPG.cc: In
>> function 'void ReplicatedPG::scan_range(int, int, PG::BackfillInterval*,
>> ThreadPool::TPHandle&)' thread 7fcf6c537700 time 2016-09-02 11:32:38.688536
>> osd/ReplicatedPG.cc: 11345: FAILED assert(r >= 0)
>>
>> 2016-09-02 11:32:38.711869 7fcf6c537700 -1 *** Caught signal (Aborted) **
>>
>> in thread 7fcf6c537700 thread_name:tp_osd_recov
>>
>> ceph version 10.2.2 (45107e21c568dd033c2f0a3107dec8f0b0e58374)
>> 1: (()+0x8ebb02) [0x55a402375b02]
>> 2: (()+0x10330) [0x7fcfa2b51330]
>> 3: (gsignal()+0x37) [0x7fcfa0bb3c37]
>> 4: (abort()+0x148) [0x7fcfa0bb7028]
>> 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char
>> const*)+0x265) [0x55a40246cf85]
>> 6: (ReplicatedPG::scan_range(int, int, PG::BackfillInterval*,
>> ThreadPool::TPHandle&)+0xad2) [0x55a401f4f482]
>> 7: (ReplicatedPG::update_range(PG::BackfillInterval*,
>> ThreadPool::TPHandle&)+0x614) [0x55a401f4fac4]
>> 8: (ReplicatedPG::recover_backfill(int, ThreadPool::TPHandle&,
>> bool*)+0x337) [0x55a401f6fc87]
>> 9: (ReplicatedPG::start_recovery_ops(int, ThreadPool::TPHandle&,
>> int*)+0x8a0) [0x55a401fa1160]
>> 10: (OSD::do_recovery(PG*, ThreadPool::TPHandle&)+0x355) [0x55a401e31555]
>> 11: (OSD::RecoveryWQ::_process(PG*, ThreadPool::TPHandle&)+0xd)
>> [0x55a401e7a0dd]
>> 12: (ThreadPool::worker(ThreadPool::WorkThread*)+0xa6e) [0x55a40245e18e]
>> 13: (ThreadPool::WorkThread::entry()+0x10) [0x55a40245f070]
>> 14: (()+0x8184) [0x7fcfa2b49184]
>> 15: (clone()+0x6d) [0x7fcfa0c7737d]
>>
>>
>> Any help with this appreciated.
>>
>> Thanks,
>>
>> Reed
>>
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
