OSD : filestore FileSystem: ext4 system: centos7.2 I use https://rook.io/ deploy ceph on kubernetes
ceph config -------------------------- mon keyvaluedb = rocksdb mon_allow_pool_delete = true mon_max_pg_per_osd = 1000 debug default = 0 debug rados = 0 debug mon = 0 debug osd = 0 debug bluestore = 0 debug filestore = 0 debug journal = 0 debug leveldb = 0 filestore_omap_backend = rocksdb osd pg bits = 11 osd pgp bits = 11 osd pool default size = 1 osd pool default min size = 1 osd pool default pg num = 100 osd pool default pgp num = 100 osd max object name len = 256 osd max object namespace len = 64 osd objectstore = filestore osd log #grep -A100 "hand-gesture-58-worker-s7u" rook-ceph-osd-bg48v.log |grep osd29 2018-09-20 08:29:28.524416 I | osd29: 2018-09-20 08:29:28.524249 7f232c452700 20 osd.29 pg_epoch: 1004 pg[6.15s0( v 1004'15526 (1004'14000,1004'15526] local-lis/les=984/985 n=2776 ec=90/90 lis/c 984/984 les/c/f 985/985/0 984/984/984) [29,18,17]p29(0) r=0 lpr=984 crt=1004'15526 lcod 1004'15525 mlcod 1004'15525 active+clean] do_op: op osd_op(client.1233661.0:1943429 6.15s0 6:a8e2d0d1:::7f3000c9-66f8-4598-9811-df3800e4469a.804194.12_train_result%!f(MISSING)mobilenetv2_160_0.35_feature16_pyramid3_minside160_lr0.01_batchsize32_steps2000_limitratio0.5625_slot_blankdata201809041612_bluedata201808302300composite_background_201809111827%!f(MISSING)201809111827%!f(MISSING)logs%!f(MISSING)events.out.tfevents.1536672273.tf-hand-gesture-58-worker-s7uc-0-jsuf7:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e1004) v8 2018-09-20 08:29:28.524425 I | osd29: 2018-09-20 08:29:28.524272 7f232c452700 20 osd.29 pg_epoch: 1004 pg[6.15s0( v 1004'15526 (1004'14000,1004'15526] local-lis/les=984/985 n=2776 ec=90/90 lis/c 984/984 les/c/f 985/985/0 984/984/984) [29,18,17]p29(0) r=0 lpr=984 crt=1004'15526 lcod 1004'15525 mlcod 1004'15525 active+clean] op_has_sufficient_caps session=0x5584235c3000 pool=6 (my-store.rgw.buckets.data ) owner=0 need_read_cap=1 need_write_cap=0 classes=[] -> yes 2018-09-20 08:29:28.524435 I | osd29: 2018-09-20 08:29:28.524287 7f232c452700 4 osd.29 pg_epoch: 1004 pg[6.15s0( v 1004'15526 (1004'14000,1004'15526] local-lis/les=984/985 n=2776 ec=90/90 lis/c 984/984 les/c/f 985/985/0 984/984/984) [29,18,17]p29(0) r=0 lpr=984 crt=1004'15526 lcod 1004'15525 mlcod 1004'15525 active+clean] do_op name is longer than 256 bytes 2018-09-20 08:29:28.524440 I | osd29: 2018-09-20 08:29:28.524322 7f232c452700 10 osd.29 1004 dequeue_op 0x558423971500 finish 2018-09-20 08:29:28.524443 I | osd29: 2018-09-20 08:29:28.524336 7f232c452700 20 osd.29 op_wq(1) _process empty q, waiting 2018-09-20 08:29:28.581474 I | osd29: 2018-09-20 08:29:28.581402 7f2346447700 20 osd.29 1004 share_map_peer 0x558422608800 already has epoch 1004 2018-09-20 08:29:28.581487 I | osd29: 2018-09-20 08:29:28.581461 7f2346c48700 20 osd.29 1004 share_map_peer 0x558422608800 already has epoch 1004 2018-09-20 08:29:28.714091 I | osd29: 2018-09-20 08:29:28.714054 7f232c452700 20 osd.29 pg_epoch: 1004 pg[6.15s0( v 1004'15526 (1004'14000,1004'15526] local-lis/les=984/985 n=2776 ec=90/90 lis/c 984/984 les/c/f 985/985/0 984/984/984) [29,18,17]p29(0) r=0 lpr=984 crt=1004'15526 lcod 1004'15525 mlcod 1004'15525 active+clean] do_op: op osd_op(client.1233661.0:1943432 6.15s0 6:a8e2d0d1:::7f3000c9-66f8-4598-9811-df3800e4469a.804194.12_train_result%!f(MISSING)mobilenetv2_160_0.35_feature16_pyramid3_minside160_lr0.01_batchsize32_steps2000_limitratio0.5625_slot_blankdata201809041612_bluedata201808302300composite_background_201809111827%!f(MISSING)201809111827%!f(MISSING)logs%!f(MISSING)events.out.tfevents.1536672273.tf-hand-gesture-58-worker-s7uc-0-jsuf7:head [getxattrs,stat] snapc 0=[] ondisk+read+known_if_redirected e1004) v8 2018-09-20 08:29:28.714098 I | osd29: 2018-09-20 08:29:28.714076 7f232c452700 20 osd.29 pg_epoch: 1004 pg[6.15s0( v 1004'15526 (1004'14000,1004'15526] local-lis/les=984/985 n=2776 ec=90/90 lis/c 984/984 les/c/f 985/985/0 984/984/984) [29,18,17]p29(0) r=0 lpr=984 crt=1004'15526 lcod 1004'15525 mlcod 1004'15525 active+clean] op_has_sufficient_caps session=0x5584235c3000 pool=6 (my-store.rgw.buckets.data ) owner=0 need_read_cap=1 need_write_cap=0 classes=[] -> yes 2018-09-20 08:29:28.714113 I | osd29: 2018-09-20 08:29:28.714089 7f232c452700 4 osd.29 pg_epoch: 1004 pg[6.15s0( v 1004'15526 (1004'14000,1004'15526] local-lis/les=984/985 n=2776 ec=90/90 lis/c 984/984 les/c/f 985/985/0 984/984/984) [29,18,17]p29(0) r=0 lpr=984 crt=1004'15526 lcod 1004'15525 mlcod 1004'15525 active+clean] do_op name is longer than 256 bytes 2018-09-20 08:29:28.714143 I | osd29: 2018-09-20 08:29:28.714121 7f232c452700 10 osd.29 1004 dequeue_op 0x55841ea82d80 finish 2018-09-20 08:29:28.714149 I | osd29: 2018-09-20 08:29:28.714135 7f232c452700 20 osd.29 op_wq(1) _process empty q, waiting 2018-09-20 08:29:28.715575 I | osd29: 2018-09-20 08:29:28.715530 7f2335c65700 20 filestore(/home7/rook/storage-dir/osd29) sync_entry(3987): woke after 5.000062 2018-09-20 08:29:28.715588 I | osd29: 2018-09-20 08:29:28.715548 7f2335c65700 15 filestore(/home7/rook/storage-dir/osd29) sync_entry(4018): committing 1230940 2018-09-20 08:29:28.715592 I | osd29: 2018-09-20 08:29:28.715559 7f2335c65700 20 filestore dbobjectmap: seq is 29685 2018-09-20 08:29:28.740460 I | osd29: 2018-09-20 08:29:28.740433 7f2335c65700 15 genericfilestorebackend(/home7/rook/storage-dir/osd29) syncfs: doing a full sync (syncfs(2) if possible) 2018-09-20 08:29:28.762017 I | osd29: 2018-09-20 08:29:28.761984 7f234247e700 10 osd.29 1004 tick_without_osd_lock 2018-09-20 08:29:28.762026 I | osd29: 2018-09-20 08:29:28.761996 7f234247e700 20 osd.29 1004 can_inc_scrubs_pending 0 -> 1 (max 1, active 0) 2018-09-20 08:29:28.762029 I | osd29: 2018-09-20 08:29:28.762000 7f234247e700 20 osd.29 1004 scrub_time_permit should run between 0 - 24 now 8 = yes 2018-09-20 08:29:28.762048 I | osd29: 2018-09-20 08:29:28.762025 7f234247e700 20 osd.29 1004 scrub_load_below_threshold loadavg 0.08 < max 0.5 = yes 2018-09-20 08:29:28.762053 I | osd29: 2018-09-20 08:29:28.762035 7f234247e700 20 osd.29 1004 sched_scrub load_is_low=1 2018-09-20 08:29:28.762056 I | osd29: 2018-09-20 08:29:28.762037 7f234247e700 10 osd.29 1004 sched_scrub 3.38 scheduled at 2018-09-20 08:37:52.103353 > 2018-09-20 08:29:28.761999 2018-09-20 08:29:28.762060 I | osd29: 2018-09-20 08:29:28.762040 7f234247e700 20 osd.29 1004 sched_scrub done 2018-09-20 08:29:28.762065 I | osd29: 2018-09-20 08:29:28.762042 7f234247e700 10 osd.29 1004 promote_throttle_recalibrate 0 attempts, promoted 0 objects and 0 bytes; target 25 obj/sec or 5120 k bytes/sec 2018-09-20 08:29:28.762068 I | osd29: 2018-09-20 08:29:28.762045 7f234247e700 20 osd.29 1004 promote_throttle_recalibrate new_prob 1000 2018-09-20 08:29:28.762071 I | osd29: 2018-09-20 08:29:28.762047 7f234247e700 10 osd.29 1004 promote_throttle_recalibrate actual 0, actual/prob ratio 1, adjusted new_prob 1000, prob 1000 -> 1000 2018-09-20 08:29:28.799899 I | osd29: 2018-09-20 08:29:28.799864 7f2335c65700 10 filestore(/home7/rook/storage-dir/osd29) sync_entry(4086): commit took 0.084321, interval was 5.084397 2018-09-20 08:29:28.799908 I | osd29: 2018-09-20 08:29:28.799884 7f2335c65700 15 filestore(/home7/rook/storage-dir/osd29) sync_entry(4118): committed to op_seq 1230940 2018-09-20 08:29:28.799913 I | osd29: 2018-09-20 08:29:28.799891 7f2335c65700 20 filestore(/home7/rook/storage-dir/osd29) sync_entry(3971): waiting for max_interval 5.000000 # grep -A50 "hand-gesture-58-worker-s7u" rook-ceph-osd-bdltb.log|grep osd22 2018-09-20 08:29:30.279985 I | osd22: 2018-09-20 08:29:30.279760 7f86d8c48700 20 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] Found key train_result/mobilenetv2_160_0.35_feature16_pyramid3_minside160_lr0.01_batchsize32_steps2000_limitratio0.5625_slot_blankdata201809041612_bluedata201808302300composite_background_201809111827/201809111827/logs/events.out.tfevents.1536672273.tf-hand-gesture-58-worker-s7uc-0-jsuf7 2018-09-20 08:29:30.279989 I | osd22: 2018-09-20 08:29:30.279807 7f86d8c48700 10 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] do_osd_op 4:da7cef2a:::.dir.7f3000c9-66f8-4598-9811-df3800e4469a.1225281.6.0:head [omap-get-vals] 2018-09-20 08:29:30.279994 I | osd22: 2018-09-20 08:29:30.279816 7f86d8c48700 10 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] do_osd_op omap-get-vals 2018-09-20 08:29:30.279996 I | osd22: 2018-09-20 08:29:30.279822 7f86d8c48700 15 filestore(/home1/rook/storage-dir/osd22) get_omap_iterator(5189): 4.5b_head/#4:da7cef2a:::.dir.7f3000c9-66f8-4598-9811-df3800e4469a.1225281.6.0:head# 2018-09-20 08:29:30.279999 I | osd22: 2018-09-20 08:29:30.279876 7f86d8c48700 10 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] method called response length=870 2018-09-20 08:29:30.280004 I | osd22: 2018-09-20 08:29:30.279887 7f86d8c48700 10 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] dropping ondisk_read_lock 2018-09-20 08:29:30.280007 I | osd22: 2018-09-20 08:29:30.279896 7f86d8c48700 15 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] do_osd_op_effects client.1233661 con 0x55580d199000 2018-09-20 08:29:30.280015 I | osd22: 2018-09-20 08:29:30.279904 7f86d8c48700 15 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] log_op_stats osd_op(client.1233661.0:1943437 4.5b 4:da7cef2a:::.dir.7f3000c9-66f8-4598-9811-df3800e4469a.1225281.6.0:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e1004) v8 inb 0 outb 870 lat 0.000635 2018-09-20 08:29:30.280020 I | osd22: 2018-09-20 08:29:30.279918 7f86d8c48700 15 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] publish_stats_to_osd 1004:3157 2018-09-20 08:29:30.280023 I | osd22: 2018-09-20 08:29:30.279946 7f86d8c48700 10 osd.22 1004 dequeue_op 0x555809a02900 finish 2018-09-20 08:29:30.280026 I | osd22: 2018-09-20 08:29:30.279964 7f86d8c48700 20 osd.22 op_wq(1) _process empty q, waiting 2018-09-20 08:29:31.604963 I | osd22: 2018-09-20 08:29:31.604859 7f86d8c48700 20 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] Found key train_result/mobilenetv2_160_0.35_feature16_pyramid3_minside160_lr0.01_batchsize32_steps2000_limitratio0.5625_slot_blankdata201809041612_bluedata201808302300composite_background_201809111827/201809111827/logs/events.out.tfevents.1536672273.tf-hand-gesture-58-worker-s7uc-0-jsuf7 2018-09-20 08:29:31.604978 I | osd22: 2018-09-20 08:29:31.604903 7f86d8c48700 10 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] do_osd_op 4:da7cef2a:::.dir.7f3000c9-66f8-4598-9811-df3800e4469a.1225281.6.0:head [omap-get-vals] 2018-09-20 08:29:31.604984 I | osd22: 2018-09-20 08:29:31.604917 7f86d8c48700 10 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] do_osd_op omap-get-vals 2018-09-20 08:29:31.604988 I | osd22: 2018-09-20 08:29:31.604924 7f86d8c48700 15 filestore(/home1/rook/storage-dir/osd22) get_omap_iterator(5189): 4.5b_head/#4:da7cef2a:::.dir.7f3000c9-66f8-4598-9811-df3800e4469a.1225281.6.0:head# 2018-09-20 08:29:31.605030 I | osd22: 2018-09-20 08:29:31.604972 7f86d8c48700 10 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] method called response length=870 2018-09-20 08:29:31.605055 I | osd22: 2018-09-20 08:29:31.604987 7f86d8c48700 10 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] dropping ondisk_read_lock 2018-09-20 08:29:31.605061 I | osd22: 2018-09-20 08:29:31.604994 7f86d8c48700 15 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] do_osd_op_effects client.1233661 con 0x55580d199000 2018-09-20 08:29:31.605065 I | osd22: 2018-09-20 08:29:31.605003 7f86d8c48700 15 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] log_op_stats osd_op(client.1233661.0:1943440 4.5b 4:da7cef2a:::.dir.7f3000c9-66f8-4598-9811-df3800e4469a.1225281.6.0:head [call rgw.bucket_list] snapc 0=[] ondisk+read+known_if_redirected e1004) v8 inb 0 outb 870 lat 0.000652 2018-09-20 08:29:31.605070 I | osd22: 2018-09-20 08:29:31.605020 7f86d8c48700 15 osd.22 pg_epoch: 1004 pg[4.5b( v 1004'1631 (1004'100,1004'1631] local-lis/les=992/993 n=2 ec=77/77 lis/c 992/992 les/c/f 993/993/0 992/992/973) [22,14,28] r=0 lpr=992 crt=1004'1631 lcod 1004'1630 mlcod 1004'1630 active+clean] publish_stats_to_osd 1004:3158 2018-09-20 08:29:31.605078 I | osd22: 2018-09-20 08:29:31.605049 7f86d8c48700 10 osd.22 1004 dequeue_op 0x555809823e00 finish 2018-09-20 08:29:31.605088 I | osd22: 2018-09-20 08:29:31.605064 7f86d8c48700 20 osd.22 op_wq(1) _process empty q, waiting Brad Hubbard <[email protected]> 于2018年9月19日周三 下午3:52写道： > Are you using filestore or bluestore on the OSDs? If filestore what is > the underlying filesystem? > > You could try setting debug_osd and debug_filestore to 20 and see if > that gives some more info? > On Wed, Sep 19, 2018 at 12:36 PM fatkun chan <[email protected]> wrote: > > > > > > ceph version 12.2.5 (cad919881333ac92274171586c827e01f554a70a) luminous > (stable) > > > > I have a file with long name , when I cat the file through minio client, > the error show. > > librados: Objecter returned from getxattrs r=-36 > > > > > > the log is come from radosgw > > > > 2018-09-15 03:38:24.763109 7f833c0ed700 2 req 20:0.000272:s3:GET > /hand-gesture/:list_bucket:verifying op params > > 2018-09-15 03:38:24.763111 7f833c0ed700 2 req 20:0.000273:s3:GET > /hand-gesture/:list_bucket:pre-executing > > 2018-09-15 03:38:24.763112 7f833c0ed700 2 req 20:0.000274:s3:GET > /hand-gesture/:list_bucket:executing > > 2018-09-15 03:38:24.763115 7f833c0ed700 10 cls_bucket_list > hand-gesture[7f3000c9-66f8-4598-9811-df3800e4469a.804194.12]) start [] > num_entries 1001 > > 2018-09-15 03:38:24.763822 7f833c0ed700 20 get_obj_state: > rctx=0x7f833c0e5790 > obj=hand-gesture:train_result/mobilenetv2_160_0.35_feature16_pyramid3_minside160_lr0.01_batchsize32_steps2000_limitratio0.5625_slot_blankdata201809041612_bluedata201808302300composite_background_201809111827/201809111827/logs/events.out.tfevents.1536672273.tf-hand-gesture-58-worker-s7uc-0-jsuf7 > state=0x7f837553c0a0 s->prefetch_data=0 > > 2018-09-15 03:38:24.763841 7f833c0ed700 10 librados: getxattrs > oid=7f3000c9-66f8-4598-9811-df3800e4469a.804194.12_train_result/mobilenetv2_160_0.35_feature16_pyramid3_minside160_lr0.01_batchsize32_steps2000_limitratio0.5625_slot_blankdata201809041612_bluedata201808302300composite_background_201809111827/201809111827/logs/events.out.tfevents.1536672273.tf-hand-gesture-58-worker-s7uc-0-jsuf7 > nspace= > > 2018-09-15 03:38:24.764283 7f833c0ed700 10 librados: Objecter returned > from getxattrs r=-36 > > 2018-09-15 03:38:24.764304 7f833c0ed700 2 req 20:0.001466:s3:GET > /hand-gesture/:list_bucket:completing > > 2018-09-15 03:38:24.764308 7f833c0ed700 0 WARNING: set_req_state_err > err_no=36 resorting to 500 > > 2018-09-15 03:38:24.764355 7f833c0ed700 2 req 20:0.001517:s3:GET > /hand-gesture/:list_bucket:op status=-36 > > 2018-09-15 03:38:24.764362 7f833c0ed700 2 req 20:0.001524:s3:GET > /hand-gesture/:list_bucket:http status=500 > > 2018-09-15 03:38:24.764364 7f833c0ed700 1 ====== req done > req=0x7f833c0e7110 op status=-36 http_status=500 ====== > > 2018-09-15 03:38:24.764371 7f833c0ed700 20 process_request() returned -36 > > _______________________________________________ > > ceph-users mailing list > > [email protected] > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Cheers, > Brad >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
