Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Glad it worked. On Mon, Apr 22, 2019 at 11:01 AM Can Zhang wrote: > > Thanks for your detailed response. > > I freshly installed a CentOS 7.6 and run install-deps.sh and > do_cmake.sh this time, and it works this time. Maybe the problem was > caused by dirty environment. > > > Best, > Can Zhang > > > On Fri, Apr 19, 2019 at 6:28 PM Brad Hubbard wrote: > > > > OK. So this works for me with master commit > > bdaac2d619d603f53a16c07f9d7bd47751137c4c on Centos 7.5.1804. > > > > I cloned the repo and ran './install-deps.sh' and './do_cmake.sh > > -DWITH_FIO=ON' then 'make all'. > > > > # find ./lib -iname '*.so*' | xargs nm -AD 2>&1 | grep > > _ZTIN13PriorityCache8PriCacheE > > ./lib/libfio_ceph_objectstore.so:018f72d0 V > > _ZTIN13PriorityCache8PriCacheE > > > > # LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > > conf: Path to a ceph configuration file > > oi_attr_len : Set OI(aka '_') attribute to specified length > > snapset_attr_len: Set 'snapset' attribute to specified length > > _fastinfo_omap_len : Set '_fastinfo' OMAP attribute to specified length > > pglog_simulation: Enables PG Log simulation behavior > > pglog_omap_len : Set pglog omap entry to specified length > > pglog_dup_omap_len : Set duplicate pglog omap entry to specified length > > single_pool_mode: Enables the mode when all jobs run against > > the same pool > > preallocate_files : Enables/disables file preallocation (touch > > and resize) on init > > > > So my result above matches your result on ubuntu but not on centos. It > > looks to me like we used to define in libceph-common but currently > > it's defined in libfio_ceph_objectstore.so. For reasons that are > > unclear you are seeing the old behaviour. Why this is and why it isn't > > working as designed is not clear to me but I suspect if you clone the > > repo again and build from scratch (maybe in a different directory if > > you wish to keep debugging, see below) you should get a working > > result. Could you try that as a test? > > > > If, on the other hand, you wish to keep debugging your current > > environment I'd suggest looking at the output of the following command > > as it may shed further light on the issue. > > > > # LD_DEBUG=all LD_LIBRARY_PATH=./lib ./bin/fio > > --enghelp=libfio_ceph_objectstore.so > > > > 'LD_DEBUG=lib' may suffice but that's difficult to judge without > > knowing what the problem is. I still suspect somehow you have > > mis-matched libraries and, if that's the case, it's probably not worth > > pursuing. If you can give me specific steps so I can reproduce this > > from a freshly cloned tree I'd be happy to look further into it. > > > > Good luck. > > > > On Thu, Apr 18, 2019 at 7:00 PM Brad Hubbard wrote: > > > > > > Let me try to reproduce this on centos 7.5 with master and I'll let > > > you know how I go. > > > > > > On Thu, Apr 18, 2019 at 3:59 PM Can Zhang wrote: > > > > > > > > Using the commands you provided, I actually find some differences: > > > > > > > > On my CentOS VM: > > > > ``` > > > > # sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > > > _ZTIN13PriorityCache8PriCacheE > > > > ./libceph-common.so:0221cc08 V _ZTIN13PriorityCache8PriCacheE > > > > ./libceph-common.so.0:0221cc08 V _ZTIN13PriorityCache8PriCacheE > > > > ./libfio_ceph_objectstore.so: U > > > > _ZTIN13PriorityCache8PriCacheE > > > > ``` > > > > ``` > > > > # ldd libfio_ceph_objectstore.so |grep common > > > > libceph-common.so.0 => /root/ceph/build/lib/libceph-common.so.0 > > > > (0x7fd13f3e7000) > > > > ``` > > > > On my Ubuntu VM: > > > > ``` > > > > $ sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > > > _ZTIN13PriorityCache8PriCacheE > > > > ./libfio_ceph_objectstore.so:019d13e0 V > > > > _ZTIN13PriorityCache8PriCacheE > > > > ``` > > > > ``` > > > > $ ldd libfio_ceph_objectstore.so |grep common > > > > libceph-common.so.0 => > > > > /home/can/work/ceph/build/lib/libceph-common.so.0 (0x7f024a89e000) > > > > ``` > > > > > > > > Notice the "U" and "V" from nm results. > > > > > > > > > > > > > > > > > > > > Best, > > > > Can Zhang > > > > > > > > On Thu, Apr 18, 2019 at 9:36 AM Brad Hubbard > > > > wrote: > > > > > > > > > > Does it define _ZTIN13PriorityCache8PriCacheE ? If it does, and all is > > > > > as you say, then it should not say that _ZTIN13PriorityCache8PriCacheE > > > > > is undefined. Does ldd show that it is finding the libraries you think > > > > > it is? Either it is finding a different version of that library > > > > > somewhere else or the version you have may not define that symbol. > > > > > > > > > > On Thu, Apr 18, 2019 at 11:12 AM Can Zhang wrote: > > > > > > > > > > > > It's already in LD_LIBRARY_PATH, under the same directory of > > > > > > libfio_ceph_objectstore.so > > > > > > > > > > > > > > > > > > $ ll lib/|grep libceph-common > > > > >
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Thanks for your detailed response. I freshly installed a CentOS 7.6 and run install-deps.sh and do_cmake.sh this time, and it works this time. Maybe the problem was caused by dirty environment. Best, Can Zhang On Fri, Apr 19, 2019 at 6:28 PM Brad Hubbard wrote: > > OK. So this works for me with master commit > bdaac2d619d603f53a16c07f9d7bd47751137c4c on Centos 7.5.1804. > > I cloned the repo and ran './install-deps.sh' and './do_cmake.sh > -DWITH_FIO=ON' then 'make all'. > > # find ./lib -iname '*.so*' | xargs nm -AD 2>&1 | grep > _ZTIN13PriorityCache8PriCacheE > ./lib/libfio_ceph_objectstore.so:018f72d0 V > _ZTIN13PriorityCache8PriCacheE > > # LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > conf: Path to a ceph configuration file > oi_attr_len : Set OI(aka '_') attribute to specified length > snapset_attr_len: Set 'snapset' attribute to specified length > _fastinfo_omap_len : Set '_fastinfo' OMAP attribute to specified length > pglog_simulation: Enables PG Log simulation behavior > pglog_omap_len : Set pglog omap entry to specified length > pglog_dup_omap_len : Set duplicate pglog omap entry to specified length > single_pool_mode: Enables the mode when all jobs run against > the same pool > preallocate_files : Enables/disables file preallocation (touch > and resize) on init > > So my result above matches your result on ubuntu but not on centos. It > looks to me like we used to define in libceph-common but currently > it's defined in libfio_ceph_objectstore.so. For reasons that are > unclear you are seeing the old behaviour. Why this is and why it isn't > working as designed is not clear to me but I suspect if you clone the > repo again and build from scratch (maybe in a different directory if > you wish to keep debugging, see below) you should get a working > result. Could you try that as a test? > > If, on the other hand, you wish to keep debugging your current > environment I'd suggest looking at the output of the following command > as it may shed further light on the issue. > > # LD_DEBUG=all LD_LIBRARY_PATH=./lib ./bin/fio > --enghelp=libfio_ceph_objectstore.so > > 'LD_DEBUG=lib' may suffice but that's difficult to judge without > knowing what the problem is. I still suspect somehow you have > mis-matched libraries and, if that's the case, it's probably not worth > pursuing. If you can give me specific steps so I can reproduce this > from a freshly cloned tree I'd be happy to look further into it. > > Good luck. > > On Thu, Apr 18, 2019 at 7:00 PM Brad Hubbard wrote: > > > > Let me try to reproduce this on centos 7.5 with master and I'll let > > you know how I go. > > > > On Thu, Apr 18, 2019 at 3:59 PM Can Zhang wrote: > > > > > > Using the commands you provided, I actually find some differences: > > > > > > On my CentOS VM: > > > ``` > > > # sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > > _ZTIN13PriorityCache8PriCacheE > > > ./libceph-common.so:0221cc08 V _ZTIN13PriorityCache8PriCacheE > > > ./libceph-common.so.0:0221cc08 V _ZTIN13PriorityCache8PriCacheE > > > ./libfio_ceph_objectstore.so: U > > > _ZTIN13PriorityCache8PriCacheE > > > ``` > > > ``` > > > # ldd libfio_ceph_objectstore.so |grep common > > > libceph-common.so.0 => /root/ceph/build/lib/libceph-common.so.0 > > > (0x7fd13f3e7000) > > > ``` > > > On my Ubuntu VM: > > > ``` > > > $ sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > > _ZTIN13PriorityCache8PriCacheE > > > ./libfio_ceph_objectstore.so:019d13e0 V > > > _ZTIN13PriorityCache8PriCacheE > > > ``` > > > ``` > > > $ ldd libfio_ceph_objectstore.so |grep common > > > libceph-common.so.0 => > > > /home/can/work/ceph/build/lib/libceph-common.so.0 (0x7f024a89e000) > > > ``` > > > > > > Notice the "U" and "V" from nm results. > > > > > > > > > > > > > > > Best, > > > Can Zhang > > > > > > On Thu, Apr 18, 2019 at 9:36 AM Brad Hubbard wrote: > > > > > > > > Does it define _ZTIN13PriorityCache8PriCacheE ? If it does, and all is > > > > as you say, then it should not say that _ZTIN13PriorityCache8PriCacheE > > > > is undefined. Does ldd show that it is finding the libraries you think > > > > it is? Either it is finding a different version of that library > > > > somewhere else or the version you have may not define that symbol. > > > > > > > > On Thu, Apr 18, 2019 at 11:12 AM Can Zhang wrote: > > > > > > > > > > It's already in LD_LIBRARY_PATH, under the same directory of > > > > > libfio_ceph_objectstore.so > > > > > > > > > > > > > > > $ ll lib/|grep libceph-common > > > > > lrwxrwxrwx. 1 root root19 Apr 17 11:15 libceph-common.so -> > > > > > libceph-common.so.0 > > > > > -rwxr-xr-x. 1 root root 211853400 Apr 17 11:15 libceph-common.so.0 > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > Can Zhang > > > > > > > > > > On Thu, Apr 18, 2019 at 7:00 AM Brad
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
OK. So this works for me with master commit bdaac2d619d603f53a16c07f9d7bd47751137c4c on Centos 7.5.1804. I cloned the repo and ran './install-deps.sh' and './do_cmake.sh -DWITH_FIO=ON' then 'make all'. # find ./lib -iname '*.so*' | xargs nm -AD 2>&1 | grep _ZTIN13PriorityCache8PriCacheE ./lib/libfio_ceph_objectstore.so:018f72d0 V _ZTIN13PriorityCache8PriCacheE # LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so conf: Path to a ceph configuration file oi_attr_len : Set OI(aka '_') attribute to specified length snapset_attr_len: Set 'snapset' attribute to specified length _fastinfo_omap_len : Set '_fastinfo' OMAP attribute to specified length pglog_simulation: Enables PG Log simulation behavior pglog_omap_len : Set pglog omap entry to specified length pglog_dup_omap_len : Set duplicate pglog omap entry to specified length single_pool_mode: Enables the mode when all jobs run against the same pool preallocate_files : Enables/disables file preallocation (touch and resize) on init So my result above matches your result on ubuntu but not on centos. It looks to me like we used to define in libceph-common but currently it's defined in libfio_ceph_objectstore.so. For reasons that are unclear you are seeing the old behaviour. Why this is and why it isn't working as designed is not clear to me but I suspect if you clone the repo again and build from scratch (maybe in a different directory if you wish to keep debugging, see below) you should get a working result. Could you try that as a test? If, on the other hand, you wish to keep debugging your current environment I'd suggest looking at the output of the following command as it may shed further light on the issue. # LD_DEBUG=all LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so 'LD_DEBUG=lib' may suffice but that's difficult to judge without knowing what the problem is. I still suspect somehow you have mis-matched libraries and, if that's the case, it's probably not worth pursuing. If you can give me specific steps so I can reproduce this from a freshly cloned tree I'd be happy to look further into it. Good luck. On Thu, Apr 18, 2019 at 7:00 PM Brad Hubbard wrote: > > Let me try to reproduce this on centos 7.5 with master and I'll let > you know how I go. > > On Thu, Apr 18, 2019 at 3:59 PM Can Zhang wrote: > > > > Using the commands you provided, I actually find some differences: > > > > On my CentOS VM: > > ``` > > # sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > _ZTIN13PriorityCache8PriCacheE > > ./libceph-common.so:0221cc08 V _ZTIN13PriorityCache8PriCacheE > > ./libceph-common.so.0:0221cc08 V _ZTIN13PriorityCache8PriCacheE > > ./libfio_ceph_objectstore.so: U > > _ZTIN13PriorityCache8PriCacheE > > ``` > > ``` > > # ldd libfio_ceph_objectstore.so |grep common > > libceph-common.so.0 => /root/ceph/build/lib/libceph-common.so.0 > > (0x7fd13f3e7000) > > ``` > > On my Ubuntu VM: > > ``` > > $ sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > _ZTIN13PriorityCache8PriCacheE > > ./libfio_ceph_objectstore.so:019d13e0 V > > _ZTIN13PriorityCache8PriCacheE > > ``` > > ``` > > $ ldd libfio_ceph_objectstore.so |grep common > > libceph-common.so.0 => > > /home/can/work/ceph/build/lib/libceph-common.so.0 (0x7f024a89e000) > > ``` > > > > Notice the "U" and "V" from nm results. > > > > > > > > > > Best, > > Can Zhang > > > > On Thu, Apr 18, 2019 at 9:36 AM Brad Hubbard wrote: > > > > > > Does it define _ZTIN13PriorityCache8PriCacheE ? If it does, and all is > > > as you say, then it should not say that _ZTIN13PriorityCache8PriCacheE > > > is undefined. Does ldd show that it is finding the libraries you think > > > it is? Either it is finding a different version of that library > > > somewhere else or the version you have may not define that symbol. > > > > > > On Thu, Apr 18, 2019 at 11:12 AM Can Zhang wrote: > > > > > > > > It's already in LD_LIBRARY_PATH, under the same directory of > > > > libfio_ceph_objectstore.so > > > > > > > > > > > > $ ll lib/|grep libceph-common > > > > lrwxrwxrwx. 1 root root19 Apr 17 11:15 libceph-common.so -> > > > > libceph-common.so.0 > > > > -rwxr-xr-x. 1 root root 211853400 Apr 17 11:15 libceph-common.so.0 > > > > > > > > > > > > > > > > > > > > Best, > > > > Can Zhang > > > > > > > > On Thu, Apr 18, 2019 at 7:00 AM Brad Hubbard > > > > wrote: > > > > > > > > > > On Wed, Apr 17, 2019 at 1:37 PM Can Zhang wrote: > > > > > > > > > > > > Thanks for your suggestions. > > > > > > > > > > > > I tried to build libfio_ceph_objectstore.so, but it fails to load: > > > > > > > > > > > > ``` > > > > > > $ LD_LIBRARY_PATH=./lib ./bin/fio > > > > > > --enghelp=libfio_ceph_objectstore.so > > > > > > > > > > > > fio: engine libfio_ceph_objectstore.so not loadable > > > > > > IO engine libfio_ceph_objectstore.so not found > >
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Let me try to reproduce this on centos 7.5 with master and I'll let you know how I go. On Thu, Apr 18, 2019 at 3:59 PM Can Zhang wrote: > > Using the commands you provided, I actually find some differences: > > On my CentOS VM: > ``` > # sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > _ZTIN13PriorityCache8PriCacheE > ./libceph-common.so:0221cc08 V _ZTIN13PriorityCache8PriCacheE > ./libceph-common.so.0:0221cc08 V _ZTIN13PriorityCache8PriCacheE > ./libfio_ceph_objectstore.so: U _ZTIN13PriorityCache8PriCacheE > ``` > ``` > # ldd libfio_ceph_objectstore.so |grep common > libceph-common.so.0 => /root/ceph/build/lib/libceph-common.so.0 > (0x7fd13f3e7000) > ``` > On my Ubuntu VM: > ``` > $ sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > _ZTIN13PriorityCache8PriCacheE > ./libfio_ceph_objectstore.so:019d13e0 V _ZTIN13PriorityCache8PriCacheE > ``` > ``` > $ ldd libfio_ceph_objectstore.so |grep common > libceph-common.so.0 => > /home/can/work/ceph/build/lib/libceph-common.so.0 (0x7f024a89e000) > ``` > > Notice the "U" and "V" from nm results. > > > > > Best, > Can Zhang > > On Thu, Apr 18, 2019 at 9:36 AM Brad Hubbard wrote: > > > > Does it define _ZTIN13PriorityCache8PriCacheE ? If it does, and all is > > as you say, then it should not say that _ZTIN13PriorityCache8PriCacheE > > is undefined. Does ldd show that it is finding the libraries you think > > it is? Either it is finding a different version of that library > > somewhere else or the version you have may not define that symbol. > > > > On Thu, Apr 18, 2019 at 11:12 AM Can Zhang wrote: > > > > > > It's already in LD_LIBRARY_PATH, under the same directory of > > > libfio_ceph_objectstore.so > > > > > > > > > $ ll lib/|grep libceph-common > > > lrwxrwxrwx. 1 root root19 Apr 17 11:15 libceph-common.so -> > > > libceph-common.so.0 > > > -rwxr-xr-x. 1 root root 211853400 Apr 17 11:15 libceph-common.so.0 > > > > > > > > > > > > > > > Best, > > > Can Zhang > > > > > > On Thu, Apr 18, 2019 at 7:00 AM Brad Hubbard wrote: > > > > > > > > On Wed, Apr 17, 2019 at 1:37 PM Can Zhang wrote: > > > > > > > > > > Thanks for your suggestions. > > > > > > > > > > I tried to build libfio_ceph_objectstore.so, but it fails to load: > > > > > > > > > > ``` > > > > > $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > > > > > > > > > > fio: engine libfio_ceph_objectstore.so not loadable > > > > > IO engine libfio_ceph_objectstore.so not found > > > > > ``` > > > > > > > > > > I managed to print the dlopen error, it said: > > > > > > > > > > ``` > > > > > dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > > > > > _ZTIN13PriorityCache8PriCacheE > > > > > > > > $ c++filt _ZTIN13PriorityCache8PriCacheE > > > > typeinfo for PriorityCache::PriCache > > > > > > > > $ sudo find /lib* /usr/lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > > > _ZTIN13PriorityCache8PriCacheE > > > > /usr/lib64/ceph/libceph-common.so:008edab0 V > > > > _ZTIN13PriorityCache8PriCacheE > > > > /usr/lib64/ceph/libceph-common.so.0:008edab0 V > > > > _ZTIN13PriorityCache8PriCacheE > > > > > > > > It needs to be able to find libceph-common, put it in your path or > > > > preload it. > > > > > > > > > ``` > > > > > > > > > > I found a not-so-relevant > > > > > issue(https://tracker.ceph.com/issues/38360), the error seems to be > > > > > caused by mixed versions. My build environment is CentOS 7.5.1804 with > > > > > SCL devtoolset-7, and ceph is latest master branch. Does someone know > > > > > about the symbol? > > > > > > > > > > > > > > > Best, > > > > > Can Zhang > > > > > > > > > > Best, > > > > > Can Zhang > > > > > > > > > > > > > > > On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > > > > > > > > > > > Besides already mentioned store_test.cc one can also use ceph > > > > > > objectstore fio plugin > > > > > > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > > > > > > standalone BlueStore instance from FIO benchmarking tool. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Igor > > > > > > > > > > > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > > > > > > Hi, > > > > > > > > > > > > > > I'd like to run a standalone Bluestore instance so as to test and > > > > > > > tune > > > > > > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Can Zhang > > > > > > > > > > > > > > ___ > > > > > > > ceph-users mailing list > > > > > > > ceph-users@lists.ceph.com > > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > ___ > > > > > ceph-users mailing list > > > > > ceph-users@lists.ceph.com > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > > > > > -- > > > > Cheers, > > > > Brad > > > > > > >
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Using the commands you provided, I actually find some differences: On my CentOS VM: ``` # sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep _ZTIN13PriorityCache8PriCacheE ./libceph-common.so:0221cc08 V _ZTIN13PriorityCache8PriCacheE ./libceph-common.so.0:0221cc08 V _ZTIN13PriorityCache8PriCacheE ./libfio_ceph_objectstore.so: U _ZTIN13PriorityCache8PriCacheE ``` ``` # ldd libfio_ceph_objectstore.so |grep common libceph-common.so.0 => /root/ceph/build/lib/libceph-common.so.0 (0x7fd13f3e7000) ``` On my Ubuntu VM: ``` $ sudo find ./lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep _ZTIN13PriorityCache8PriCacheE ./libfio_ceph_objectstore.so:019d13e0 V _ZTIN13PriorityCache8PriCacheE ``` ``` $ ldd libfio_ceph_objectstore.so |grep common libceph-common.so.0 => /home/can/work/ceph/build/lib/libceph-common.so.0 (0x7f024a89e000) ``` Notice the "U" and "V" from nm results. Best, Can Zhang On Thu, Apr 18, 2019 at 9:36 AM Brad Hubbard wrote: > > Does it define _ZTIN13PriorityCache8PriCacheE ? If it does, and all is > as you say, then it should not say that _ZTIN13PriorityCache8PriCacheE > is undefined. Does ldd show that it is finding the libraries you think > it is? Either it is finding a different version of that library > somewhere else or the version you have may not define that symbol. > > On Thu, Apr 18, 2019 at 11:12 AM Can Zhang wrote: > > > > It's already in LD_LIBRARY_PATH, under the same directory of > > libfio_ceph_objectstore.so > > > > > > $ ll lib/|grep libceph-common > > lrwxrwxrwx. 1 root root19 Apr 17 11:15 libceph-common.so -> > > libceph-common.so.0 > > -rwxr-xr-x. 1 root root 211853400 Apr 17 11:15 libceph-common.so.0 > > > > > > > > > > Best, > > Can Zhang > > > > On Thu, Apr 18, 2019 at 7:00 AM Brad Hubbard wrote: > > > > > > On Wed, Apr 17, 2019 at 1:37 PM Can Zhang wrote: > > > > > > > > Thanks for your suggestions. > > > > > > > > I tried to build libfio_ceph_objectstore.so, but it fails to load: > > > > > > > > ``` > > > > $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > > > > > > > > fio: engine libfio_ceph_objectstore.so not loadable > > > > IO engine libfio_ceph_objectstore.so not found > > > > ``` > > > > > > > > I managed to print the dlopen error, it said: > > > > > > > > ``` > > > > dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > > > > _ZTIN13PriorityCache8PriCacheE > > > > > > $ c++filt _ZTIN13PriorityCache8PriCacheE > > > typeinfo for PriorityCache::PriCache > > > > > > $ sudo find /lib* /usr/lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > > _ZTIN13PriorityCache8PriCacheE > > > /usr/lib64/ceph/libceph-common.so:008edab0 V > > > _ZTIN13PriorityCache8PriCacheE > > > /usr/lib64/ceph/libceph-common.so.0:008edab0 V > > > _ZTIN13PriorityCache8PriCacheE > > > > > > It needs to be able to find libceph-common, put it in your path or > > > preload it. > > > > > > > ``` > > > > > > > > I found a not-so-relevant > > > > issue(https://tracker.ceph.com/issues/38360), the error seems to be > > > > caused by mixed versions. My build environment is CentOS 7.5.1804 with > > > > SCL devtoolset-7, and ceph is latest master branch. Does someone know > > > > about the symbol? > > > > > > > > > > > > Best, > > > > Can Zhang > > > > > > > > Best, > > > > Can Zhang > > > > > > > > > > > > On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > > > > > > > > > Besides already mentioned store_test.cc one can also use ceph > > > > > objectstore fio plugin > > > > > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > > > > > standalone BlueStore instance from FIO benchmarking tool. > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Igor > > > > > > > > > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > > > > > Hi, > > > > > > > > > > > > I'd like to run a standalone Bluestore instance so as to test and > > > > > > tune > > > > > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > Can Zhang > > > > > > > > > > > > ___ > > > > > > ceph-users mailing list > > > > > > ceph-users@lists.ceph.com > > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > ___ > > > > ceph-users mailing list > > > > ceph-users@lists.ceph.com > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > > > > > -- > > > Cheers, > > > Brad > > > > -- > Cheers, > Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Does it define _ZTIN13PriorityCache8PriCacheE ? If it does, and all is as you say, then it should not say that _ZTIN13PriorityCache8PriCacheE is undefined. Does ldd show that it is finding the libraries you think it is? Either it is finding a different version of that library somewhere else or the version you have may not define that symbol. On Thu, Apr 18, 2019 at 11:12 AM Can Zhang wrote: > > It's already in LD_LIBRARY_PATH, under the same directory of > libfio_ceph_objectstore.so > > > $ ll lib/|grep libceph-common > lrwxrwxrwx. 1 root root19 Apr 17 11:15 libceph-common.so -> > libceph-common.so.0 > -rwxr-xr-x. 1 root root 211853400 Apr 17 11:15 libceph-common.so.0 > > > > > Best, > Can Zhang > > On Thu, Apr 18, 2019 at 7:00 AM Brad Hubbard wrote: > > > > On Wed, Apr 17, 2019 at 1:37 PM Can Zhang wrote: > > > > > > Thanks for your suggestions. > > > > > > I tried to build libfio_ceph_objectstore.so, but it fails to load: > > > > > > ``` > > > $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > > > > > > fio: engine libfio_ceph_objectstore.so not loadable > > > IO engine libfio_ceph_objectstore.so not found > > > ``` > > > > > > I managed to print the dlopen error, it said: > > > > > > ``` > > > dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > > > _ZTIN13PriorityCache8PriCacheE > > > > $ c++filt _ZTIN13PriorityCache8PriCacheE > > typeinfo for PriorityCache::PriCache > > > > $ sudo find /lib* /usr/lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > > _ZTIN13PriorityCache8PriCacheE > > /usr/lib64/ceph/libceph-common.so:008edab0 V > > _ZTIN13PriorityCache8PriCacheE > > /usr/lib64/ceph/libceph-common.so.0:008edab0 V > > _ZTIN13PriorityCache8PriCacheE > > > > It needs to be able to find libceph-common, put it in your path or preload > > it. > > > > > ``` > > > > > > I found a not-so-relevant > > > issue(https://tracker.ceph.com/issues/38360), the error seems to be > > > caused by mixed versions. My build environment is CentOS 7.5.1804 with > > > SCL devtoolset-7, and ceph is latest master branch. Does someone know > > > about the symbol? > > > > > > > > > Best, > > > Can Zhang > > > > > > Best, > > > Can Zhang > > > > > > > > > On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > > > > > > > Besides already mentioned store_test.cc one can also use ceph > > > > objectstore fio plugin > > > > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > > > > standalone BlueStore instance from FIO benchmarking tool. > > > > > > > > > > > > Thanks, > > > > > > > > Igor > > > > > > > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > > > > Hi, > > > > > > > > > > I'd like to run a standalone Bluestore instance so as to test and tune > > > > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > > > > > > > > > > > > > Best, > > > > > Can Zhang > > > > > > > > > > ___ > > > > > ceph-users mailing list > > > > > ceph-users@lists.ceph.com > > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > ___ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > > > > > -- > > Cheers, > > Brad -- Cheers, Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
It's already in LD_LIBRARY_PATH, under the same directory of libfio_ceph_objectstore.so $ ll lib/|grep libceph-common lrwxrwxrwx. 1 root root19 Apr 17 11:15 libceph-common.so -> libceph-common.so.0 -rwxr-xr-x. 1 root root 211853400 Apr 17 11:15 libceph-common.so.0 Best, Can Zhang On Thu, Apr 18, 2019 at 7:00 AM Brad Hubbard wrote: > > On Wed, Apr 17, 2019 at 1:37 PM Can Zhang wrote: > > > > Thanks for your suggestions. > > > > I tried to build libfio_ceph_objectstore.so, but it fails to load: > > > > ``` > > $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > > > > fio: engine libfio_ceph_objectstore.so not loadable > > IO engine libfio_ceph_objectstore.so not found > > ``` > > > > I managed to print the dlopen error, it said: > > > > ``` > > dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > > _ZTIN13PriorityCache8PriCacheE > > $ c++filt _ZTIN13PriorityCache8PriCacheE > typeinfo for PriorityCache::PriCache > > $ sudo find /lib* /usr/lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep > _ZTIN13PriorityCache8PriCacheE > /usr/lib64/ceph/libceph-common.so:008edab0 V > _ZTIN13PriorityCache8PriCacheE > /usr/lib64/ceph/libceph-common.so.0:008edab0 V > _ZTIN13PriorityCache8PriCacheE > > It needs to be able to find libceph-common, put it in your path or preload it. > > > ``` > > > > I found a not-so-relevant > > issue(https://tracker.ceph.com/issues/38360), the error seems to be > > caused by mixed versions. My build environment is CentOS 7.5.1804 with > > SCL devtoolset-7, and ceph is latest master branch. Does someone know > > about the symbol? > > > > > > Best, > > Can Zhang > > > > Best, > > Can Zhang > > > > > > On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > > > > > Besides already mentioned store_test.cc one can also use ceph > > > objectstore fio plugin > > > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > > > standalone BlueStore instance from FIO benchmarking tool. > > > > > > > > > Thanks, > > > > > > Igor > > > > > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > > > Hi, > > > > > > > > I'd like to run a standalone Bluestore instance so as to test and tune > > > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > > > > > > > > > Best, > > > > Can Zhang > > > > > > > > ___ > > > > ceph-users mailing list > > > > ceph-users@lists.ceph.com > > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > > > -- > Cheers, > Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
On Wed, Apr 17, 2019 at 1:37 PM Can Zhang wrote: > > Thanks for your suggestions. > > I tried to build libfio_ceph_objectstore.so, but it fails to load: > > ``` > $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > > fio: engine libfio_ceph_objectstore.so not loadable > IO engine libfio_ceph_objectstore.so not found > ``` > > I managed to print the dlopen error, it said: > > ``` > dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > _ZTIN13PriorityCache8PriCacheE $ c++filt _ZTIN13PriorityCache8PriCacheE typeinfo for PriorityCache::PriCache $ sudo find /lib* /usr/lib* -iname '*.so*' | xargs nm -AD 2>&1 | grep _ZTIN13PriorityCache8PriCacheE /usr/lib64/ceph/libceph-common.so:008edab0 V _ZTIN13PriorityCache8PriCacheE /usr/lib64/ceph/libceph-common.so.0:008edab0 V _ZTIN13PriorityCache8PriCacheE It needs to be able to find libceph-common, put it in your path or preload it. > ``` > > I found a not-so-relevant > issue(https://tracker.ceph.com/issues/38360), the error seems to be > caused by mixed versions. My build environment is CentOS 7.5.1804 with > SCL devtoolset-7, and ceph is latest master branch. Does someone know > about the symbol? > > > Best, > Can Zhang > > Best, > Can Zhang > > > On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > > > Besides already mentioned store_test.cc one can also use ceph > > objectstore fio plugin > > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > > standalone BlueStore instance from FIO benchmarking tool. > > > > > > Thanks, > > > > Igor > > > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > > Hi, > > > > > > I'd like to run a standalone Bluestore instance so as to test and tune > > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > > > > > Best, > > > Can Zhang > > > > > > ___ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -- Cheers, Brad ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
I fully rebuilt libfio_ceph_objectstore file on my Ubuntu VM. Best, Can Zhang On Wed, Apr 17, 2019 at 10:39 PM Igor Fedotov wrote: > > Or try full rebuild? > > On 4/17/2019 5:37 PM, Igor Fedotov wrote: > > Could you please check if libfio_ceph_objectstore.so has been rebuilt > > with your last build? > > > > On 4/17/2019 6:37 AM, Can Zhang wrote: > >> Thanks for your suggestions. > >> > >> I tried to build libfio_ceph_objectstore.so, but it fails to load: > >> > >> ``` > >> $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > >> > >> fio: engine libfio_ceph_objectstore.so not loadable > >> IO engine libfio_ceph_objectstore.so not found > >> ``` > >> > >> I managed to print the dlopen error, it said: > >> > >> ``` > >> dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > >> _ZTIN13PriorityCache8PriCacheE > >> ``` > >> > >> I found a not-so-relevant > >> issue(https://tracker.ceph.com/issues/38360), the error seems to be > >> caused by mixed versions. My build environment is CentOS 7.5.1804 with > >> SCL devtoolset-7, and ceph is latest master branch. Does someone know > >> about the symbol? > >> > >> > >> Best, > >> Can Zhang > >> > >> Best, > >> Can Zhang > >> > >> > >> On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > >>> Besides already mentioned store_test.cc one can also use ceph > >>> objectstore fio plugin > >>> (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > >>> standalone BlueStore instance from FIO benchmarking tool. > >>> > >>> > >>> Thanks, > >>> > >>> Igor > >>> > >>> On 4/16/2019 7:58 AM, Can ZHANG wrote: > Hi, > > I'd like to run a standalone Bluestore instance so as to test and tune > its performance. Are there any tools about it, or any suggestions? > > > > Best, > Can Zhang > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Or try full rebuild? On 4/17/2019 5:37 PM, Igor Fedotov wrote: Could you please check if libfio_ceph_objectstore.so has been rebuilt with your last build? On 4/17/2019 6:37 AM, Can Zhang wrote: Thanks for your suggestions. I tried to build libfio_ceph_objectstore.so, but it fails to load: ``` $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so fio: engine libfio_ceph_objectstore.so not loadable IO engine libfio_ceph_objectstore.so not found ``` I managed to print the dlopen error, it said: ``` dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: _ZTIN13PriorityCache8PriCacheE ``` I found a not-so-relevant issue(https://tracker.ceph.com/issues/38360), the error seems to be caused by mixed versions. My build environment is CentOS 7.5.1804 with SCL devtoolset-7, and ceph is latest master branch. Does someone know about the symbol? Best, Can Zhang Best, Can Zhang On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: Besides already mentioned store_test.cc one can also use ceph objectstore fio plugin (https://github.com/ceph/ceph/tree/master/src/test/fio) to access standalone BlueStore instance from FIO benchmarking tool. Thanks, Igor On 4/16/2019 7:58 AM, Can ZHANG wrote: Hi, I'd like to run a standalone Bluestore instance so as to test and tune its performance. Are there any tools about it, or any suggestions? Best, Can Zhang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Could you please check if libfio_ceph_objectstore.so has been rebuilt with your last build? On 4/17/2019 6:37 AM, Can Zhang wrote: Thanks for your suggestions. I tried to build libfio_ceph_objectstore.so, but it fails to load: ``` $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so fio: engine libfio_ceph_objectstore.so not loadable IO engine libfio_ceph_objectstore.so not found ``` I managed to print the dlopen error, it said: ``` dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: _ZTIN13PriorityCache8PriCacheE ``` I found a not-so-relevant issue(https://tracker.ceph.com/issues/38360), the error seems to be caused by mixed versions. My build environment is CentOS 7.5.1804 with SCL devtoolset-7, and ceph is latest master branch. Does someone know about the symbol? Best, Can Zhang Best, Can Zhang On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: Besides already mentioned store_test.cc one can also use ceph objectstore fio plugin (https://github.com/ceph/ceph/tree/master/src/test/fio) to access standalone BlueStore instance from FIO benchmarking tool. Thanks, Igor On 4/16/2019 7:58 AM, Can ZHANG wrote: Hi, I'd like to run a standalone Bluestore instance so as to test and tune its performance. Are there any tools about it, or any suggestions? Best, Can Zhang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Then I tried to build libfio_ceph_objectstore.so on a Ubuntu 18.04 vm, it seems to be working now. Best, Can Zhang On Wed, Apr 17, 2019 at 11:37 AM Can Zhang wrote: > > Thanks for your suggestions. > > I tried to build libfio_ceph_objectstore.so, but it fails to load: > > ``` > $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so > > fio: engine libfio_ceph_objectstore.so not loadable > IO engine libfio_ceph_objectstore.so not found > ``` > > I managed to print the dlopen error, it said: > > ``` > dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: > _ZTIN13PriorityCache8PriCacheE > ``` > > I found a not-so-relevant > issue(https://tracker.ceph.com/issues/38360), the error seems to be > caused by mixed versions. My build environment is CentOS 7.5.1804 with > SCL devtoolset-7, and ceph is latest master branch. Does someone know > about the symbol? > > > Best, > Can Zhang > > Best, > Can Zhang > > > On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > > > Besides already mentioned store_test.cc one can also use ceph > > objectstore fio plugin > > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > > standalone BlueStore instance from FIO benchmarking tool. > > > > > > Thanks, > > > > Igor > > > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > > Hi, > > > > > > I'd like to run a standalone Bluestore instance so as to test and tune > > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > > > > > Best, > > > Can Zhang > > > > > > ___ > > > ceph-users mailing list > > > ceph-users@lists.ceph.com > > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Thanks for your suggestions. I tried to build libfio_ceph_objectstore.so, but it fails to load: ``` $ LD_LIBRARY_PATH=./lib ./bin/fio --enghelp=libfio_ceph_objectstore.so fio: engine libfio_ceph_objectstore.so not loadable IO engine libfio_ceph_objectstore.so not found ``` I managed to print the dlopen error, it said: ``` dlopen error: ./lib/libfio_ceph_objectstore.so: undefined symbol: _ZTIN13PriorityCache8PriCacheE ``` I found a not-so-relevant issue(https://tracker.ceph.com/issues/38360), the error seems to be caused by mixed versions. My build environment is CentOS 7.5.1804 with SCL devtoolset-7, and ceph is latest master branch. Does someone know about the symbol? Best, Can Zhang Best, Can Zhang On Tue, Apr 16, 2019 at 8:37 PM Igor Fedotov wrote: > > Besides already mentioned store_test.cc one can also use ceph > objectstore fio plugin > (https://github.com/ceph/ceph/tree/master/src/test/fio) to access > standalone BlueStore instance from FIO benchmarking tool. > > > Thanks, > > Igor > > On 4/16/2019 7:58 AM, Can ZHANG wrote: > > Hi, > > > > I'd like to run a standalone Bluestore instance so as to test and tune > > its performance. Are there any tools about it, or any suggestions? > > > > > > > > Best, > > Can Zhang > > > > ___ > > ceph-users mailing list > > ceph-users@lists.ceph.com > > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Besides already mentioned store_test.cc one can also use ceph objectstore fio plugin (https://github.com/ceph/ceph/tree/master/src/test/fio) to access standalone BlueStore instance from FIO benchmarking tool. Thanks, Igor On 4/16/2019 7:58 AM, Can ZHANG wrote: Hi, I'd like to run a standalone Bluestore instance so as to test and tune its performance. Are there any tools about it, or any suggestions? Best, Can Zhang ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Is it possible to run a standalone Bluestore instance?
Depending on how low-level you are willing to go, a good place to start would be the unit test for the various object store implementations: https://github.com/ceph/ceph/blob/master/src/test/objectstore/store_test.cc Paul -- Paul Emmerich Looking for help with your Ceph cluster? Contact us at https://croit.io croit GmbH Freseniusstr. 31h 81247 München www.croit.io Tel: +49 89 1896585 90 On Tue, Apr 16, 2019 at 6:58 AM Can ZHANG wrote: > > Hi, > > I'd like to run a standalone Bluestore instance so as to test and tune its > performance. Are there any tools about it, or any suggestions? > > > > Best, > Can Zhang > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
