Re: issues with adjusting the crushmap in 0.51
Hi Tommi.
On 6 Sep 2012, at 21:31, Tommi Virtanen wrote:
On Thu, Sep 6, 2012 at 11:51 AM, Jimmy Tang jt...@tchpc.tcd.ie wrote:
Also, the ceph osd setcrushmap... command doesn't up when a ceph
--help is run in the 0.51 release, however it is documented on the
wiki as far as I recall. It'd be real nice if the applications emitted
all the available commands, it would make experimenting much nicer and
fun.
ceph is just a client app that sends (most of) the commands to
ceph-mon for execution; it's --help is problematic to keep up to
date.. We're trying to do better. Patches appreciated, see ticket at
the end..
Just to scratch an itch, here's some documentation additions from testing
locally,
[jtang@x00 ceph (master)]$ git diff
diff --git a/doc/cluster-ops/control.rst b/doc/cluster-ops/control.rst
index 9af4562..9946229 100644
--- a/doc/cluster-ops/control.rst
+++ b/doc/cluster-ops/control.rst
@@ -293,6 +293,10 @@ Enables debug messages. ::
Displays the status of all metadata servers.
+Make a pool usable by the fs with::
+
+ ceph mds add_data_pool {pool-name}
+
.. todo:: ``ceph mds`` subcommands missing docs: set_max_mds, dump, getmap,
stop, setmap
diff --git a/src/tools/ceph.cc b/src/tools/ceph.cc
index 0435771..306d67a 100644
--- a/src/tools/ceph.cc
+++ b/src/tools/ceph.cc
@@ -62,6 +62,7 @@ static void usage()
cout \n;
cout METADATA SERVER (MDS) COMMANDS\n;
coutceph mds stat\n;
+ coutceph mds add_data_pool pool\n;
coutceph mds tell mds-id or * injectargs '--switch value
[--switch value...]'\n;
cout \n;
cout MONITOR (MON) COMMANDS\n;
@@ -82,6 +83,7 @@ static void usage()
coutceph osd unpause\n;
coutceph osd tell osd-id or * injectargs '--switch value
[--switch value...]'\n;
coutceph osd getcrushmap -o file\n;
+ coutceph osd setcrushmap -i file\n;
coutceph osd getmap -o file\n;
coutceph osd crush set osd-id weight loc1 [loc2 ...]\n;
coutceph osd crush move bucketname loc1 [loc2 ...]\n;
Regards,
Jimmy Tang
--
Senior Software Engineer, Digital Repository of Ireland (DRI)
Trinity Centre for High Performance Computing,
Lloyd Building, Trinity College Dublin, Dublin 2, Ireland.
http://www.tchpc.tcd.ie/
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
rbd map error with new rbd format
Hi, whilst testing the new rbd layering feature I found a problem with rbd map. It seems rbd map doesn't support the new format. -martin ceph -v ceph version 0.51-265-gc7d11cd (commit:c7d11cd7b813a47167108c160358f70ec1aab7d6) rbd create --size 10 --new-format new rbd map new add failed: (2) No such file or directory rbd create --size 10 old rbd map old rbd showmapped id poolimage snapdevice 1 rbd old - /dev/rbd1 rbd info new rbd image 'new': size 10 MB in 25000 objects order 22 (4096 KB objects) block_name_prefix: rbd_data.101e1a89b511 old format: False features: layering rbd info old rbd image 'old': size 10 MB in 25000 objects order 22 (4096 KB objects) block_name_prefix: rb.0.1021.23697452 old format: True features: -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
[Solved] Re: Access Dienied for bucket upload - 403 code
Problem solved. Everything because Nginx, and a request_uri in fcgi params to radosgw. Now request_uri is ok, and problem disappear. Big thanks for help Yehuda. Regards Dnia 12 wrz 2012 o godz. 01:27 Yehuda Sadeh yeh...@inktank.com napisał(a): On Tue, Sep 11, 2012 at 1:41 PM, Sławomir Skowron szi...@gmail.com wrote: And more logs: 2012-09-11 21:03:38.357304 7faf0bf4f700 1 == req done req=0x141a650 http_status=403 == 2012-09-11 21:23:54.423185 7faf0bf4f700 20 dequeued request req=0x139a3d0 2012-09-11 21:23:54.423192 7faf0bf4f700 20 RGWWQ: empty 2012-09-11 21:23:54.423198 7faf0bf4f700 1 == starting new request req=0x139a3d0 = 2012-09-11 21:23:54.423237 7faf0bf4f700 2 req 58098:0.39initializing 2012-09-11 21:23:54.423258 7faf0bf4f700 10 s-object=NULL s-bucket=NULL 2012-09-11 21:23:54.423265 7faf0bf4f700 20 FCGI_ROLE=RESPONDER 2012-09-11 21:23:54.423267 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi 2012-09-11 21:23:54.423269 7faf0bf4f700 20 QUERY_STRING= 2012-09-11 21:23:54.423270 7faf0bf4f700 20 REQUEST_METHOD=GET 2012-09-11 21:23:54.423272 7faf0bf4f700 20 CONTENT_TYPE= 2012-09-11 21:23:54.423273 7faf0bf4f700 20 CONTENT_LENGTH= 2012-09-11 21:23:54.423274 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= 2012-09-11 21:23:54.423276 7faf0bf4f700 20 SCRIPT_NAME=/ 2012-09-11 21:23:54.423277 7faf0bf4f700 20 REQUEST_URI=/ 2012-09-11 21:23:54.423279 7faf0bf4f700 20 DOCUMENT_URI=/ 2012-09-11 21:23:54.423280 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www 2012-09-11 21:23:54.423282 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.0 2012-09-11 21:23:54.423283 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 2012-09-11 21:23:54.423284 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 2012-09-11 21:23:54.423286 7faf0bf4f700 20 REMOTE_ADDR=10.177.95.19 2012-09-11 21:23:54.423287 7faf0bf4f700 20 REMOTE_PORT=60477 2012-09-11 21:23:54.423289 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 2012-09-11 21:23:54.423290 7faf0bf4f700 20 SERVER_PORT=80 ...skipping... 2012-09-11 22:23:44.530567 7faf0bf4f700 10 s-object=images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg s-bucket=ocdn 2012-09-11 22:23:44.530586 7faf0bf4f700 20 FCGI_ROLE=RESPONDER 2012-09-11 22:23:44.530588 7faf0bf4f700 20 SCRIPT_FILENAME=/var/www/radosgw.fcgi 2012-09-11 22:23:44.530589 7faf0bf4f700 20 QUERY_STRING= 2012-09-11 22:23:44.530591 7faf0bf4f700 20 REQUEST_METHOD=GET 2012-09-11 22:23:44.530592 7faf0bf4f700 20 CONTENT_TYPE= 2012-09-11 22:23:44.530593 7faf0bf4f700 20 CONTENT_LENGTH= 2012-09-11 22:23:44.530594 7faf0bf4f700 20 HTTP_CONTENT_LENGTH= 2012-09-11 22:23:44.530595 7faf0bf4f700 20 SCRIPT_NAME=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg 2012-09-11 22:23:44.530596 7faf0bf4f700 20 REQUEST_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg 2012-09-11 22:23:44.530598 7faf0bf4f700 20 DOCUMENT_URI=/ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg 2012-09-11 22:23:44.530600 7faf0bf4f700 20 DOCUMENT_ROOT=/var/www 2012-09-11 22:23:44.530603 7faf0bf4f700 20 SERVER_PROTOCOL=HTTP/1.1 2012-09-11 22:23:44.530604 7faf0bf4f700 20 GATEWAY_INTERFACE=CGI/1.1 2012-09-11 22:23:44.530605 7faf0bf4f700 20 SERVER_SOFTWARE=nginx/1.2.0 2012-09-11 22:23:44.530606 7faf0bf4f700 20 REMOTE_ADDR=10.167.14.53 2012-09-11 22:23:44.530607 7faf0bf4f700 20 REMOTE_PORT=62145 2012-09-11 22:23:44.530608 7faf0bf4f700 20 SERVER_ADDR=10.177.64.4 2012-09-11 22:23:44.530609 7faf0bf4f700 20 SERVER_PORT=80 2012-09-11 22:23:44.530610 7faf0bf4f700 20 SERVER_NAME= 2012-09-11 22:23:44.530610 7faf0bf4f700 20 REDIRECT_STATUS=200 2012-09-11 22:23:44.530611 7faf0bf4f700 20 RGW_SHOULD_LOG=no 2012-09-11 22:23:44.530612 7faf0bf4f700 20 HTTP_HOST=10.177.64.4 2012-09-11 22:23:44.530613 7faf0bf4f700 20 HTTP_CONNECTION=keep-alive 2012-09-11 22:23:44.530614 7faf0bf4f700 20 HTTP_USER_AGENT=Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/537.1 (KHTML, like Gecko) Chrome/21.0.1180.89 Safari/537.1 2012-09-11 22:23:44.530615 7faf0bf4f700 20 HTTP_ACCEPT=text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 2012-09-11 22:23:44.530616 7faf0bf4f700 20 HTTP_ACCEPT_ENCODING=gzip,deflate,sdch 2012-09-11 22:23:44.530617 7faf0bf4f700 20 HTTP_ACCEPT_LANGUAGE=en-US,en;q=0.8 2012-09-11 22:23:44.530618 7faf0bf4f700 20 HTTP_ACCEPT_CHARSET=ISO-8859-1,utf-8;q=0.7,*;q=0.3 2012-09-11 22:23:44.530620 7faf0bf4f700 2 req 68518:0.000117:s3:GET /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg::getting op 2012-09-11 22:23:44.530626 7faf0bf4f700 2 req 68518:0.000123:s3:GET /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:authorizing 2012-09-11 22:23:44.530630 7faf0bf4f700 2 req 68518:0.000127:s3:GET /ocdn/images/pulscms/NjQ7MDMsMWUwLDAsMCwx/0a9915212e85062de6134566905cf252.jpg:get_obj:reading permissions 2012-09-11 22:23:44.530646 7faf0bf4f700 20 get_obj_state:
Re: rbd map error with new rbd format
On 09/12/2012 05:56 AM, Martin Mailand wrote: Hi, whilst testing the new rbd layering feature I found a problem with rbd map. It seems rbd map doesn't support the new format. Yeah, format 2 and layering support is in progress for kernel rbd, but not ready yet. The userspace side is all ready in the master branch, but it takes more time to implement in the kernel. Btw, instead of --new-format you should use --format 2. It's in the man page in the master branch. Josh -martin ceph -v ceph version 0.51-265-gc7d11cd (commit:c7d11cd7b813a47167108c160358f70ec1aab7d6) rbd create --size 10 --new-format new rbd map new add failed: (2) No such file or directory rbd create --size 10 old rbd map old rbd showmapped id poolimage snapdevice 1 rbd old - /dev/rbd1 rbd info new rbd image 'new': size 10 MB in 25000 objects order 22 (4096 KB objects) block_name_prefix: rbd_data.101e1a89b511 old format: False features: layering rbd info old rbd image 'old': size 10 MB in 25000 objects order 22 (4096 KB objects) block_name_prefix: rb.0.1021.23697452 old format: True features: -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: enabling cephx by default
On 09/12/2012 02:25 AM, Sage Weil wrote:
The next stable release will have cephx authentication enabled by default.
We will probably do it in the next development release (v0.53) to work out
any upgrade kinks well before that. The process for setting up teh
authentication keys on an existing cluster is at
http://ceph.com/docs/master/cluster-ops/authentication/
This needs a few eyeballs to make sure the upgrade process makes sense...
Generate a secret key for every OSD, where {$id} is the OSD number:
Where does {$id} come from? I know it's just a variable which the users
needs to fill in, but it could be somewhat confusing.
You could do:
for id in {0..10}; do
ceph auth get-or-create osd.${id} mon 'allow rwx' osd 'allow *' -o
/var/lib/ceph/osd/ceph-${id}/keyring;
done
I know this doesn't work for the mds which uses alpha-numeric names, but
imho the {$id} variable seems to come from nowhere.
Maybe an example to make it more clear, because later in the page $id is
used without the brackets ( { } )
Later on, this command won't work:
$ sudo ceph auth get-or-create client.admin mds 'allow' osd 'allow *'
mon 'allow *' /etc/ceph/keyring
The ceph command gets executed as root, but the output won't, so
writing to /etc/ceph/keyring will fail.
We could assume everybody executes these commands as root, but it might
be somewhat confusing if one command has sudo prefixed and other
don't. That might suggest it's somewhat special.
The same goes for a couple of commands after the one mentioned above.
I haven't tested the upgrade itself, but this is what I noticed while
reading the docs.
Wido
Thanks!
sage
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
--
To unsubscribe from this list: send the line unsubscribe ceph-devel in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Quick CentOS/RHEL question ...
On Wed, 12 Sep 2012, Jimmy Tang wrote: On Tue, Jul 31, 2012 at 06:09:59PM -0700, Yehuda Sadeh wrote: On Tue, Jul 31, 2012 at 3:17 PM, Joe Landman land...@scalableinformatics.com wrote: Hi folks I was struggling and failing to get Ceph properly built/installed for CentOS 6 (and 5) last week. Is this simply not a recommended platform? Please advise. Thanks! As mentioned by the other responses, it is possible to get Ceph built for RHEL/CentOS even though it's probably not trivial. Getting proper packaging for these distributions is pretty high on our list. We'd love to hear about specific the issues that you have had. Will there also be backports of the kernel modules for cephfs and rbd? That's probably a make or break feature for anyone using RHEL based distros as they are not likely to want to upgrade their kernels manually and maintain them for security fixes. I guess if there are no kernel modules for cephfs and rbd RHEL could still be used as a osd, mon or mds. We do not have any kernel backports planned at this point. The RHEL kernels also tend to be pretty old, so it's likely a fair bit of work. That said, the server stuff should all run fine on RHEL/CentOS. We are now doing builds on centos 6 as part of our continuous builds (ceph.com/gitbuilders.cgi), and the next release will include centos rpms (and hopefully Fedora). Looking forward, we'll be adding opensuse and sles rpms as well. sage -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: ceph-level build system fixes cleanup
On Tue, Sep 11, 2012 at 7:22 PM, Jan Engelhardt jeng...@inai.de wrote: per our short exchange in private, I am taking to spruce up the ceph build system definitions. First comes leveldb, more shall follow soonish. Thanks! Patches #1–2 are mandatory for continued successful operation, #3–5 cosmetical, and #6 a proposal. You are free to merge, pick or give blame :) I'd like to see a more explanatory commit message for 6/6, but a quick skim of all of these guys looked fine. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 6/6] build: provide libleveldb as a shared library
On Wed, 12 Sep 2012, Jan Engelhardt wrote: The library is used by ceph (main repo), and code share is a good thing, is it not? I'm not sure making this a .so has much value in this case. We can link against an installed libleveldb (and the upstream debian package does that). It's present here to make build painless in other environments, and to get the libleveldb unit test stuff coverage in our automated builds. Linking it statically to ceph-osd in that case makes sense. I wouldn't want this to interfere at all with an installed libleveldb.so... would it? The other patches look good, thanks! I'd be particularly happy to see if similar cleanups are possible with the ceph Makefile.am.. :) sage Signed-off-by: Jan Engelhardt jeng...@inai.de --- .gitignore|6 +- Makefile.am | 10 ++ configure.ac |4 +++- m4/.gitignore |2 ++ 4 files changed, 16 insertions(+), 6 deletions(-) create mode 100644 m4/.gitignore diff --git a/.gitignore b/.gitignore index bf40c93..972e902 100644 --- a/.gitignore +++ b/.gitignore @@ -7,14 +7,18 @@ /configure # created by ./configure -/.deps/ +.deps/ +.libs/ /Makefile /config.h /config.log /config.status +/libtool /stamp-h1 # created by make +*.la +*.lo *.o *.a /TAGS diff --git a/Makefile.am b/Makefile.am index 19b52a8..1d62bad 100644 --- a/Makefile.am +++ b/Makefile.am @@ -9,15 +9,15 @@ OPT ?= -O2 -DNDEBUG # (A) Production use (optimized mode) # TODO maybe support android chromium platforms via automake too? +ACLOCAL_AMFLAGS = -I m4 AM_CFLAGS = -I$(srcdir)/include $(OPT) -pthread -fno-builtin-memcmp -DLEVELDB_PLATFORM_POSIX AM_CXXFLAGS = -I$(srcdir)/include $(OPT) -pthread -fno-builtin-memcmp -DLEVELDB_PLATFORM_POSIX if CSTDATOMIC AM_CXXFLAGS += -DLEVELDB_CSTDATOMIC_PRESENT -std=c++0x endif -AM_LDFLAGS = -pthread -LDADD = libleveldb.a +LDADD = libleveldb.la if WITH_TCMALLOC LDADD += -ltcmalloc endif @@ -30,7 +30,7 @@ check_PROGRAMS = $(TESTS) # target by target TESTS = -noinst_LIBRARIES = libleveldb.a +lib_LTLIBRARIES = libleveldb.la noinst_HEADERS = \ util/random.h \ util/coding.h \ @@ -86,7 +86,7 @@ noinst_HEADERS = \ db/log_reader.h \ db/version_edit.h -libleveldb_a_SOURCES = \ +libleveldb_la_SOURCES = \ db/builder.cc \ db/c.cc \ db/db_impl.cc \ @@ -123,6 +123,8 @@ libleveldb_a_SOURCES = \ util/options.cc \ util/status.cc +libleveldb_la_LIBADD = -lpthread + TESTUTIL = util/testutil.cc TESTHARNESS = util/testharness.cc $(TESTUTIL) diff --git a/configure.ac b/configure.ac index 6a26ab9..6b29da4 100644 --- a/configure.ac +++ b/configure.ac @@ -1,5 +1,6 @@ AC_INIT([leveldb], [0.1], [leve...@googlegroups.com]) AC_CONFIG_AUX_DIR([build-aux]) +AC_CONFIG_MACRO_DIR([m4]) AM_INIT_AUTOMAKE([-Wall -Werror foreign]) m4_ifdef([AM_SILENT_RULES],[AM_SILENT_RULES([yes])]) AC_PROG_CC @@ -10,7 +11,8 @@ AC_CONFIG_FILES([ Makefile ]) m4_ifdef([AM_PROG_AR], [AM_PROG_AR]) -AC_PROG_RANLIB +AC_DISABLE_STATIC +AC_PROG_LIBTOOL AC_MSG_CHECKING(whether compiler supports C++11 cstdatomic) OLD_CXXFLAGS=$CXXFLAGS diff --git a/m4/.gitignore b/m4/.gitignore new file mode 100644 index 000..64d9bbc --- /dev/null +++ b/m4/.gitignore @@ -0,0 +1,2 @@ +/libtool.m4 +/lt*.m4 -- 1.7.10.4 -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Collection of strange lockups on 0.51
Hi, This is completely off-list, but I`m asking because only ceph trigger such a bug :) . With 0.51, following happens: if I kill an osd, one or more neighbor nodes may go to hanged state with cpu lockups, not related to temperature or overall interrupt count or la and it happens randomly over 16-node cluster. Almost sure that ceph triggerizing some hardware bug, but I don`t quite sure of which origin. Also after a short time after reset from such crash a new lockup may be created by any action. Before blaming system drivers and continuing to investigate a problem, may I ask if someone faced similar problem? I am using 802.ad on pair intel 350 for general connectivity. I have attached a bit of traces which was pushed to netconsole(in some cases, machine died hardly, e.g. not even sending a final bye over netconsole, so it is not complete). netcon.log.gz Description: GNU Zip compressed data
Re: messaging/IO/radosbench results
On Mon, Sep 10, 2012 at 10:39:58PM +0200, Mark Nelson wrote: On 09/10/2012 03:15 PM, Mike Ryan wrote: *Disclaimer*: these results are an investigation into potential bottlenecks in RADOS. I appreciate this investigation very much ! The test setup is wholly unrealistic, and these numbers SHOULD NOT be used as an indication of the performance of OSDs, messaging, RADOS, or ceph in general. Executive summary: rados bench has some internal bottleneck. Once that's cleared up, we're still having some issues saturating a single connection to an OSD. Having 2-3 connection in parallel alleviates that (either by having 1 OSD or having multiple bencher clients). I've run three separate tests: msbench, smalliobench, and rados bench. In all cases I was trying to determine where bottleneck(s) exist. All the tests were run on a machine with 192 GB of RAM. The backing stores for all OSDs and journals are RAMdisks. The stores are running XFS. smalliobench: I ran tests varying the number of OSDs and bencher clients. In all cases, the number of PG's per OSD is 100. OSD Bencher Throughput (mbyte/sec) 1 1 510 1 2 800 1 3 850 2 1 640 2 2 660 2 3 670 3 1 780 3 2 820 3 3 870 4 1 850 4 2 970 4 3 990 Note: these numbers are fairly fuzzy. I eyeballed them and they're only really accurate to about 10 mbyte/sec. The small IO bencher was run with 100 ops in flight, 4 mbyte io's, 4 mbyte files. msbench: ran tests trying to determine max throughput of raw messaging layer. Varied the number of concurrently connected msbench clients and measured aggregate throughput. Take-away: a messaging client can very consistently push 400-500 mbytes/sec through a single socket. Clients Throughput (mbyte/sec) 1 520 2 880 3 1300 4 1900 Finally, rados bench, which seems to have its own bottleneck. Running varying numbers of these, each client seems to get 250 mbyte/sec up till the aggregate rate is around 1000 mbyte/sec (appx line speed as measured by iperf). These were run on a pool with 100 PGs/OSD. Clients Throughput (mbyte/sec) 1 250 2 500 3 750 4 1000 (very fuzzy, probably 1000 +/- 75) 5 1000, seems to level out here -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hi guys, Some background on all of this: We've been doing some performance testing at Inktank and noticed that performance with a single rados bench instance was plateauing at between 600-700MB/s. 4-nodes with 10GbE interconnect; journals in RAM-Disk; replica=2 # rados bench -p pbench 20 write Maintaining 16 concurrent writes of 4194304 bytes for at least 20 seconds. sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 16 288 272 1087.81 1088 0.051123 0.0571643 2 16 579 563 1125.85 1164 0.045729 0.0561784 3 16 863 847 1129.19 1136 0.042012 0.0560869 4 16 1150 1134 1133.87 1148 0.05466 0.0559281 5 16 1441 1425 1139.87 1164 0.036852 0.0556809 6 16 1733 1717 1144.54 1168 0.054594 0.0556124 7 16 2007 1991 1137.59 1096 0.04454 0.0556698 8 16 2290 2274 1136.88 1132 0.046777 0.0560103 9 16 2580 2564 1139.44 1160 0.073328 0.0559353 10 16 2871 2855 1141.88 1164 0.034091 0.0558576 11 16 3158 3142 1142.43 1148 0.250688 0.0558404 12 16 3445 3429 1142.88 1148 0.046941 0.0558071 13 16 3726 3710 1141.42 1124 0.0540920.0559 14 16 4014 3998 1142.17 1152 0.03531 0.0558533 15 16 4298 4282 1141.75 1136 0.040005 0.0559383 16 16 4582 4566 1141.39 1136 0.048431 0.0559162 17 16 4859 4843 1139.42 1108 0.045805 0.0559891 18 16 5145 5129 1139.66 1144 0.046805 0.0560177 19 16 5422 5406 1137.99 1108 0.037295 0.0561341 2012-09-08 14:36:32.460311min lat: 0.029503 max lat: 0.47757 avg lat: 0.0561424 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 20 16 5701 5685 1136.89 1116 0.041493 0.0561424 Total time run: 20.197129 Total writes made: 5702 Write
Re: does ceph consider the device performance for distributing data?
Hi, may i ask which part of codes within ceph deal with the distribution of workloads among the OSDs? i am interested in ceph's source codes and want to understand that part of codes. Thanks, Sheng On Tue, Sep 11, 2012 at 11:51 AM, Tommi Virtanen t...@inktank.com wrote: On Mon, Sep 10, 2012 at 8:54 PM, sheng qiu herbert1984...@gmail.com wrote: i have a simple question. for distribution workload among OSDs, does ceph do any online modeling for OSDs, i.e. collect the online IO latency and try to distribute more workloads to lower latency OSDs? or only based on capacity/utilization balance? I've heard this question now several times. Ceph does not contain the logic to adjust OSD weights based on such measurements. Ceph will collect some measurements that are useful for this (perf counters, mostly). For a holistic systems view, the metrics Ceph collects aren't enough. You also want to take into account things like CPU load, available memory, hardware temperature sensors, network link utilization, network error rate, etc. And once you start looking at that angle, you realize that this is a completely generic machine health indicator, where Ceph is just one of the data sources. Hence, I view this as something that ultimately consumes more information than just Ceph. Ceph should play nice, and both feed information into such a system, and let the system set OSD weights; but I do believe it belongs outside of Ceph core. -- Sheng Qiu Texas A M University Room 332B Wisenbaker email: herbert1984...@gmail.com College Station, TX 77843-3259 -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: does ceph consider the device performance for distributing data?
On Wed, Sep 12, 2012 at 1:15 PM, sheng qiu herbert1984...@gmail.com wrote: may i ask which part of codes within ceph deal with the distribution of workloads among the OSDs? i am interested in ceph's source codes and want to understand that part of codes. You want to read the academic papers on CRUSH (or just the whole Ceph paper), to properly understand that part. http://ceph.com/resources/publications/ -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Fwd: does ceph consider the device performance for distributing data?
-- Forwarded message -- From: sheng qiu herbert1984...@gmail.com Date: Wed, Sep 12, 2012 at 3:26 PM Subject: Re: does ceph consider the device performance for distributing data? To: Tommi Virtanen t...@inktank.com actually i have read both these two papers. i understand the theory but just curious about how is it implemented within the codes. i looked into the ceph source codes, it seems very complicated for me. i want to read the codes about distribution workloads among OSDs, can you give some clue about which files are related with, If it is convenient? i am a Ph.d student and just for my research cares. Thanks, Sheng On Wed, Sep 12, 2012 at 3:20 PM, Tommi Virtanen t...@inktank.com wrote: On Wed, Sep 12, 2012 at 1:15 PM, sheng qiu herbert1984...@gmail.com wrote: may i ask which part of codes within ceph deal with the distribution of workloads among the OSDs? i am interested in ceph's source codes and want to understand that part of codes. You want to read the academic papers on CRUSH (or just the whole Ceph paper), to properly understand that part. http://ceph.com/resources/publications/ -- Sheng Qiu Texas A M University Room 332B Wisenbaker email: herbert1984...@gmail.com College Station, TX 77843-3259 -- Sheng Qiu Texas A M University Room 332B Wisenbaker email: herbert1984...@gmail.com College Station, TX 77843-3259 -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: does ceph consider the device performance for distributing data?
On Wed, Sep 12, 2012 at 1:27 PM, sheng qiu herbert1984...@gmail.com wrote: actually i have read both these two papers. i understand the theory but just curious about how is it implemented within the codes. i looked into the ceph source codes, it seems very complicated for me. The CRUSH library is in src/crush. It's tightly integrated into RADOS and, as you put it, the system as a whole is very complicated. Not sure what we can do about that; it's fundamentally a complex distributed programming problem space. The internal notes at http://ceph.com/docs/master/dev/ may be helpful. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Collection of strange lockups on 0.51
On Wed, Sep 12, 2012 at 10:33 AM, Andrey Korolyov and...@xdel.ru wrote: Hi, This is completely off-list, but I`m asking because only ceph trigger such a bug :) . With 0.51, following happens: if I kill an osd, one or more neighbor nodes may go to hanged state with cpu lockups, not related to temperature or overall interrupt count or la and it happens randomly over 16-node cluster. Almost sure that ceph triggerizing some hardware bug, but I don`t quite sure of which origin. Also after a short time after reset from such crash a new lockup may be created by any action. From the log, it looks like your ethernet driver is crapping out. [172517.057886] NETDEV WATCHDOG: eth0 (igb): transmit queue 7 timed out ... [172517.058622] [812b2975] ? netif_tx_lock+0x40/0x76 etc. The later oopses are talking about paravirt_write_msr etc, which makes me thing you're using Xen? You probably don't want to run Ceph servers inside virtualization (for production). [172696.503900] [8100d025] ? paravirt_write_msr+0xb/0xe [172696.503942] [810325f3] ? leave_mm+0x3e/0x3e and *then* you get [172695.041709] sd 0:2:0:0: [sda] megasas: RESET cmd=2a retries=0 [172695.041745] megasas: [ 0]waiting for 35 commands to complete [172696.045602] megaraid_sas: no pending cmds after reset [172696.045644] megasas: reset successful which just adds more awesomeness to the soup -- though I do wonder if this could be caused by the soft hang from earlier. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH] osd/ReplicatedPG: set truncate_seq when handling CEPH_OSD_OP_APPEND
Hi, On Tue, 11 Sep 2012, Yan, Zheng wrote: From: Yan, Zheng zheng.z@intel.com We need set truncate_seq when redirect the newop to CEPH_OSD_OP_WRITE, otherwise the code handles CEPH_OSD_OP_WRITE may quietly drop the data. Signed-off-by: Yan, Zheng zheng.z@intel.com Applying. This is correct, but I don't think there are any APPEND users currently in the tree. Did you run across this via inspection, or were you using this code in some other way? Thanks! sage --- src/osd/ReplicatedPG.cc | 1 + 1 file changed, 1 insertion(+) diff --git a/src/osd/ReplicatedPG.cc b/src/osd/ReplicatedPG.cc index f353090..fcd8be7 100644 --- a/src/osd/ReplicatedPG.cc +++ b/src/osd/ReplicatedPG.cc @@ -2310,6 +2310,7 @@ int ReplicatedPG::do_osd_ops(OpContext *ctx, vectorOSDOp ops) newop.op.op = CEPH_OSD_OP_WRITE; newop.op.extent.offset = oi.size; newop.op.extent.length = op.extent.length; + newop.op.extent.truncate_seq = oi.truncate_seq; newop.indata = osd_op.indata; do_osd_ops(ctx, nops); osd_op.outdata.claim(newop.outdata); -- 1.7.11.4 -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Collection of strange lockups on 0.51
On Thu, Sep 13, 2012 at 1:09 AM, Tommi Virtanen t...@inktank.com wrote: On Wed, Sep 12, 2012 at 10:33 AM, Andrey Korolyov and...@xdel.ru wrote: Hi, This is completely off-list, but I`m asking because only ceph trigger such a bug :) . With 0.51, following happens: if I kill an osd, one or more neighbor nodes may go to hanged state with cpu lockups, not related to temperature or overall interrupt count or la and it happens randomly over 16-node cluster. Almost sure that ceph triggerizing some hardware bug, but I don`t quite sure of which origin. Also after a short time after reset from such crash a new lockup may be created by any action. From the log, it looks like your ethernet driver is crapping out. [172517.057886] NETDEV WATCHDOG: eth0 (igb): transmit queue 7 timed out ... [172517.058622] [812b2975] ? netif_tx_lock+0x40/0x76 etc. The later oopses are talking about paravirt_write_msr etc, which makes me thing you're using Xen? You probably don't want to run Ceph servers inside virtualization (for production). NOPE. Xen was my choice for almost five years, but right now I am replaced it with kvm everywhere due to buggy 4.1 '-stable'. 4.0 has same poor network performance as 3.x but can be really named stable. All those backtraces comes from bare hardware. At the end you can see nice backtrace which comes out soon after end of the boot sequence when I manually typed 'modprobe rbd', it may be any other command assuming from experience. As soon as I don`t know anything about long-lasting states in intel, especially of those which will survive ipmi reset button, I think that first-sight complain about igb may be not quite right. If there cards may save some of runtime states to EEPROM and pull them back then I`m wrong. [172696.503900] [8100d025] ? paravirt_write_msr+0xb/0xe [172696.503942] [810325f3] ? leave_mm+0x3e/0x3e and *then* you get [172695.041709] sd 0:2:0:0: [sda] megasas: RESET cmd=2a retries=0 [172695.041745] megasas: [ 0]waiting for 35 commands to complete [172696.045602] megaraid_sas: no pending cmds after reset [172696.045644] megasas: reset successful which just adds more awesomeness to the soup -- though I do wonder if this could be caused by the soft hang from earlier. -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: messaging/IO/radosbench results
On 13 September 2012 08:25, Mark Nelson mark.nel...@inktank.com wrote: On 09/12/2012 03:08 PM, Dieter Kasper wrote: On Mon, Sep 10, 2012 at 10:39:58PM +0200, Mark Nelson wrote: On 09/10/2012 03:15 PM, Mike Ryan wrote: *Disclaimer*: these results are an investigation into potential bottlenecks in RADOS. I appreciate this investigation very much ! The test setup is wholly unrealistic, and these numbers SHOULD NOT be used as an indication of the performance of OSDs, messaging, RADOS, or ceph in general. Executive summary: rados bench has some internal bottleneck. Once that's cleared up, we're still having some issues saturating a single connection to an OSD. Having 2-3 connection in parallel alleviates that (either by having 1 OSD or having multiple bencher clients). I've run three separate tests: msbench, smalliobench, and rados bench. In all cases I was trying to determine where bottleneck(s) exist. All the tests were run on a machine with 192 GB of RAM. The backing stores for all OSDs and journals are RAMdisks. The stores are running XFS. smalliobench: I ran tests varying the number of OSDs and bencher clients. In all cases, the number of PG's per OSD is 100. OSD Bencher Throughput (mbyte/sec) 1 1 510 1 2 800 1 3 850 2 1 640 2 2 660 2 3 670 3 1 780 3 2 820 3 3 870 4 1 850 4 2 970 4 3 990 Note: these numbers are fairly fuzzy. I eyeballed them and they're only really accurate to about 10 mbyte/sec. The small IO bencher was run with 100 ops in flight, 4 mbyte io's, 4 mbyte files. msbench: ran tests trying to determine max throughput of raw messaging layer. Varied the number of concurrently connected msbench clients and measured aggregate throughput. Take-away: a messaging client can very consistently push 400-500 mbytes/sec through a single socket. Clients Throughput (mbyte/sec) 1 520 2 880 3 1300 4 1900 Finally, rados bench, which seems to have its own bottleneck. Running varying numbers of these, each client seems to get 250 mbyte/sec up till the aggregate rate is around 1000 mbyte/sec (appx line speed as measured by iperf). These were run on a pool with 100 PGs/OSD. Clients Throughput (mbyte/sec) 1 250 2 500 3 750 4 1000 (very fuzzy, probably 1000 +/- 75) 5 1000, seems to level out here -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hi guys, Some background on all of this: We've been doing some performance testing at Inktank and noticed that performance with a single rados bench instance was plateauing at between 600-700MB/s. 4-nodes with 10GbE interconnect; journals in RAM-Disk; replica=2 # rados bench -p pbench 20 write Maintaining 16 concurrent writes of 4194304 bytes for at least 20 seconds. sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 16 288 272 1087.81 1088 0.051123 0.0571643 2 16 579 563 1125.85 1164 0.045729 0.0561784 3 16 863 847 1129.19 1136 0.042012 0.0560869 4 16 1150 1134 1133.87 1148 0.05466 0.0559281 5 16 1441 1425 1139.87 1164 0.036852 0.0556809 6 16 1733 1717 1144.54 1168 0.054594 0.0556124 7 16 2007 1991 1137.59 1096 0.04454 0.0556698 8 16 2290 2274 1136.88 1132 0.046777 0.0560103 9 16 2580 2564 1139.44 1160 0.073328 0.0559353 10 16 2871 2855 1141.88 1164 0.034091 0.0558576 11 16 3158 3142 1142.43 1148 0.250688 0.0558404 12 16 3445 3429 1142.88 1148 0.046941 0.0558071 13 16 3726 3710 1141.42 1124 0.054092 0.0559 14 16 4014 3998 1142.17 1152 0.03531 0.0558533 15 16 4298 4282 1141.75 1136 0.040005 0.0559383 16 16 4582 4566 1141.39 1136 0.048431 0.0559162 17 16 4859 4843 1139.42 1108 0.045805 0.0559891 18 16 5145 5129 1139.66 1144 0.046805 0.0560177 19 16 5422 5406 1137.99 1108 0.037295 0.0561341 2012-09-08 14:36:32.460311min lat: 0.029503 max lat: 0.47757 avg lat: 0.0561424 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 20 16 5701
Re: messaging/IO/radosbench results
On 09/12/2012 06:24 PM, Joseph Glanville wrote: On 13 September 2012 08:25, Mark Nelsonmark.nel...@inktank.com wrote: On 09/12/2012 03:08 PM, Dieter Kasper wrote: On Mon, Sep 10, 2012 at 10:39:58PM +0200, Mark Nelson wrote: On 09/10/2012 03:15 PM, Mike Ryan wrote: *Disclaimer*: these results are an investigation into potential bottlenecks in RADOS. I appreciate this investigation very much ! The test setup is wholly unrealistic, and these numbers SHOULD NOT be used as an indication of the performance of OSDs, messaging, RADOS, or ceph in general. Executive summary: rados bench has some internal bottleneck. Once that's cleared up, we're still having some issues saturating a single connection to an OSD. Having 2-3 connection in parallel alleviates that (either by having1 OSD or having multiple bencher clients). I've run three separate tests: msbench, smalliobench, and rados bench. In all cases I was trying to determine where bottleneck(s) exist. All the tests were run on a machine with 192 GB of RAM. The backing stores for all OSDs and journals are RAMdisks. The stores are running XFS. smalliobench: I ran tests varying the number of OSDs and bencher clients. In all cases, the number of PG's per OSD is 100. OSD Bencher Throughput (mbyte/sec) 1 1 510 1 2 800 1 3 850 2 1 640 2 2 660 2 3 670 3 1 780 3 2 820 3 3 870 4 1 850 4 2 970 4 3 990 Note: these numbers are fairly fuzzy. I eyeballed them and they're only really accurate to about 10 mbyte/sec. The small IO bencher was run with 100 ops in flight, 4 mbyte io's, 4 mbyte files. msbench: ran tests trying to determine max throughput of raw messaging layer. Varied the number of concurrently connected msbench clients and measured aggregate throughput. Take-away: a messaging client can very consistently push 400-500 mbytes/sec through a single socket. Clients Throughput (mbyte/sec) 1 520 2 880 3 1300 4 1900 Finally, rados bench, which seems to have its own bottleneck. Running varying numbers of these, each client seems to get 250 mbyte/sec up till the aggregate rate is around 1000 mbyte/sec (appx line speed as measured by iperf). These were run on a pool with 100 PGs/OSD. Clients Throughput (mbyte/sec) 1 250 2 500 3 750 4 1000 (very fuzzy, probably 1000 +/- 75) 5 1000, seems to level out here -- To unsubscribe from this list: send the line unsubscribe ceph-devel in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Hi guys, Some background on all of this: We've been doing some performance testing at Inktank and noticed that performance with a single rados bench instance was plateauing at between 600-700MB/s. 4-nodes with 10GbE interconnect; journals in RAM-Disk; replica=2 # rados bench -p pbench 20 write Maintaining 16 concurrent writes of 4194304 bytes for at least 20 seconds. sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 0 0 0 0 0 0 - 0 1 16 288 272 1087.81 1088 0.051123 0.0571643 2 16 579 563 1125.85 1164 0.045729 0.0561784 3 16 863 847 1129.19 1136 0.042012 0.0560869 4 16 1150 1134 1133.87 1148 0.05466 0.0559281 5 16 1441 1425 1139.87 1164 0.036852 0.0556809 6 16 1733 1717 1144.54 1168 0.054594 0.0556124 7 16 2007 1991 1137.59 1096 0.04454 0.0556698 8 16 2290 2274 1136.88 1132 0.046777 0.0560103 9 16 2580 2564 1139.44 1160 0.073328 0.0559353 10 16 2871 2855 1141.88 1164 0.034091 0.0558576 11 16 3158 3142 1142.43 1148 0.250688 0.0558404 12 16 3445 3429 1142.88 1148 0.046941 0.0558071 13 16 3726 3710 1141.42 1124 0.054092 0.0559 14 16 4014 3998 1142.17 1152 0.03531 0.0558533 15 16 4298 4282 1141.75 1136 0.040005 0.0559383 16 16 4582 4566 1141.39 1136 0.048431 0.0559162 17 16 4859 4843 1139.42 1108 0.045805 0.0559891 18 16 5145 5129 1139.66 1144 0.046805 0.0560177 19 16 5422 5406 1137.99 1108 0.037295 0.0561341 2012-09-08 14:36:32.460311min lat: 0.029503 max lat: 0.47757 avg lat: 0.0561424 sec Cur ops started finished avg MB/s cur MB/s last lat avg lat 20 16 5701 5685 1136.89 1116
