date:20160427

Re: [ceph-users] osds udev rules not triggered on reboot (jewel, jessie)

2016-04-27 Thread Alexandre DERUMIER

Hi, 
they are missing target files in debian packages

http://tracker.ceph.com/issues/15573
https://github.com/ceph/ceph/pull/8700

I have also done some other trackers about packaging bug

jewel: debian package: wrong /etc/default/ceph/ceph location
http://tracker.ceph.com/issues/15587

debian/ubuntu : TCMALLOC_MAX_TOTAL_THREAD_CACHE_BYTES not specified in 
/etc/default/cep
http://tracker.ceph.com/issues/15588

jewel: debian package: init.d script bug
http://tracker.ceph.com/issues/15585


@CC loic dachary, maybe he could help to speed up packaging fixes

- Mail original -
De: "Karsten Heymann" 
À: "Loris Cuoghi" 
Cc: "ceph-users" 
Envoyé: Mercredi 27 Avril 2016 15:20:29
Objet: Re: [ceph-users] osds udev rules not triggered on reboot (jewel, jessie)

2016-04-27 15:18 GMT+02:00 Loris Cuoghi : 
> Le 27/04/2016 14:45, Karsten Heymann a écrit : 
>> one workaround I found was to add 
>> 
>> [Install] 
>> WantedBy=ceph-osd.target 
>> 
>> to /lib/systemd/system/ceph-disk@.service and then manually enable my 
>> disks with 
>> 
>> # systemctl enable ceph-disk\@dev-sdi1 
>> # systemctl start ceph-disk\@dev-sdi1 
>> 
>> That way they at least are started at boot time. 

> Great! But only if the disks keep their device names, right ? 

Exactly. It's just a little workaround until the real issue is fixed. 

+Karsten 
___ 
ceph-users mailing list 
ceph-users@lists.ceph.com 
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw crash - Infernalis

2016-04-27 Thread Ben Hines

Aha, i see how to use the debuginfo - trying it by running through gdb.


On Wed, Apr 27, 2016 at 10:09 PM, Ben Hines  wrote:

> Got it again - however, the stack is exactly the same, no symbols -
> debuginfo didn't resolve. Do i need to do something to enable that?
>
> The server in 'debug ms=10' this time, so there is a bit more spew:
>
>-14> 2016-04-27 21:59:58.811919 7f9e817fa700  1 --
> 10.30.1.8:0/3291985349 --> 10.30.2.13:6805/27519 --
> osd_op(client.44936150.0:223 obj_delete_at_hint.55 [call
> timeindex.list] 10.2c88dbcf ack+read+known_if_redirected e100564) v6 -- ?+0
> 0x7f9f140dc5f0 con 0x7f9f1410ed10
>-13> 2016-04-27 21:59:58.812039 7f9e3fa6b700 10 --
> 10.30.1.8:0/3291985349 >> 10.30.2.13:6805/27519 pipe(0x7f9f14110010
> sd=153 :10861 s=2 pgs=725914 cs=1 l=1 c=0x7f9f1410ed10).writer: state =
> open policy.server=0
>-12> 2016-04-27 21:59:58.812096 7f9e3fa6b700 10 --
> 10.30.1.8:0/3291985349 >> 10.30.2.13:6805/27519 pipe(0x7f9f14110010
> sd=153 :10861 s=2 pgs=725914 cs=1 l=1 c=0x7f9f1410ed10).writer: state =
> open policy.server=0
>-11> 2016-04-27 21:59:58.814343 7f9e3f96a700 10 --
> 10.30.1.8:0/3291985349 >> 10.30.2.13:6805/27519 pipe(0x7f9f14110010
> sd=153 :10861 s=2 pgs=725914 cs=1 l=1 c=0x7f9f1410ed10).reader wants 211
> from dispatch throttler 0/104857600
>-10> 2016-04-27 21:59:58.814375 7f9e3f96a700 10 --
> 10.30.1.8:0/3291985349 >> 10.30.2.13:6805/27519 pipe(0x7f9f14110010
> sd=153 :10861 s=2 pgs=725914 cs=1 l=1 c=0x7f9f1410ed10).aborted = 0
> -9> 2016-04-27 21:59:58.814405 7f9e3f96a700 10 --
> 10.30.1.8:0/3291985349 >> 10.30.2.13:6805/27519 pipe(0x7f9f14110010
> sd=153 :10861 s=2 pgs=725914 cs=1 l=1 c=0x7f9f1410ed10).reader got message
> 2 0x7f9ec0009250 osd_op_reply(223 obj_delete_at_hint.55 [call] v0'0
> uv1448004 ondisk = 0) v6
> -8> 2016-04-27 21:59:58.814428 7f9e3f96a700  1 --
> 10.30.1.8:0/3291985349 <== osd.6 10.30.2.13:6805/27519 2 
> osd_op_reply(223 obj_delete_at_hint.55 [call] v0'0 uv1448004 ondisk
> = 0) v6  196+0+15 (3849172018 0 2149983739) 0x7f9ec0009250 con
> 0x7f9f1410ed10
> -7> 2016-04-27 21:59:58.814472 7f9e3f96a700 10 --
> 10.30.1.8:0/3291985349 dispatch_throttle_release 211 to dispatch
> throttler 211/104857600
> -6> 2016-04-27 21:59:58.814470 7f9e3fa6b700 10 --
> 10.30.1.8:0/3291985349 >> 10.30.2.13:6805/27519 pipe(0x7f9f14110010
> sd=153 :10861 s=2 pgs=725914 cs=1 l=1 c=0x7f9f1410ed10).writer: state =
> open policy.server=0
> -5> 2016-04-27 21:59:58.814511 7f9e3fa6b700 10 --
> 10.30.1.8:0/3291985349 >> 10.30.2.13:6805/27519 pipe(0x7f9f14110010
> sd=153 :10861 s=2 pgs=725914 cs=1 l=1 c=0x7f9f1410ed10).write_ack 2
> -4> 2016-04-27 21:59:58.814528 7f9e3fa6b700 10 --
> 10.30.1.8:0/3291985349 >> 10.30.2.13:6805/27519 pipe(0x7f9f14110010
> sd=153 :10861 s=2 pgs=725914 cs=1 l=1 c=0x7f9f1410ed10).writer: state =
> open policy.server=0
> -3> 2016-04-27 21:59:58.814607 7f9e817fa700  1 --
> 10.30.1.8:0/3291985349 --> 10.30.2.13:6805/27519 --
> osd_op(client.44936150.0:224 obj_delete_at_hint.55 [call
> lock.unlock] 10.2c88dbcf ondisk+write+known_if_redirected e100564) v6 --
> ?+0 0x7f9f140dc5f0 con 0x7f9f1410ed10
> -2> 2016-04-27 21:59:58.814718 7f9e3fa6b700 10 --
> 10.30.1.8:0/3291985349 >> 10.30.2.13:6805/27519 pipe(0x7f9f14110010
> sd=153 :10861 s=2 pgs=725914 cs=1 l=1 c=0x7f9f1410ed10).writer: state =
> open policy.server=0
> -1> 2016-04-27 21:59:58.814778 7f9e3fa6b700 10 --
> 10.30.1.8:0/3291985349 >> 10.30.2.13:6805/27519 pipe(0x7f9f14110010
> sd=153 :10861 s=2 pgs=725914 cs=1 l=1 c=0x7f9f1410ed10).writer: state =
> open policy.server=0
>  0> 2016-04-27 21:59:58.826494 7f9e7e7f4700 -1 *** Caught signal
> (Segmentation fault) **
>  in thread 7f9e7e7f4700
>
>  ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
>  1: (()+0x30b0a2) [0x7fa11c5030a2]
>  2: (()+0xf100) [0x7fa1183fe100]
>  NOTE: a copy of the executable, or `objdump -rdS ` is needed
> to interpret this.
>
> --- logging levels ---
> 
>
>
> On Wed, Apr 27, 2016 at 9:39 PM, Ben Hines  wrote:
>
>> Yes, CentOS 7.2. Happened twice in a row, both times shortly after a
>> restart, so i expect i'll be able to reproduce it. However, i've now tried
>> a bunch of times and it's not happening again.
>>
>> In any case i have glibc + ceph-debuginfo installed so we can get more
>> info if it does happen.
>>
>> thanks!
>>
>> On Wed, Apr 27, 2016 at 8:40 PM, Brad Hubbard 
>> wrote:
>>
>>> - Original Message -
>>> > From: "Karol Mroz" 
>>> > To: "Ben Hines" 
>>> > Cc: "ceph-users" 
>>> > Sent: Wednesday, 27 April, 2016 7:06:56 PM
>>> > Subject: Re: [ceph-users] radosgw crash - Infernalis
>>> >
>>> > On Tue, Apr 26, 2016 at 10:17:31PM -0700, Ben Hines wrote:
>>> > [...]
>>> > > --> 10.30.1.6:6800/10350 -- osd_op(client.44852756.0:79
>>> > > default.42048218. [getxattrs,stat,read 0~524288]
>>>

Re: [ceph-users] radosgw crash - Infernalis

2016-04-27 Thread Brad Hubbard

- Original Message - 

> From: "Ben Hines" 
> To: "Brad Hubbard" 
> Cc: "Karol Mroz" , "ceph-users" 
> Sent: Thursday, 28 April, 2016 3:09:16 PM
> Subject: Re: [ceph-users] radosgw crash - Infernalis

> Got it again - however, the stack is exactly the same, no symbols - debuginfo
> didn't resolve. Do i need to do something to enable that?

It's possible we are in a library for which you don't have debuginfo loaded.
Given the list of libraries that radosgw links to getting all debuginfo loaded
may be a daunting prospect. The other possibility is the stack is badly
corrupted as Karol suggested.

Any chance you can capture a core?

You could try setting "ulimit -c unlimited" and starting the osd from the
command line.

HTH,
Brad

> The server in 'debug ms=10' this time, so there is a bit more spew:

> -14> 2016-04-27 21:59:58.811919 7f9e817fa700 1 -- 10.30.1.8:0/3291985349 -->
> 10.30.2.13:6805/27519 -- osd_op(client.44936150.0:223
> obj_delete_at_hint.55 [call timeindex.list] 10.2c88dbcf
> ack+read+known_if_redirected e100564) v6 -- ?+0 0x7f9f140dc5f0 con
> 0x7f9f1410ed10
> -13> 2016-04-27 21:59:58.812039 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349 >>
> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914 cs=1
> l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
> -12> 2016-04-27 21:59:58.812096 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349 >>
> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914 cs=1
> l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
> -11> 2016-04-27 21:59:58.814343 7f9e3f96a700 10 -- 10.30.1.8:0/3291985349 >>
> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914 cs=1
> l=1 c=0x7f9f1410ed10).reader wants 211 from dispatch throttler 0/104857600
> -10> 2016-04-27 21:59:58.814375 7f9e3f96a700 10 -- 10.30.1.8:0/3291985349 >>
> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914 cs=1
> l=1 c=0x7f9f1410ed10).aborted = 0
> -9> 2016-04-27 21:59:58.814405 7f9e3f96a700 10 -- 10.30.1.8:0/3291985349 >>
> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914 cs=1
> l=1 c=0x7f9f1410ed10).reader got message 2 0x7f9ec0009250 osd_op_reply(223
> obj_delete_at_hint.55 [call] v0'0 uv1448004 ondisk = 0) v6
> -8> 2016-04-27 21:59:58.814428 7f9e3f96a700 1 -- 10.30.1.8:0/3291985349 <==
> osd.6 10.30.2.13:6805/27519 2  osd_op_reply(223
> obj_delete_at_hint.55 [call] v0'0 uv1448004 ondisk = 0) v6 
> 196+0+15 (3849172018 0 2149983739) 0x7f9ec0009250 con 0x7f9f1410ed10
> -7> 2016-04-27 21:59:58.814472 7f9e3f96a700 10 -- 10.30.1.8:0/3291985349
> dispatch_throttle_release 211 to dispatch throttler 211/104857600
> -6> 2016-04-27 21:59:58.814470 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349 >>
> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914 cs=1
> l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
> -5> 2016-04-27 21:59:58.814511 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349 >>
> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914 cs=1
> l=1 c=0x7f9f1410ed10).write_ack 2
> -4> 2016-04-27 21:59:58.814528 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349 >>
> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914 cs=1
> l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
> -3> 2016-04-27 21:59:58.814607 7f9e817fa700 1 -- 10.30.1.8:0/3291985349 -->
> 10.30.2.13:6805/27519 -- osd_op(client.44936150.0:224
> obj_delete_at_hint.55 [call lock.unlock] 10.2c88dbcf
> ondisk+write+known_if_redirected e100564) v6 -- ?+0 0x7f9f140dc5f0 con
> 0x7f9f1410ed10
> -2> 2016-04-27 21:59:58.814718 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349 >>
> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914 cs=1
> l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
> -1> 2016-04-27 21:59:58.814778 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349 >>
> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914 cs=1
> l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
> 0> 2016-04-27 21:59:58.826494 7f9e7e7f4700 -1 *** Caught signal (Segmentation
> fault) **
> in thread 7f9e7e7f4700

> ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
> 1: (()+0x30b0a2) [0x7fa11c5030a2]
> 2: (()+0xf100) [0x7fa1183fe100]
> NOTE: a copy of the executable, or `objdump -rdS ` is needed to
> interpret this.

> --- logging levels ---
> 

> On Wed, Apr 27, 2016 at 9:39 PM, Ben Hines < bhi...@gmail.com > wrote:

> > Yes, CentOS 7.2. Happened twice in a row, both times shortly after a
> > restart,
> > so i expect i'll be able to reproduce it. However, i've now tried a bunch
> > of
> > times and it's not happening again.
> 

> > In any case i have glibc + ceph-debuginfo installed so we can get more info
> > if it does happen.
> 

> > thanks!
> 

> > On Wed, Apr 27, 2016 at 8:40 PM, Brad Hubbard < bhubb...@redhat.com >
> > wrote:
> 

> > > -

Re: [ceph-users] radosgw crash - Infernalis

2016-04-27 Thread Ben Hines

Got it again - however, the stack is exactly the same, no symbols -
debuginfo didn't resolve. Do i need to do something to enable that?

The server in 'debug ms=10' this time, so there is a bit more spew:

   -14> 2016-04-27 21:59:58.811919 7f9e817fa700  1 -- 10.30.1.8:0/3291985349
--> 10.30.2.13:6805/27519 -- osd_op(client.44936150.0:223
obj_delete_at_hint.55 [call timeindex.list] 10.2c88dbcf
ack+read+known_if_redirected e100564) v6 -- ?+0 0x7f9f140dc5f0 con
0x7f9f1410ed10
   -13> 2016-04-27 21:59:58.812039 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349
>> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914
cs=1 l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
   -12> 2016-04-27 21:59:58.812096 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349
>> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914
cs=1 l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
   -11> 2016-04-27 21:59:58.814343 7f9e3f96a700 10 -- 10.30.1.8:0/3291985349
>> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914
cs=1 l=1 c=0x7f9f1410ed10).reader wants 211 from dispatch throttler
0/104857600
   -10> 2016-04-27 21:59:58.814375 7f9e3f96a700 10 -- 10.30.1.8:0/3291985349
>> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914
cs=1 l=1 c=0x7f9f1410ed10).aborted = 0
-9> 2016-04-27 21:59:58.814405 7f9e3f96a700 10 -- 10.30.1.8:0/3291985349
>> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914
cs=1 l=1 c=0x7f9f1410ed10).reader got message 2 0x7f9ec0009250
osd_op_reply(223 obj_delete_at_hint.55 [call] v0'0 uv1448004 ondisk
= 0) v6
-8> 2016-04-27 21:59:58.814428 7f9e3f96a700  1 -- 10.30.1.8:0/3291985349
<== osd.6 10.30.2.13:6805/27519 2  osd_op_reply(223
obj_delete_at_hint.55 [call] v0'0 uv1448004 ondisk = 0) v6 
196+0+15 (3849172018 0 2149983739) 0x7f9ec0009250 con 0x7f9f1410ed10
-7> 2016-04-27 21:59:58.814472 7f9e3f96a700 10 -- 10.30.1.8:0/3291985349
dispatch_throttle_release 211 to dispatch throttler 211/104857600
-6> 2016-04-27 21:59:58.814470 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349
>> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914
cs=1 l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
-5> 2016-04-27 21:59:58.814511 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349
>> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914
cs=1 l=1 c=0x7f9f1410ed10).write_ack 2
-4> 2016-04-27 21:59:58.814528 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349
>> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914
cs=1 l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
-3> 2016-04-27 21:59:58.814607 7f9e817fa700  1 -- 10.30.1.8:0/3291985349
--> 10.30.2.13:6805/27519 -- osd_op(client.44936150.0:224
obj_delete_at_hint.55 [call lock.unlock] 10.2c88dbcf
ondisk+write+known_if_redirected e100564) v6 -- ?+0 0x7f9f140dc5f0 con
0x7f9f1410ed10
-2> 2016-04-27 21:59:58.814718 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349
>> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914
cs=1 l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
-1> 2016-04-27 21:59:58.814778 7f9e3fa6b700 10 -- 10.30.1.8:0/3291985349
>> 10.30.2.13:6805/27519 pipe(0x7f9f14110010 sd=153 :10861 s=2 pgs=725914
cs=1 l=1 c=0x7f9f1410ed10).writer: state = open policy.server=0
 0> 2016-04-27 21:59:58.826494 7f9e7e7f4700 -1 *** Caught signal
(Segmentation fault) **
 in thread 7f9e7e7f4700

 ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
 1: (()+0x30b0a2) [0x7fa11c5030a2]
 2: (()+0xf100) [0x7fa1183fe100]
 NOTE: a copy of the executable, or `objdump -rdS ` is needed
to interpret this.

--- logging levels ---



On Wed, Apr 27, 2016 at 9:39 PM, Ben Hines  wrote:

> Yes, CentOS 7.2. Happened twice in a row, both times shortly after a
> restart, so i expect i'll be able to reproduce it. However, i've now tried
> a bunch of times and it's not happening again.
>
> In any case i have glibc + ceph-debuginfo installed so we can get more
> info if it does happen.
>
> thanks!
>
> On Wed, Apr 27, 2016 at 8:40 PM, Brad Hubbard  wrote:
>
>> - Original Message -
>> > From: "Karol Mroz" 
>> > To: "Ben Hines" 
>> > Cc: "ceph-users" 
>> > Sent: Wednesday, 27 April, 2016 7:06:56 PM
>> > Subject: Re: [ceph-users] radosgw crash - Infernalis
>> >
>> > On Tue, Apr 26, 2016 at 10:17:31PM -0700, Ben Hines wrote:
>> > [...]
>> > > --> 10.30.1.6:6800/10350 -- osd_op(client.44852756.0:79
>> > > default.42048218. [getxattrs,stat,read 0~524288] 12.aa730416
>> > > ack+read+known_if_redirected e100207) v6 -- ?+0 0x7f49c41880b0 con
>> > > 0x7f49c4145eb0
>> > >  0> 2016-04-26 22:07:59.685615 7f49a07f0700 -1 *** Caught signal
>> > > (Segmentation fault) **
>> > >  in thread 7f49a07f0700
>> > >
>> > >  ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
>> >

Re: [ceph-users] radosgw crash - Infernalis

2016-04-27 Thread Ben Hines

Yes, CentOS 7.2. Happened twice in a row, both times shortly after a
restart, so i expect i'll be able to reproduce it. However, i've now tried
a bunch of times and it's not happening again.

In any case i have glibc + ceph-debuginfo installed so we can get more info
if it does happen.

thanks!

On Wed, Apr 27, 2016 at 8:40 PM, Brad Hubbard  wrote:

> - Original Message -
> > From: "Karol Mroz" 
> > To: "Ben Hines" 
> > Cc: "ceph-users" 
> > Sent: Wednesday, 27 April, 2016 7:06:56 PM
> > Subject: Re: [ceph-users] radosgw crash - Infernalis
> >
> > On Tue, Apr 26, 2016 at 10:17:31PM -0700, Ben Hines wrote:
> > [...]
> > > --> 10.30.1.6:6800/10350 -- osd_op(client.44852756.0:79
> > > default.42048218. [getxattrs,stat,read 0~524288] 12.aa730416
> > > ack+read+known_if_redirected e100207) v6 -- ?+0 0x7f49c41880b0 con
> > > 0x7f49c4145eb0
> > >  0> 2016-04-26 22:07:59.685615 7f49a07f0700 -1 *** Caught signal
> > > (Segmentation fault) **
> > >  in thread 7f49a07f0700
> > >
> > >  ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
> > >  1: (()+0x30b0a2) [0x7f4c4907f0a2]
> > >  2: (()+0xf100) [0x7f4c44f7a100]
> > >  NOTE: a copy of the executable, or `objdump -rdS ` is
> needed
> > > to interpret this.
> >
> > Hi Ben,
> >
> > I sense a pretty badly corrupted stack. From the radosgw-9.2.1 (obtained
> from
> > a downloaded rpm):
> >
> > 0030a810 <_Z13pidfile_writePK11md_config_t@@Base>:
> > ...
> >   30b09d:   e8 0e 40 e4 ff  callq  14f0b0 
> >   30b0a2:   4c 89 efmov%r13,%rdi
> >   ---
> > ...
> >
> > So either we tripped backtrace() code from pidfile_write() _or_ we can't
> > trust the stack. From the log snippet, it looks that we're far past the
> point
> > at which we would write a pidfile to disk (ie. at process start during
> > global_init()).
> > Rather, we're actually handling a request and outputting some bit of
> debug
> > message
> > via MSDOp::print() and beyond...
>
> It would help to know what binary this is and what OS.
>
> We know the offset into the function is 0x30b0a2 but we don't know which
> function yet AFAICT. Karol, how did you arrive at pidfile_write? Purely
> from
> the offset? I'm not sure that would be reliable...
>
> This is a segfault so the address of the frame where we crashed should be
> the
> exact instruction where we crashed. I don't believe a mov from one
> register to
> another that does not involve a dereference ((%r13) as opposed to %r13) can
> cause a segfault so I don't think we are on the right instruction but
> then, as
> you say, the stack may be corrupt.
>
> >
> > Is this something you're able to easily reproduce? More logs with higher
> log
> > levels
> > would be helpful... a coredump with radosgw compiled with -g would be
> > excellent :)
>
> Agreed, although if this is an rpm based system it should be sufficient to
> run the following.
>
> # debuginfo-install ceph glibc
>
> That may give us the name of the function depending on where we are (if we
> are
> in a library it may require the debuginfo for that library be loaded.
>
> Karol is right that a coredump would be a good idea in this case and will
> give
> us maximum information about the issue you are seeing.
>
> Cheers,
> Brad
>
> >
> > --
> > Regards,
> > Karol
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] about slides on VAULT of 2016

2016-04-27 Thread 席智勇

hi sage:

   I find the slides of VAULT of 2016 on this page(
http://events.linuxfoundation.org/events/vault/program/slides), it seems
not the whole accoding to the schedule info, and I didn't find yours. Can
you share your slides or any things usefull on VAULT about BlueStore.


regards~
zhiyong
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw crash - Infernalis

2016-04-27 Thread Brad Hubbard

- Original Message -
> From: "Karol Mroz" 
> To: "Ben Hines" 
> Cc: "ceph-users" 
> Sent: Wednesday, 27 April, 2016 7:06:56 PM
> Subject: Re: [ceph-users] radosgw crash - Infernalis
> 
> On Tue, Apr 26, 2016 at 10:17:31PM -0700, Ben Hines wrote:
> [...]
> > --> 10.30.1.6:6800/10350 -- osd_op(client.44852756.0:79
> > default.42048218. [getxattrs,stat,read 0~524288] 12.aa730416
> > ack+read+known_if_redirected e100207) v6 -- ?+0 0x7f49c41880b0 con
> > 0x7f49c4145eb0
> >  0> 2016-04-26 22:07:59.685615 7f49a07f0700 -1 *** Caught signal
> > (Segmentation fault) **
> >  in thread 7f49a07f0700
> > 
> >  ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
> >  1: (()+0x30b0a2) [0x7f4c4907f0a2]
> >  2: (()+0xf100) [0x7f4c44f7a100]
> >  NOTE: a copy of the executable, or `objdump -rdS ` is needed
> > to interpret this.
> 
> Hi Ben,
> 
> I sense a pretty badly corrupted stack. From the radosgw-9.2.1 (obtained from
> a downloaded rpm):
> 
> 0030a810 <_Z13pidfile_writePK11md_config_t@@Base>:
> ...
>   30b09d:   e8 0e 40 e4 ff  callq  14f0b0 
>   30b0a2:   4c 89 efmov%r13,%rdi
>   ---
> ...
> 
> So either we tripped backtrace() code from pidfile_write() _or_ we can't
> trust the stack. From the log snippet, it looks that we're far past the point
> at which we would write a pidfile to disk (ie. at process start during
> global_init()).
> Rather, we're actually handling a request and outputting some bit of debug
> message
> via MSDOp::print() and beyond...

It would help to know what binary this is and what OS.

We know the offset into the function is 0x30b0a2 but we don't know which
function yet AFAICT. Karol, how did you arrive at pidfile_write? Purely from
the offset? I'm not sure that would be reliable...

This is a segfault so the address of the frame where we crashed should be the
exact instruction where we crashed. I don't believe a mov from one register to
another that does not involve a dereference ((%r13) as opposed to %r13) can
cause a segfault so I don't think we are on the right instruction but then, as
you say, the stack may be corrupt.

> 
> Is this something you're able to easily reproduce? More logs with higher log
> levels
> would be helpful... a coredump with radosgw compiled with -g would be
> excellent :)

Agreed, although if this is an rpm based system it should be sufficient to
run the following.

# debuginfo-install ceph glibc

That may give us the name of the function depending on where we are (if we are
in a library it may require the debuginfo for that library be loaded.

Karol is right that a coredump would be a good idea in this case and will give
us maximum information about the issue you are seeing.

Cheers,
Brad

> 
> --
> Regards,
> Karol
> 
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Jewel Compilaton Error

2016-04-27 Thread Dyweni - Ceph-Users


Hi List,

Ceph 10.2.0 errors out during compilation when compiling without radowgw 
support.



./configure --prefix=/usr --build=i686-pc-linux-gnu 
--host=i686-pc-linux-gnu --mandir=/usr/share/man 
--infodir=/usr/share/info --datadir=/usr/share --sysconfdir=/etc 
--localstatedir=/var/lib --disable-dependency-tracking 
--disable-silent-rules --libdir=/usr/lib --without-hadoop 
--docdir=/usr/share/doc/ceph-10.2.0 --includedir=/usr/include 
--without-debug --without-fuse --with-libaio --without-libatomic-ops 
--with-nss --without-cryptopp --without-radosgw --without-gtk2 
--disable-static --with-jemalloc --without-libxfs --without-libzfs 
--without-lttng --without-babeltrace --with-eventfd --with-python 
--without-kinetic --without-librocksdb 
--with-systemdsystemunitdir=/usr/lib/systemd/system




rgw/ceph_dencoder-rgw_dencoder.o: In function 
`RGWZoneGroup::generate_test_instances(std::list >&)':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:1096: 
undefined reference to `vtable for RGWZoneGroup'
rgw/ceph_dencoder-rgw_dencoder.o: In function 
`RGWZoneParams::generate_test_instances(std::list >&)':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:871: 
undefined reference to `vtable for RGWZoneParams'
rgw/ceph_dencoder-rgw_json_enc.o: In function 
`decode_zonegroups(std::map > >&, JSONObj*)':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:1096: 
undefined reference to `vtable for RGWZoneGroup'
rgw/ceph_dencoder-rgw_json_enc.o: In function 
`RGWSystemMetaObj::~RGWSystemMetaObj()':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:764: 
undefined reference to `vtable for RGWSystemMetaObj'
rgw/ceph_dencoder-rgw_json_enc.o: In function 
`RGWSystemMetaObj::~RGWSystemMetaObj()':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:764: 
undefined reference to `vtable for RGWSystemMetaObj'
rgw/ceph_dencoder-rgw_json_enc.o: In function 
`std::_Rb_tree, 
std::less, std::allocator > >::_M_erase(std::_Rb_tree_node >*)':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:1064: 
undefined reference to `vtable for RGWZoneGroup'
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:764: 
undefined reference to `vtable for RGWSystemMetaObj'
rgw/ceph_dencoder-rgw_json_enc.o: In function 
`std::_Rb_tree_iterator 
std::_Rb_tree, 
std::less, std::allocator > >::_M_emplace_hint_uniqueconst&, std::tuple, std::tuple<> 
>(std::_Rb_tree_const_iterator, std::piecewise_construct_t const&, std::tuple&&, std::tuple<>&&)':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:1096: 
undefined reference to `vtable for RGWZoneGroup'
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:764: 
undefined reference to `vtable for RGWSystemMetaObj'
rgw/ceph_dencoder-rgw_json_enc.o: In function 
`std::_Rb_tree_iterator 
std::_Rb_tree, 
std::less, std::allocator > >::_M_insert_&>(std::_Rb_tree_node_base*, std::_Rb_tree_node_base*, 
std::pair&)':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:734: 
undefined reference to `vtable for RGWSystemMetaObj'
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:1064: 
undefined reference to `vtable for RGWZoneGroup'
rgw/ceph_dencoder-rgw_json_enc.o: In function 
`RGWZoneGroup::~RGWZoneGroup()':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:1064: 
undefined reference to `vtable for RGWZoneGroup'
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:764: 
undefined reference to `vtable for RGWSystemMetaObj'
rgw/ceph_dencoder-rgw_json_enc.o: In function `bool 
JSONDecoder::decode_json(char const*, RGWZoneGroup&, 
JSONObj*, bool)':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:1096: 
undefined reference to `vtable for RGWZoneGroup'
rgw/ceph_dencoder-rgw_json_enc.o: In function `void 
decode_json_obj(std::map >&, 
JSONObj*)':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:1096: 
undefined reference to `vtable for RGWZoneGroup'
rgw/ceph_dencoder-rgw_json_enc.o: In function 
`RGWZoneGroup::~RGWZoneGroup()':
/var/tmp/portage/sys-cluster/ceph-10.2.0/work/ceph-10.2.0/src/rgw/rgw_rados.h:1064: 
undefined reference to `vtable for

Re: [ceph-users] pgnum warning and decrease

2016-04-27 Thread Christian Balzer


Hello,

On Wed, 27 Apr 2016 22:55:35 + Carlos M. Perez wrote:

> Hi,
> 
> My current setup is running on 12 OSD's split between 3 hosts.  We're
> using this for VM's (Proxmox) and nothing else.
> 
I assume evenly split (4 OSDs per host)?

> According to:
> http://docs.ceph.com/docs/master/rados/operations/placement-groups/ - my
> pg_num should be set to 4096
> 

That's what you get when people try to simplify what is a rather
complicated matter.
I would have just put the link to PGcalc there.

> If I use the calculator, and put in Size 3, OSD 12, and 200PG target, I
> get 1024.
> 
That's the correct answer and unless you plan to grow (double) your
cluster size around 100PGs per OSD is a good idea.

> So I decided to split the difference, and use 2048, but ceph is warning
> me that I have too many 512 (2048/4)
> 
That's not how ceph gets to that result, it's 
2048(PGs)*3(replication)/12(OSDs).

> root@pve151201:~# ceph -w
> cluster 9005acf0-17a2-4973-bfe0-55dc9f23786c
>  health HEALTH_WARN
> too many PGs per OSD (512 > max 300)
>  monmap e3: 3 mons at
> {0=172.31.31.21:6789/0,1=172.31.31.22:6789/0,2=172.31.31.23:6789/0}
> election epoch 8310, quorum 0,1,2 0,1,2 osdmap e32336: 12 osds: 12 up,
> 12 in pgmap v9908729: 2048 pgs, 1 pools, 237 GB data, 62340 objects
> 719 GB used, 10453 GB / 11172 GB avail
> 2048 active+clean
> 
> # ceph osd pool get rbd pg_num
> pg_num: 2048
> # ceph osd pool get rbd pgp_num
> pgp_num: 2048
> # ceph osd lspools
> 3 rbd,
> # ceph -v
> ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)
> 
> Safe to ignore?
> 
If you have enough resources (CPU/RAM mostly) yes.

> If I were to change it to decrease it to 1024, is this a safe way:
> http://www.sebastien-han.fr/blog/2013/03/12/ceph-change-pg-number-on-the-fly/
> seems to make sense, but I don't have enough ceph experience (and guts)
> to give it a go...
>
You can't decrease PGs, ever. 
The only way is to destroy and re-create the pool(s).
 

Christian
-- 
Christian BalzerNetwork/Systems Engineer
ch...@gol.com   Global OnLine Japan/Rakuten Communications
http://www.gol.com/
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] "rbd diff" disparity vs mounted usage

2016-04-27 Thread Tyler Wilson

Hello Jason,

Yes, I believe that is my question. Is there any way I can either reclaim
the space for this disk?

On Wed, Apr 27, 2016 at 1:25 PM, Jason Dillaman  wrote:

> The image size (50G) minus the fstrim size (1.7G) approximately equals
> the actual usage (48.19G).  Therefore, I guess the question is why
> doesn't fstrim think it can discard more space?
>
> On a semi-related note, we should probably improve the rbd copy
> sparsify logic.  Right now it requires the full stripe period (or
> object size if striping is disabled) to be zeroes before it skips the
> write operation to the destination.
>
> On Wed, Apr 27, 2016 at 2:26 PM, Tyler Wilson 
> wrote:
> > Hello Jason,
> >
> > Thanks for the quick reply, this was copied from an VM instance snapshot
> to
> > my backup pool (rbd snap create, rbd cp (to backup pool), rbd snap rm).
> I've
> > tried piping through grep per your recommendation and it still reports
> the
> > same usage
> >
> > $ rbd diff backup/cd4e5d37-3023-4640-be5a-5577d3f9307e | grep data | awk
> '{
> > SUM += $2 } END { print SUM/1024/1024 " MB" }'
> > 49345.4 MB
> >
> > Thanks for the help.
> >
> > On Wed, Apr 27, 2016 at 12:22 PM, Jason Dillaman 
> > wrote:
> >>
> >> On Wed, Apr 27, 2016 at 2:07 PM, Tyler Wilson 
> >> wrote:
> >> > $ rbd diff backup/cd4e5d37-3023-4640-be5a-5577d3f9307e | awk '{ SUM +=
> >> > $2 }
> >> > END { print SUM/1024/1024 " MB" }'
> >> > 49345.4 MB
> >>
> >> Is this a cloned image?  That awk trick doesn't account for discarded
> >> regions (i.e. when column three says "zero" instead of "data"). Does
> >> the number change when you pipe the "rbd diff" results through "grep
> >> data" before piping to awk?
> >>
> >> > Could this be affected by replica counts some how? It seems to be
> twice
> >> > as
> >> > large as what is reported in the filesystem which matches my replica
> >> > count.
> >>
> >> No, the "rbd diff" output is only reporting image data and zeroed
> >> extents -- so the replication factor is not included.
> >>
> >> --
> >> Jason
> >
> >
>
>
>
> --
> Jason
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] pgnum warning and decrease

2016-04-27 Thread Carlos M. Perez

Hi,

My current setup is running on 12 OSD's split between 3 hosts.  We're using 
this for VM's (Proxmox) and nothing else.

According to: 
http://docs.ceph.com/docs/master/rados/operations/placement-groups/ - my pg_num 
should be set to 4096

If I use the calculator, and put in Size 3, OSD 12, and 200PG target, I get 
1024.

So I decided to split the difference, and use 2048, but ceph is warning me that 
I have too many 512 (2048/4)

root@pve151201:~# ceph -w
cluster 9005acf0-17a2-4973-bfe0-55dc9f23786c
 health HEALTH_WARN
too many PGs per OSD (512 > max 300)
 monmap e3: 3 mons at 
{0=172.31.31.21:6789/0,1=172.31.31.22:6789/0,2=172.31.31.23:6789/0}
election epoch 8310, quorum 0,1,2 0,1,2
 osdmap e32336: 12 osds: 12 up, 12 in
  pgmap v9908729: 2048 pgs, 1 pools, 237 GB data, 62340 objects
719 GB used, 10453 GB / 11172 GB avail
2048 active+clean

# ceph osd pool get rbd pg_num
pg_num: 2048
# ceph osd pool get rbd pgp_num
pgp_num: 2048
# ceph osd lspools
3 rbd,
# ceph -v
ceph version 0.94.3 (95cefea9fd9ab740263bf8bb4796fd864d9afe2b)

Safe to ignore?

If I were to change it to decrease it to 1024, is this a safe way: 
http://www.sebastien-han.fr/blog/2013/03/12/ceph-change-pg-number-on-the-fly/ 
seems to make sense, but I don't have enough ceph experience (and guts) to give 
it a go...

Thanks in advance,

Carlos M. Perez
CMP Consulting Services
305-669-1515

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mount -t ceph

2016-04-27 Thread David Disseldorp

Hi Tom,

On Wed, 27 Apr 2016 20:17:51 +, Deneau, Tom wrote:

> I was using SLES 12, SP1 which has 3.12.49 
> 
> It did have a /usr/sbin/mount.ceph command but using it gave 
>   modprobe: FATAL: Module ceph not found.
>   failed to load ceph kernel module (1)

The SLES 12 SP1 kernel doesn't currently ship with CephFS support.

Regards, David
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osd problem upgrading from hammer to jewel

2016-04-27 Thread Randy Orr

Hi,

I have a small dev/test ceph cluster that sat neglected for quite some
time. It was on the firefly release until very recently. I successfully
upgraded from firefly to hammer without issue as an intermediate step to
get to the latest jewel release.

This cluster has 3 ubuntu 14.04 hosts with kernel 3.13.0-40-generic. MONs
and OSDs are colocated on the same hosts with 11 total OSDs across the 3
hosts.

The 3 MONs have been updated to jewel and are running successfully. I set
noout on the cluster and shutdown the first 3 OSD processes, ran chown -R
ceph:ceph on /var/lib/ceph/osd. The OSD processes start and run, but never
show as UP. After setting debug osd = 20 I see the following in the logs:

2016-04-27 15:55:19.042230 7fd3854c7700 10 osd.1 13324 tick
2016-04-27 15:55:19.042244 7fd3854c7700 10 osd.1 13324 do_waiters -- start
2016-04-27 15:55:19.042247 7fd3854c7700 10 osd.1 13324 do_waiters -- finish
2016-04-27 15:55:19.061083 7fd384cc6700 10 osd.1 13324 tick_without_osd_lock
2016-04-27 15:55:19.061096 7fd384cc6700 20 osd.1 13324 scrub_random_backoff
lost coin flip, randomly backing off
2016-04-27 15:55:20.042351 7fd3854c7700 10 osd.1 13324 tick
2016-04-27 15:55:20.042364 7fd3854c7700 10 osd.1 13324 do_waiters -- start
2016-04-27 15:55:20.042368 7fd3854c7700 10 osd.1 13324 do_waiters -- finish
2016-04-27 15:55:20.061192 7fd384cc6700 10 osd.1 13324 tick_without_osd_lock
2016-04-27 15:55:20.061206 7fd384cc6700 20 osd.1 13324
can_inc_scrubs_pending0 -> 1 (max 1, active 0)
2016-04-27 15:55:20.061212 7fd384cc6700 20 osd.1 13324 scrub_time_permit
should run between 0 - 24 now 15 = yes
2016-04-27 15:55:20.061247 7fd384cc6700 20 osd.1 13324
scrub_load_below_threshold loadavg 0.04 < max 0.5 = yes
2016-04-27 15:55:20.061259 7fd384cc6700 20 osd.1 13324 sched_scrub
load_is_low=1
2016-04-27 15:55:20.061261 7fd384cc6700 20 osd.1 13324 sched_scrub done
2016-04-27 15:55:20.861872 7fd368ded700 20 osd.1 13324 update_osd_stat
osd_stat(61789 MB used, 865 GB avail, 926 GB total, peers []/[] op hist [])
2016-04-27 15:55:20.861886 7fd368ded700  5 osd.1 13324 heartbeat:
osd_stat(61789 MB used, 865 GB avail, 926 GB total, peers []/[] op hist [])

The fact that no peers show up in the heartbeat seems problematic, but I
can't see why the OSDs are failing to start correctly.

A ceph status gives this:

cluster 9e3f9cab-6f1b-4c7c-ab13-e01cb774f752
 health HEALTH_WARN
725 pgs degraded
3584 pgs stuck unclean
725 pgs undersized
recovery 23363/180420 objects degraded (12.949%)
recovery 49218/180420 objects misplaced (27.280%)
too many PGs per OSD (651 > max 300)
3/11 in osds are down
noout flag(s) set
 monmap e3: 3 mons at {DAL1S4UTIL6=
10.2.0.116:6789/0,DAL1S4UTIL7=10.2.0.117:6789/0,DAL1S4UTIL8=10.2.0.118:6789/0
}
election epoch 32, quorum 0,1,2
DAL1S4UTIL6,DAL1S4UTIL7,DAL1S4UTIL8
 osdmap e13324: 11 osds: 8 up, 11 in; 2859 remapped pgs
flags noout
  pgmap v6332775: 3584 pgs, 7 pools, 180 GB data, 60140 objects
703 GB used, 9483 GB / 10186 GB avail
23363/180420 objects degraded (12.949%)
49218/180420 objects misplaced (27.280%)
2238 active+remapped
 725 active+undersized+degraded
 621 active

Disk utilization is low. Nothing interesting in syslog or dmesg. Any ideas
or suggestions on where to start debugging this?

Thanks,
Randy
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] "rbd diff" disparity vs mounted usage

2016-04-27 Thread Jason Dillaman

The image size (50G) minus the fstrim size (1.7G) approximately equals
the actual usage (48.19G).  Therefore, I guess the question is why
doesn't fstrim think it can discard more space?

On a semi-related note, we should probably improve the rbd copy
sparsify logic.  Right now it requires the full stripe period (or
object size if striping is disabled) to be zeroes before it skips the
write operation to the destination.

On Wed, Apr 27, 2016 at 2:26 PM, Tyler Wilson  wrote:
> Hello Jason,
>
> Thanks for the quick reply, this was copied from an VM instance snapshot to
> my backup pool (rbd snap create, rbd cp (to backup pool), rbd snap rm). I've
> tried piping through grep per your recommendation and it still reports the
> same usage
>
> $ rbd diff backup/cd4e5d37-3023-4640-be5a-5577d3f9307e | grep data | awk '{
> SUM += $2 } END { print SUM/1024/1024 " MB" }'
> 49345.4 MB
>
> Thanks for the help.
>
> On Wed, Apr 27, 2016 at 12:22 PM, Jason Dillaman 
> wrote:
>>
>> On Wed, Apr 27, 2016 at 2:07 PM, Tyler Wilson 
>> wrote:
>> > $ rbd diff backup/cd4e5d37-3023-4640-be5a-5577d3f9307e | awk '{ SUM +=
>> > $2 }
>> > END { print SUM/1024/1024 " MB" }'
>> > 49345.4 MB
>>
>> Is this a cloned image?  That awk trick doesn't account for discarded
>> regions (i.e. when column three says "zero" instead of "data"). Does
>> the number change when you pipe the "rbd diff" results through "grep
>> data" before piping to awk?
>>
>> > Could this be affected by replica counts some how? It seems to be twice
>> > as
>> > large as what is reported in the filesystem which matches my replica
>> > count.
>>
>> No, the "rbd diff" output is only reporting image data and zeroed
>> extents -- so the replication factor is not included.
>>
>> --
>> Jason
>
>



-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mount -t ceph

2016-04-27 Thread Gregory Farnum

On Wed, Apr 27, 2016 at 3:17 PM, Deneau, Tom  wrote:
> I was using SLES 12, SP1 which has 3.12.49
>
> It did have a /usr/sbin/mount.ceph command but using it gave
>   modprobe: FATAL: Module ceph not found.
>   failed to load ceph kernel module (1)

So that's about what is in your distro kernel, not upstream. Afraid I
don't know what goes on in...well, any of them, but certainly not
SLES. ;)
-Greg

>
> -- Tom
>
>
>> -Original Message-
>> From: Gregory Farnum [mailto:gfar...@redhat.com]
>> Sent: Wednesday, April 27, 2016 2:59 PM
>> To: Deneau, Tom 
>> Cc: ceph-users 
>> Subject: Re: [ceph-users] mount -t ceph
>>
>> On Wed, Apr 27, 2016 at 2:55 PM, Deneau, Tom  wrote:
>> > What kernel versions are required to be able to use CephFS thru mount -t
>> ceph?
>>
>> The CephFS kernel client has been in for ages (2.6.34, I think?), but you
>> want the absolute latest you can make happen if you're going to try it
>> out.
>> The actual mount command requires you have mount.ceph, which is in
>> different places/availabilities depending on your distro.
>> -Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mount -t ceph

2016-04-27 Thread Deneau, Tom

I was using SLES 12, SP1 which has 3.12.49 

It did have a /usr/sbin/mount.ceph command but using it gave 
  modprobe: FATAL: Module ceph not found.
  failed to load ceph kernel module (1)

-- Tom


> -Original Message-
> From: Gregory Farnum [mailto:gfar...@redhat.com]
> Sent: Wednesday, April 27, 2016 2:59 PM
> To: Deneau, Tom 
> Cc: ceph-users 
> Subject: Re: [ceph-users] mount -t ceph
> 
> On Wed, Apr 27, 2016 at 2:55 PM, Deneau, Tom  wrote:
> > What kernel versions are required to be able to use CephFS thru mount -t
> ceph?
> 
> The CephFS kernel client has been in for ages (2.6.34, I think?), but you
> want the absolute latest you can make happen if you're going to try it
> out.
> The actual mount command requires you have mount.ceph, which is in
> different places/availabilities depending on your distro.
> -Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] mount -t ceph

2016-04-27 Thread Gregory Farnum

On Wed, Apr 27, 2016 at 2:55 PM, Deneau, Tom  wrote:
> What kernel versions are required to be able to use CephFS thru mount -t ceph?

The CephFS kernel client has been in for ages (2.6.34, I think?), but
you want the absolute latest you can make happen if you're going to
try it out.
The actual mount command requires you have mount.ceph, which is in
different places/availabilities depending on your distro.
-Greg
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] mount -t ceph

2016-04-27 Thread Deneau, Tom

What kernel versions are required to be able to use CephFS thru mount -t ceph?

-- Tom Deneau
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] "rbd diff" disparity vs mounted usage

2016-04-27 Thread Tyler Wilson

Hello Jason,

Thanks for the quick reply, this was copied from an VM instance snapshot to
my backup pool (rbd snap create, rbd cp (to backup pool), rbd snap rm).
I've tried piping through grep per your recommendation and it still reports
the same usage

$ rbd diff backup/cd4e5d37-3023-4640-be5a-5577d3f9307e | grep data | awk '{
SUM += $2 } END { print SUM/1024/1024 " MB" }'
49345.4 MB

Thanks for the help.

On Wed, Apr 27, 2016 at 12:22 PM, Jason Dillaman 
wrote:

> On Wed, Apr 27, 2016 at 2:07 PM, Tyler Wilson 
> wrote:
> > $ rbd diff backup/cd4e5d37-3023-4640-be5a-5577d3f9307e | awk '{ SUM +=
> $2 }
> > END { print SUM/1024/1024 " MB" }'
> > 49345.4 MB
>
> Is this a cloned image?  That awk trick doesn't account for discarded
> regions (i.e. when column three says "zero" instead of "data"). Does
> the number change when you pipe the "rbd diff" results through "grep
> data" before piping to awk?
>
> > Could this be affected by replica counts some how? It seems to be twice
> as
> > large as what is reported in the filesystem which matches my replica
> count.
>
> No, the "rbd diff" output is only reporting image data and zeroed
> extents -- so the replication factor is not included.
>
> --
> Jason
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] "rbd diff" disparity vs mounted usage

2016-04-27 Thread Jason Dillaman

On Wed, Apr 27, 2016 at 2:07 PM, Tyler Wilson  wrote:
> $ rbd diff backup/cd4e5d37-3023-4640-be5a-5577d3f9307e | awk '{ SUM += $2 }
> END { print SUM/1024/1024 " MB" }'
> 49345.4 MB

Is this a cloned image?  That awk trick doesn't account for discarded
regions (i.e. when column three says "zero" instead of "data"). Does
the number change when you pipe the "rbd diff" results through "grep
data" before piping to awk?

> Could this be affected by replica counts some how? It seems to be twice as
> large as what is reported in the filesystem which matches my replica count.

No, the "rbd diff" output is only reporting image data and zeroed
extents -- so the replication factor is not included.

-- 
Jason
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] "rbd diff" disparity vs mounted usage

2016-04-27 Thread Tyler Wilson

Hello All,

I am currently trying to get an accurate count of bytes used for an rbd
image. I've tried trimming the filesystem which relieves about 1.7gb
however there is still a huge disparity of size reported in the filesystem
vs what 'rbd diff' shows;

$ rbd map backup/cd4e5d37-3023-4640-be5a-5577d3f9307e
/dev/rbd3

$ mount -o discard /dev/rbd3p1 /tmp/cd4e5d37-3023-4640-be5a-5577d3f9307e/

$ df -h /tmp/cd4e5d37-3023-4640-be5a-5577d3f9307e/
Filesystem  Size  Used Avail Use% Mounted on
/dev/rbd3p1  50G   24G   24G  50%
/tmp/cd4e5d37-3023-4640-be5a-5577d3f9307e

$ df -i /tmp/cd4e5d37-3023-4640-be5a-5577d3f9307e/
Filesystem  Inodes  IUsed   IFree IUse% Mounted on
/dev/rbd3p13276800 582930 2693870   18%
/tmp/cd4e5d37-3023-4640-be5a-5577d3f9307e

$ fstrim -v /tmp/cd4e5d37-3023-4640-be5a-5577d3f9307e/
/tmp/cd4e5d37-3023-4640-be5a-5577d3f9307e/: 1.7 GiB (1766875136 bytes)
trimmed

$ rbd diff backup/cd4e5d37-3023-4640-be5a-5577d3f9307e | awk '{ SUM += $2 }
END { print SUM/1024/1024 " MB" }'
49345.4 MB

Could this be affected by replica counts some how? It seems to be twice as
large as what is reported in the filesystem which matches my replica count.

Thanks for any and all assistance!
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NO mon start after Jewel Upgrade using systemctl

2016-04-27 Thread Iban Cabrillo

Hi Karsten,
   I have checked taht files arethe same that git ones.

  -rw-r--r-- 1 root root 810 Apr 20 18:45 /lib/systemd/system/ceph-mon@
.service
  -rw-r--r-- 1 root root 162 Apr 20 18:45
/lib/systemd/system/ceph-mon.target

[root@cephmon03 ~]# cat /lib/systemd/system/ceph-mon.target
[Unit]
Description=ceph target allowing to start/stop all ceph-mon@.service
instances at once
PartOf=ceph.target
[Install]
WantedBy=multi-user.target ceph.target

[root@cephmon03 ~]# systemctl list-unit-files|grep ceph
ceph-create-keys@.service  static
ceph-disk@.service static
ceph-mds@.service  disabled
ceph-mon@.service  disabled
ceph-osd@.service  disabled
ceph-radosgw@.service  disabled
ceph.service   masked
ceph-mds.targetdisabled
ceph-mon.targetenabled
ceph-osd.targetdisabled
ceph-radosgw.targetdisabled
ceph.targetdisabled

But still doesn't work (The upgrade was made from latest Hammer version )
and it is running on CentOS 7. This instance is running a mon service only.

[root@cephmon03 ~]# rpm -qa | grep ceph
ceph-release-1-1.el7.noarch
ceph-common-10.2.0-0.el7.x86_64
ceph-mds-10.2.0-0.el7.x86_64
libcephfs1-10.2.0-0.el7.x86_64
python-cephfs-10.2.0-0.el7.x86_64
ceph-selinux-10.2.0-0.el7.x86_64
ceph-mon-10.2.0-0.el7.x86_64
ceph-osd-10.2.0-0.el7.x86_64
ceph-radosgw-10.2.0-0.el7.x86_64
ceph-base-10.2.0-0.el7.x86_64
ceph-10.2.0-0.el7.x86_64

I have test it with ceph.target, with the same result.

regards, I




2016-04-27 15:13 GMT+02:00 Karsten Heymann :

> Hi Iban,
>
> the current jewel packages seem to be missing some important systemd
> files. Try to copy
> https://github.com/ceph/ceph/blob/master/systemd/ceph-mon.target to
> /lib/systemd/system and enable it:
>
> systemctl enable ceph-mon.target
>
> I also would disable the legacy init script with
>
> systemctl mask ceph.service
>
> There are already several open pull requests regarding this issue
> (
> https://github.com/ceph/ceph/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+systemd
> ),
> so I hope it will be fixed with the next point release.
>
> Best regards
> Karsten
>
> 2016-04-27 14:18 GMT+02:00 Iban Cabrillo :
> > Hi cephers,
> >   I've been following the upgrade intrucctions...but..I sure I did
> something
> > wrong.
> >
> >   I just upgrade using ceph-deploy on one monitor (after ofcourse down de
> > mon service).
> >   Then  the chow to var/lib/ceph and /var/log/ceph for ceph user
> >
> >   [root@cephmon03 ~]# systemctl start ceph.target
> > [root@cephmon03 ~]#
> > [root@cephmon03 ~]#
> > [root@cephmon03 ~]# systemctl status ceph.target
> > ● ceph.target - ceph target allowing to start/stop all ceph*@.service
> > instances at once
> >Loaded: loaded (/usr/lib/systemd/system/ceph.target; disabled; vendor
> > preset: disabled)
> >Active: active since mié 2016-04-27 13:43:24 CEST; 10min ago
> >
> > abr 27 13:43:24 cephmon03.ifca.es systemd[1]: Reached target ceph target
> > allowing to start/stop all ceph*@.service instances at once.
> > abr 27 13:43:24 cephmon03.ifca.es systemd[1]: Starting ceph target
> allowing
> > to start/stop all ceph*@.service instances at once.
> > abr 27 13:44:17 cephmon03.ifca.es systemd[1]: Reached target ceph target
> > allowing to start/stop all ceph*@.service instances at once.
> > abr 27 13:47:09 cephmon03.ifca.es systemd[1]: Reached target ceph target
> > allowing to start/stop all ceph*@.service instances at once.
> > abr 27 13:53:36 cephmon03.ifca.es systemd[1]: Reached target ceph target
> > allowing to start/stop all ceph*@.service instances at once.
> >
> >  [root@cephmon03 ~]# systemctl status ceph-mon@3
> > ● ceph-mon@3.service - Ceph cluster monitor daemon
> >Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; disabled;
> > vendor preset: disabled)
> >Active: inactive (dead)
> >
> > abr 27 13:55:44 cephmon03 systemd[1]:
> > [/usr/lib/systemd/system/ceph-mon@.service:24] Unknown lvalue
> 'TasksMax' in
> > section 'Service'
> >
> >
> > looking at systemctl i see:
> >
> >   ceph-mon.cephmon03.1456312447.168540372.service  loadedactive
> exited
> > /usr/bin/bash -c ulimit -n 32768;  /usr/bin/ceph-mon -i cephmon03
> --pid-file
> > /var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster ceph -f
> > ● ceph.service not-found active
> exited
> > ceph.service
> >
> > but:
> >
> > [root@cephmon03 ~]# systemctl start
> > ceph-mon.cephmon03.1456312447.168540372.service
> > [root@cephmon03 ~]# systemctl status
> > ceph-mon.cephmon03.1456312447.168540372.service
> > ● ceph-mon.cephmon03.1456312447.168540372.service - /usr/bin/bash -c
> ulimit
> > -n 32768;  /usr/bin/ceph-mon -i cephmon03 --pid-file
> > /var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster

Re: [ceph-users] Any Docs to configure NFS to access RADOSGW buckets on Jewel

2016-04-27 Thread Matt Benjamin

Hi WD,

No, it's not the same.  The new mechanism uses an nfs-ganesha server to export 
the RGW namespace.  Some upstream documentation will be forthcoming...

Regards,

Matt

- Original Message -
> From: "WD Hwang" 
> To: "a jazdzewski" 
> Cc: ceph-users@lists.ceph.com
> Sent: Wednesday, April 27, 2016 5:03:12 AM
> Subject: Re: [ceph-users] Any Docs to configure NFS to access RADOSGW buckets 
> on Jewel
> 
> Hi Ansgar,
>   Thanks for your information.
>   I have tried 's3fs-fuse' to mount RADOSGW buckets on Ubuntu client node. It
>   works.
>   But I am not sure this is the technique that access RADOSGW buckets via NFS
>   on Jewel.
> 
> Best Regards,
> WD
> 
> -Original Message-
> From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of
> Ansgar Jazdzewski
> Sent: Wednesday, April 27, 2016 4:32 PM
> To: WD Hwang/WHQ/Wistron
> Cc: ceph-users@lists.ceph.com
> Subject: Re: [ceph-users] Any Docs to configure NFS to access RADOSGW buckets
> on Jewel
> 
> all informations i have so far are from the FOSDEM
> 
> https://fosdem.org/2016/schedule/event/virt_iaas_ceph_rados_gateway_overview/attachments/audio/1079/export/events/attachments/virt_iaas_ceph_rados_gateway_overview/audio/1079/Fosdem_RGW.pdf
> 
> Cheers,
> Ansgar
> 
> 2016-04-27 2:28 GMT+02:00  :
> > Hello:
> >
> >   Are there any documents or examples to explain the configuration of
> > NFS to access RADOSGW buckets on Jewel?
> >
> > Thanks a lot.
> >
> >
> >
> > Best Regards,
> >
> > WD
> >
> > --
> > --
> > ---
> >
> > This email contains confidential or legally privileged information and
> > is for the sole use of its intended recipient.
> >
> > Any unauthorized review, use, copying or distribution of this email or
> > the content of this email is strictly prohibited.
> >
> > If you are not the intended recipient, you may reply to the sender and
> > should delete this e-mail immediately.
> >
> > --
> > --
> > ---
> >
> >
> > ___
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> >
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> 

-- 
Matt Benjamin
Red Hat, Inc.
315 West Huron Street, Suite 140A
Ann Arbor, Michigan 48103

http://www.redhat.com/en/technologies/storage

tel.  734-707-0660
fax.  734-769-8938
cel.  734-216-5309
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osds udev rules not triggered on reboot (jewel, jessie)

2016-04-27 Thread Karsten Heymann

2016-04-27 15:18 GMT+02:00 Loris Cuoghi :
> Le 27/04/2016 14:45, Karsten Heymann a écrit :
>> one workaround I found was to add
>>
>> [Install]
>> WantedBy=ceph-osd.target
>>
>> to /lib/systemd/system/ceph-disk@.service and then manually enable my
>> disks with
>>
>> # systemctl enable ceph-disk\@dev-sdi1
>> # systemctl start ceph-disk\@dev-sdi1
>>
>> That way they at least are started at boot time.

> Great! But only if the disks keep their device names, right ?

Exactly. It's just a little workaround until the real issue is fixed.

+Karsten
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osds udev rules not triggered on reboot (jewel, jessie)

2016-04-27 Thread Loris Cuoghi




Le 27/04/2016 14:45, Karsten Heymann a écrit :

Hi,

one workaround I found was to add

[Install]
WantedBy=ceph-osd.target

to /lib/systemd/system/ceph-disk@.service and then manually enable my disks with

# systemctl enable ceph-disk\@dev-sdi1
# systemctl start ceph-disk\@dev-sdi1

That way they at least are started at boot time.


Great! But only if the disks keep their device names, right ?



Best regards,
Karsten

2016-04-27 13:57 GMT+02:00 Loris Cuoghi :


Le 27/04/2016 13:51, Karsten Heymann a écrit :


Hi Loris,

thank you for your feedback. As I plan to go productive with the
cluster later this year I'm really hesitant to update udev and systemd
to a version newer than jessie, especially as there is no official
backport for those packages yet. I really would expect ceph to work
out of the box with the version in jessie, so I'd rather try to find
the cause of the problem and help fixing it.



That's my plan too, updating was just a troubleshoting method. ;)





best regards

2016-04-27 13:36 GMT+02:00 Loris Cuoghi :


Hi Karsten,

I've had the same experience updating our test cluster (Debian 8) from
Infernalis to Jewel.

I've update udev/systemd to the one in testing (so, from 215 to 229), and
it
worked much better at reboot.

So... Are the udev rules written for the udev version in RedHat (219) or
greater versions ?

Thanks in advance :)


Le 27/04/2016 09:33, Karsten Heymann a écrit :



Hi!

the last days, I updated my jessie evaluation cluster to jewel and now
osds are not started automatically after reboot because they are not
mounted. This is the output of ceph-disk list after boot:

/dev/sdh :
/dev/sdh1 ceph data, prepared, cluster ceph, osd.47, journal
/dev/sde1
/dev/sdi :
/dev/sdi1 ceph data, prepared, cluster ceph, osd.48, journal
/dev/sde2
/dev/sdj :
/dev/sdj1 ceph data, prepared, cluster ceph, osd.49, journal
/dev/sde3

and so on.

systemd tried to start the units:

# systemctl | grep osd
● ceph-osd@47.service
loaded failed failedCeph object
storage daemon
● ceph-osd@48.service
loaded failed failedCeph object
storage daemon
● ceph-osd@49.service
loaded failed failedCeph object
storage daemon

# systemctl status ceph-osd@47.service
● ceph-osd@47.service - Ceph object storage daemon
  Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled)
  Active: failed (Result: start-limit) since Wed 2016-04-27 08:50:07
CEST; 21min ago
 Process: 3139 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
 Process: 2682 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
(code=exited, status=0/SUCCESS)
Main PID: 3139 (code=exited, status=1/FAILURE)

Apr 27 08:50:06 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
entered failed state.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: ceph-osd@47.service start
request repeated too quickly, refusing to start.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Failed to start Ceph object
storage daemon.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
entered failed state.

Which is no suprise as the osd is not mounted:

# ls -l /var/lib/ceph/osd/ceph-47
total 0

The weird thing is running the following starts the osd:

# echo add > /sys/class/block/sdr1/uevents

so the udev rules to mount the osds seem to work.

Any ideas on how to debug this?

Best regards
Karsten
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] NO mon start after Jewel Upgrade using systemctl

2016-04-27 Thread Karsten Heymann

Hi Iban,

the current jewel packages seem to be missing some important systemd
files. Try to copy
https://github.com/ceph/ceph/blob/master/systemd/ceph-mon.target to
/lib/systemd/system and enable it:

systemctl enable ceph-mon.target

I also would disable the legacy init script with

systemctl mask ceph.service

There are already several open pull requests regarding this issue
(https://github.com/ceph/ceph/pulls?utf8=%E2%9C%93=is%3Apr+is%3Aopen+systemd),
so I hope it will be fixed with the next point release.

Best regards
Karsten

2016-04-27 14:18 GMT+02:00 Iban Cabrillo :
> Hi cephers,
>   I've been following the upgrade intrucctions...but..I sure I did something
> wrong.
>
>   I just upgrade using ceph-deploy on one monitor (after ofcourse down de
> mon service).
>   Then  the chow to var/lib/ceph and /var/log/ceph for ceph user
>
>   [root@cephmon03 ~]# systemctl start ceph.target
> [root@cephmon03 ~]#
> [root@cephmon03 ~]#
> [root@cephmon03 ~]# systemctl status ceph.target
> ● ceph.target - ceph target allowing to start/stop all ceph*@.service
> instances at once
>Loaded: loaded (/usr/lib/systemd/system/ceph.target; disabled; vendor
> preset: disabled)
>Active: active since mié 2016-04-27 13:43:24 CEST; 10min ago
>
> abr 27 13:43:24 cephmon03.ifca.es systemd[1]: Reached target ceph target
> allowing to start/stop all ceph*@.service instances at once.
> abr 27 13:43:24 cephmon03.ifca.es systemd[1]: Starting ceph target allowing
> to start/stop all ceph*@.service instances at once.
> abr 27 13:44:17 cephmon03.ifca.es systemd[1]: Reached target ceph target
> allowing to start/stop all ceph*@.service instances at once.
> abr 27 13:47:09 cephmon03.ifca.es systemd[1]: Reached target ceph target
> allowing to start/stop all ceph*@.service instances at once.
> abr 27 13:53:36 cephmon03.ifca.es systemd[1]: Reached target ceph target
> allowing to start/stop all ceph*@.service instances at once.
>
>  [root@cephmon03 ~]# systemctl status ceph-mon@3
> ● ceph-mon@3.service - Ceph cluster monitor daemon
>Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; disabled;
> vendor preset: disabled)
>Active: inactive (dead)
>
> abr 27 13:55:44 cephmon03 systemd[1]:
> [/usr/lib/systemd/system/ceph-mon@.service:24] Unknown lvalue 'TasksMax' in
> section 'Service'
>
>
> looking at systemctl i see:
>
>   ceph-mon.cephmon03.1456312447.168540372.service  loadedactive exited
> /usr/bin/bash -c ulimit -n 32768;  /usr/bin/ceph-mon -i cephmon03 --pid-file
> /var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster ceph -f
> ● ceph.service not-found active exited
> ceph.service
>
> but:
>
> [root@cephmon03 ~]# systemctl start
> ceph-mon.cephmon03.1456312447.168540372.service
> [root@cephmon03 ~]# systemctl status
> ceph-mon.cephmon03.1456312447.168540372.service
> ● ceph-mon.cephmon03.1456312447.168540372.service - /usr/bin/bash -c ulimit
> -n 32768;  /usr/bin/ceph-mon -i cephmon03 --pid-file
> /var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster ceph -f
>Loaded: loaded
> (/run/systemd/system/ceph-mon.cephmon03.1456312447.168540372.service;
> static; vendor preset: disabled)
>   Drop-In:
> /run/systemd/system/ceph-mon.cephmon03.1456312447.168540372.service.d
>└─50-Description.conf, 50-ExecStart.conf, 50-RemainAfterExit.conf
>Active: active (exited) since mié 2016-02-24 12:14:07 CET; 2 months 2
> days ago
>  Main PID: 1017 (code=exited, status=0/SUCCESS)
>CGroup: /system.slice/ceph-mon.cephmon03.1456312447.168540372.service
>
> abr 27 13:35:33 cephmon03 bash[1017]: 2016-04-27 13:35:33.913841
> 7f16dba5b700 -1 mon.cephmon03@2(peon) e1 *** Got Signal Terminated ***
> abr 27 14:02:37 cephmon03 systemd[1]: Started /usr/bin/bash -c ulimit -n
> 32768;  /usr/bin/ceph-mon -i cephmon03 --pid-file
> /var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster ceph -f.
> abr 27 14:04:37 cephmon03 systemd[1]: Started /usr/bin/bash -c ulimit -n
> 32768;  /usr/bin/ceph-mon -i cephmon03 --pid-file
> /var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster ceph -f.
>
> Nothing happens...
>
>
> But running the command in line (as root):
>
> /usr/bin/bash -c ulimit -n 32768;  /usr/bin/ceph-mon -i cephmon03 --pid-file
> /var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster ceph -f
>
> the mon starts well ( health HEALTH_OK)
>
> Any idea about this??
>
> regards, I
>
> --
> 
> Iban Cabrillo Bartolome
> Instituto de Fisica de Cantabria (IFCA)
> Santander, Spain
> Tel: +34942200969
> PGP PUBLIC KEY:
> http://pgp.mit.edu/pks/lookup?op=get=0xD9DF0B3D6C8C08AC
> 
> Bertrand Russell:
> "El problema con el mundo es que los estúpidos están seguros de todo y los
> inteligentes están llenos de dudas"
>
> ___
>

Re: [ceph-users] osds udev rules not triggered on reboot (jewel, jessie)

2016-04-27 Thread Karsten Heymann

Hi,

one workaround I found was to add

[Install]
WantedBy=ceph-osd.target

to /lib/systemd/system/ceph-disk@.service and then manually enable my disks with

# systemctl enable ceph-disk\@dev-sdi1
# systemctl start ceph-disk\@dev-sdi1

That way they at least are started at boot time.

Best regards,
Karsten

2016-04-27 13:57 GMT+02:00 Loris Cuoghi :
>
> Le 27/04/2016 13:51, Karsten Heymann a écrit :
>>
>> Hi Loris,
>>
>> thank you for your feedback. As I plan to go productive with the
>> cluster later this year I'm really hesitant to update udev and systemd
>> to a version newer than jessie, especially as there is no official
>> backport for those packages yet. I really would expect ceph to work
>> out of the box with the version in jessie, so I'd rather try to find
>> the cause of the problem and help fixing it.
>
>
> That's my plan too, updating was just a troubleshoting method. ;)
>
>
>
>>
>> best regards
>>
>> 2016-04-27 13:36 GMT+02:00 Loris Cuoghi :
>>>
>>> Hi Karsten,
>>>
>>> I've had the same experience updating our test cluster (Debian 8) from
>>> Infernalis to Jewel.
>>>
>>> I've update udev/systemd to the one in testing (so, from 215 to 229), and
>>> it
>>> worked much better at reboot.
>>>
>>> So... Are the udev rules written for the udev version in RedHat (219) or
>>> greater versions ?
>>>
>>> Thanks in advance :)
>>>
>>>
>>> Le 27/04/2016 09:33, Karsten Heymann a écrit :


 Hi!

 the last days, I updated my jessie evaluation cluster to jewel and now
 osds are not started automatically after reboot because they are not
 mounted. This is the output of ceph-disk list after boot:

 /dev/sdh :
/dev/sdh1 ceph data, prepared, cluster ceph, osd.47, journal
 /dev/sde1
 /dev/sdi :
/dev/sdi1 ceph data, prepared, cluster ceph, osd.48, journal
 /dev/sde2
 /dev/sdj :
/dev/sdj1 ceph data, prepared, cluster ceph, osd.49, journal
 /dev/sde3

 and so on.

 systemd tried to start the units:

 # systemctl | grep osd
 ● ceph-osd@47.service
loaded failed failedCeph object
 storage daemon
 ● ceph-osd@48.service
loaded failed failedCeph object
 storage daemon
 ● ceph-osd@49.service
loaded failed failedCeph object
 storage daemon

 # systemctl status ceph-osd@47.service
 ● ceph-osd@47.service - Ceph object storage daemon
  Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled)
  Active: failed (Result: start-limit) since Wed 2016-04-27 08:50:07
 CEST; 21min ago
 Process: 3139 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
 --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
 Process: 2682 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh
 --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
 (code=exited, status=0/SUCCESS)
Main PID: 3139 (code=exited, status=1/FAILURE)

 Apr 27 08:50:06 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
 entered failed state.
 Apr 27 08:50:07 ceph-cap1-02 systemd[1]: ceph-osd@47.service start
 request repeated too quickly, refusing to start.
 Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Failed to start Ceph object
 storage daemon.
 Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
 entered failed state.

 Which is no suprise as the osd is not mounted:

 # ls -l /var/lib/ceph/osd/ceph-47
 total 0

 The weird thing is running the following starts the osd:

 # echo add > /sys/class/block/sdr1/uevents

 so the udev rules to mount the osds seem to work.

 Any ideas on how to debug this?

 Best regards
 Karsten
 ___
 ceph-users mailing list
 ceph-users@lists.ceph.com
 http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

>>> ___
>>> ceph-users mailing list
>>> ceph-users@lists.ceph.com
>>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Slow read on RBD mount, Hammer 0.94.5

2016-04-27 Thread Mike Miller


Nick, all,

fantastic, that did it!

I installed kernel 4.5.2 on the client, now the single threaded read 
performance using krbd mount is up to about 370 MB/s with standard 256 
readahead size, and touching 400 MB/s with larger readahead sizes.

All single threaded.

Multi-threaded krbd read on the same mount also improved, a 10 GBit/s 
network connection is easily saturated.


Thank you all so much for the discussion and the hints.

Regards,

Mike

On 4/23/16 9:51 AM, n...@fisk.me.uk wrote:

I've just looked through github for the Linux kernel and it looks like
that read ahead fix was introduced in 4.4, so I'm not sure if it's worth
trying a slightly newer kernel?

Sent from Nine


*From:* Mike Miller 
*Sent:* 21 Apr 2016 2:20 pm
*To:* ceph-users@lists.ceph.com
*Subject:* Re: [ceph-users] Slow read on RBD mount, Hammer 0.94.5

Hi Udo,

thanks, just to make sure, further increased the readahead:

$ sudo blockdev --getra /dev/rbd0
1048576

$ cat /sys/block/rbd0/queue/read_ahead_kb
524288

No difference here. First one is sectors (512 bytes), second one KB.

The second read (after drop cache) is somewhat faster (10%-20%) but not
much.

I also found this info
http://tracker.ceph.com/issues/9192

Maybe Ilya can help us, he knows probably best how this can be improved.

Thanks and cheers,

Mike


On 4/21/16 4:32 PM, Udo Lembke wrote:

Hi Mike,

Am 21.04.2016 um 09:07 schrieb Mike Miller:

Hi Nick and Udo,

thanks, very helpful, I tweaked some of the config parameters along
the line Udo suggests, but still only some 80 MB/s or so.

this mean you have reached factor 3 (this are round about the value I
see with single thread on RBD too). Better than nothing.



Kernel 4.3.4 running on the client machine and comfortable readahead
configured

$ sudo blockdev --getra /dev/rbd0
262144

Still not more than about 80-90 MB/s.

they are two possibilities for read-ahead.
Take a look here (and change with echo)
cat /sys/block/rbd0/queue/read_ahead_kb

Perhaps there are slightly differences?



For writing the parallelization is amazing and I see very impressive
speeds, but why is reading performance so much behind? Why is it not
parallelized the same way writing is? Is this something coming up in
the jewel release? Or is it planned further down the road?

If you read an big file and clear your cache ("echo 3 >
/proc/sys/vm/drop_caches") on the client, is the second read very fast?
I assume yes.
In this case the readed data is in the cache on the osd-nodes... so
tuning must be there (and I'm very interesting in improvements).

Udo

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com



___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] NO mon start after Jewel Upgrade using systemctl

2016-04-27 Thread Iban Cabrillo

Hi cephers,
  I've been following the upgrade intrucctions...but..I sure I did
something wrong.

  I just upgrade using ceph-deploy on one monitor (after ofcourse down de
mon service).
  Then  the chow to var/lib/ceph and /var/log/ceph for ceph user

  [root@cephmon03 ~]# systemctl start ceph.target
[root@cephmon03 ~]#
[root@cephmon03 ~]#
[root@cephmon03 ~]# systemctl status ceph.target
● ceph.target - ceph target allowing to start/stop all ceph*@.service
instances at once
   Loaded: loaded (/usr/lib/systemd/system/ceph.target; disabled; vendor
preset: disabled)
   Active: active since mié 2016-04-27 13:43:24 CEST; 10min ago

abr 27 13:43:24 cephmon03.ifca.es systemd[1]: Reached target ceph target
allowing to start/stop all ceph*@.service instances at once.
abr 27 13:43:24 cephmon03.ifca.es systemd[1]: Starting ceph target allowing
to start/stop all ceph*@.service instances at once.
abr 27 13:44:17 cephmon03.ifca.es systemd[1]: Reached target ceph target
allowing to start/stop all ceph*@.service instances at once.
abr 27 13:47:09 cephmon03.ifca.es systemd[1]: Reached target ceph target
allowing to start/stop all ceph*@.service instances at once.
abr 27 13:53:36 cephmon03.ifca.es systemd[1]: Reached target ceph target
allowing to start/stop all ceph*@.service instances at once.

 [root@cephmon03 ~]# systemctl status ceph-mon@3
● ceph-mon@3.service - Ceph cluster monitor daemon
   Loaded: loaded (/usr/lib/systemd/system/ceph-mon@.service; disabled;
vendor preset: disabled)
   Active: inactive (dead)

abr 27 13:55:44 cephmon03 systemd[1]:
[/usr/lib/systemd/system/ceph-mon@.service:24]
Unknown lvalue 'TasksMax' in section 'Service'


looking at systemctl i see:

  *ceph-mon.cephmon03.1456312447.168540372.service  loadedactive
exited/usr/bin/bash -c ulimit -n 32768;  /usr/bin/ceph-mon -i cephmon03
--pid-file /var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster
ceph -f*
● ceph.service* not-found active exited*
ceph.service

but:

[root@cephmon03 ~]# systemctl start
ceph-mon.cephmon03.1456312447.168540372.service
[root@cephmon03 ~]# systemctl status
ceph-mon.cephmon03.1456312447.168540372.service
● ceph-mon.cephmon03.1456312447.168540372.service - /usr/bin/bash -c ulimit
-n 32768;  /usr/bin/ceph-mon -i cephmon03 --pid-file
/var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster ceph -f
   Loaded: loaded
(/run/systemd/system/ceph-mon.cephmon03.1456312447.168540372.service;
static; vendor preset: disabled)
  Drop-In:
/run/systemd/system/ceph-mon.cephmon03.1456312447.168540372.service.d
   └─50-Description.conf, 50-ExecStart.conf, 50-RemainAfterExit.conf
   Active: active (exited) since mié 2016-02-24 12:14:07 CET; 2 months 2
days ago
 Main PID: 1017 (code=exited, status=0/SUCCESS)
   CGroup: /system.slice/ceph-mon.cephmon03.1456312447.168540372.service

abr 27 13:35:33 cephmon03 bash[1017]: 2016-04-27 13:35:33.913841
7f16dba5b700 -1 mon.cephmon03@2(peon) e1 *** Got Signal Terminated ***
abr 27 14:02:37 cephmon03 systemd[1]: Started /usr/bin/bash -c ulimit -n
32768;  /usr/bin/ceph-mon -i cephmon03 --pid-file
/var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster ceph -f.
abr 27 14:04:37 cephmon03 systemd[1]: Started /usr/bin/bash -c ulimit -n
32768;  /usr/bin/ceph-mon -i cephmon03 --pid-file
/var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster ceph -f.

Nothing happens...


But running the command in line (as root):

/usr/bin/bash -c ulimit -n 32768;  /usr/bin/ceph-mon -i cephmon03
--pid-file /var/run/ceph/mon.cephmon03.pid -c /etc/ceph/ceph.conf --cluster
ceph -f

the mon starts well ( health HEALTH_OK)

Any idea about this??

regards, I

-- 

Iban Cabrillo Bartolome
Instituto de Fisica de Cantabria (IFCA)
Santander, Spain
Tel: +34942200969
PGP PUBLIC KEY:
http://pgp.mit.edu/pks/lookup?op=get=0xD9DF0B3D6C8C08AC

Bertrand Russell:
*"El problema con el mundo es que los estúpidos están seguros de todo y los
inteligentes están llenos de dudas*"
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osds udev rules not triggered on reboot (jewel, jessie)

2016-04-27 Thread Loris Cuoghi



Le 27/04/2016 13:51, Karsten Heymann a écrit :

Hi Loris,

thank you for your feedback. As I plan to go productive with the
cluster later this year I'm really hesitant to update udev and systemd
to a version newer than jessie, especially as there is no official
backport for those packages yet. I really would expect ceph to work
out of the box with the version in jessie, so I'd rather try to find
the cause of the problem and help fixing it.


That's my plan too, updating was just a troubleshoting method. ;)




best regards

2016-04-27 13:36 GMT+02:00 Loris Cuoghi :

Hi Karsten,

I've had the same experience updating our test cluster (Debian 8) from
Infernalis to Jewel.

I've update udev/systemd to the one in testing (so, from 215 to 229), and it
worked much better at reboot.

So... Are the udev rules written for the udev version in RedHat (219) or
greater versions ?

Thanks in advance :)


Le 27/04/2016 09:33, Karsten Heymann a écrit :


Hi!

the last days, I updated my jessie evaluation cluster to jewel and now
osds are not started automatically after reboot because they are not
mounted. This is the output of ceph-disk list after boot:

/dev/sdh :
   /dev/sdh1 ceph data, prepared, cluster ceph, osd.47, journal /dev/sde1
/dev/sdi :
   /dev/sdi1 ceph data, prepared, cluster ceph, osd.48, journal /dev/sde2
/dev/sdj :
   /dev/sdj1 ceph data, prepared, cluster ceph, osd.49, journal /dev/sde3

and so on.

systemd tried to start the units:

# systemctl | grep osd
● ceph-osd@47.service
   loaded failed failedCeph object
storage daemon
● ceph-osd@48.service
   loaded failed failedCeph object
storage daemon
● ceph-osd@49.service
   loaded failed failedCeph object
storage daemon

# systemctl status ceph-osd@47.service
● ceph-osd@47.service - Ceph object storage daemon
 Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled)
 Active: failed (Result: start-limit) since Wed 2016-04-27 08:50:07
CEST; 21min ago
Process: 3139 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
Process: 2682 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
(code=exited, status=0/SUCCESS)
   Main PID: 3139 (code=exited, status=1/FAILURE)

Apr 27 08:50:06 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
entered failed state.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: ceph-osd@47.service start
request repeated too quickly, refusing to start.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Failed to start Ceph object
storage daemon.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
entered failed state.

Which is no suprise as the osd is not mounted:

# ls -l /var/lib/ceph/osd/ceph-47
total 0

The weird thing is running the following starts the osd:

# echo add > /sys/class/block/sdr1/uevents

so the udev rules to mount the osds seem to work.

Any ideas on how to debug this?

Best regards
Karsten
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osds udev rules not triggered on reboot (jewel, jessie)

2016-04-27 Thread Karsten Heymann

Hi Loris,

thank you for your feedback. As I plan to go productive with the
cluster later this year I'm really hesitant to update udev and systemd
to a version newer than jessie, especially as there is no official
backport for those packages yet. I really would expect ceph to work
out of the box with the version in jessie, so I'd rather try to find
the cause of the problem and help fixing it.

best regards

2016-04-27 13:36 GMT+02:00 Loris Cuoghi :
> Hi Karsten,
>
> I've had the same experience updating our test cluster (Debian 8) from
> Infernalis to Jewel.
>
> I've update udev/systemd to the one in testing (so, from 215 to 229), and it
> worked much better at reboot.
>
> So... Are the udev rules written for the udev version in RedHat (219) or
> greater versions ?
>
> Thanks in advance :)
>
>
> Le 27/04/2016 09:33, Karsten Heymann a écrit :
>>
>> Hi!
>>
>> the last days, I updated my jessie evaluation cluster to jewel and now
>> osds are not started automatically after reboot because they are not
>> mounted. This is the output of ceph-disk list after boot:
>>
>> /dev/sdh :
>>   /dev/sdh1 ceph data, prepared, cluster ceph, osd.47, journal /dev/sde1
>> /dev/sdi :
>>   /dev/sdi1 ceph data, prepared, cluster ceph, osd.48, journal /dev/sde2
>> /dev/sdj :
>>   /dev/sdj1 ceph data, prepared, cluster ceph, osd.49, journal /dev/sde3
>>
>> and so on.
>>
>> systemd tried to start the units:
>>
>> # systemctl | grep osd
>> ● ceph-osd@47.service
>>   loaded failed failedCeph object
>> storage daemon
>> ● ceph-osd@48.service
>>   loaded failed failedCeph object
>> storage daemon
>> ● ceph-osd@49.service
>>   loaded failed failedCeph object
>> storage daemon
>>
>> # systemctl status ceph-osd@47.service
>> ● ceph-osd@47.service - Ceph object storage daemon
>> Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled)
>> Active: failed (Result: start-limit) since Wed 2016-04-27 08:50:07
>> CEST; 21min ago
>>Process: 3139 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
>> --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
>>Process: 2682 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh
>> --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
>> (code=exited, status=0/SUCCESS)
>>   Main PID: 3139 (code=exited, status=1/FAILURE)
>>
>> Apr 27 08:50:06 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
>> entered failed state.
>> Apr 27 08:50:07 ceph-cap1-02 systemd[1]: ceph-osd@47.service start
>> request repeated too quickly, refusing to start.
>> Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Failed to start Ceph object
>> storage daemon.
>> Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
>> entered failed state.
>>
>> Which is no suprise as the osd is not mounted:
>>
>> # ls -l /var/lib/ceph/osd/ceph-47
>> total 0
>>
>> The weird thing is running the following starts the osd:
>>
>> # echo add > /sys/class/block/sdr1/uevents
>>
>> so the udev rules to mount the osds seem to work.
>>
>> Any ideas on how to debug this?
>>
>> Best regards
>> Karsten
>> ___
>> ceph-users mailing list
>> ceph-users@lists.ceph.com
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Unable to unmap rbd device (Jewel)

2016-04-27 Thread Diego Castro

Here's one of my nodes that can't unmap a device:

[root@nodebr6 ~]# rbd unmap /dev/rbd0
2016-04-27 11:36:56.975668 7fcd61ae67c0 -1 did not load config file, using
default settings.
Option -q no longer supported.
rbd: run_cmd(udevadm): exited with status 1
rbd: sysfs write failed
rbd: unmap failed: (16) Device or resource busy


[root@nodebr6 ~]# fuser -amv /dev/rbd0
 USERPID ACCESS COMMAND
/dev/rbd0:

[root@nodebr6 ~]# ll /sys/block/rbd0/holders/
total 0


This is a Centos 7.2 (3.10.0-327.4.5.el7.x86_64)


---
Diego Castro / The CloudFather
GetupCloud.com - Eliminamos a Gravidade
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osds udev rules not triggered on reboot (jewel, jessie)

2016-04-27 Thread Loris Cuoghi


Hi Karsten,

I've had the same experience updating our test cluster (Debian 8) from 
Infernalis to Jewel.


I've update udev/systemd to the one in testing (so, from 215 to 229), 
and it worked much better at reboot.


So... Are the udev rules written for the udev version in RedHat (219) or 
greater versions ?


Thanks in advance :)

Le 27/04/2016 09:33, Karsten Heymann a écrit :

Hi!

the last days, I updated my jessie evaluation cluster to jewel and now
osds are not started automatically after reboot because they are not
mounted. This is the output of ceph-disk list after boot:

/dev/sdh :
  /dev/sdh1 ceph data, prepared, cluster ceph, osd.47, journal /dev/sde1
/dev/sdi :
  /dev/sdi1 ceph data, prepared, cluster ceph, osd.48, journal /dev/sde2
/dev/sdj :
  /dev/sdj1 ceph data, prepared, cluster ceph, osd.49, journal /dev/sde3

and so on.

systemd tried to start the units:

# systemctl | grep osd
● ceph-osd@47.service
  loaded failed failedCeph object
storage daemon
● ceph-osd@48.service
  loaded failed failedCeph object
storage daemon
● ceph-osd@49.service
  loaded failed failedCeph object
storage daemon

# systemctl status ceph-osd@47.service
● ceph-osd@47.service - Ceph object storage daemon
Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled)
Active: failed (Result: start-limit) since Wed 2016-04-27 08:50:07
CEST; 21min ago
   Process: 3139 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
   Process: 2682 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
(code=exited, status=0/SUCCESS)
  Main PID: 3139 (code=exited, status=1/FAILURE)

Apr 27 08:50:06 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
entered failed state.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: ceph-osd@47.service start
request repeated too quickly, refusing to start.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Failed to start Ceph object
storage daemon.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
entered failed state.

Which is no suprise as the osd is not mounted:

# ls -l /var/lib/ceph/osd/ceph-47
total 0

The weird thing is running the following starts the osd:

# echo add > /sys/class/block/sdr1/uevents

so the udev rules to mount the osds seem to work.

Any ideas on how to debug this?

Best regards
Karsten
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com


___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] Fwd: google perftools on ceph-osd

2016-04-27 Thread 席智勇

Anyone can give me some advice?
-- Forwarded message --
From: 
Date: 2016-04-26 18:50 GMT+08:00
Subject: google perftools on ceph-osd
To: Stefan Priebe - Profihost AG 


hi Stefan:

  When We are using ceph, I found osd process use much more CPU,
especially when small rand write. So I want to do some analysis to find the
slow point or bottleneck things.First I used the perf (record/report), and
found the memery alloc and free of tcmalloc use more CPU, but there is no
more info I want about the ceph-osd itself.
  I did some search and found this 'extreme ceph-osd cpu load for rand.
4k write' (
http://comments.gmane.org/gmane.comp.file-systems.ceph.devel/10315)
reported by you some years ago.In the mails you had atttached some files
gen by google-perftools, and got some detail about the ceph-osd.
  Below is how I did.
  First I add some code in ceph_osd.cc, to add a signal handler(code
like below),then I start a I/O bench on ceph, and wait the CPU usage of
ceph-osd get high, the send a SIGUSR1 to the process(kill -s SIGUSR1
pid),later send a SIGUSR2 signal(kill -s SIGUSR2 pid), now I get the perf
file, but when I use pprof tools to do some analysis, I found there is no
perf data.(Total: 0 samples).
  Can your share me more detail how did you do with google-perftools on
ceph-osd which is a daemon process.
---code-
diff --git a/src/ceph_osd.cc b/src/ceph_osd.cc
index a2f4542..1ed654e 100644
--- a/src/ceph_osd.cc
+++ b/src/ceph_osd.cc
@@ -12,6 +12,7 @@
  *
  */

+#include 
 #include 
 #include 
 #include 
@@ -83,6 +84,20 @@ int preload_erasure_code()
   return r;
 }

+void gprof_callback(int signum)
+{
+  if (signum == SIGUSR1)
+  {
+dout(0) << "Catch the signal ProfilerStart\n" << dendl;
+ProfilerStart("/tmp/bs.prof");
+  }
+  else if (signum == SIGUSR2)
+  {
+dout(0) << "Catch the signal ProfilerStop\n" << dendl;
+ProfilerStop();
+  }
+}
+
 int main(int argc, const char **argv)
 {
   vector args;
@@ -511,6 +526,11 @@ int main(int argc, const char **argv)
   // install signal handlers
   init_async_signal_handler();
   register_async_signal_handler(SIGHUP, sighup_handler);
+
+  register_async_signal_handler(SIGUSR1, gprof_callback);
+
+  register_async_signal_handler(SIGUSR2, gprof_callback);
+
   register_async_signal_handler_oneshot(SIGINT, handle_osd_signal);
   register_async_signal_handler_oneshot(SIGTERM, handle_osd_signal);
--

some log
2016-04-26 17:49:28.836289 7fd653475700  0 Catch the signal ProfilerStart

2016-04-26 17:52:44.877696 7fd653475700  0 Catch the signal ProfilerStop


regards~

Zhiyong Xi
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] radosgw crash - Infernalis

2016-04-27 Thread Karol Mroz

On Tue, Apr 26, 2016 at 10:17:31PM -0700, Ben Hines wrote:
[...]
> --> 10.30.1.6:6800/10350 -- osd_op(client.44852756.0:79
> default.42048218. [getxattrs,stat,read 0~524288] 12.aa730416
> ack+read+known_if_redirected e100207) v6 -- ?+0 0x7f49c41880b0 con
> 0x7f49c4145eb0
>  0> 2016-04-26 22:07:59.685615 7f49a07f0700 -1 *** Caught signal
> (Segmentation fault) **
>  in thread 7f49a07f0700
> 
>  ceph version 9.2.1 (752b6a3020c3de74e07d2a8b4c5e48dab5a6b6fd)
>  1: (()+0x30b0a2) [0x7f4c4907f0a2]
>  2: (()+0xf100) [0x7f4c44f7a100]
>  NOTE: a copy of the executable, or `objdump -rdS ` is needed
> to interpret this.

Hi Ben,

I sense a pretty badly corrupted stack. From the radosgw-9.2.1 (obtained from
a downloaded rpm):

0030a810 <_Z13pidfile_writePK11md_config_t@@Base>:
...
  30b09d:   e8 0e 40 e4 ff  callq  14f0b0 
  30b0a2:   4c 89 efmov%r13,%rdi
  ---
...

So either we tripped backtrace() code from pidfile_write() _or_ we can't
trust the stack. From the log snippet, it looks that we're far past the point
at which we would write a pidfile to disk (ie. at process start during 
global_init()).
Rather, we're actually handling a request and outputting some bit of debug 
message
via MSDOp::print() and beyond...

Is this something you're able to easily reproduce? More logs with higher log 
levels
would be helpful... a coredump with radosgw compiled with -g would be excellent 
:)

-- 
Regards,
Karol

signature.asc
Description: Digital signature
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Any Docs to configure NFS to access RADOSGW buckets on Jewel

2016-04-27 Thread WD_Hwang

Hi Ansgar,
  Thanks for your information.
  I have tried 's3fs-fuse' to mount RADOSGW buckets on Ubuntu client node. It 
works.
  But I am not sure this is the technique that access RADOSGW buckets via NFS 
on Jewel.

Best Regards,
WD

-Original Message-
From: ceph-users [mailto:ceph-users-boun...@lists.ceph.com] On Behalf Of Ansgar 
Jazdzewski
Sent: Wednesday, April 27, 2016 4:32 PM
To: WD Hwang/WHQ/Wistron
Cc: ceph-users@lists.ceph.com
Subject: Re: [ceph-users] Any Docs to configure NFS to access RADOSGW buckets 
on Jewel

all informations i have so far are from the FOSDEM

https://fosdem.org/2016/schedule/event/virt_iaas_ceph_rados_gateway_overview/attachments/audio/1079/export/events/attachments/virt_iaas_ceph_rados_gateway_overview/audio/1079/Fosdem_RGW.pdf

Cheers,
Ansgar

2016-04-27 2:28 GMT+02:00  :
> Hello:
>
>   Are there any documents or examples to explain the configuration of 
> NFS to access RADOSGW buckets on Jewel?
>
> Thanks a lot.
>
>
>
> Best Regards,
>
> WD
>
> --
> --
> ---
>
> This email contains confidential or legally privileged information and 
> is for the sole use of its intended recipient.
>
> Any unauthorized review, use, copying or distribution of this email or 
> the content of this email is strictly prohibited.
>
> If you are not the intended recipient, you may reply to the sender and 
> should delete this e-mail immediately.
>
> --
> --
> ---
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] Any Docs to configure NFS to access RADOSGW buckets on Jewel

2016-04-27 Thread Ansgar Jazdzewski

all informations i have so far are from the FOSDEM

https://fosdem.org/2016/schedule/event/virt_iaas_ceph_rados_gateway_overview/attachments/audio/1079/export/events/attachments/virt_iaas_ceph_rados_gateway_overview/audio/1079/Fosdem_RGW.pdf

Cheers,
Ansgar

2016-04-27 2:28 GMT+02:00  :
> Hello:
>
>   Are there any documents or examples to explain the configuration of NFS to
> access RADOSGW buckets on Jewel?
>
> Thanks a lot.
>
>
>
> Best Regards,
>
> WD
>
> ---
>
> This email contains confidential or legally privileged information and is
> for the sole use of its intended recipient.
>
> Any unauthorized review, use, copying or distribution of this email or the
> content of this email is strictly prohibited.
>
> If you are not the intended recipient, you may reply to the sender and
> should delete this e-mail immediately.
>
> ---
>
>
> ___
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

Re: [ceph-users] osds udev rules not triggered on reboot (jewel, jessie)

2016-04-27 Thread Karsten Heymann

Add-on-Question:

What is the intended purpose of the ceph-disk@.service? I can run

systemctl start ceph-disk@/dev/sdr1

but I can't 'enable' it like the the ceph-osd@.service so why is it there?

Best regards
Karsten

2016-04-27 9:33 GMT+02:00 Karsten Heymann :
> Hi!
>
> the last days, I updated my jessie evaluation cluster to jewel and now
> osds are not started automatically after reboot because they are not
> mounted. This is the output of ceph-disk list after boot:
>
> /dev/sdh :
>  /dev/sdh1 ceph data, prepared, cluster ceph, osd.47, journal /dev/sde1
> /dev/sdi :
>  /dev/sdi1 ceph data, prepared, cluster ceph, osd.48, journal /dev/sde2
> /dev/sdj :
>  /dev/sdj1 ceph data, prepared, cluster ceph, osd.49, journal /dev/sde3
>
> and so on.
>
> systemd tried to start the units:
>
> # systemctl | grep osd
> ● ceph-osd@47.service
>  loaded failed failedCeph object
> storage daemon
> ● ceph-osd@48.service
>  loaded failed failedCeph object
> storage daemon
> ● ceph-osd@49.service
>  loaded failed failedCeph object
> storage daemon
>
> # systemctl status ceph-osd@47.service
> ● ceph-osd@47.service - Ceph object storage daemon
>Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled)
>Active: failed (Result: start-limit) since Wed 2016-04-27 08:50:07
> CEST; 21min ago
>   Process: 3139 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
> --id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
>   Process: 2682 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh
> --cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
> (code=exited, status=0/SUCCESS)
>  Main PID: 3139 (code=exited, status=1/FAILURE)
>
> Apr 27 08:50:06 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
> entered failed state.
> Apr 27 08:50:07 ceph-cap1-02 systemd[1]: ceph-osd@47.service start
> request repeated too quickly, refusing to start.
> Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Failed to start Ceph object
> storage daemon.
> Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
> entered failed state.
>
> Which is no suprise as the osd is not mounted:
>
> # ls -l /var/lib/ceph/osd/ceph-47
> total 0
>
> The weird thing is running the following starts the osd:
>
> # echo add > /sys/class/block/sdr1/uevents
>
> so the udev rules to mount the osds seem to work.
>
> Any ideas on how to debug this?
>
> Best regards
> Karsten
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

[ceph-users] osds udev rules not triggered on reboot (jewel, jessie)

2016-04-27 Thread Karsten Heymann

Hi!

the last days, I updated my jessie evaluation cluster to jewel and now
osds are not started automatically after reboot because they are not
mounted. This is the output of ceph-disk list after boot:

/dev/sdh :
 /dev/sdh1 ceph data, prepared, cluster ceph, osd.47, journal /dev/sde1
/dev/sdi :
 /dev/sdi1 ceph data, prepared, cluster ceph, osd.48, journal /dev/sde2
/dev/sdj :
 /dev/sdj1 ceph data, prepared, cluster ceph, osd.49, journal /dev/sde3

and so on.

systemd tried to start the units:

# systemctl | grep osd
● ceph-osd@47.service
 loaded failed failedCeph object
storage daemon
● ceph-osd@48.service
 loaded failed failedCeph object
storage daemon
● ceph-osd@49.service
 loaded failed failedCeph object
storage daemon

# systemctl status ceph-osd@47.service
● ceph-osd@47.service - Ceph object storage daemon
   Loaded: loaded (/lib/systemd/system/ceph-osd@.service; enabled)
   Active: failed (Result: start-limit) since Wed 2016-04-27 08:50:07
CEST; 21min ago
  Process: 3139 ExecStart=/usr/bin/ceph-osd -f --cluster ${CLUSTER}
--id %i --setuser ceph --setgroup ceph (code=exited, status=1/FAILURE)
  Process: 2682 ExecStartPre=/usr/lib/ceph/ceph-osd-prestart.sh
--cluster ${CLUSTER} --id %i --setuser ceph --setgroup ceph
(code=exited, status=0/SUCCESS)
 Main PID: 3139 (code=exited, status=1/FAILURE)

Apr 27 08:50:06 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
entered failed state.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: ceph-osd@47.service start
request repeated too quickly, refusing to start.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Failed to start Ceph object
storage daemon.
Apr 27 08:50:07 ceph-cap1-02 systemd[1]: Unit ceph-osd@47.service
entered failed state.

Which is no suprise as the osd is not mounted:

# ls -l /var/lib/ceph/osd/ceph-47
total 0

The weird thing is running the following starts the osd:

# echo add > /sys/class/block/sdr1/uevents

so the udev rules to mount the osds seem to work.

Any ideas on how to debug this?

Best regards
Karsten
___
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com

39 matches

Mail list logo