Hello Mark,
As this issue noticed while giving write via librados {C API} only , the
same can't be reproduce with rados user space utility.
Ref;- http://docs.ceph.com/docs/master/rados/api/librados/
Jack, I guess you also creating load via librados.
Thanks
Jayaram
On Thu, Jun 8, 2017 at 5:46 PM, Mark Nelson <mnel...@redhat.com> wrote:
> Hi Jayaram,
>
> Thanks for creating a tracker entry! Any chance you could add a note about
> how you are generating the 200MB/s client workload? I've not seen this
> problem in the lab, but any details you could give that would help us
> reproduce the problem would be much appreciated!
>
> Mark
>
> On 06/08/2017 06:08 AM, nokia ceph wrote:
>
>> Hello Mark,
>>
>> Raised tracker for the issue -- http://tracker.ceph.com/issues/20222
>>
>> Jake can you share the restart_OSD_and_log-this.sh script
>>
>> Thanks
>> Jayaram
>>
>> On Wed, Jun 7, 2017 at 9:40 PM, Jake Grimmett <j...@mrc-lmb.cam.ac.uk
>> <mailto:j...@mrc-lmb.cam.ac.uk>> wrote:
>>
>> Hi Mark & List,
>>
>> Unfortunately, even when using yesterdays master version of ceph,
>> I'm still seeing OSDs go down, same error as before:
>>
>> OSD log shows lots of entries like this:
>>
>> (osd38)
>> 2017-06-07 16:48:46.070564 7f90b58c3700 1 heartbeat_map is_healthy
>> 'tp_osd_tp thread tp_osd_tp' had timed out after 60
>>
>> (osd3)
>> 2017-06-07 17:01:25.391075 7f62de6c3700 1 heartbeat_map is_healthy
>> 'tp_osd_tp thread tp_osd_tp' had timed out after 60
>> 2017-06-07 17:01:26.276881 7f62dbe86700 -1 osd.3 6165 heartbeat_check:
>> no reply from 10.1.0.86:6811 <http://10.1.0.86:6811> osd.2 since
>>
>> back 2017-06-07 17:00:19.640002
>> front 2017-06-07 17:01:21.950160 (cutoff 2017-06-07 17:01:06.276881)
>>
>>
>> [root@ceph4 ceph]# ceph -v
>> ceph version 12.0.2-2399-ge38ca14
>> (e38ca14914340d65ea8001c7bd6e0ff769f3eb2e) luminous (dev)
>>
>>
>> I'll continue running the cluster with my
>> "restart_OSD_and_log-this.sh"
>> workaround...
>>
>> thanks again for your help,
>>
>> Jake
>>
>> On 06/06/17 15:52, Jake Grimmett wrote:
>> > Hi Mark,
>> >
>> > OK, I'll upgrade to the current master and retest...
>> >
>> > best,
>> >
>> > Jake
>> >
>> > On 06/06/17 15:46, Mark Nelson wrote:
>> >> Hi Jake,
>> >>
>> >> I just happened to notice this was on 12.0.3. Would it be
>> possible to
>> >> test this out with current master and see if it still is a problem?
>> >>
>> >> Mark
>> >>
>> >> On 06/06/2017 09:10 AM, Mark Nelson wrote:
>> >>> Hi Jake,
>> >>>
>> >>> Thanks much. I'm guessing at this point this is probably a
>> bug. Would
>> >>> you (or nokiauser) mind creating a bug in the tracker with a short
>> >>> description of what's going on and the collectl sample showing
>> this is
>> >>> not IOs backing up on the disk?
>> >>>
>> >>> If you want to try it, we have a gdb based wallclock profiler
>> that might
>> >>> be interesting to run while it's in the process of timing out.
>> It tries
>> >>> to grab 2000 samples from the osd process which typically takes
>> about 10
>> >>> minutes or so. You'll need to either change the number of
>> samples to be
>> >>> lower in the python code (maybe like 50-100), or change the
>> timeout to
>> >>> be something longer.
>> >>>
>> >>> You can find the code here:
>> >>>
>> >>> https://github.com/markhpc/gdbprof
>> <https://github.com/markhpc/gdbprof>
>> >>>
>> >>> and invoke it like:
>> >>>
>> >>> udo gdb -ex 'set pagination off' -ex 'attach 27962' -ex 'source
>> >>> ./gdbprof.py' -ex 'profile begin' -ex 'quit'
>> >>>
>> >>> where 27962 in this case is the PID of the ceph-osd process.
>> You'll
>> >>> need gdb with the python bindings and the ceph debug symbols for
>> it to
>> >>> work.
>> >>>
>> >>> This might tell us over time if the tp_osd_tp processes are just
>> sitting
>> >>> on pg::locks.
>> >>>
>> >>> Mark
>> >>>
>> >>> On 06/06/2017 05:34 AM, Jake Grimmett wrote:
>> >>>> Hi Mark,
>> >>>>
>> >>>> Thanks again for looking into this problem.
>> >>>>
>> >>>> I ran the cluster overnight, with a script checking for dead
>> OSDs every
>> >>>> second, and restarting them.
>> >>>>
>> >>>> 40 OSD failures occurred in 12 hours, some OSDs failed multiple
>> times,
>> >>>> (there are 50 OSDs in the EC tier).
>> >>>>
>> >>>> Unfortunately, the output of collectl doesn't appear to show any
>> >>>> increase in disk queue depth and service times before the OSDs
>> die.
>> >>>>
>> >>>> I've put a couple of examples of collectl output for the disks
>> >>>> associated with the OSDs here:
>> >>>>
>> >>>> https://hastebin.com/icuvotemot.scala
>> <https://hastebin.com/icuvotemot.scala>
>> >>>>
>> >>>> please let me know if you need more info...
>> >>>>
>> >>>> best regards,
>> >>>>
>> >>>> Jake
>> >>>>
>> >>>>
>> >
>> _______________________________________________
>> ceph-users mailing list
>> ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com>
>> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>> <http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com>
>>
>>
>>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
