On Wed, Mar 1, 2017 at 11:34 PM, Sage Weil <[email protected]> wrote:
> On Wed, 1 Mar 2017, Stephen Blinick wrote: > > Hello, I'm chasing down a situation where there's periodic slow requests > > occurring. While the specific version in this case is 0.80.7 Firefly, I > > think this log format is the same in newer versions. I can verify. > > There's a host of symptoms going on, but one strange anomaly I found > that I > > wasn't able to chase down or find in search has to do with the Op > parameters > > for an osd_op being logged as a slow request. > > > > Specifically, in some cases the byte range for an op of various types > (i.e. > > read, writefull) is sometimes negative. Here's an example of two slow > > request log entries: > > > > #1 > > 2017-02-28 18:39:09.943169 osd.27 10.0.1.84:6822/2402845 5255 : [WRN] > slow > > request 16.440574 seconds old, received at 2017-02-28 18:38:53.502539: > > osd_op(client.2000529.0:496 ObjectNameOne [writefull 0~4194304] > 3.3f26fcc9 > > ondisk+write e691) v4 currently commit sent > > > > #2 > > 2017-02-28 18:39:05.959253 osd.40 10.0.1.88:6831/2187230 6180 : [WRN] > slow > > request 8.470175 seconds old, received at 2017-02-28 18:38:57.489045: > > osd_op(client.1941470.0:21164213 ObjectNameTwo [read 3670016~524288] > > 3.3a50c331 ack+read e691) v4 currently started > > > > As you can see, some of them show the byte range A~B where B is lower > than > > A. I'm mostly interested to find out if this is an indication of any > > This is quirky Ceph convention for printing extents as offset~length (it's > not start~end). So these look fine. > Ahh I should have guessed. This makes a lot more sense. I'll update the parser accordingly. Thanks! > > Firefly 0.80.7? You should really upgrade. Twice (to hammer and then to > jewel). > > Indeed! We already have for the most part, but this system is in production, so that always adds friction to any upgrade. Definitely makes problem debug more of a 'forensic' exercise :) > sage > > > problem. This is an EC pool, 3+2. > > > > Thanks, > > > > Stephen > > > > >
_______________________________________________ ceph-users mailing list [email protected] http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
