Hi Jan, Thanks for your response.
> *How exactly do you know this is the cause? This is usually just an effect > of something going wrong and part of error recovery process.**Preceding > this event should be the real error/root cause...* We have been working with LSI/Avago to resolve this. We get a bunch of these type log events: 2015-09-04T14:58:59.169677+12:00 <server_name> ceph-osd: - ceph-osd: 2015-09-04 14:58:59.168444 7fbc5ec71700 0 log [WRN] : slow request 30.894936 seconds old, received at 2015-09-04 14:58:28.272976: osd_op(client.42319583.0:1185218039 rbd_data.1d8a5a92eb141f2.00000000000056a0 [read 3579392~8192] 4.f9f016cb ack+read e66603) v4 currently no flag points reached Followed by the task abort I mentioned: sd 11:0:4:0: attempting task abort! scmd(ffff8804c07d0480) sd 11:0:4:0: [sdf] CDB: Write(10): 2a 00 24 6f 01 a8 00 00 08 00 scsi target11:0:4: handle(0x000d), sas_address(0x4433221104000000), phy(4) scsi target11:0:4: enclosure_logical_id(0x5003048000000000), slot(4) sd 11:0:4:0: task abort: SUCCESS scmd(ffff8804c07d0480) LSI had us enable debugging on our card and send them many logs and debugging data. Their response was: Please do not send in the Synchronize cache command(35h). That’s the one > causing the drive from not responding to Read/write commands quick enough. A Synchronize cache command instructs the ATA device to flush the cache > contents to medium and so while the disk is in the process of doing it, > it’s probably causing the read/write commands to take longer time to > complete. LSI/Avago believe this to be the root cause of the IO delay based on the debugging info. *and from what I've seen it is not necessary with fast drives (such as > S3700).* While I agree with you that it should not be necessary as the S3700's should be very fast, our current experience does not show this to be the case. Just a little more about our setup. We're using Ceph Firefly (0.80.10) on Ubuntu 14.04. We see this same thing on every S3700/10 on four hosts. We do not see this happening on the spinning disks in the same cluster but different pool on similar hardware. If you know of any other reason this may be happening, we would appreciate it. Otherwise we will need to continue investigating the possibility of setting nobarriers. Regards, Richard On 5 September 2015 at 09:32, Jan Schermer <j...@schermer.cz> wrote: > We are seeing some significant I/O delays on the disks causing a “SCSI > Task Abort” from the OS. This seems to be triggered by the drive receiving > a “Synchronize cache command”. > > > How exactly do you know this is the cause? This is usually just an effect > of something going wrong and part of error recovery process. > Preceding this event should be the real error/root cause... > > It is _supposedly_ safe to disable barriers in this scenario, but IMO the > assumptions behind that are deeply flawed, and from what I've seen it is > not necessary with fast drives (such as S3700). > > Take a look in the mailing list archives, I elaborated on this quite a bit > in the past, including my experience with Kingston drives + XFS + LSI (and > the effect is present even on Intels, but because they are much faster it > shouldn't cause any real problems). > > Jan > > > On 04 Sep 2015, at 21:55, Richard Bade <hitr...@gmail.com> wrote: > > Hi Everyone, > > We have a Ceph pool that is entirely made up of Intel S3700/S3710 > enterprise SSD's. > > We are seeing some significant I/O delays on the disks causing a “SCSI > Task Abort” from the OS. This seems to be triggered by the drive receiving > a “Synchronize cache command”. > > My current thinking is that setting nobarriers in XFS will stop the drive > receiving a sync command and therefore stop the I/O delay associated with > it. > > In the XFS FAQ it looks like the recommendation is that if you have a > Battery Backed raid controller you should set nobarriers for performance > reasons. > > Our LSI card doesn’t have battery backed cache as it’s configured in HBA > mode (IT) rather than Raid (IR). Our Intel s37xx SSD’s do have a capacitor > backed cache though. > > So is it recommended that barriers are turned off as the drive has a safe > cache (I am confident that the cache will write out to disk on power > failure)? > > Has anyone else encountered this issue? > > Any info or suggestions about this would be appreciated. > > Regards, > > Richard > _______________________________________________ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > > >
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
