On Thu, 2017-03-16 at 10:39 -0400, Aaron Knister wrote: > In all seriousness, I'd love to be wrong about this. I'm not sure > which part(s) of what I said was inaccurate-- the vendor lock in > and/or that GNR is the only way to get end to end checksums.
The end to end checksums, or at least the ability to protect against silent corruption by having the data checksummed at all stages. > > When I say end to end checksums I mean that from the moment an FS > write is submitted to mmfsd a checksum is calculated that is passed > through every layer (memory, network, sas, fibre channel etc.) down to > individual disk drives (understanding that the RAID controller may > need to derive the checksum based on whatever RAID algorithm it's > using). It's my understanding that the only way to achieve this with > GPFS today is with GNR which is only available via purchasing a GSS or > ESS solution from IBM/Lenovo. One is of course free to by hardware > from any vendor but you don't get GNR. I should really have said "if > you want GNR you have to buy hardware from IBM or Lenovo" which to me > is being locked in to a vendor as long as you'd like end to end > checksums. > > If there's another way to get end-to-end checksums, could you help me > understand how? > Bear in mind the purpose of the checksums is to ensure the data is not corrupted. Noting you don't get true end to end because the checksums are never exposed to the application even in ZFS, at least as I understand it; you just get a read failure. > Regarding DIF/DIX, it's my understanding that I can/could use T10-DIF > today (with the correct hardware) without purchasing any standards > which would checksum data from the HBA to the RAID controller (and in > theory disks). However, in an environment with NSD servers the origin > of a read/write could in theory be quite a bit further away from the > HBA in terms of hops. T10-DIF would be completely separate from GPFS > as I understand it. I'm not aware of any integration (T10-DIF + DIX). > What I'm really looking for, I think, is T10-DIF + DIX where the > application (GPFS) generates protection information that's then passed > along to the layers below it. > Well yes an end user does not need to purchase a copy of the standard. However some people take the view that as you need to spend money to get the standard it is not truly open, so I was just making that clear. Right, so bearing in mind that the purpose of the checksums is to ensure data is not corrupted if the NSD servers are using DIF/DIX, then the options for silent data corruption are very limited if you are using ECC memory and you are using standard TCP/IP to communicate between the nodes. That is the TCP/IP checksums will protect your data between the clients and the NSD nodes, and once at the NSD node the DIF/DIX will protect your data as it makes it way to the disk. In between all that the ECC memory in your machines will protect everything once in RAM, and the PCIe bus has a 32bit CRC protecting everything that moves over the PCIe bus. The only "hole" in this is as far as I am aware is that the CPU itself, because at least x86 ones (as far as I am aware) do not "checksum" themselves as they do calculations, but you have the same problem with ZFS for exactly the same reason. So the point is that what you should be interested in as an admin; ensuring your data is protected against silent corruption, is achievable without hardware vendor lock in using open standards. The advantage of DIF/DIX over ZFS is as I understand it unless you do an immediate read on written blocks with ZFS and take a large performance penalty on writes you have no idea if your data has been corrupted until you try and read it back which could be months if not years down the line. Where DIF/DIX should try and fix it immediately by a retry and if that does not work pass the failure back up the line immediately. JAB. -- Jonathan A. Buzzard Email: jonathan (at) buzzard.me.uk Fife, United Kingdom. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
