(I'm slowly catching up on a backlog of e-mail, sorry for the delayed
reply).
Thanks, Sven. I recognize the complexity and appreciate your explanation.
In my mind I had envisioned either the block integrity information being
stored as a new metadata structure or stored leveraging T10-DIX/DIF
(perhaps configurable on a per-pool basis) to pass the checksums down to
the RAID controller.
I would quite like to run GNR as software on generic hardware and in
fact voted, along with 26 other customers, on an RFE
(https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090)
requesting this but the request was declined. I think customers spoke
pretty loudly there and IBM gave it the kibosh.
-Aaron
On 06/30/2017 02:25 PM, Sven Oehme wrote:
end-to-end data integrity is very important and the reason it hasn't
been done in Scale is not because its not important, its because its
very hard to do without impacting performance in a very dramatic way.
imagine your raid controller blocksize is 1mb and your filesystem
blocksize is 1MB . if your application does a 1 MB write this ends up
being a perfect full block , full track de-stage to your raid layer
and everything works fine and fast. as soon as you add checksum
support you need to add data somehow into this, means your 1MB is no
longer 1 MB but 1 MB+checksum.
to store this additional data you have multiple options, inline ,
outside the data block or some combination ,the net is either you need
to do more physical i/o's to different places to get both the data and
the corresponding checksum or your per block on disc structure becomes
bigger than than what your application reads/or writes, both put
massive burden on the Storage layer as e.g. a 1 MB write will now,
even the blocks are all aligned from the application down to the raid
layer, cause a read/modify/write on the raid layer as the data is
bigger than the physical track size.
so to get end-to-end checksum in Scale outside of ESS the best way is
to get GNR as SW to run on generic HW, this is what people should vote
for as RFE if they need that functionality. beside end-to-end
checksums you get read/write cache and acceleration , fast rebuild and
many other goodies as a added bonus.
Sven
On Fri, Jun 30, 2017 at 10:53 AM Aaron Knister
<[email protected] <mailto:[email protected]>> wrote:
In fact the answer was quite literally "no":
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=84523
(the RFE was declined and the answer was that the "function is already
available in GNR environments").
Regarding GNR, see this RFE request
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=95090
requesting the use of GNR outside of an ESS/GSS environment. It's
interesting to note this is the highest voted Public RFE for GPFS
that I
can see, at least. It too was declined.
-Aaron
On 6/30/17 1:41 PM, Aaron Knister wrote:
> Thanks Olaf, that's good to know (and is kind of what I
suspected). I've
> requested a number of times this capability for those of us who
can't
> use or aren't using GNR and the answer is effectively "no". This
> response is curious to me because I'm sure IBM doesn't believe
that data
> integrity is only important and of value to customers who
purchase their
> hardware *and* software.
>
> -Aaron
>
> On Fri, Jun 30, 2017 at 1:37 PM, Olaf Weiser
<[email protected] <mailto:[email protected]>
> <mailto:[email protected] <mailto:[email protected]>>>
wrote:
>
> yes.. in case of GNR (GPFS native raid) .. we do end-to-end
> check-summing ... client --> server --> downToDisk
> GNR writes down a chksum to disk (to all pdisks /all "raid"
segments
> ) so that dropped writes can be detected as well as miss-done
> writes (bit flips..)
>
>
>
> From: Aaron Knister <[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> To: gpfsug main discussion list
<[email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>>
> Date: 06/30/2017 07:15 PM
> Subject: [gpfsug-discuss] Fwd: FLASH: IBM Spectrum Scale (GPFS):
> RDMA-enabled network adapter failure on the NSD server may
result in
> file IO error (2017.06.30)
> Sent by: [email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>
>
------------------------------------------------------------------------
>
>
>
> I'm curious to know why this doesn't affect GSS/ESS? Is it a
feature of
> the additional check-summing done on those platforms?
>
>
> -------- Forwarded Message --------
> Subject: FLASH: IBM Spectrum Scale (GPFS): RDMA-enabled network
> adapter
> failure on the NSD server may result in file IO error
(2017.06.30)
> Date: Fri, 30 Jun 2017 14:19:02 +0000
> From: IBM My Notifications
> <[email protected]
<mailto:[email protected]>
<mailto:[email protected]
<mailto:[email protected]>>>
> To: [email protected]
<mailto:[email protected]> <mailto:[email protected]
<mailto:[email protected]>>
>
>
>
>
> My Notifications for Storage - 30 Jun 2017
>
> Dear Subscriber ([email protected]
<mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>>),
>
> Here are your updates from IBM My Notifications.
>
> Your support Notifications display in English by default.
Machine
> translation based on your IBM profile
> language setting is added if you specify this option in My
defaults
> within My Notifications.
> (Note: Not all languages are available at this time, and the
English
> version always takes precedence
> over the machine translated version.)
>
>
------------------------------------------------------------------------------
> 1. IBM Spectrum Scale
>
> - TITLE: IBM Spectrum Scale (GPFS): RDMA-enabled network adapter
> failure
> on the NSD server may result in file IO error
> - URL:
>
http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E
>
<http://www.ibm.com/support/docview.wss?uid=ssg1S1010233&myns=s033&mynp=OCSTXKQY&mynp=OCSWJ00&mync=E&cm_sp=s033-_-OCSTXKQY-OCSWJ00-_-E>
> - ABSTRACT: IBM has identified an issue with all IBM GPFS
and IBM
> Spectrum Scale versions where the NSD server is enabled to
use RDMA for
> file IO and the storage used in your GPFS cluster accessed
via NSD
> servers (not fully SAN accessible) includes anything other
than IBM
> Elastic Storage Server (ESS) or GPFS Storage Server (GSS);
under these
> conditions, when the RDMA-enabled network adapter fails, the
issue may
> result in undetected data corruption for file write or read
operations.
>
>
------------------------------------------------------------------------------
> Manage your My Notifications subscriptions, or send
questions and
> comments.
> - Subscribe or Unsubscribe -
> https://www.ibm.com/support/mynotifications
> <https://www.ibm.com/support/mynotifications>
> - Feedback -
>
https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html
>
<https://www-01.ibm.com/support/feedback/techFeedbackCardContentMyNotifications.html>
>
> - Follow us on Twitter - https://twitter.com/IBMStorageSupt
> <https://twitter.com/IBMStorageSupt>
>
>
>
> To ensure proper delivery please add
[email protected] <mailto:[email protected]>
> <mailto:[email protected]
<mailto:[email protected]>> to
> your address book.
> You received this email because you are subscribed to IBM My
> Notifications as:
> [email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>
>
> Please do not reply to this message as it is generated by an
automated
> service machine.
>
> (C) International Business Machines Corporation 2017. All rights
> reserved.
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
<http://spectrumscale.org> <http://spectrumscale.org>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org
<http://spectrumscale.org> <http://spectrumscale.org>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
> <http://gpfsug.org/mailman/listinfo/gpfsug-discuss>
>
>
>
>
> _______________________________________________
> gpfsug-discuss mailing list
> gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
> http://gpfsug.org/mailman/listinfo/gpfsug-discuss
>
--
Aaron Knister
NASA Center for Climate Simulation (Code 606.2)
Goddard Space Flight Center
(301) 286-2776 <tel:%28301%29%20286-2776>
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org <http://spectrumscale.org>
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
_______________________________________________
gpfsug-discuss mailing list
gpfsug-discuss at spectrumscale.org
http://gpfsug.org/mailman/listinfo/gpfsug-discuss
