Hi Alex, did you try mmhealth ? It should detect stale file handles of the gpfs filesystems already and report a "stale_mount" event.
Mit freundlichen Grüßen / Kind regards Mathias Dietz Spectrum Scale Development - Release Lead Architect (4.2.x) Spectrum Scale RAS Architect --------------------------------------------------------------------------- IBM Deutschland Am Weiher 24 65451 Kelsterbach Phone: +49 70342744105 Mobile: +49-15152801035 E-Mail: [email protected] ----------------------------------------------------------------------------- IBM Deutschland Research & Development GmbH Vorsitzender des Aufsichtsrats: Martina Koederitz, Geschäftsführung: Dirk WittkoppSitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht Stuttgart, HRB 243294 From: Alexander John Mamach <[email protected]> To: gpfsug main discussion list <[email protected]> Cc: "[email protected]" <[email protected]> Date: 09/08/2019 22:33 Subject: [EXTERNAL] Re: [gpfsug-discuss] Checking for Stale File Handles Sent by: [email protected] Hi Fred, We sometimes find a node will show that GPFS is active when running mmgetstate, but one of our GPFS filesystems, (such as our home or projects filesystems) are inaccessible to users, while the other GPFS-mounted filesystems behave as expected. Our current node health checks don?t always detect this, especially when it?s for a resource-based mount that doesn?t impact the node but would impact jobs trying to run on the node. If there is something native to GPFS that can detect this, all the better, but I?m simply unaware of how to do so. Thanks, Alex Senior Systems Administrator Research Computing Infrastructure Northwestern University Information Technology (NUIT) 2020 Ridge Ave Evanston, IL 60208-4311 O: (847) 491-2219 M: (312) 887-1881 www.it.northwestern.edu From: [email protected] <[email protected]> on behalf of Frederick Stock <[email protected]> Sent: Friday, August 9, 2019 1:03:09 PM To: [email protected] <[email protected]> Cc: [email protected] <[email protected]> Subject: Re: [gpfsug-discuss] Checking for Stale File Handles Are you able to explain why you want to check for stale file handles? Are you attempting to detect failures of some sort, and why do the existing mechanisms in GPFS not provide the functionality you require? Fred __________________________________________________ Fred Stock | IBM Pittsburgh Lab | 720-430-8821 [email protected] ----- Original message ----- From: Alexander John Mamach <[email protected]> Sent by: [email protected] To: "[email protected]" <[email protected]> Cc: Subject: [EXTERNAL] [gpfsug-discuss] Checking for Stale File Handles Date: Fri, Aug 9, 2019 1:46 PM Hi folks, We?re currently investigating a way to check for stale file handles on the nodes across our cluster in a way that minimizes impact to the filesystem and performance. Has anyone found a direct way of doing so? We considered a few methods, including simply attempting to ls a GPFS filesystem from each node, but that might have false positives, (detecting slowdowns as stale file handles), and could negatively impact performance with hundreds of nodes doing this simultaneously. Thanks, Alex Senior Systems Administrator Research Computing Infrastructure Northwestern University Information Technology (NUIT) 2020 Ridge Ave Evanston, IL 60208-4311 O: (847) 491-2219 M: (312) 887-1881 www.it.northwestern.edu _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.proofpoint.com/v2/url?u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=DwICAg&c=jf_iaSHvJObTbx-siA1ZOg&r=9dCEbNr27klWay2AcOfvOE1xq50K-CyRUu4qQx4HOlk&m=sUjgq9g2p2ncIpALAqAhOqt7blwynTJmgmFdYYik7MI&s=EFC3lNuf6koYPMPSWuYCNhwmIMUKKZ9mCQFhxVCYWLQ&e=
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
