Thanks for your insights. Alex, we did as you mentioned but after using mmimportfs there are a lot of errors on all commands having to do with filesystems:
GPFS: 6027-419 Failed to read a file system descriptor. There is an input or output error that occurs for: mmlsfs mmlsdisk mmdf obviously, fs won't mount. 2018-01-27 0:02 GMT-05:00 Alex Levin <[email protected]>: > Steve, > I've read that "mmfsctl suspend or suspend-write" should be executed, > but in real life it is impossible for DR scenario. > > We tested both cases, > the graceful one when the failover to another site is planned, > applications stopped and i/o suspended > and the case when there was no advanced notice about the disaster in the > primary site. > > Both worked and for the second case various loads were simulated including > heavy writes and reads/writes. > In disaster model as expected some data were lost (due to incomplete > writes, replication latency ... ), > but mmfsck was always able to repair and the applications databases > located on the affected filesystem were in an acceptable state. > > > it is possible that we were just lucky and the test case didn't cover all > possible scenarios. > > Harold, > In our case, it is Linux, not AIX, but it shouldn't matter > And our DR cluster is fully configured ( different IPs, hostnames and > cluster name ) and running without filesystems at all or with > a differently named filesystem. > > Before running mmimport , make sure that all expected LUNs are visible > and writable. > You can verify if the device is correct by reading first blocks of the > device ( for example dd if=/dev/NSD_LUN_device bs=1M count=16 | strings ) > > not sure where you are " moved the mmsdrfs" > you don't need to move/modify mmsdrfs file on the target ( disaster > recovery ) cluster > > Just copy the one from primary to /tmp or /var/tmp and try to run mmimportfs > fs_name -i copy_of_mmsdrfs /tmp/mmsdrfs > > > > Glen, as Harold has no IP connectivity between the clusters "mmfsctl > syncFSconfig" is not an option ... > > Thanks > --Alex > > > > > On Fri, Jan 26, 2018 at 4:04 PM, Steve Xiao <[email protected]> wrote: > >> When using this method of replication, you need to either issue "mmfsctl >> suspend or suspend-write" command before replication or setup a single >> consistency group for all LUNs. This is needed to ensure replica contain >> a consistent copy of GPFS data. >> >> Steve Y. Xiao >> >> [email protected] wrote on 01/26/2018 03:21:23 PM: >> >> > From: [email protected] >> > To: [email protected] >> > Date: 01/26/2018 03:21 PM >> > Subject: gpfsug-discuss Digest, Vol 72, Issue 69 >> > Sent by: [email protected] >> > >> > Send gpfsug-discuss mailing list submissions to >> > [email protected] >> > >> > To subscribe or unsubscribe via the World Wide Web, visit >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > or, via email, send a message with subject or body 'help' to >> > [email protected] >> > >> > You can reach the person managing the list at >> > [email protected] >> > >> > When replying, please edit your Subject line so it is more specific >> > than "Re: Contents of gpfsug-discuss digest..." >> > >> > >> > Today's Topics: >> > >> > 1. Re: storage-based replication for Spectrum Scale (Harold Morales) >> > 2. Re: storage-based replication for Spectrum Scale (Glen Corneau) >> > >> > >> > ---------------------------------------------------------------------- >> > >> > Message: 1 >> > Date: Fri, 26 Jan 2018 13:29:09 -0500 >> > From: Harold Morales <[email protected]> >> > To: gpfsug main discussion list <[email protected]> >> > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum >> > Scale >> > Message-ID: >> > <cagqnkwaajaufxyt2pzjmw47vuosmg4evuat8+sv49rrvkkq...@mail.gmail.com> >> > Content-Type: text/plain; charset="utf-8" >> >> > >> > Hi Alex, This set up seems close to what I am trying to achieve. >> > >> > With regards to this kind of replication: any prereqs need to be met in >> the >> > target environment for this to work? for example, should disk devices >> > naming on AIX be the same as in the source environment? when importing >> the >> > mmsdrfs file, how is Scale going to know which disks should it assign to >> > the cluster? by their hdisk name alone? >> > >> > Thanks again, >> > >> > >> > >> > 2018-01-24 2:30 GMT-05:00 Alex Levin <[email protected]>: >> > >> > > Hi, >> > > >> > > We are using a similar type of replication. >> > > I assume the site B is the cold site prepared for DR >> > > >> > > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. >> > > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX >> replication >> > > group to ensure consistency. >> > > >> > > The cluster name, IP addresses , hostnames of the cluster nodes are >> > > different on another site - it can be a pre-configured cluster without >> > > gpfs filesystems or with another filesystem. >> > > Same names and addresses shouldn't be a problem. >> > > >> > > Additionally to the replicated LUNs/NSDs you need to deliver copy >> > > of /var/mmfs/gen/mmsdrfs file from A to B site. >> > > There is no need to replicate it in real-time, only after the change >> of >> > > the cluster configuration. >> > > >> > > To activate site B - present replicated LUNs to the nodes in the DR >> > > cluster and run mmimportfs as "mmimportfs fs_name -i >> copy_of_mmsdrfs" >> > > >> > > Tested with multiples LUNs and filesystems on various workloads - >> seems >> > > to be working >> > > >> > > --Alex >> > > >> > > >> > > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales < >> [email protected]> >> > > wrote: >> > > >> > >> Thanks for answering. >> > >> >> > >> Essentially, the idea being explored is to replicate LUNs between >> > >> identical storage hardware (HP 3PAR volumesrein) on both sites. >> There is an >> > >> IP connection between storage boxes but not between servers on both >> sites, >> > >> there is a dark fiber connecting both sites. Here they dont want to >> explore >> > >> the idea of a scaled-based. >> > >> >> > >> >> > >> _______________________________________________ >> > >> gpfsug-discuss mailing list >> > >> gpfsug-discuss at spectrumscale.org >> > >> https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> >> > >> >> > > >> > > _______________________________________________ >> > > gpfsug-discuss mailing list >> > > gpfsug-discuss at spectrumscale.org >> > > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > > >> > > >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: <https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_1503995b_attachment-2D0001.html&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=SKdMmQae8uzHNWZq3vuRTp5UVwYFeeusLAxtbaposX0&e=> >> > >> > ------------------------------ >> > >> > Message: 2 >> > Date: Fri, 26 Jan 2018 14:21:15 -0600 >> > From: "Glen Corneau" <[email protected]> >> > To: gpfsug main discussion list <[email protected]> >> > Subject: Re: [gpfsug-discuss] storage-based replication for Spectrum >> > Scale >> > Message-ID: >> > <OFEAC48C60.2890CAEF-ON86258221.006F3F43-86258221. >> > [email protected]> >> > >> > Content-Type: text/plain; charset="us-ascii" >> > >> > Scale will walk across all discovered disks upon start time and attempt >> to >> > read the NSD identifiers from the disks. Once it finds them, it makes >> a >> > local map file that correlates the NSD id and the hdiskX identifier. >> The >> > names do not have to be the same as either the source cluster or even >> from >> > node-to-node. >> > >> > The main thing to keep in mind is to keep the file system definitions >> in >> > sync between the source and destination clusters. The "syncFSconfig" >> user >> > exit is the best way to do it because it's automatic. You generally >> > shouldn't be shuffling the mmsdrfs file between sites, that's what the >> > "mmfsctl syncFSconfig" does for you, on a per-file system basis. >> > >> > GPFS+AIX customers have been using this kind of storage replication for >> > over 10 years, it's business as usual. >> > >> > ------------------ >> > Glen Corneau >> > Power Systems >> > Washington Systems Center >> > [email protected] >> > >> > >> > >> > >> > >> > From: Harold Morales <[email protected]> >> > To: gpfsug main discussion list <[email protected]> >> > Date: 01/26/2018 12:30 PM >> > Subject: Re: [gpfsug-discuss] storage-based replication for >> > Spectrum Scale >> > Sent by: [email protected] >> > >> > >> > >> > Hi Alex, This set up seems close to what I am trying to achieve. >> > >> > With regards to this kind of replication: any prereqs need to be met in >> > the target environment for this to work? for example, should disk >> devices >> > naming on AIX be the same as in the source environment? when importing >> the >> > mmsdrfs file, how is Scale going to know which disks should it assign >> to >> > the cluster? by their hdisk name alone? >> > >> > Thanks again, >> > >> > >> > >> > 2018-01-24 2:30 GMT-05:00 Alex Levin <[email protected]>: >> > Hi, >> > >> > We are using a similar type of replication. >> > I assume the site B is the cold site prepared for DR >> > >> > The storage layer is EMC VMAX and the LUNs are replicated with SRDF. >> > All LUNs ( NSDs ) of the gpfs filesystem are in the same VMAX >> replication >> > group to ensure consistency. >> > >> > The cluster name, IP addresses , hostnames of the cluster nodes are >> > different on another site - it can be a pre-configured cluster without >> > gpfs filesystems or with another filesystem. >> > Same names and addresses shouldn't be a problem. >> > >> > Additionally to the replicated LUNs/NSDs you need to deliver copy >> > of /var/mmfs/gen/mmsdrfs file from A to B site. >> > There is no need to replicate it in real-time, only after the change of >> > the cluster configuration. >> > >> > To activate site B - present replicated LUNs to the nodes in the DR >> > cluster and run mmimportfs as "mmimportfs fs_name -i copy_of_mmsdrfs" >> > >> > Tested with multiples LUNs and filesystems on various workloads - >> seems >> > to be working >> > >> > --Alex >> > >> > >> > >> > On Wed, Jan 24, 2018 at 1:33 AM, Harold Morales <[email protected]> >> >> > wrote: >> > Thanks for answering. >> > >> > Essentially, the idea being explored is to replicate LUNs between >> > identical storage hardware (HP 3PAR volumesrein) on both sites. There >> is >> > an IP connection between storage boxes but not between servers on both >> > sites, there is a dark fiber connecting both sites. Here they dont want >> to >> > explore the idea of a scaled-based. >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=d- >> > vphLEe_UlGazP6RdYAyyAA3Qv5S9IRVNuO1i9vjJc&m=VbfWaftYSVjx8fMb >> 2vHGBi6XUhDJOsKf_dKOX3J8s1A&s=p2mGvPrlPLO1oyEh- >> > GeVJiVS49opBwwCFs-FKQrQ7rc&e= >> > >> > >> > >> > >> > >> > -------------- next part -------------- >> > An HTML attachment was scrubbed... >> > URL: <https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_e291af63_attachment.html&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=bYnf-7v0CxYUkGth-QaVeUQdIlG8f1Gro-hwOxok7Qw&e=> >> > -------------- next part -------------- >> > A non-text attachment was scrubbed... >> > Name: not available >> > Type: image/jpeg >> > Size: 26117 bytes >> > Desc: not available >> > URL: <https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_pipermail_gpfsug-2Ddiscuss_attachments >> _20180126_e291af63_attachment.jpe&d=DwICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=jYdnqhQBlnpf58oxunzBcTs9XdcbeOtLDQdgnASidDA&e=> >> > >> > ------------------------------ >> > >> > _______________________________________________ >> > gpfsug-discuss mailing list >> > gpfsug-discuss at spectrumscale.org >> > https://urldefense.proofpoint.com/v2/url? >> > u=http-3A__gpfsug.org_mailman_listinfo_gpfsug-2Ddiscuss&d=Dw >> ICAg&c=jf_iaSHvJObTbx- >> > siA1ZOg&r=ck4PYlaRFvCcNKlfHMPhoA&m=ddkAIwRPQmQKBLLh6nBzhdt- >> > OKoJwOucQ8oaQet8mkE&s=Emkr3VzCYTecA6E61hAk1AeB6ka34dGAYip6fGaKuwU&e= >> > >> > >> > End of gpfsug-discuss Digest, Vol 72, Issue 69 >> > ********************************************** >> > >> >> >> _______________________________________________ >> gpfsug-discuss mailing list >> gpfsug-discuss at spectrumscale.org >> http://gpfsug.org/mailman/listinfo/gpfsug-discuss >> >> > > _______________________________________________ > gpfsug-discuss mailing list > gpfsug-discuss at spectrumscale.org > http://gpfsug.org/mailman/listinfo/gpfsug-discuss > >
_______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
