Re: nfstimeout on server ISILON storage
Quick question - what's the data link protocol (Ethernet, IB, etc.) and link rate that you're using? On Tue, Sep 04, 2018 at 02:05:33PM -0400, Zoltan Forray wrote: > We are still fighting issues with ISILON storage. Our current issue is with > NFS timeouts for the storage a server is using. We see message like these > in the server /var/log > > Sep 4 13:21:49 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu > not responding, still trying > Sep 4 13:21:49 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu > not responding, still trying > Sep 4 13:21:49 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu > not responding, still trying > Sep 4 13:21:49 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu > not responding, still trying > Sep 4 13:22:14 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu > not responding, still trying > Sep 4 13:22:15 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu > not responding, still trying > Sep 4 13:22:16 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu OK > Sep 4 13:22:16 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu OK > Sep 4 13:22:16 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu OK > > OS folks say the NFS mount is setup as IBM recommends in various documents. > So they asked us to implement the nfstimeout option from this document ( > https://www.ibm.com/support/knowledgecenter/en/SSGSG7_7.1.0/com.ibm.itsm.client.doc/r_opt_nfstimeout.html). > Yes I realize it is primarily for a client backup of an NFS mount, but the > statement: > > Supported Clients This option is for all UNIX and Linux clients. *The > server can also define this option*. > > throws us - kind-of implying I can use this from the server perspective? > But I can't find any documentation to support using it from the server. > > For you Linux guru's - this is what the mount says: > > hhisilonnfs23.rams.adp.vcu.edu:/ifs/NFS/TSM on /tsmnfs type nfs > (rw,relatime,sync,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.19.12,mountvers=3,mountport=300,mountproto=udp,local_lock=none,addr=192.168.19.12) > > Any thoughts? Suggestion? Are we simply expecting too much from NFS? > > My OS person also asks why ISP is so slow to write to NFS? When they did a > test copy of a large file to the NFS mount, they were getting upwards of 8G/s > vs 1.5-3G/s when TSM/ISP writes to it (via EMC monitoring tools). > > -- > *Zoltan Forray* > Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator > Xymon Monitor Administrator > VMware Administrator > Virginia Commonwealth University > UCC/Office of Technology Services > www.ucc.vcu.edu > zfor...@vcu.edu - 804-828-4807 > Don't be a phishing victim - VCU and other reputable organizations will > never use email to request that you reply with your password, social > security number or confidential personal information. For more details > visit http://phishing.vcu.edu/ -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine
Re: MariaDB backups using modern MariaDB methods and high performance restores
We do this with PostgreSQL - take a snapshot and mount it with a preschedulecmd, run an incremental backup on the snapshot, and then unmount and destroy it with a postschedulecmd. The complication for Stefan would be that he would have to restore the entire snapshot in order to have a usable database, which might take too long for his restore objective. PostgreSQL also supports continuous backups of the WAL (journal)[1] which allow for more fine-grained point-in-time restores, but I'm not sure if MySQL/MariaDB have an equivalent solution. [1] https://www.postgresql.org/docs/current/static/continuous-archiving.html On Tue, Sep 04, 2018 at 05:29:18PM +0200, Remco Post wrote: > Just a thought. This is a linux server, right? So you have linux LVM. I think > it should be possible to make a consistent snapshot using MariaDB and LVM. > Then you can backup the snapshot and in case of a disaster restore that. Now, > I’ve never attempted this, and I don’t know how to do it, but it seems to be > the only viable acceptable solution. > > > Op 4 sep. 2018, om 09:49 heeft Stefan Folkerts > > het volgende geschreven: > > > > Hi all, > > > > I'm currently looking for the best backup option for a large and extremely > > transaction-heavy MariaDB database environment. I'm talking about up to > > 100.000.000 transactions a year (payment industry). > > > > It needs to connect to Spectrum Protect to store it's database data, it is > > acceptable if this is a two stage backup solution but not for restores due > > to the duration of a two stage restore. > > > > We have looked at one option but that used the traditional mysqldump > > methods that have proven to be unusable for this customer because the > > restore is up to 8 times slower than the backup and during the backup all > > transactions are stored to be committed later, this is an issue with this > > many transactions. > > > > zmanda seems to offer newer backup mechanics for MariaDB, i'm wondering if > > anybody used this with Spectrum Protect that can share some experiences > > with this solution. > > Also, any other ideas for solutions that are officially supporting Spectrum > > Protect would be great. > > > > Thanks in advance, > > Stefan > > -- > > Met vriendelijke groeten/Kind Regards, > > Remco Post > r.p...@plcs.nl > +31 6 248 21 622 -- -- Skylar Thompson (skyl...@u.washington.edu) -- Genome Sciences Department, System Administrator -- Foege Building S046, (206)-685-7354 -- University of Washington School of Medicine
nfstimeout on server ISILON storage
We are still fighting issues with ISILON storage. Our current issue is with NFS timeouts for the storage a server is using. We see message like these in the server /var/log Sep 4 13:21:49 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu not responding, still trying Sep 4 13:21:49 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu not responding, still trying Sep 4 13:21:49 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu not responding, still trying Sep 4 13:21:49 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu not responding, still trying Sep 4 13:22:14 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu not responding, still trying Sep 4 13:22:15 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu not responding, still trying Sep 4 13:22:16 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu OK Sep 4 13:22:16 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu OK Sep 4 13:22:16 earth kernel: nfs: server hhisilonnfs23.rams.adp.vcu.edu OK OS folks say the NFS mount is setup as IBM recommends in various documents. So they asked us to implement the nfstimeout option from this document ( https://www.ibm.com/support/knowledgecenter/en/SSGSG7_7.1.0/com.ibm.itsm.client.doc/r_opt_nfstimeout.html). Yes I realize it is primarily for a client backup of an NFS mount, but the statement: Supported Clients This option is for all UNIX and Linux clients. *The server can also define this option*. throws us - kind-of implying I can use this from the server perspective? But I can't find any documentation to support using it from the server. For you Linux guru's - this is what the mount says: hhisilonnfs23.rams.adp.vcu.edu:/ifs/NFS/TSM on /tsmnfs type nfs (rw,relatime,sync,vers=3,rsize=131072,wsize=524288,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=192.168.19.12,mountvers=3,mountport=300,mountproto=udp,local_lock=none,addr=192.168.19.12) Any thoughts? Suggestion? Are we simply expecting too much from NFS? My OS person also asks why ISP is so slow to write to NFS? When they did a test copy of a large file to the NFS mount, they were getting upwards of 8G/s vs 1.5-3G/s when TSM/ISP writes to it (via EMC monitoring tools). -- *Zoltan Forray* Spectrum Protect (p.k.a. TSM) Software & Hardware Administrator Xymon Monitor Administrator VMware Administrator Virginia Commonwealth University UCC/Office of Technology Services www.ucc.vcu.edu zfor...@vcu.edu - 804-828-4807 Don't be a phishing victim - VCU and other reputable organizations will never use email to request that you reply with your password, social security number or confidential personal information. For more details visit http://phishing.vcu.edu/
Re: Download SP VE 8.1.6
Hi Christian, the plannes availability date for version 8.1.6 is the 21th of September 2018. Before that date you can‘t download that version from PA. Regards Uwe > Am 04.09.2018 um 16:31 schrieb Hans Christian Riksheim : > > We need that version to support vSphere 6.7 but on PA only 8.12 and 8.1.4 > exists. No one at PA support or IBM support can help so I address the > community. > > (I have navigated ibm.com for half a day and feel 5 years older). > > Hans Chr.
Re: MariaDB backups using modern MariaDB methods and high performance restores
Just a thought. This is a linux server, right? So you have linux LVM. I think it should be possible to make a consistent snapshot using MariaDB and LVM. Then you can backup the snapshot and in case of a disaster restore that. Now, I’ve never attempted this, and I don’t know how to do it, but it seems to be the only viable acceptable solution. > Op 4 sep. 2018, om 09:49 heeft Stefan Folkerts > het volgende geschreven: > > Hi all, > > I'm currently looking for the best backup option for a large and extremely > transaction-heavy MariaDB database environment. I'm talking about up to > 100.000.000 transactions a year (payment industry). > > It needs to connect to Spectrum Protect to store it's database data, it is > acceptable if this is a two stage backup solution but not for restores due > to the duration of a two stage restore. > > We have looked at one option but that used the traditional mysqldump > methods that have proven to be unusable for this customer because the > restore is up to 8 times slower than the backup and during the backup all > transactions are stored to be committed later, this is an issue with this > many transactions. > > zmanda seems to offer newer backup mechanics for MariaDB, i'm wondering if > anybody used this with Spectrum Protect that can share some experiences > with this solution. > Also, any other ideas for solutions that are officially supporting Spectrum > Protect would be great. > > Thanks in advance, > Stefan -- Met vriendelijke groeten/Kind Regards, Remco Post r.p...@plcs.nl +31 6 248 21 622
Re: Download SP VE 8.1.6
Thanks, I overlooked that info in the client requirements. Hans Chr. On Tue, Sep 4, 2018 at 4:39 PM Del Hoobler wrote: > HI Hans Chr. > > As shown in the announcement letter, Spectrum Protect 8.1.6 will be > available on September 21. > > > https://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/6/897/ENUS218-316/index.html_locale=ja > > > Del > > > > "ADSM: Dist Stor Manager" wrote on 09/04/2018 > 10:31:57 AM: > > > From: Hans Christian Riksheim > > To: ADSM-L@VM.MARIST.EDU > > Date: 09/04/2018 10:33 AM > > Subject: Download SP VE 8.1.6 > > Sent by: "ADSM: Dist Stor Manager" > > > > We need that version to support vSphere 6.7 but on PA only 8.12 and > 8.1.4 > > exists. No one at PA support or IBM support can help so I address the > > community. > > > > (I have navigated ibm.com for half a day and feel 5 years older). > > > > Hans Chr. > > >
Re: Download SP VE 8.1.6
HI Hans Chr. As shown in the announcement letter, Spectrum Protect 8.1.6 will be available on September 21. https://www-01.ibm.com/common/ssi/ShowDoc.wss?docURL=/common/ssi/rep_ca/6/897/ENUS218-316/index.html_locale=ja Del "ADSM: Dist Stor Manager" wrote on 09/04/2018 10:31:57 AM: > From: Hans Christian Riksheim > To: ADSM-L@VM.MARIST.EDU > Date: 09/04/2018 10:33 AM > Subject: Download SP VE 8.1.6 > Sent by: "ADSM: Dist Stor Manager" > > We need that version to support vSphere 6.7 but on PA only 8.12 and 8.1.4 > exists. No one at PA support or IBM support can help so I address the > community. > > (I have navigated ibm.com for half a day and feel 5 years older). > > Hans Chr. >
Download SP VE 8.1.6
We need that version to support vSphere 6.7 but on PA only 8.12 and 8.1.4 exists. No one at PA support or IBM support can help so I address the community. (I have navigated ibm.com for half a day and feel 5 years older). Hans Chr.
Re: MariaDB backups using modern MariaDB methods and high performance restores
> 100.000.000 transactions/year is a little over 3 transactions per second. With transaction I mean payment, not a single database transaction, I don't know how many database transactions a single payment creates. What I do know is that they are running a high database load, especially during peak hours. I'm sure there are databases doing many many more transactions than this but it's enough to rule out the mysqldump backup solution for them knowing what they know about how it operates. On Tue, Sep 4, 2018 at 1:12 PM Loon, Eric van (ITOPT3) - KLM < eric-van.l...@klm.com> wrote: > Hi Stefan, > Just out of curiosity: 100.000.000 transactions/year is a little over 3 > transactions per second. I would not call that heavily used, are these > figures correct? > Kind regards, > Eric van Loon > Air France/KLM Storage Engineering > > -Original Message- > From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of > Stefan Folkerts > Sent: dinsdag 4 september 2018 11:29 > To: ADSM-L@VM.MARIST.EDU > Subject: Re: MariaDB backups using modern MariaDB methods and high > performance restores > > Yes we did, Repostor uses (at least the version we tested) mysql tools to > backup and restore the database, the backup-impact and restore performance > of the tool doesn't suite this customer environment. > Now, let me be clear, I don't want to be negative about Repostor data > protector because it worked great and is very easy to setup and use, it's > just that the very high amount of transactions that are done at this site > eliminate this tool as a solution. > > I will for sure advise Repostor data protector solutions when the demands > are not so extreme as they are at this site. > > > > > On Tue, Sep 4, 2018 at 11:20 AM Uwe Schreiber > > wrote: > > > Hi Stefan, > > > > did you have a look on Repostor DATA Protector? > > > > Regards Uwe > > > > > Am 04.09.2018 um 09:49 schrieb Stefan Folkerts < > > stefan.folke...@gmail.com>: > > > > > > Hi all, > > > > > > I'm currently looking for the best backup option for a large and > > extremely > > > transaction-heavy MariaDB database environment. I'm talking about up > > > to > > > 100.000.000 transactions a year (payment industry). > > > > > > It needs to connect to Spectrum Protect to store it's database data, > > > it > > is > > > acceptable if this is a two stage backup solution but not for > > > restores > > due > > > to the duration of a two stage restore. > > > > > > We have looked at one option but that used the traditional mysqldump > > > methods that have proven to be unusable for this customer because > > > the restore is up to 8 times slower than the backup and during the > > > backup all transactions are stored to be committed later, this is an > > > issue with this many transactions. > > > > > > zmanda seems to offer newer backup mechanics for MariaDB, i'm > > > wondering > > if > > > anybody used this with Spectrum Protect that can share some > > > experiences with this solution. > > > Also, any other ideas for solutions that are officially supporting > > Spectrum > > > Protect would be great. > > > > > > Thanks in advance, > > > Stefan > > > > For information, services and offers, please visit our web site: > http://www.klm.com. This e-mail and any attachment may contain > confidential and privileged material intended for the addressee only. If > you are not the addressee, you are notified that no part of the e-mail or > any attachment may be disclosed, copied or distributed, and that any other > action related to this e-mail or attachment is strictly prohibited, and may > be unlawful. If you have received this e-mail by error, please notify the > sender immediately by return e-mail, and delete this message. > > Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its > employees shall not be liable for the incorrect or incomplete transmission > of this e-mail or any attachments, nor responsible for any delay in receipt. > Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch > Airlines) is registered in Amstelveen, The Netherlands, with registered > number 33014286 > >
Re: MariaDB backups using modern MariaDB methods and high performance restores
Hi Stefan, Just out of curiosity: 100.000.000 transactions/year is a little over 3 transactions per second. I would not call that heavily used, are these figures correct? Kind regards, Eric van Loon Air France/KLM Storage Engineering -Original Message- From: ADSM: Dist Stor Manager [mailto:ADSM-L@VM.MARIST.EDU] On Behalf Of Stefan Folkerts Sent: dinsdag 4 september 2018 11:29 To: ADSM-L@VM.MARIST.EDU Subject: Re: MariaDB backups using modern MariaDB methods and high performance restores Yes we did, Repostor uses (at least the version we tested) mysql tools to backup and restore the database, the backup-impact and restore performance of the tool doesn't suite this customer environment. Now, let me be clear, I don't want to be negative about Repostor data protector because it worked great and is very easy to setup and use, it's just that the very high amount of transactions that are done at this site eliminate this tool as a solution. I will for sure advise Repostor data protector solutions when the demands are not so extreme as they are at this site. On Tue, Sep 4, 2018 at 11:20 AM Uwe Schreiber wrote: > Hi Stefan, > > did you have a look on Repostor DATA Protector? > > Regards Uwe > > > Am 04.09.2018 um 09:49 schrieb Stefan Folkerts < > stefan.folke...@gmail.com>: > > > > Hi all, > > > > I'm currently looking for the best backup option for a large and > extremely > > transaction-heavy MariaDB database environment. I'm talking about up > > to > > 100.000.000 transactions a year (payment industry). > > > > It needs to connect to Spectrum Protect to store it's database data, > > it > is > > acceptable if this is a two stage backup solution but not for > > restores > due > > to the duration of a two stage restore. > > > > We have looked at one option but that used the traditional mysqldump > > methods that have proven to be unusable for this customer because > > the restore is up to 8 times slower than the backup and during the > > backup all transactions are stored to be committed later, this is an > > issue with this many transactions. > > > > zmanda seems to offer newer backup mechanics for MariaDB, i'm > > wondering > if > > anybody used this with Spectrum Protect that can share some > > experiences with this solution. > > Also, any other ideas for solutions that are officially supporting > Spectrum > > Protect would be great. > > > > Thanks in advance, > > Stefan > For information, services and offers, please visit our web site: http://www.klm.com. This e-mail and any attachment may contain confidential and privileged material intended for the addressee only. If you are not the addressee, you are notified that no part of the e-mail or any attachment may be disclosed, copied or distributed, and that any other action related to this e-mail or attachment is strictly prohibited, and may be unlawful. If you have received this e-mail by error, please notify the sender immediately by return e-mail, and delete this message. Koninklijke Luchtvaart Maatschappij NV (KLM), its subsidiaries and/or its employees shall not be liable for the incorrect or incomplete transmission of this e-mail or any attachments, nor responsible for any delay in receipt. Koninklijke Luchtvaart Maatschappij N.V. (also known as KLM Royal Dutch Airlines) is registered in Amstelveen, The Netherlands, with registered number 33014286
Re: MariaDB backups using modern MariaDB methods and high performance restores
Yes we did, Repostor uses (at least the version we tested) mysql tools to backup and restore the database, the backup-impact and restore performance of the tool doesn't suite this customer environment. Now, let me be clear, I don't want to be negative about Repostor data protector because it worked great and is very easy to setup and use, it's just that the very high amount of transactions that are done at this site eliminate this tool as a solution. I will for sure advise Repostor data protector solutions when the demands are not so extreme as they are at this site. On Tue, Sep 4, 2018 at 11:20 AM Uwe Schreiber wrote: > Hi Stefan, > > did you have a look on Repostor DATA Protector? > > Regards Uwe > > > Am 04.09.2018 um 09:49 schrieb Stefan Folkerts < > stefan.folke...@gmail.com>: > > > > Hi all, > > > > I'm currently looking for the best backup option for a large and > extremely > > transaction-heavy MariaDB database environment. I'm talking about up to > > 100.000.000 transactions a year (payment industry). > > > > It needs to connect to Spectrum Protect to store it's database data, it > is > > acceptable if this is a two stage backup solution but not for restores > due > > to the duration of a two stage restore. > > > > We have looked at one option but that used the traditional mysqldump > > methods that have proven to be unusable for this customer because the > > restore is up to 8 times slower than the backup and during the backup all > > transactions are stored to be committed later, this is an issue with this > > many transactions. > > > > zmanda seems to offer newer backup mechanics for MariaDB, i'm wondering > if > > anybody used this with Spectrum Protect that can share some experiences > > with this solution. > > Also, any other ideas for solutions that are officially supporting > Spectrum > > Protect would be great. > > > > Thanks in advance, > > Stefan >
Re: MariaDB backups using modern MariaDB methods and high performance restores
Hi Stefan, did you have a look on Repostor DATA Protector? Regards Uwe > Am 04.09.2018 um 09:49 schrieb Stefan Folkerts : > > Hi all, > > I'm currently looking for the best backup option for a large and extremely > transaction-heavy MariaDB database environment. I'm talking about up to > 100.000.000 transactions a year (payment industry). > > It needs to connect to Spectrum Protect to store it's database data, it is > acceptable if this is a two stage backup solution but not for restores due > to the duration of a two stage restore. > > We have looked at one option but that used the traditional mysqldump > methods that have proven to be unusable for this customer because the > restore is up to 8 times slower than the backup and during the backup all > transactions are stored to be committed later, this is an issue with this > many transactions. > > zmanda seems to offer newer backup mechanics for MariaDB, i'm wondering if > anybody used this with Spectrum Protect that can share some experiences > with this solution. > Also, any other ideas for solutions that are officially supporting Spectrum > Protect would be great. > > Thanks in advance, > Stefan
MariaDB backups using modern MariaDB methods and high performance restores
Hi all, I'm currently looking for the best backup option for a large and extremely transaction-heavy MariaDB database environment. I'm talking about up to 100.000.000 transactions a year (payment industry). It needs to connect to Spectrum Protect to store it's database data, it is acceptable if this is a two stage backup solution but not for restores due to the duration of a two stage restore. We have looked at one option but that used the traditional mysqldump methods that have proven to be unusable for this customer because the restore is up to 8 times slower than the backup and during the backup all transactions are stored to be committed later, this is an issue with this many transactions. zmanda seems to offer newer backup mechanics for MariaDB, i'm wondering if anybody used this with Spectrum Protect that can share some experiences with this solution. Also, any other ideas for solutions that are officially supporting Spectrum Protect would be great. Thanks in advance, Stefan
