RE: DB Communication Link Failure
Hello All, My DB is now up and running. i just executed the following commands : mysql > alter table cloud.event engine = innoDB mysql> check table cloud.event The check command give me an "OK", so i restarted the cloudstack-management. Now, all is good. Thanks you all for your help ! Nicolas Bouige DIMSI cloud.dimsi.fr<http://www.cloud.dimsi.fr> 4, avenue Laurent Cely Tour d’Asnière – 92600 Asnière sur Seine T/ +33 (0)6 28 98 53 40 De : Nicolas Bouige Envoyé : lundi 18 juin 2018 14:38:28 À : users@cloudstack.apache.org Objet : RE: DB Communication Link Failure Hi Leandro, i will take a look to this tool, thanks for the information ! Nicolas Bouige DIMSI cloud.dimsi.fr<http://www.cloud.dimsi.fr> 4, avenue Laurent Cely Tour d’Asnière – 92600 Asnière sur Seine T/ +33 (0)6 28 98 53 40 De : Leandro Mendes Envoyé : lundi 18 juin 2018 14:32:17 À : users@cloudstack.apache.org Objet : Re: DB Communication Link Failure Nicolas, I did not follow the thread properly, but as i saw this line about no backup and mysql corrupted files it got my attention. I had a similar problem once and i had used the Percona toolkit. It will not fix your DB but dump the data so you can reimport it Good luck. On Mon, Jun 18, 2018 at 2:18 PM Nicolas Bouige wrote: > Stephan, > > > Thanks for your help, > > Unfortunately, the --auto-repair switch doesnt work as it's not support > by the storage engine...and yes i dont have any backup without the > corrupted tables. > > Nicolas Bouige > DIMSI > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > 4, avenue Laurent Cely > Tour d’Asnière – 92600 Asnière sur Seine > T/ +33 (0)6 28 98 53 40 > > > > De : Stephan Seitz > Envoyé : lundi 18 juin 2018 14:07:50 > À : users@cloudstack.apache.org > Objet : Re: DB Communication Link Failure > > Hi! > > there's also a --auto-repair switch that could be added to mysqlcheck > --all-databases. > > But to be honest, you can't guarantee the content will match. So > references to > other tablefields might not match afterwards (well, i expect these > references don't match right now either) > > As far as your resultset shows, the corruption "only" happened to the > index-space so your data > "could" be fine. > > Normally, I'ld suggest to revert to a backup, but as this question has > been around for a few days here, > I assume your last uncorrupted backup could be far too old. > > > > Am Montag, den 18.06.2018, 11:56 + schrieb Nicolas Bouige: > > Hi Stephan, > > > > > > thanks for the command, i could spot which tables is corrupted : > > > > > > cloud.event > > Warning : InnoDB: Index 'i_event__created' contains 548 entries, should > be 542. > > Warning : InnoDB: Index 'i_event__user_id' contains 547 entries, should > be 542. > > Warning : InnoDB: Index 'i_event__account_id' contains 547 entries, > should be 542. > > Warning : InnoDB: Index 'i_event__level_id' contains 547 entries, > should be 542. > > Warning : InnoDB: Index 'i_event__type_id' contains 548 entries, should > be 542. > > error: Corrupt > > > > Now, i supposed i have to delete the entries > > > > Nicolas Bouige > > DIMSI > > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > > 4, avenue Laurent Cely > > Tour d’Asnière – 92600 Asnière sur Seine > > T/ +33 (0)6 28 98 53 40 > > > > > > > > De : Stephan Seitz > > Envoyé : lundi 18 juin 2018 13:30:43 > > À : users@cloudstack.apache.org > > Objet : Re: DB Communication Link Failure > > > > Hi! > > > > This sound's like a corrupted database table. It's not that unusual > mysqld are > > restarting after a query reqeuests values from a corrupted table space. > That > > behaviour subsequently results in aborted connections. > > > > I'ld double check database consistency. The easist way to check against > > (at least physical) corruption should be mysqlcheckk --all-databases > > > > cheers, > > > > Stephan > > > > > > Am Montag, den 18.06.2018, 12:47 +0200 schrieb Rafael Weingärtner: > > > > > > Your timeout configuration seems fine. There must be something wrong in > > > your network. Or maybe in your MySQL service; as you said, it is > restarting > > > when you run commands against it. Therefore, it might be better to > > > e
RE: DB Communication Link Failure
Hi Leandro, i will take a look to this tool, thanks for the information ! Nicolas Bouige DIMSI cloud.dimsi.fr<http://www.cloud.dimsi.fr> 4, avenue Laurent Cely Tour d’Asnière – 92600 Asnière sur Seine T/ +33 (0)6 28 98 53 40 De : Leandro Mendes Envoyé : lundi 18 juin 2018 14:32:17 À : users@cloudstack.apache.org Objet : Re: DB Communication Link Failure Nicolas, I did not follow the thread properly, but as i saw this line about no backup and mysql corrupted files it got my attention. I had a similar problem once and i had used the Percona toolkit. It will not fix your DB but dump the data so you can reimport it Good luck. On Mon, Jun 18, 2018 at 2:18 PM Nicolas Bouige wrote: > Stephan, > > > Thanks for your help, > > Unfortunately, the --auto-repair switch doesnt work as it's not support > by the storage engine...and yes i dont have any backup without the > corrupted tables. > > Nicolas Bouige > DIMSI > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > 4, avenue Laurent Cely > Tour d’Asnière – 92600 Asnière sur Seine > T/ +33 (0)6 28 98 53 40 > > > > De : Stephan Seitz > Envoyé : lundi 18 juin 2018 14:07:50 > À : users@cloudstack.apache.org > Objet : Re: DB Communication Link Failure > > Hi! > > there's also a --auto-repair switch that could be added to mysqlcheck > --all-databases. > > But to be honest, you can't guarantee the content will match. So > references to > other tablefields might not match afterwards (well, i expect these > references don't match right now either) > > As far as your resultset shows, the corruption "only" happened to the > index-space so your data > "could" be fine. > > Normally, I'ld suggest to revert to a backup, but as this question has > been around for a few days here, > I assume your last uncorrupted backup could be far too old. > > > > Am Montag, den 18.06.2018, 11:56 + schrieb Nicolas Bouige: > > Hi Stephan, > > > > > > thanks for the command, i could spot which tables is corrupted : > > > > > > cloud.event > > Warning : InnoDB: Index 'i_event__created' contains 548 entries, should > be 542. > > Warning : InnoDB: Index 'i_event__user_id' contains 547 entries, should > be 542. > > Warning : InnoDB: Index 'i_event__account_id' contains 547 entries, > should be 542. > > Warning : InnoDB: Index 'i_event__level_id' contains 547 entries, > should be 542. > > Warning : InnoDB: Index 'i_event__type_id' contains 548 entries, should > be 542. > > error: Corrupt > > > > Now, i supposed i have to delete the entries > > > > Nicolas Bouige > > DIMSI > > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > > 4, avenue Laurent Cely > > Tour d’Asnière – 92600 Asnière sur Seine > > T/ +33 (0)6 28 98 53 40 > > > > > > > > De : Stephan Seitz > > Envoyé : lundi 18 juin 2018 13:30:43 > > À : users@cloudstack.apache.org > > Objet : Re: DB Communication Link Failure > > > > Hi! > > > > This sound's like a corrupted database table. It's not that unusual > mysqld are > > restarting after a query reqeuests values from a corrupted table space. > That > > behaviour subsequently results in aborted connections. > > > > I'ld double check database consistency. The easist way to check against > > (at least physical) corruption should be mysqlcheckk --all-databases > > > > cheers, > > > > Stephan > > > > > > Am Montag, den 18.06.2018, 12:47 +0200 schrieb Rafael Weingärtner: > > > > > > Your timeout configuration seems fine. There must be something wrong in > > > your network. Or maybe in your MySQL service; as you said, it is > restarting > > > when you run commands against it. Therefore, it might be better to > > > eliminate these issues first. > > > > > > On Mon, Jun 18, 2018 at 11:56 AM, Nicolas Bouige > wrote: > > > > > > > > > > > > > > > Hello Dag, > > > > > > > > Im not trying to do a multi-master setup, just recover my DB :/ > > > > I have installed a second node and connect it to the DB and it's not > > > > possible to connect to the database server automatically (but > manually > > > > yes..) > > > > On the first node at each sql query sent, the service mysql restart > on db > > > > server... > > > > > > > >
Re: DB Communication Link Failure
Nicolas, I did not follow the thread properly, but as i saw this line about no backup and mysql corrupted files it got my attention. I had a similar problem once and i had used the Percona toolkit. It will not fix your DB but dump the data so you can reimport it Good luck. On Mon, Jun 18, 2018 at 2:18 PM Nicolas Bouige wrote: > Stephan, > > > Thanks for your help, > > Unfortunately, the --auto-repair switch doesnt work as it's not support > by the storage engine...and yes i dont have any backup without the > corrupted tables. > > Nicolas Bouige > DIMSI > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > 4, avenue Laurent Cely > Tour d’Asnière – 92600 Asnière sur Seine > T/ +33 (0)6 28 98 53 40 > > > > De : Stephan Seitz > Envoyé : lundi 18 juin 2018 14:07:50 > À : users@cloudstack.apache.org > Objet : Re: DB Communication Link Failure > > Hi! > > there's also a --auto-repair switch that could be added to mysqlcheck > --all-databases. > > But to be honest, you can't guarantee the content will match. So > references to > other tablefields might not match afterwards (well, i expect these > references don't match right now either) > > As far as your resultset shows, the corruption "only" happened to the > index-space so your data > "could" be fine. > > Normally, I'ld suggest to revert to a backup, but as this question has > been around for a few days here, > I assume your last uncorrupted backup could be far too old. > > > > Am Montag, den 18.06.2018, 11:56 + schrieb Nicolas Bouige: > > Hi Stephan, > > > > > > thanks for the command, i could spot which tables is corrupted : > > > > > > cloud.event > > Warning : InnoDB: Index 'i_event__created' contains 548 entries, should > be 542. > > Warning : InnoDB: Index 'i_event__user_id' contains 547 entries, should > be 542. > > Warning : InnoDB: Index 'i_event__account_id' contains 547 entries, > should be 542. > > Warning : InnoDB: Index 'i_event__level_id' contains 547 entries, > should be 542. > > Warning : InnoDB: Index 'i_event__type_id' contains 548 entries, should > be 542. > > error: Corrupt > > > > Now, i supposed i have to delete the entries > > > > Nicolas Bouige > > DIMSI > > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > > 4, avenue Laurent Cely > > Tour d’Asnière – 92600 Asnière sur Seine > > T/ +33 (0)6 28 98 53 40 > > > > > > > > De : Stephan Seitz > > Envoyé : lundi 18 juin 2018 13:30:43 > > À : users@cloudstack.apache.org > > Objet : Re: DB Communication Link Failure > > > > Hi! > > > > This sound's like a corrupted database table. It's not that unusual > mysqld are > > restarting after a query reqeuests values from a corrupted table space. > That > > behaviour subsequently results in aborted connections. > > > > I'ld double check database consistency. The easist way to check against > > (at least physical) corruption should be mysqlcheckk --all-databases > > > > cheers, > > > > Stephan > > > > > > Am Montag, den 18.06.2018, 12:47 +0200 schrieb Rafael Weingärtner: > > > > > > Your timeout configuration seems fine. There must be something wrong in > > > your network. Or maybe in your MySQL service; as you said, it is > restarting > > > when you run commands against it. Therefore, it might be better to > > > eliminate these issues first. > > > > > > On Mon, Jun 18, 2018 at 11:56 AM, Nicolas Bouige > wrote: > > > > > > > > > > > > > > > Hello Dag, > > > > > > > > Im not trying to do a multi-master setup, just recover my DB :/ > > > > I have installed a second node and connect it to the DB and it's not > > > > possible to connect to the database server automatically (but > manually > > > > yes..) > > > > On the first node at each sql query sent, the service mysql restart > on db > > > > server... > > > > > > > > > > > > @Rafael, the timeout value is 28800 > > > > > > > > > > > > mysql> SHOW VARIABLES LIKE 'wait_timeout'; > > > > +---+---+ > > > > > > > > > > > > > > > Variable_name | Value | > > > > +---+---+ > > > > >
RE: DB Communication Link Failure
Stephan, Thanks for your help, Unfortunately, the --auto-repair switch doesnt work as it's not support by the storage engine...and yes i dont have any backup without the corrupted tables. Nicolas Bouige DIMSI cloud.dimsi.fr<http://www.cloud.dimsi.fr> 4, avenue Laurent Cely Tour d’Asnière – 92600 Asnière sur Seine T/ +33 (0)6 28 98 53 40 De : Stephan Seitz Envoyé : lundi 18 juin 2018 14:07:50 À : users@cloudstack.apache.org Objet : Re: DB Communication Link Failure Hi! there's also a --auto-repair switch that could be added to mysqlcheck --all-databases. But to be honest, you can't guarantee the content will match. So references to other tablefields might not match afterwards (well, i expect these references don't match right now either) As far as your resultset shows, the corruption "only" happened to the index-space so your data "could" be fine. Normally, I'ld suggest to revert to a backup, but as this question has been around for a few days here, I assume your last uncorrupted backup could be far too old. Am Montag, den 18.06.2018, 11:56 + schrieb Nicolas Bouige: > Hi Stephan, > > > thanks for the command, i could spot which tables is corrupted : > > > cloud.event > Warning : InnoDB: Index 'i_event__created' contains 548 entries, should be > 542. > Warning : InnoDB: Index 'i_event__user_id' contains 547 entries, should be > 542. > Warning : InnoDB: Index 'i_event__account_id' contains 547 entries, should > be 542. > Warning : InnoDB: Index 'i_event__level_id' contains 547 entries, should be > 542. > Warning : InnoDB: Index 'i_event__type_id' contains 548 entries, should be > 542. > error: Corrupt > > Now, i supposed i have to delete the entries > > Nicolas Bouige > DIMSI > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > 4, avenue Laurent Cely > Tour d’Asnière – 92600 Asnière sur Seine > T/ +33 (0)6 28 98 53 40 > > > > De : Stephan Seitz > Envoyé : lundi 18 juin 2018 13:30:43 > À : users@cloudstack.apache.org > Objet : Re: DB Communication Link Failure > > Hi! > > This sound's like a corrupted database table. It's not that unusual mysqld are > restarting after a query reqeuests values from a corrupted table space. That > behaviour subsequently results in aborted connections. > > I'ld double check database consistency. The easist way to check against > (at least physical) corruption should be mysqlcheckk --all-databases > > cheers, > > Stephan > > > Am Montag, den 18.06.2018, 12:47 +0200 schrieb Rafael Weingärtner: > > > > Your timeout configuration seems fine. There must be something wrong in > > your network. Or maybe in your MySQL service; as you said, it is restarting > > when you run commands against it. Therefore, it might be better to > > eliminate these issues first. > > > > On Mon, Jun 18, 2018 at 11:56 AM, Nicolas Bouige wrote: > > > > > > > > > > > Hello Dag, > > > > > > Im not trying to do a multi-master setup, just recover my DB :/ > > > I have installed a second node and connect it to the DB and it's not > > > possible to connect to the database server automatically (but manually > > > yes..) > > > On the first node at each sql query sent, the service mysql restart on db > > > server... > > > > > > > > > @Rafael, the timeout value is 28800 > > > > > > > > > mysql> SHOW VARIABLES LIKE 'wait_timeout'; > > > +---+---+ > > > > > > > > > > > > Variable_name | Value | > > > +---+---+ > > > > > > > > > > > > wait_timeout | 28800 | > > > +---+---+ > > > > > > Best regards, > > > > > > > > > Nicolas Bouige > > > DIMSI > > > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > > > 4, avenue Laurent Cely > > > Tour d’Asnière – 92600 Asnière sur Seine > > > T/ +33 (0)6 28 98 53 40 > > > > > > > > > > > > De : Dag Sonstebo > > > Envoyé : jeudi 14 juin 2018 10:32:13 > > > À : users@cloudstack.apache.org > > > Objet : Re: DB Communication Link Failure > > > > > > What Rafael said… > > > > > > In addition – can you confirm you aren’t trying something like a > > > multi-master MySQL setup? I have seen this cause similar issues. > > > > > > Regard
Re: DB Communication Link Failure
Hi! there's also a --auto-repair switch that could be added to mysqlcheck --all-databases. But to be honest, you can't guarantee the content will match. So references to other tablefields might not match afterwards (well, i expect these references don't match right now either) As far as your resultset shows, the corruption "only" happened to the index-space so your data "could" be fine. Normally, I'ld suggest to revert to a backup, but as this question has been around for a few days here, I assume your last uncorrupted backup could be far too old. Am Montag, den 18.06.2018, 11:56 + schrieb Nicolas Bouige: > Hi Stephan, > > > thanks for the command, i could spot which tables is corrupted : > > > cloud.event > Warning : InnoDB: Index 'i_event__created' contains 548 entries, should be > 542. > Warning : InnoDB: Index 'i_event__user_id' contains 547 entries, should be > 542. > Warning : InnoDB: Index 'i_event__account_id' contains 547 entries, should > be 542. > Warning : InnoDB: Index 'i_event__level_id' contains 547 entries, should be > 542. > Warning : InnoDB: Index 'i_event__type_id' contains 548 entries, should be > 542. > error: Corrupt > > Now, i supposed i have to delete the entries > > Nicolas Bouige > DIMSI > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > 4, avenue Laurent Cely > Tour d’Asnière – 92600 Asnière sur Seine > T/ +33 (0)6 28 98 53 40 > > > > De : Stephan Seitz > Envoyé : lundi 18 juin 2018 13:30:43 > À : users@cloudstack.apache.org > Objet : Re: DB Communication Link Failure > > Hi! > > This sound's like a corrupted database table. It's not that unusual mysqld are > restarting after a query reqeuests values from a corrupted table space. That > behaviour subsequently results in aborted connections. > > I'ld double check database consistency. The easist way to check against > (at least physical) corruption should be mysqlcheckk --all-databases > > cheers, > > Stephan > > > Am Montag, den 18.06.2018, 12:47 +0200 schrieb Rafael Weingärtner: > > > > Your timeout configuration seems fine. There must be something wrong in > > your network. Or maybe in your MySQL service; as you said, it is restarting > > when you run commands against it. Therefore, it might be better to > > eliminate these issues first. > > > > On Mon, Jun 18, 2018 at 11:56 AM, Nicolas Bouige wrote: > > > > > > > > > > > Hello Dag, > > > > > > Im not trying to do a multi-master setup, just recover my DB :/ > > > I have installed a second node and connect it to the DB and it's not > > > possible to connect to the database server automatically (but manually > > > yes..) > > > On the first node at each sql query sent, the service mysql restart on db > > > server... > > > > > > > > > @Rafael, the timeout value is 28800 > > > > > > > > > mysql> SHOW VARIABLES LIKE 'wait_timeout'; > > > +---+---+ > > > > > > > > > > > > Variable_name | Value | > > > +-------+---+ > > > > > > > > > > > > wait_timeout | 28800 | > > > +---+---+ > > > > > > Best regards, > > > > > > > > > Nicolas Bouige > > > DIMSI > > > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > > > 4, avenue Laurent Cely > > > Tour d’Asnière – 92600 Asnière sur Seine > > > T/ +33 (0)6 28 98 53 40 > > > > > > > > > > > > De : Dag Sonstebo > > > Envoyé : jeudi 14 juin 2018 10:32:13 > > > À : users@cloudstack.apache.org > > > Objet : Re: DB Communication Link Failure > > > > > > What Rafael said… > > > > > > In addition – can you confirm you aren’t trying something like a > > > multi-master MySQL setup? I have seen this cause similar issues. > > > > > > Regards, > > > Dag Sonstebo > > > Cloud Architect > > > ShapeBlue > > > > > > On 13/06/2018, 18:44, "Rafael Weingärtner" > > > wrote: > > > > > > In this case, I would say that you might be either having some problem > > > in > > > your network, or maybe some timeout in the mysql server. > > > Can you check the following variable? > > > > > >
RE: DB Communication Link Failure
Hi Stephan, thanks for the command, i could spot which tables is corrupted : cloud.event Warning : InnoDB: Index 'i_event__created' contains 548 entries, should be 542. Warning : InnoDB: Index 'i_event__user_id' contains 547 entries, should be 542. Warning : InnoDB: Index 'i_event__account_id' contains 547 entries, should be 542. Warning : InnoDB: Index 'i_event__level_id' contains 547 entries, should be 542. Warning : InnoDB: Index 'i_event__type_id' contains 548 entries, should be 542. error: Corrupt Now, i supposed i have to delete the entries Nicolas Bouige DIMSI cloud.dimsi.fr<http://www.cloud.dimsi.fr> 4, avenue Laurent Cely Tour d’Asnière – 92600 Asnière sur Seine T/ +33 (0)6 28 98 53 40 De : Stephan Seitz Envoyé : lundi 18 juin 2018 13:30:43 À : users@cloudstack.apache.org Objet : Re: DB Communication Link Failure Hi! This sound's like a corrupted database table. It's not that unusual mysqld are restarting after a query reqeuests values from a corrupted table space. That behaviour subsequently results in aborted connections. I'ld double check database consistency. The easist way to check against (at least physical) corruption should be mysqlcheckk --all-databases cheers, Stephan Am Montag, den 18.06.2018, 12:47 +0200 schrieb Rafael Weingärtner: > Your timeout configuration seems fine. There must be something wrong in > your network. Or maybe in your MySQL service; as you said, it is restarting > when you run commands against it. Therefore, it might be better to > eliminate these issues first. > > On Mon, Jun 18, 2018 at 11:56 AM, Nicolas Bouige wrote: > > > > > Hello Dag, > > > > Im not trying to do a multi-master setup, just recover my DB :/ > > I have installed a second node and connect it to the DB and it's not > > possible to connect to the database server automatically (but manually > > yes..) > > On the first node at each sql query sent, the service mysql restart on db > > server... > > > > > > @Rafael, the timeout value is 28800 > > > > > > mysql> SHOW VARIABLES LIKE 'wait_timeout'; > > +---+---+ > > > > > > Variable_name | Value | > > +---+---+ > > > > > > wait_timeout | 28800 | > > +---+---+ > > > > Best regards, > > > > > > Nicolas Bouige > > DIMSI > > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > > 4, avenue Laurent Cely > > Tour d’Asnière – 92600 Asnière sur Seine > > T/ +33 (0)6 28 98 53 40 > > > > > > > > De : Dag Sonstebo > > Envoyé : jeudi 14 juin 2018 10:32:13 > > À : users@cloudstack.apache.org > > Objet : Re: DB Communication Link Failure > > > > What Rafael said… > > > > In addition – can you confirm you aren’t trying something like a > > multi-master MySQL setup? I have seen this cause similar issues. > > > > Regards, > > Dag Sonstebo > > Cloud Architect > > ShapeBlue > > > > On 13/06/2018, 18:44, "Rafael Weingärtner" > > wrote: > > > > In this case, I would say that you might be either having some problem > > in > > your network, or maybe some timeout in the mysql server. > > Can you check the following variable? > > > > > > show variables like "%timeout%"; > > > > > > > > > > dag.sonst...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com>; > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > On Wed, Jun 13, 2018 at 6:05 PM, Nicolas Bouige wrote: > > > > > Hello Dag, Rafael, > > > > > > Thanks for your answer, i knonw this seems to be a "simple" issue > > but, as > > > i said in my previous mail, i checked connectivity between both > > server > > > without any problem (ping, telnet, firewall policies, remote mysql > > > connection...) and yes i restarted mgmt server (and it's not a silly > > > question...;)..) > > > > > > Someone knows something about the rollback transactions ? > > > > > > Best regards, > > > > > > N.B > > > > > > -Message d'origine- > > > De : Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > > Envoyé : mercredi 13 juin 2018 16:59 > > > À : users > > > Objet : Re:
Re: DB Communication Link Failure
Hi! This sound's like a corrupted database table. It's not that unusual mysqld are restarting after a query reqeuests values from a corrupted table space. That behaviour subsequently results in aborted connections. I'ld double check database consistency. The easist way to check against (at least physical) corruption should be mysqlcheckk --all-databases cheers, Stephan Am Montag, den 18.06.2018, 12:47 +0200 schrieb Rafael Weingärtner: > Your timeout configuration seems fine. There must be something wrong in > your network. Or maybe in your MySQL service; as you said, it is restarting > when you run commands against it. Therefore, it might be better to > eliminate these issues first. > > On Mon, Jun 18, 2018 at 11:56 AM, Nicolas Bouige wrote: > > > > > Hello Dag, > > > > Im not trying to do a multi-master setup, just recover my DB :/ > > I have installed a second node and connect it to the DB and it's not > > possible to connect to the database server automatically (but manually > > yes..) > > On the first node at each sql query sent, the service mysql restart on db > > server... > > > > > > @Rafael, the timeout value is 28800 > > > > > > mysql> SHOW VARIABLES LIKE 'wait_timeout'; > > +---+---+ > > > > > > Variable_name | Value | > > +---+---+ > > > > > > wait_timeout | 28800 | > > +---+---+ > > > > Best regards, > > > > > > Nicolas Bouige > > DIMSI > > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > > 4, avenue Laurent Cely > > Tour d’Asnière – 92600 Asnière sur Seine > > T/ +33 (0)6 28 98 53 40 > > > > > > > > De : Dag Sonstebo > > Envoyé : jeudi 14 juin 2018 10:32:13 > > À : users@cloudstack.apache.org > > Objet : Re: DB Communication Link Failure > > > > What Rafael said… > > > > In addition – can you confirm you aren’t trying something like a > > multi-master MySQL setup? I have seen this cause similar issues. > > > > Regards, > > Dag Sonstebo > > Cloud Architect > > ShapeBlue > > > > On 13/06/2018, 18:44, "Rafael Weingärtner" > > wrote: > > > > In this case, I would say that you might be either having some problem > > in > > your network, or maybe some timeout in the mysql server. > > Can you check the following variable? > > > > > > show variables like "%timeout%"; > > > > > > > > > > dag.sonst...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com>; > > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > > @shapeblue > > > > > > > > On Wed, Jun 13, 2018 at 6:05 PM, Nicolas Bouige wrote: > > > > > Hello Dag, Rafael, > > > > > > Thanks for your answer, i knonw this seems to be a "simple" issue > > but, as > > > i said in my previous mail, i checked connectivity between both > > server > > > without any problem (ping, telnet, firewall policies, remote mysql > > > connection...) and yes i restarted mgmt server (and it's not a silly > > > question...;)..) > > > > > > Someone knows something about the rollback transactions ? > > > > > > Best regards, > > > > > > N.B > > > > > > -Message d'origine- > > > De : Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > > Envoyé : mercredi 13 juin 2018 16:59 > > > À : users > > > Objet : Re: DB Communication Link Failure > > > > > > This might be a silly question, but... Have you restarted the > > management > > > server? > > > This problem may be caused by the connection pool. In theory, it > > should > > > re-create the connection, but you know, sometimes things just break. > > > > > > On Wed, Jun 13, 2018 at 4:56 PM, Dag Sonstebo < > > dag.sonst...@shapeblue.com> > > > wrote: > > > > > > > Hi Nicolas – not to dig too far into your analysis or log entries – > > > > but in my experience any time we see “Communications link failure” > > the > > > > problem is exactly that – a communications issue between management > > > > and DB, rather than an internal DB issue. > >
Re: DB Communication Link Failure
Your timeout configuration seems fine. There must be something wrong in your network. Or maybe in your MySQL service; as you said, it is restarting when you run commands against it. Therefore, it might be better to eliminate these issues first. On Mon, Jun 18, 2018 at 11:56 AM, Nicolas Bouige wrote: > Hello Dag, > > Im not trying to do a multi-master setup, just recover my DB :/ > I have installed a second node and connect it to the DB and it's not > possible to connect to the database server automatically (but manually > yes..) > On the first node at each sql query sent, the service mysql restart on db > server... > > > @Rafael, the timeout value is 28800 > > > mysql> SHOW VARIABLES LIKE 'wait_timeout'; > +---+---+ > | Variable_name | Value | > +---+---+ > | wait_timeout | 28800 | > +---+---+ > > Best regards, > > > Nicolas Bouige > DIMSI > cloud.dimsi.fr<http://www.cloud.dimsi.fr> > 4, avenue Laurent Cely > Tour d’Asnière – 92600 Asnière sur Seine > T/ +33 (0)6 28 98 53 40 > > > ________ > De : Dag Sonstebo > Envoyé : jeudi 14 juin 2018 10:32:13 > À : users@cloudstack.apache.org > Objet : Re: DB Communication Link Failure > > What Rafael said… > > In addition – can you confirm you aren’t trying something like a > multi-master MySQL setup? I have seen this cause similar issues. > > Regards, > Dag Sonstebo > Cloud Architect > ShapeBlue > > On 13/06/2018, 18:44, "Rafael Weingärtner" > wrote: > > In this case, I would say that you might be either having some problem > in > your network, or maybe some timeout in the mysql server. > Can you check the following variable? > > > > show variables like "%timeout%"; > > > > > > dag.sonst...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > On Wed, Jun 13, 2018 at 6:05 PM, Nicolas Bouige wrote: > > > Hello Dag, Rafael, > > > > Thanks for your answer, i knonw this seems to be a "simple" issue > but, as > > i said in my previous mail, i checked connectivity between both > server > > without any problem (ping, telnet, firewall policies, remote mysql > > connection...) and yes i restarted mgmt server (and it's not a silly > > question...;)..) > > > > Someone knows something about the rollback transactions ? > > > > Best regards, > > > > N.B > > > > -Message d'origine- > > De : Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > > Envoyé : mercredi 13 juin 2018 16:59 > > À : users > > Objet : Re: DB Communication Link Failure > > > > This might be a silly question, but... Have you restarted the > management > > server? > > This problem may be caused by the connection pool. In theory, it > should > > re-create the connection, but you know, sometimes things just break. > > > > On Wed, Jun 13, 2018 at 4:56 PM, Dag Sonstebo < > dag.sonst...@shapeblue.com> > > wrote: > > > > > Hi Nicolas – not to dig too far into your analysis or log entries – > > > but in my experience any time we see “Communications link failure” > the > > > problem is exactly that – a communications issue between management > > > and DB, rather than an internal DB issue. > > > > > > Regards, > > > Dag Sonstebo > > > Cloud Architect > > > ShapeBlue > > > > > > From: Nicolas Bouige > > > Reply-To: "users@cloudstack.apache.org" < > users@cloudstack.apache.org> > > > Date: Wednesday, 13 June 2018 at 15:04 > > > To: "users@cloudstack.apache.org" > > > Subject: DB Communication Link Failure > > > > > > Hello All, > > > > > > Tonight du to a network issue our management cloudstack server and > > > MariaDb server have been shutdowned and restarted this morning. > > > > > > Unfortunately, we get some issues of « connectivity » between the > > > mgmt server and db server. > > > Both service (cloudstack-management/mysqld) are up and running > without > > > any errors. > > > Db server are available from mgmt server (telnet OK, mysql –h > > > dbserver…ok
RE: DB Communication Link Failure
Hello Dag, Im not trying to do a multi-master setup, just recover my DB :/ I have installed a second node and connect it to the DB and it's not possible to connect to the database server automatically (but manually yes..) On the first node at each sql query sent, the service mysql restart on db server... @Rafael, the timeout value is 28800 mysql> SHOW VARIABLES LIKE 'wait_timeout'; +---+---+ | Variable_name | Value | +---+---+ | wait_timeout | 28800 | +---+---+ Best regards, Nicolas Bouige DIMSI cloud.dimsi.fr<http://www.cloud.dimsi.fr> 4, avenue Laurent Cely Tour d’Asnière – 92600 Asnière sur Seine T/ +33 (0)6 28 98 53 40 De : Dag Sonstebo Envoyé : jeudi 14 juin 2018 10:32:13 À : users@cloudstack.apache.org Objet : Re: DB Communication Link Failure What Rafael said… In addition – can you confirm you aren’t trying something like a multi-master MySQL setup? I have seen this cause similar issues. Regards, Dag Sonstebo Cloud Architect ShapeBlue On 13/06/2018, 18:44, "Rafael Weingärtner" wrote: In this case, I would say that you might be either having some problem in your network, or maybe some timeout in the mysql server. Can you check the following variable? > > show variables like "%timeout%"; > > dag.sonst...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue On Wed, Jun 13, 2018 at 6:05 PM, Nicolas Bouige wrote: > Hello Dag, Rafael, > > Thanks for your answer, i knonw this seems to be a "simple" issue but, as > i said in my previous mail, i checked connectivity between both server > without any problem (ping, telnet, firewall policies, remote mysql > connection...) and yes i restarted mgmt server (and it's not a silly > question...;)..) > > Someone knows something about the rollback transactions ? > > Best regards, > > N.B > > -Message d'origine- > De : Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > Envoyé : mercredi 13 juin 2018 16:59 > À : users > Objet : Re: DB Communication Link Failure > > This might be a silly question, but... Have you restarted the management > server? > This problem may be caused by the connection pool. In theory, it should > re-create the connection, but you know, sometimes things just break. > > On Wed, Jun 13, 2018 at 4:56 PM, Dag Sonstebo > wrote: > > > Hi Nicolas – not to dig too far into your analysis or log entries – > > but in my experience any time we see “Communications link failure” the > > problem is exactly that – a communications issue between management > > and DB, rather than an internal DB issue. > > > > Regards, > > Dag Sonstebo > > Cloud Architect > > ShapeBlue > > > > From: Nicolas Bouige > > Reply-To: "users@cloudstack.apache.org" > > Date: Wednesday, 13 June 2018 at 15:04 > > To: "users@cloudstack.apache.org" > > Subject: DB Communication Link Failure > > > > Hello All, > > > > Tonight du to a network issue our management cloudstack server and > > MariaDb server have been shutdowned and restarted this morning. > > > > Unfortunately, we get some issues of « connectivity » between the > > mgmt server and db server. > > Both service (cloudstack-management/mysqld) are up and running without > > any errors. > > Db server are available from mgmt server (telnet OK, mysql –h > > dbserver…ok, firewall policies are good from both sides) > > > > However, when we want to login to the GUI cloudstack an execption is > > raised in mgmt-server.log : > > > > 2018-06-13 15:28:41,019 DEBUG [c.c.u.d.T.Transaction] > > (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Rolling back the > > transaction: Time = 97 Name = qtp1796488937-13; called by > > -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832- > > TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36- > > ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor. > > invoke:92-ReflectiveMethodInvocation.proceed:185- > > JdkDynamicAopProxy.invoke:212-$Proxy121.persist:-1-ActionEventUtils. > > persistActionEvent:186-ActionEventUtils.onActionEvent:98- > > AccountManagerImpl.logoutUser:2096 > > 2018-06-13 15:28:41,020 WARN [c.c.u.d.T.Transa
Re: DB Communication Link Failure
What Rafael said… In addition – can you confirm you aren’t trying something like a multi-master MySQL setup? I have seen this cause similar issues. Regards, Dag Sonstebo Cloud Architect ShapeBlue On 13/06/2018, 18:44, "Rafael Weingärtner" wrote: In this case, I would say that you might be either having some problem in your network, or maybe some timeout in the mysql server. Can you check the following variable? > > show variables like "%timeout%"; > > dag.sonst...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue On Wed, Jun 13, 2018 at 6:05 PM, Nicolas Bouige wrote: > Hello Dag, Rafael, > > Thanks for your answer, i knonw this seems to be a "simple" issue but, as > i said in my previous mail, i checked connectivity between both server > without any problem (ping, telnet, firewall policies, remote mysql > connection...) and yes i restarted mgmt server (and it's not a silly > question...;)..) > > Someone knows something about the rollback transactions ? > > Best regards, > > N.B > > -Message d'origine- > De : Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > Envoyé : mercredi 13 juin 2018 16:59 > À : users > Objet : Re: DB Communication Link Failure > > This might be a silly question, but... Have you restarted the management > server? > This problem may be caused by the connection pool. In theory, it should > re-create the connection, but you know, sometimes things just break. > > On Wed, Jun 13, 2018 at 4:56 PM, Dag Sonstebo > wrote: > > > Hi Nicolas – not to dig too far into your analysis or log entries – > > but in my experience any time we see “Communications link failure” the > > problem is exactly that – a communications issue between management > > and DB, rather than an internal DB issue. > > > > Regards, > > Dag Sonstebo > > Cloud Architect > > ShapeBlue > > > > From: Nicolas Bouige > > Reply-To: "users@cloudstack.apache.org" > > Date: Wednesday, 13 June 2018 at 15:04 > > To: "users@cloudstack.apache.org" > > Subject: DB Communication Link Failure > > > > Hello All, > > > > Tonight du to a network issue our management cloudstack server and > > MariaDb server have been shutdowned and restarted this morning. > > > > Unfortunately, we get some issues of « connectivity » between the > > mgmt server and db server. > > Both service (cloudstack-management/mysqld) are up and running without > > any errors. > > Db server are available from mgmt server (telnet OK, mysql –h > > dbserver…ok, firewall policies are good from both sides) > > > > However, when we want to login to the GUI cloudstack an execption is > > raised in mgmt-server.log : > > > > 2018-06-13 15:28:41,019 DEBUG [c.c.u.d.T.Transaction] > > (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Rolling back the > > transaction: Time = 97 Name = qtp1796488937-13; called by > > -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832- > > TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36- > > ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor. > > invoke:92-ReflectiveMethodInvocation.proceed:185- > > JdkDynamicAopProxy.invoke:212-$Proxy121.persist:-1-ActionEventUtils. > > persistActionEvent:186-ActionEventUtils.onActionEvent:98- > > AccountManagerImpl.logoutUser:2096 > > 2018-06-13 15:28:41,020 WARN [c.c.u.d.T.Transaction] > > (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Unable to rollback > > com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: > > Communications link failure during rollback(). Transaction resolution > > unknown. > > at > > sun.reflect.GeneratedConstructorAccessor98.newInstance(Unknown > > Source) > > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > > DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java: > 423) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at com.mysql.jdbc.Util.getInstance(Util.java:386) > > at com.mysql.jdbc.SQLError.createSQLException(SQLError. > java:1015) > >
Re: DB Communication Link Failure
In this case, I would say that you might be either having some problem in your network, or maybe some timeout in the mysql server. Can you check the following variable? > > show variables like "%timeout%"; > > On Wed, Jun 13, 2018 at 6:05 PM, Nicolas Bouige wrote: > Hello Dag, Rafael, > > Thanks for your answer, i knonw this seems to be a "simple" issue but, as > i said in my previous mail, i checked connectivity between both server > without any problem (ping, telnet, firewall policies, remote mysql > connection...) and yes i restarted mgmt server (and it's not a silly > question...;)..) > > Someone knows something about the rollback transactions ? > > Best regards, > > N.B > > -Message d'origine- > De : Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] > Envoyé : mercredi 13 juin 2018 16:59 > À : users > Objet : Re: DB Communication Link Failure > > This might be a silly question, but... Have you restarted the management > server? > This problem may be caused by the connection pool. In theory, it should > re-create the connection, but you know, sometimes things just break. > > On Wed, Jun 13, 2018 at 4:56 PM, Dag Sonstebo > wrote: > > > Hi Nicolas – not to dig too far into your analysis or log entries – > > but in my experience any time we see “Communications link failure” the > > problem is exactly that – a communications issue between management > > and DB, rather than an internal DB issue. > > > > Regards, > > Dag Sonstebo > > Cloud Architect > > ShapeBlue > > > > From: Nicolas Bouige > > Reply-To: "users@cloudstack.apache.org" > > Date: Wednesday, 13 June 2018 at 15:04 > > To: "users@cloudstack.apache.org" > > Subject: DB Communication Link Failure > > > > Hello All, > > > > Tonight du to a network issue our management cloudstack server and > > MariaDb server have been shutdowned and restarted this morning. > > > > Unfortunately, we get some issues of « connectivity » between the > > mgmt server and db server. > > Both service (cloudstack-management/mysqld) are up and running without > > any errors. > > Db server are available from mgmt server (telnet OK, mysql –h > > dbserver…ok, firewall policies are good from both sides) > > > > However, when we want to login to the GUI cloudstack an execption is > > raised in mgmt-server.log : > > > > 2018-06-13 15:28:41,019 DEBUG [c.c.u.d.T.Transaction] > > (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Rolling back the > > transaction: Time = 97 Name = qtp1796488937-13; called by > > -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832- > > TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36- > > ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor. > > invoke:92-ReflectiveMethodInvocation.proceed:185- > > JdkDynamicAopProxy.invoke:212-$Proxy121.persist:-1-ActionEventUtils. > > persistActionEvent:186-ActionEventUtils.onActionEvent:98- > > AccountManagerImpl.logoutUser:2096 > > 2018-06-13 15:28:41,020 WARN [c.c.u.d.T.Transaction] > > (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Unable to rollback > > com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: > > Communications link failure during rollback(). Transaction resolution > > unknown. > > at > > sun.reflect.GeneratedConstructorAccessor98.newInstance(Unknown > > Source) > > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > > DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java: > 423) > > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > > at com.mysql.jdbc.Util.getInstance(Util.java:386) > > at com.mysql.jdbc.SQLError.createSQLException(SQLError. > java:1015) > > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989) > > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975) > > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920) > > at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java: > > 5067) > > at org.apache.commons.dbcp.DelegatingConnection.rollback( > > DelegatingConnection.java:368) > > at org.apache.commons.dbcp.PoolingDataSource$ > > PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:323) > > at com.cloud.utils.db.TransactionLegacy.rollbackTransaction( > > TransactionLegacy.java:851) > > at com
RE: DB Communication Link Failure
Hello Dag, Rafael, Thanks for your answer, i knonw this seems to be a "simple" issue but, as i said in my previous mail, i checked connectivity between both server without any problem (ping, telnet, firewall policies, remote mysql connection...) and yes i restarted mgmt server (and it's not a silly question...;)..) Someone knows something about the rollback transactions ? Best regards, N.B -Message d'origine- De : Rafael Weingärtner [mailto:rafaelweingart...@gmail.com] Envoyé : mercredi 13 juin 2018 16:59 À : users Objet : Re: DB Communication Link Failure This might be a silly question, but... Have you restarted the management server? This problem may be caused by the connection pool. In theory, it should re-create the connection, but you know, sometimes things just break. On Wed, Jun 13, 2018 at 4:56 PM, Dag Sonstebo wrote: > Hi Nicolas – not to dig too far into your analysis or log entries – > but in my experience any time we see “Communications link failure” the > problem is exactly that – a communications issue between management > and DB, rather than an internal DB issue. > > Regards, > Dag Sonstebo > Cloud Architect > ShapeBlue > > From: Nicolas Bouige > Reply-To: "users@cloudstack.apache.org" > Date: Wednesday, 13 June 2018 at 15:04 > To: "users@cloudstack.apache.org" > Subject: DB Communication Link Failure > > Hello All, > > Tonight du to a network issue our management cloudstack server and > MariaDb server have been shutdowned and restarted this morning. > > Unfortunately, we get some issues of « connectivity » between the > mgmt server and db server. > Both service (cloudstack-management/mysqld) are up and running without > any errors. > Db server are available from mgmt server (telnet OK, mysql –h > dbserver…ok, firewall policies are good from both sides) > > However, when we want to login to the GUI cloudstack an execption is > raised in mgmt-server.log : > > 2018-06-13 15:28:41,019 DEBUG [c.c.u.d.T.Transaction] > (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Rolling back the > transaction: Time = 97 Name = qtp1796488937-13; called by > -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832- > TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36- > ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor. > invoke:92-ReflectiveMethodInvocation.proceed:185- > JdkDynamicAopProxy.invoke:212-$Proxy121.persist:-1-ActionEventUtils. > persistActionEvent:186-ActionEventUtils.onActionEvent:98- > AccountManagerImpl.logoutUser:2096 > 2018-06-13 15:28:41,020 WARN [c.c.u.d.T.Transaction] > (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Unable to rollback > com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: > Communications link failure during rollback(). Transaction resolution > unknown. > at > sun.reflect.GeneratedConstructorAccessor98.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at com.mysql.jdbc.Util.getInstance(Util.java:386) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920) > at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java: > 5067) > at org.apache.commons.dbcp.DelegatingConnection.rollback( > DelegatingConnection.java:368) > at org.apache.commons.dbcp.PoolingDataSource$ > PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:323) > at com.cloud.utils.db.TransactionLegacy.rollbackTransaction( > TransactionLegacy.java:851) > at com.cloud.utils.db.TransactionLegacy.rollback( > TransactionLegacy.java:889) > at com.cloud.utils.db.TransactionLegacy.removeUpTo( > TransactionLegacy.java:832) > at com.cloud.utils.db.TransactionLegacy.close( > TransactionLegacy.java:656) > at com.cloud.utils.db.TransactionContextInterceptor.invoke( > TransactionContextInterceptor.java:36) > at org.springframework.aop.framework.ReflectiveMethodInvocation. > proceed(ReflectiveMethodInvocation.java:174) > at org.springframework.aop.interceptor. > ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > at org.springframework.aop.framework.ReflectiveMethodInvocation. > proceed(ReflectiveMethodInvocation.jav
Re: DB Communication Link Failure
This might be a silly question, but... Have you restarted the management server? This problem may be caused by the connection pool. In theory, it should re-create the connection, but you know, sometimes things just break. On Wed, Jun 13, 2018 at 4:56 PM, Dag Sonstebo wrote: > Hi Nicolas – not to dig too far into your analysis or log entries – but in > my experience any time we see “Communications link failure” the problem is > exactly that – a communications issue between management and DB, rather > than an internal DB issue. > > Regards, > Dag Sonstebo > Cloud Architect > ShapeBlue > > From: Nicolas Bouige > Reply-To: "users@cloudstack.apache.org" > Date: Wednesday, 13 June 2018 at 15:04 > To: "users@cloudstack.apache.org" > Subject: DB Communication Link Failure > > Hello All, > > Tonight du to a network issue our management cloudstack server and MariaDb > server have been shutdowned and restarted this morning. > > Unfortunately, we get some issues of « connectivity » between the mgmt > server and db server. > Both service (cloudstack-management/mysqld) are up and running without any > errors. > Db server are available from mgmt server (telnet OK, mysql –h dbserver…ok, > firewall policies are good from both sides) > > However, when we want to login to the GUI cloudstack an execption is > raised in mgmt-server.log : > > 2018-06-13 15:28:41,019 DEBUG [c.c.u.d.T.Transaction] > (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Rolling back the > transaction: Time = 97 Name = qtp1796488937-13; called by > -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832- > TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36- > ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor. > invoke:92-ReflectiveMethodInvocation.proceed:185- > JdkDynamicAopProxy.invoke:212-$Proxy121.persist:-1-ActionEventUtils. > persistActionEvent:186-ActionEventUtils.onActionEvent:98- > AccountManagerImpl.logoutUser:2096 > 2018-06-13 15:28:41,020 WARN [c.c.u.d.T.Transaction] > (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Unable to rollback > com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: > Communications link failure during rollback(). Transaction resolution > unknown. > at sun.reflect.GeneratedConstructorAccessor98.newInstance(Unknown > Source) > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance( > DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) > at com.mysql.jdbc.Util.getInstance(Util.java:386) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975) > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920) > at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java: > 5067) > at org.apache.commons.dbcp.DelegatingConnection.rollback( > DelegatingConnection.java:368) > at org.apache.commons.dbcp.PoolingDataSource$ > PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:323) > at com.cloud.utils.db.TransactionLegacy.rollbackTransaction( > TransactionLegacy.java:851) > at com.cloud.utils.db.TransactionLegacy.rollback( > TransactionLegacy.java:889) > at com.cloud.utils.db.TransactionLegacy.removeUpTo( > TransactionLegacy.java:832) > at com.cloud.utils.db.TransactionLegacy.close( > TransactionLegacy.java:656) > at com.cloud.utils.db.TransactionContextInterceptor.invoke( > TransactionContextInterceptor.java:36) > at org.springframework.aop.framework.ReflectiveMethodInvocation. > proceed(ReflectiveMethodInvocation.java:174) > at org.springframework.aop.interceptor. > ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) > at org.springframework.aop.framework.ReflectiveMethodInvocation. > proceed(ReflectiveMethodInvocation.java:185) > at org.springframework.aop.framework.JdkDynamicAopProxy. > invoke(JdkDynamicAopProxy.java:212) > at com.sun.proxy.$Proxy121.persist(Unknown Source) > at com.cloud.event.ActionEventUtils.persistActionEvent( > ActionEventUtils.java:186) > at com.cloud.event.ActionEventUtils.onActionEvent( > ActionEventUtils.java:98) > at com.cloud.user.AccountManagerImpl.logoutUser( > AccountManagerImpl.java:2096) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:62) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at org.springframework.aop.support.AopUtils. > invokeJoinpointUsingReflection(AopUtils.java:338) > at org.
Re: DB Communication Link Failure
Hi Nicolas – not to dig too far into your analysis or log entries – but in my experience any time we see “Communications link failure” the problem is exactly that – a communications issue between management and DB, rather than an internal DB issue. Regards, Dag Sonstebo Cloud Architect ShapeBlue From: Nicolas Bouige Reply-To: "users@cloudstack.apache.org" Date: Wednesday, 13 June 2018 at 15:04 To: "users@cloudstack.apache.org" Subject: DB Communication Link Failure Hello All, Tonight du to a network issue our management cloudstack server and MariaDb server have been shutdowned and restarted this morning. Unfortunately, we get some issues of « connectivity » between the mgmt server and db server. Both service (cloudstack-management/mysqld) are up and running without any errors. Db server are available from mgmt server (telnet OK, mysql –h dbserver…ok, firewall policies are good from both sides) However, when we want to login to the GUI cloudstack an execption is raised in mgmt-server.log : 2018-06-13 15:28:41,019 DEBUG [c.c.u.d.T.Transaction] (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Rolling back the transaction: Time = 97 Name = qtp1796488937-13; called by -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:174-ExposeInvocationInterceptor.invoke:92-ReflectiveMethodInvocation.proceed:185-JdkDynamicAopProxy.invoke:212-$Proxy121.persist:-1-ActionEventUtils.persistActionEvent:186-ActionEventUtils.onActionEvent:98-AccountManagerImpl.logoutUser:2096 2018-06-13 15:28:41,020 WARN [c.c.u.d.T.Transaction] (qtp1796488937-13:ctx-88e441c4) (logid:f9e9b399) Unable to rollback com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during rollback(). Transaction resolution unknown. at sun.reflect.GeneratedConstructorAccessor98.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:411) at com.mysql.jdbc.Util.getInstance(Util.java:386) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920) at com.mysql.jdbc.ConnectionImpl.rollback(ConnectionImpl.java:5067) at org.apache.commons.dbcp.DelegatingConnection.rollback(DelegatingConnection.java:368) at org.apache.commons.dbcp.PoolingDataSource$PoolGuardConnectionWrapper.rollback(PoolingDataSource.java:323) at com.cloud.utils.db.TransactionLegacy.rollbackTransaction(TransactionLegacy.java:851) at com.cloud.utils.db.TransactionLegacy.rollback(TransactionLegacy.java:889) at com.cloud.utils.db.TransactionLegacy.removeUpTo(TransactionLegacy.java:832) at com.cloud.utils.db.TransactionLegacy.close(TransactionLegacy.java:656) at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:36) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) at com.sun.proxy.$Proxy121.persist(Unknown Source) at com.cloud.event.ActionEventUtils.persistActionEvent(ActionEventUtils.java:186) at com.cloud.event.ActionEventUtils.onActionEvent(ActionEventUtils.java:98) at com.cloud.user.AccountManagerImpl.logoutUser(AccountManagerImpl.java:2096) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.j
