xCAT only added perl module for Mysql and Postgresql into the default package list for service node osimage, there is no rpm dependency as not all users use mysql/postgresql.
I think this could work for most of the case, but it is better for xCAT to check the failure as soon as quickly with error message or `xcatprobe`, I'll log a ticket in github to track it.
Bin Xu
HPC Software Development
Software Defined Infrastructure, IBM Systems
Software Defined Infrastructure, IBM Systems
Phone: 86-010-82454067
E-mail: bx...@cn.ibm.com
----- Original message -----
From: "Song BJ Yang" <yang...@cn.ibm.com>
To: xcat-user@lists.sourceforge.net
Cc: xcat-user@lists.sourceforge.net
Subject: Re: [xcat-user] New Service node cannot connect to Mariadb when starting xcatd
Date: Wed, Oct 24, 2018 10:25 AM
Hi Brian,Cong!Please tell us the os version and arch of your service node, or better the osimage definition of the service node, so that we can confirm whether `perl-DBD-MySQL` is missing somewhere, thanks------------------------------------------------------------------------------
YANG Song (杨嵩)
IBM China System Technology Laboratory
Tel: 86-10-82452903
Email: yang...@cn.ibm.com
Address: Building 28, ZhongGuanCun Software Park,
No.8, Dong Bei Wang West Road, Haidian District Beijing 100193, PRC
北京市海淀区东北旺西路8号中关村软件园28号楼
邮编: 100193----- Original message -----
From: Brian Joiner <martinitime1...@gmail.com>
To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
Cc:
Subject: Re: [xcat-user] New Service node cannot connect to Mariadb when starting xcatd
Date: Wed, Oct 24, 2018 6:46 AM
My Coworker Calvin found the solution! There was a missing package he noticed in the strace output for xcatd. There was a file mentioned over and over, he did a lookup for the package that provides the file(stat("/usr/local/share/perl5/DBD/mysql.pmc", 0x7ffcec9af410) = -1 ENOENT (No such file or directory)(stat("/usr/local/share/perl5/DBD/mysql.pm", 0x7ffcec9af350) = -1 ENOENT (No such file or directory)The missing package was: perl-DBD-MySQLOnce I installed that, xcatd started and I can run lsdef and other xCAT commands. If I missed this from the documentation, a thousand pardons :) However, if this is a bug with the service node package list, it needs to be remedied by the devs.Thanks everyone for all your help!On Tue, Oct 23, 2018 at 3:14 PM Kevin Keane <kke...@sandiego.edu> wrote:Somebody else on this list may be better able to answer that. My tool of choice would be tcpdump - monitor all traffic between 10.0.4.1 and 10.0.4.2 (which may be quite a bit), then try to connect again, and see if you can tease out the related packets.Another possibility is to try and use netstat immediately after you try to connect, and see if there are any connections in TIME_WAIT or CLOSE_WAIT or one of the FIN_WAIT statuses. But that is probably a matter of luck catching it._______________________________________________________________________
Kevin Keane | Systems Architect | University of San Diego ITS | kke...@sandiego.edu
Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 | 619.260.6859REMEMBER! No one from IT at USD will ever ask to confirm or supply your password.
These messages are an attempt to steal your username and password. Please do not reply to, click the links within, or open the attachments of these messages. Delete them!
_______________________________________________On Tue, Oct 23, 2018 at 11:39 AM Brian Joiner <martinitime1...@gmail.com> wrote:Ok, all of those commands from the service node work fine. It connects, switches to xcatdb and lists all the tables without any error.You're correct, the listening port for Mysql on the head node is 3306, must have fat fingered that one. Is there a way to see if the xcatd service is trying to connect to the right port?On Tue, Oct 23, 2018 at 12:06 PM Kevin Keane <kke...@sandiego.edu> wrote:Just to double-check: when you log in to the DB from the service node, you can not only access MariaDB, but it will now let you see the xcatdb as well? Try this on the service node:mysql -h 10.4.0.1 -u xcatadmin -D xcatdb -pAnd thenshow databases;use xcatdb;show tables;The first should show you mysql and xcatdb (maybe additional databases)The second should complete without error. This is the command you use to actually access a database.The third one shows you all the tables - that's just to verify that the use command didn't simply silently fail.One other thing I noticed in your initial post: at one point you mentioned port 3006, but MySQL usually listens on 3306. And the fact that the mysql command on the command line can connect without you specifying a port number suggests that MariaDB on the management node really uses the default value.So double-check whether MariaDB really listens on 3306 on the MN (it should), and then check that there is no typo in any of the configuration files.
______________________________________________________________________Kevin Keane | Systems Architect | University of San Diego ITS | kke...@sandiego.edu
Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 | 619.260.6859REMEMBER! No one from IT at USD will ever ask to confirm or supply your password.
These messages are an attempt to steal your username and password. Please do not reply to, click the links within, or open the attachments of these messages. Delete them!
_______________________________________________On Tue, Oct 23, 2018 at 9:29 AM Brian Joiner <martinitime1...@gmail.com> wrote:Kevin,I logged in to mysql as root user and ran the command you provided. It took and heres the output of show grants (from the head node)MariaDB [(none)]> show grants for xcatadmin@10.4.0.2;+--------------------------------------------------------------+| Grants for xcatadmin@10.4.0.2 |+--------------------------------------------------------------+| GRANT USAGE ON *.* TO 'xcatadmin'@'10.4.0.2' || GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.4.0.2' |+--------------------------------------------------------------+2 rows in set (0.00 sec)When trying to login to the database from the service node, the password wasn't working anymore, so I re-entered the command adding 'IDENTIFIED BY '<password>' to the end and that allowed me to connect as xcatadmin from the servicenode manually. However, starting up the service still fails with the same error: Oct 23 11:15:31 service01 xcat[18341]: Could not connect to the database. Database handle not defined.On Mon, Oct 22, 2018 at 7:31 PM Kevin Keane <kke...@sandiego.edu> wrote:That just says that any computer on your network is allowed to access MariaDB itself (which explains why you can log on), but you do not have any privileges on the xcatdb database.You need to see a line like this as well:GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.4.0.2'In fact, this is actually the exact command you are going to need to type to grant the required permission (you probably need to be the MariaDB root user, and probably on the management node, to do this).From a security standpoint, I would actually recommend removing the GRANT USAGE for anything with a %, and instead only allow specific IPs, but I'm not sure if xcat likes that._______________________________________________________________________
Kevin Keane | Systems Architect | University of San Diego ITS | kke...@sandiego.edu
Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 | 619.260.6859REMEMBER! No one from IT at USD will ever ask to confirm or supply your password.
These messages are an attempt to steal your username and password. Please do not reply to, click the links within, or open the attachments of these messages. Delete them!
_______________________________________________On Mon, Oct 22, 2018 at 5:15 PM Brian Joiner <martinitime1...@gmail.com> wrote:The plot thickens!MariaDB [xcatdb]> show grants for xcatadmin@10.4.0.2;ERROR 1141 (42000): There is no such grant defined for user 'xcatadmin' on host '10.4.0.2'But shouldn't this cover it:MariaDB [xcatdb]> show grants for xcatadmin-> ;+----------------------------------------------------------------------------------------------------------+| Grants for xcatadmin@% |+----------------------------------------------------------------------------------------------------------+| GRANT USAGE ON *.* TO 'xcatadmin'@'%' IDENTIFIED BY PASSWORD '*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19' |+----------------------------------------------------------------------------------------------------------+1 row in set (0.00 sec)On Mon, Oct 22, 2018 at 5:21 PM Kevin Keane <kke...@sandiego.edu> wrote:I might also try "show grants for xcatadmin@10.4.0.2" - the show grants you used will only tell you what permissions the MN has, but you are interested in the service node._______________________________________________________________________
Kevin Keane | Systems Architect | University of San Diego ITS | kke...@sandiego.edu
Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 | 619.260.6859REMEMBER! No one from IT at USD will ever ask to confirm or supply your password.
These messages are an attempt to steal your username and password. Please do not reply to, click the links within, or open the attachments of these messages. Delete them!
_______________________________________________On Mon, Oct 22, 2018 at 3:06 PM Brian Joiner <martinitime1...@gmail.com> wrote:It looks like everything is setup correctly:[root@service01 ~]# mysql -h 10.4.0.1 -u xcatadmin -D xcatdb -pEnter password:Reading table information for completion of table and column namesYou can turn off this feature to get a quicker startup with -AWelcome to the MariaDB monitor. Commands end with ; or \g.Your MariaDB connection id is 5Server version: 5.5.56-MariaDB MariaDB ServerCopyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.MariaDB [xcatdb]> show grants for xcatadmin@10.4.0.1-> ;+-----------------------------------------------------------------------------------------------------------------+| Grants for xcatadmin@10.4.0.1 |+-----------------------------------------------------------------------------------------------------------------+| GRANT USAGE ON *.* TO 'xcatadmin'@'10.4.0.1' IDENTIFIED BY PASSWORD '*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19' || GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.4.0.1' |+-----------------------------------------------------------------------------------------------------------------+2 rows in set (0.00 sec)MariaDB [xcatdb]>On Sun, Oct 21, 2018 at 10:40 PM Bin XA Xu <bx...@cn.ibm.com> wrote:Okay, it means you cannot access your mariadb server on MN from this service node.You can use `mysql -h 10.3.5.20 -u xcatadmin -D xcatdb -p` to verify it.Are you configure your mariadb well to GRANT your service node IPs? (https://xcat-docs.readthedocs.io/en/stable/advanced/hierarchy/databases/mysql_configure.html?highlight=mysqlsetup)You can use below command in mariadb shell to query the GRANT:```show grants for xcatadmin@10.3.5.20;
+------------------------------------------------------------------------------------------------------------------+
| Grants for xcatadmin@10.3.5.20 |
+------------------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'xcatadmin'@'10.3.5.20' IDENTIFIED BY PASSWORD '*359860933C260AB0F1316209DC2846250BAD59E3' |
| GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.3.5.20' |
+------------------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)```You can use `mysqlsetup -u -f <clientip_list filename>` to add your service nodes, you just follow the mysql steps to do that.Bin XuHPC Software Development
Software Defined Infrastructure, IBM SystemsPhone: 86-010-82454067E-mail: bx...@cn.ibm.com----- Original message -----
From: Brian Joiner <martinitime1...@gmail.com>
To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
Cc:
Subject: Re: [xcat-user] New Service node cannot connect to Mariadb when starting xcatd
Date: Fri, Oct 19, 2018 11:27 PM
Bin, heres the output:[root@service01 ~]# XCATBYPASS=1 lsxcatd -aCould not connect to the database. Database handle not defined.Could not read the site tableCould not connect to the database. Database handle not defined.Could not read the site tableCould not connect to the database. Database handle not defined.Could not read the site tableCould not connect to the database. Database handle not defined.Could not read the site tableCould not connect to the database. Database handle not defined.Could not read the site tableCould not connect to the database. Database handle not defined.Could not read the site tableCould not connect to the database. Database handle not defined.Could not read the site tableCould not connect to the database. Database handle not defined.Could not read the site tableCould not connect to the database. Database handle not defined.Could not read the site tableCould not connect to the database. Database handle not defined.Error loading module /opt/xcat/lib/perl/xCAT_plugin/zvmdiscovery.pm ...skippingCould not connect to the database. Database handle not defined.ERROR: Unable to open basic site table for configurationVersion 2.14.1 (git commit 70d6e7f93cc9714a127c22df2e7ca53d4996a34c, built Fri Jun 1 03:00:53 EDT 2018)This is a Service Nodecfgloc=mysql:dbname=xcatdb;host=10.4.0.1|xcatadmindbengine=mysqldbname=xcatdbdbhost=10.4.0.1dbadmin=xcatadminOn Thu, Oct 18, 2018 at 9:34 PM Bin XA Xu <bx...@cn.ibm.com> wrote:could you run `XCATBYPASS=1 lsxcatd -a` on your service node and paste the output here?Bin XuHPC Software Development
Software Defined Infrastructure, IBM SystemsPhone: 86-010-82454067E-mail: bx...@cn.ibm.com----- Original message -----
From: Brian Joiner <martinitime1...@gmail.com>
To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
Cc:
Subject: [xcat-user] New Service node cannot connect to Mariadb when starting xcatd
Date: Fri, Oct 19, 2018 5:45 AM
Hello,I have deployed a service node, to the best of my knowledge, using the online documentation. I've decided to rsync the install and tftpboot folders to deploy nodes directly from it. rpm -qa shows the service node rpm was appliedAttempting to start the xcatd daemon on the service node hangs, the errors being"Oct 18 16:22:18 service01 xcat[7109]: Could not connect to the database. Database handle not defined."headnode hostname: xcat (I know, super original) 10.4.0.1/24service node hostname: service01 10.4.0.2/24I have verified:Mariadb table setup is correctI can connect manually to the database from the service node and update tables (using xcatadmin account)DNS works/etc/xcat/cfgloc file exactly the same as head nodetcpdump on port 3006 (on head node) shows no activity when starting xcatd on service node, but does show activity on above mentioned manual test/etc/xCATSN exists[root@xcat ~]# lsxcatd -dcfgloc=mysql:dbname=xcatdb;host=10.4.0.1|xcatadmindbengine=mysqldbname=xcatdbdbhost=10.4.0.1dbadmin=xcatadminWhat am I missing? Let me know if you need to see my tables.--Brian Joiner_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user--Brian Joiner_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user--Brian Joiner_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user--Brian Joiner_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user--Brian Joiner_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user--Brian Joiner_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user--Brian Joiner_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
_______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
