Somebody else on this list may be better able to answer that. My tool of choice would be tcpdump - monitor all traffic between 10.0.4.1 and 10.0.4.2 (which may be quite a bit), then try to connect again, and see if you can tease out the related packets.
Another possibility is to try and use netstat immediately after you try to connect, and see if there are any connections in TIME_WAIT or CLOSE_WAIT or one of the FIN_WAIT statuses. But that is probably a matter of luck catching it. _______________________________________________________________________ Kevin Keane | Systems Architect | University of San Diego ITS | kke...@sandiego.edu Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 | 619.260.6859 *REMEMBER! **No one from IT at USD will ever ask to confirm or supply your password*. These messages are an attempt to steal your username and password. Please do not reply to, click the links within, or open the attachments of these messages. Delete them! On Tue, Oct 23, 2018 at 11:39 AM Brian Joiner <martinitime1...@gmail.com> wrote: > Ok, all of those commands from the service node work fine. It connects, > switches to xcatdb and lists all the tables without any error. > > You're correct, the listening port for Mysql on the head node is 3306, > must have fat fingered that one. Is there a way to see if the xcatd > service is trying to connect to the right port? > > On Tue, Oct 23, 2018 at 12:06 PM Kevin Keane <kke...@sandiego.edu> wrote: > >> Just to double-check: when you log in to the DB from the service node, >> you can not only access MariaDB, but it will now let you see the xcatdb as >> well? Try this on the service node: >> >> mysql -h 10.4.0.1 -u xcatadmin -D xcatdb -p >> >> And then >> >> show databases; >> use xcatdb; >> show tables; >> >> The first should show you mysql and xcatdb (maybe additional databases) >> >> The second should complete without error. This is the command you use to >> actually access a database. >> >> The third one shows you all the tables - that's just to verify that the >> use command didn't simply silently fail. >> >> One other thing I noticed in your initial post: at one point you >> mentioned port 3006, but MySQL usually listens on 3306. And the fact that >> the mysql command on the command line can connect without you specifying a >> port number suggests that MariaDB on the management node really uses the >> default value. >> >> So double-check whether MariaDB really listens on 3306 on the MN (it >> should), and then check that there is no typo in any of the configuration >> files. >> >> ______________________________________________________________________ >> >> Kevin Keane | Systems Architect | University of San Diego ITS | >> kke...@sandiego.edu >> Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 | >> 619.260.6859 >> >> *REMEMBER! **No one from IT at USD will ever ask to confirm or supply >> your password*. >> These messages are an attempt to steal your username and password. Please >> do not reply to, click the links within, or open the attachments of these >> messages. Delete them! >> >> >> >> >> On Tue, Oct 23, 2018 at 9:29 AM Brian Joiner <martinitime1...@gmail.com> >> wrote: >> >>> Kevin, >>> >>> I logged in to mysql as root user and ran the command you provided. It >>> took and heres the output of show grants (from the head node) >>> >>> MariaDB [(none)]> show grants for xcatadmin@10.4.0.2; >>> +--------------------------------------------------------------+ >>> | Grants for xcatadmin@10.4.0.2 | >>> +--------------------------------------------------------------+ >>> | GRANT USAGE ON *.* TO 'xcatadmin'@'10.4.0.2' | >>> | GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.4.0.2' | >>> +--------------------------------------------------------------+ >>> 2 rows in set (0.00 sec) >>> >>> When trying to login to the database from the service node, the password >>> wasn't working anymore, so I re-entered the command adding 'IDENTIFIED BY >>> '<password>' to the end and that allowed me to connect as xcatadmin from >>> the servicenode manually. However, starting up the service still fails >>> with the same error: Oct 23 11:15:31 service01 xcat[18341]: Could not >>> connect to the database. Database handle not defined. >>> >>> >>> On Mon, Oct 22, 2018 at 7:31 PM Kevin Keane <kke...@sandiego.edu> wrote: >>> >>>> That just says that any computer on your network is allowed to access >>>> MariaDB itself (which explains why you can log on), but you do not have any >>>> privileges on the xcatdb database. >>>> >>>> You need to see a line like this as well: >>>> GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.4.0.2' >>>> >>>> In fact, this is actually the exact command you are going to need to >>>> type to grant the required permission (you probably need to be the MariaDB >>>> root user, and probably on the management node, to do this). >>>> >>>> From a security standpoint, I would actually recommend removing the >>>> GRANT USAGE for anything with a %, and instead only allow specific IPs, but >>>> I'm not sure if xcat likes that. >>>> >>>> _______________________________________________________________________ >>>> Kevin Keane | Systems Architect | University of San Diego ITS | >>>> kke...@sandiego.edu >>>> Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 | >>>> 619.260.6859 >>>> >>>> *REMEMBER! **No one from IT at USD will ever ask to confirm or supply >>>> your password*. >>>> These messages are an attempt to steal your username and password. >>>> Please do not reply to, click the links within, or open the attachments of >>>> these messages. Delete them! >>>> >>>> >>>> >>>> >>>> On Mon, Oct 22, 2018 at 5:15 PM Brian Joiner <martinitime1...@gmail.com> >>>> wrote: >>>> >>>>> The plot thickens! >>>>> >>>>> MariaDB [xcatdb]> show grants for xcatadmin@10.4.0.2; >>>>> ERROR 1141 (42000): There is no such grant defined for user >>>>> 'xcatadmin' on host '10.4.0.2' >>>>> >>>>> But shouldn't this cover it: >>>>> MariaDB [xcatdb]> show grants for xcatadmin >>>>> -> ; >>>>> >>>>> +----------------------------------------------------------------------------------------------------------+ >>>>> | Grants for xcatadmin@% >>>>> | >>>>> >>>>> +----------------------------------------------------------------------------------------------------------+ >>>>> | GRANT USAGE ON *.* TO 'xcatadmin'@'%' IDENTIFIED BY PASSWORD >>>>> '*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19' | >>>>> >>>>> +----------------------------------------------------------------------------------------------------------+ >>>>> 1 row in set (0.00 sec) >>>>> >>>>> >>>>> On Mon, Oct 22, 2018 at 5:21 PM Kevin Keane <kke...@sandiego.edu> >>>>> wrote: >>>>> >>>>>> I might also try "show grants for xcatadmin@10.4.0.2" - the show >>>>>> grants you used will only tell you what permissions the MN has, but you >>>>>> are >>>>>> interested in the service node. >>>>>> >>>>>> >>>>>> _______________________________________________________________________ >>>>>> Kevin Keane | Systems Architect | University of San Diego ITS | >>>>>> kke...@sandiego.edu >>>>>> Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 | >>>>>> 619.260.6859 >>>>>> >>>>>> *REMEMBER! **No one from IT at USD will ever ask to confirm or >>>>>> supply your password*. >>>>>> These messages are an attempt to steal your username and password. >>>>>> Please do not reply to, click the links within, or open the attachments >>>>>> of >>>>>> these messages. Delete them! >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Mon, Oct 22, 2018 at 3:06 PM Brian Joiner < >>>>>> martinitime1...@gmail.com> wrote: >>>>>> >>>>>>> It looks like everything is setup correctly: >>>>>>> >>>>>>> >>>>>>> [root@service01 ~]# mysql -h 10.4.0.1 -u xcatadmin -D xcatdb -p >>>>>>> Enter password: >>>>>>> Reading table information for completion of table and column names >>>>>>> You can turn off this feature to get a quicker startup with -A >>>>>>> >>>>>>> Welcome to the MariaDB monitor. Commands end with ; or \g. >>>>>>> Your MariaDB connection id is 5 >>>>>>> Server version: 5.5.56-MariaDB MariaDB Server >>>>>>> >>>>>>> Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others. >>>>>>> >>>>>>> Type 'help;' or '\h' for help. Type '\c' to clear the current input >>>>>>> statement. >>>>>>> >>>>>>> MariaDB [xcatdb]> show grants for xcatadmin@10.4.0.1 >>>>>>> -> ; >>>>>>> >>>>>>> +-----------------------------------------------------------------------------------------------------------------+ >>>>>>> | Grants for xcatadmin@10.4.0.1 >>>>>>> | >>>>>>> >>>>>>> +-----------------------------------------------------------------------------------------------------------------+ >>>>>>> | GRANT USAGE ON *.* TO 'xcatadmin'@'10.4.0.1' IDENTIFIED BY >>>>>>> PASSWORD '*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19' | >>>>>>> | GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.4.0.1' >>>>>>> | >>>>>>> >>>>>>> +-----------------------------------------------------------------------------------------------------------------+ >>>>>>> 2 rows in set (0.00 sec) >>>>>>> >>>>>>> MariaDB [xcatdb]> >>>>>>> >>>>>>> On Sun, Oct 21, 2018 at 10:40 PM Bin XA Xu <bx...@cn.ibm.com> wrote: >>>>>>> >>>>>>>> Okay, it means you cannot access your mariadb server on MN from >>>>>>>> this service node. >>>>>>>> You can use `mysql -h 10.3.5.20 -u xcatadmin -D xcatdb -p` to >>>>>>>> verify it. >>>>>>>> >>>>>>>> Are you configure your mariadb well to GRANT your service node IPs? >>>>>>>> ( >>>>>>>> https://xcat-docs.readthedocs.io/en/stable/advanced/hierarchy/databases/mysql_configure.html?highlight=mysqlsetup >>>>>>>> ) >>>>>>>> >>>>>>>> You can use below command in mariadb shell to query the GRANT: >>>>>>>> ``` >>>>>>>> show grants for xcatadmin@10.3.5.20; >>>>>>>> >>>>>>>> +------------------------------------------------------------------------------------------------------------------+ >>>>>>>> | Grants for xcatadmin@10.3.5.20 >>>>>>>> | >>>>>>>> >>>>>>>> +------------------------------------------------------------------------------------------------------------------+ >>>>>>>> | GRANT USAGE ON *.* TO 'xcatadmin'@'10.3.5.20' IDENTIFIED BY >>>>>>>> PASSWORD '*359860933C260AB0F1316209DC2846250BAD59E3' | >>>>>>>> | GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.3.5.20' >>>>>>>> | >>>>>>>> >>>>>>>> +------------------------------------------------------------------------------------------------------------------+ >>>>>>>> 2 rows in set (0.00 sec) >>>>>>>> ``` >>>>>>>> >>>>>>>> You can use `mysqlsetup -u -f <clientip_list filename>` to add >>>>>>>> your service nodes, you just follow the mysql steps to do that. >>>>>>>> >>>>>>>> Bin Xu >>>>>>>> HPC Software Development >>>>>>>> Software Defined Infrastructure, IBM Systems >>>>>>>> Phone: 86-010-82454067 >>>>>>>> E-mail: bx...@cn.ibm.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ----- Original message ----- >>>>>>>> From: Brian Joiner <martinitime1...@gmail.com> >>>>>>>> To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net> >>>>>>>> Cc: >>>>>>>> Subject: Re: [xcat-user] New Service node cannot connect to Mariadb >>>>>>>> when starting xcatd >>>>>>>> Date: Fri, Oct 19, 2018 11:27 PM >>>>>>>> >>>>>>>> Bin, heres the output: >>>>>>>> >>>>>>>> [root@service01 ~]# XCATBYPASS=1 lsxcatd -a >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> Could not read the site table >>>>>>>> >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> Could not read the site table >>>>>>>> >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> Could not read the site table >>>>>>>> >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> Could not read the site table >>>>>>>> >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> Could not read the site table >>>>>>>> >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> Could not read the site table >>>>>>>> >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> Could not read the site table >>>>>>>> >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> Could not read the site table >>>>>>>> >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> Could not read the site table >>>>>>>> >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> Error loading module /opt/xcat/lib/perl/xCAT_plugin/zvmdiscovery.pm >>>>>>>> ...skipping >>>>>>>> Could not connect to the database. Database handle not defined. >>>>>>>> ERROR: Unable to open basic site table for configuration >>>>>>>> Version 2.14.1 (git commit >>>>>>>> 70d6e7f93cc9714a127c22df2e7ca53d4996a34c, built Fri Jun 1 03:00:53 EDT >>>>>>>> 2018) >>>>>>>> This is a Service Node >>>>>>>> cfgloc=mysql:dbname=xcatdb;host=10.4.0.1|xcatadmin >>>>>>>> dbengine=mysql >>>>>>>> dbname=xcatdb >>>>>>>> dbhost=10.4.0.1 >>>>>>>> dbadmin=xcatadmin >>>>>>>> >>>>>>>> >>>>>>>> On Thu, Oct 18, 2018 at 9:34 PM Bin XA Xu <bx...@cn.ibm.com> wrote: >>>>>>>> >>>>>>>> could you run `XCATBYPASS=1 lsxcatd -a` on your service node and >>>>>>>> paste the output here? >>>>>>>> >>>>>>>> Bin Xu >>>>>>>> HPC Software Development >>>>>>>> Software Defined Infrastructure, IBM Systems >>>>>>>> Phone: 86-010-82454067 >>>>>>>> E-mail: bx...@cn.ibm.com >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> ----- Original message ----- >>>>>>>> From: Brian Joiner <martinitime1...@gmail.com> >>>>>>>> To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net> >>>>>>>> Cc: >>>>>>>> Subject: [xcat-user] New Service node cannot connect to Mariadb >>>>>>>> when starting xcatd >>>>>>>> Date: Fri, Oct 19, 2018 5:45 AM >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> I have deployed a service node, to the best of my knowledge, using >>>>>>>> the online documentation. I've decided to rsync the install and >>>>>>>> tftpboot >>>>>>>> folders to deploy nodes directly from it. rpm -qa shows the service >>>>>>>> node >>>>>>>> rpm was applied >>>>>>>> >>>>>>>> Attempting to start the xcatd daemon on the service node hangs, the >>>>>>>> errors being >>>>>>>> >>>>>>>> "Oct 18 16:22:18 service01 xcat[7109]: Could not connect to the >>>>>>>> database. Database handle not defined." >>>>>>>> >>>>>>>> headnode hostname: xcat (I know, super original) 10.4.0.1/24 >>>>>>>> service node hostname: service01 10.4.0.2/24 >>>>>>>> >>>>>>>> I have verified: >>>>>>>> Mariadb table setup is correct >>>>>>>> I can connect manually to the database from the service node and >>>>>>>> update tables (using xcatadmin account) >>>>>>>> DNS works >>>>>>>> /etc/xcat/cfgloc file exactly the same as head node >>>>>>>> tcpdump on port 3006 (on head node) shows no activity when starting >>>>>>>> xcatd on service node, but does show activity on above mentioned manual >>>>>>>> test >>>>>>>> /etc/xCATSN exists >>>>>>>> >>>>>>>> [root@xcat ~]# lsxcatd -d >>>>>>>> cfgloc=mysql:dbname=xcatdb;host=10.4.0.1|xcatadmin >>>>>>>> dbengine=mysql >>>>>>>> dbname=xcatdb >>>>>>>> dbhost=10.4.0.1 >>>>>>>> dbadmin=xcatadmin >>>>>>>> >>>>>>>> >>>>>>>> What am I missing? Let me know if you need to see my tables. >>>>>>>> -- >>>>>>>> Brian Joiner >>>>>>>> _______________________________________________ >>>>>>>> xCAT-user mailing list >>>>>>>> xCAT-user@lists.sourceforge.net >>>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> xCAT-user mailing list >>>>>>>> xCAT-user@lists.sourceforge.net >>>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Brian Joiner >>>>>>>> _______________________________________________ >>>>>>>> xCAT-user mailing list >>>>>>>> xCAT-user@lists.sourceforge.net >>>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> xCAT-user mailing list >>>>>>>> xCAT-user@lists.sourceforge.net >>>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Brian Joiner >>>>>>> _______________________________________________ >>>>>>> xCAT-user mailing list >>>>>>> xCAT-user@lists.sourceforge.net >>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>>>>>> >>>>>> _______________________________________________ >>>>>> xCAT-user mailing list >>>>>> xCAT-user@lists.sourceforge.net >>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>>>>> >>>>> >>>>> >>>>> -- >>>>> Brian Joiner >>>>> _______________________________________________ >>>>> xCAT-user mailing list >>>>> xCAT-user@lists.sourceforge.net >>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>>>> >>>> _______________________________________________ >>>> xCAT-user mailing list >>>> xCAT-user@lists.sourceforge.net >>>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>>> >>> >>> >>> -- >>> Brian Joiner >>> _______________________________________________ >>> xCAT-user mailing list >>> xCAT-user@lists.sourceforge.net >>> https://lists.sourceforge.net/lists/listinfo/xcat-user >>> >> _______________________________________________ >> xCAT-user mailing list >> xCAT-user@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/xcat-user >> > > > -- > Brian Joiner > _______________________________________________ > xCAT-user mailing list > xCAT-user@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/xcat-user >
_______________________________________________ xCAT-user mailing list xCAT-user@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/xcat-user
