My Coworker Calvin found the solution! There was a missing package he
noticed in the strace output for xcatd. There was a file mentioned over
and over, he did a lookup for the package that provides the file
(stat("/usr/local/share/perl5/DBD/mysql.pmc", 0x7ffcec9af410) = -1 ENOENT
(No such file or directory)
(stat("/usr/local/share/perl5/DBD/mysql.pm", 0x7ffcec9af350) = -1 ENOENT
(No such file or directory)
The missing package was: perl-DBD-MySQL
Once I installed that, xcatd started and I can run lsdef and other xCAT
commands. If I missed this from the documentation, a thousand pardons :)
However, if this is a bug with the service node package list, it needs to
be remedied by the devs.
Thanks everyone for all your help!
On Tue, Oct 23, 2018 at 3:14 PM Kevin Keane <kke...@sandiego.edu> wrote:
> Somebody else on this list may be better able to answer that. My tool of
> choice would be tcpdump - monitor all traffic between 10.0.4.1 and 10.0.4.2
> (which may be quite a bit), then try to connect again, and see if you can
> tease out the related packets.
>
> Another possibility is to try and use netstat immediately after you try to
> connect, and see if there are any connections in TIME_WAIT or CLOSE_WAIT or
> one of the FIN_WAIT statuses. But that is probably a matter of luck
> catching it.
>
> _______________________________________________________________________
> Kevin Keane | Systems Architect | University of San Diego ITS |
> kke...@sandiego.edu
> Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 |
> 619.260.6859
>
> *REMEMBER! **No one from IT at USD will ever ask to confirm or supply
> your password*.
> These messages are an attempt to steal your username and password. Please
> do not reply to, click the links within, or open the attachments of these
> messages. Delete them!
>
>
>
>
> On Tue, Oct 23, 2018 at 11:39 AM Brian Joiner <martinitime1...@gmail.com>
> wrote:
>
>> Ok, all of those commands from the service node work fine. It connects,
>> switches to xcatdb and lists all the tables without any error.
>>
>> You're correct, the listening port for Mysql on the head node is 3306,
>> must have fat fingered that one. Is there a way to see if the xcatd
>> service is trying to connect to the right port?
>>
>> On Tue, Oct 23, 2018 at 12:06 PM Kevin Keane <kke...@sandiego.edu> wrote:
>>
>>> Just to double-check: when you log in to the DB from the service node,
>>> you can not only access MariaDB, but it will now let you see the xcatdb as
>>> well? Try this on the service node:
>>>
>>> mysql -h 10.4.0.1 -u xcatadmin -D xcatdb -p
>>>
>>> And then
>>>
>>> show databases;
>>> use xcatdb;
>>> show tables;
>>>
>>> The first should show you mysql and xcatdb (maybe additional databases)
>>>
>>> The second should complete without error. This is the command you use to
>>> actually access a database.
>>>
>>> The third one shows you all the tables - that's just to verify that the
>>> use command didn't simply silently fail.
>>>
>>> One other thing I noticed in your initial post: at one point you
>>> mentioned port 3006, but MySQL usually listens on 3306. And the fact that
>>> the mysql command on the command line can connect without you specifying a
>>> port number suggests that MariaDB on the management node really uses the
>>> default value.
>>>
>>> So double-check whether MariaDB really listens on 3306 on the MN (it
>>> should), and then check that there is no typo in any of the configuration
>>> files.
>>>
>>> ______________________________________________________________________
>>>
>>> Kevin Keane | Systems Architect | University of San Diego ITS |
>>> kke...@sandiego.edu
>>> Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 |
>>> 619.260.6859
>>>
>>> *REMEMBER! **No one from IT at USD will ever ask to confirm or supply
>>> your password*.
>>> These messages are an attempt to steal your username and password.
>>> Please do not reply to, click the links within, or open the attachments of
>>> these messages. Delete them!
>>>
>>>
>>>
>>>
>>> On Tue, Oct 23, 2018 at 9:29 AM Brian Joiner <martinitime1...@gmail.com>
>>> wrote:
>>>
>>>> Kevin,
>>>>
>>>> I logged in to mysql as root user and ran the command you provided. It
>>>> took and heres the output of show grants (from the head node)
>>>>
>>>> MariaDB [(none)]> show grants for xcatadmin@10.4.0.2;
>>>> +--------------------------------------------------------------+
>>>> | Grants for xcatadmin@10.4.0.2 |
>>>> +--------------------------------------------------------------+
>>>> | GRANT USAGE ON *.* TO 'xcatadmin'@'10.4.0.2' |
>>>> | GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.4.0.2' |
>>>> +--------------------------------------------------------------+
>>>> 2 rows in set (0.00 sec)
>>>>
>>>> When trying to login to the database from the service node, the
>>>> password wasn't working anymore, so I re-entered the command adding
>>>> 'IDENTIFIED BY '<password>' to the end and that allowed me to connect as
>>>> xcatadmin from the servicenode manually. However, starting up the service
>>>> still fails with the same error: Oct 23 11:15:31 service01 xcat[18341]:
>>>> Could not connect to the database. Database handle not defined.
>>>>
>>>>
>>>> On Mon, Oct 22, 2018 at 7:31 PM Kevin Keane <kke...@sandiego.edu>
>>>> wrote:
>>>>
>>>>> That just says that any computer on your network is allowed to access
>>>>> MariaDB itself (which explains why you can log on), but you do not have
>>>>> any
>>>>> privileges on the xcatdb database.
>>>>>
>>>>> You need to see a line like this as well:
>>>>> GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.4.0.2'
>>>>>
>>>>> In fact, this is actually the exact command you are going to need to
>>>>> type to grant the required permission (you probably need to be the MariaDB
>>>>> root user, and probably on the management node, to do this).
>>>>>
>>>>> From a security standpoint, I would actually recommend removing the
>>>>> GRANT USAGE for anything with a %, and instead only allow specific IPs,
>>>>> but
>>>>> I'm not sure if xcat likes that.
>>>>>
>>>>> _______________________________________________________________________
>>>>> Kevin Keane | Systems Architect | University of San Diego ITS |
>>>>> kke...@sandiego.edu
>>>>> Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 |
>>>>> 619.260.6859
>>>>>
>>>>> *REMEMBER! **No one from IT at USD will ever ask to confirm or supply
>>>>> your password*.
>>>>> These messages are an attempt to steal your username and password.
>>>>> Please do not reply to, click the links within, or open the attachments of
>>>>> these messages. Delete them!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Mon, Oct 22, 2018 at 5:15 PM Brian Joiner <
>>>>> martinitime1...@gmail.com> wrote:
>>>>>
>>>>>> The plot thickens!
>>>>>>
>>>>>> MariaDB [xcatdb]> show grants for xcatadmin@10.4.0.2;
>>>>>> ERROR 1141 (42000): There is no such grant defined for user
>>>>>> 'xcatadmin' on host '10.4.0.2'
>>>>>>
>>>>>> But shouldn't this cover it:
>>>>>> MariaDB [xcatdb]> show grants for xcatadmin
>>>>>> -> ;
>>>>>>
>>>>>> +----------------------------------------------------------------------------------------------------------+
>>>>>> | Grants for xcatadmin@%
>>>>>> |
>>>>>>
>>>>>> +----------------------------------------------------------------------------------------------------------+
>>>>>> | GRANT USAGE ON *.* TO 'xcatadmin'@'%' IDENTIFIED BY PASSWORD
>>>>>> '*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19' |
>>>>>>
>>>>>> +----------------------------------------------------------------------------------------------------------+
>>>>>> 1 row in set (0.00 sec)
>>>>>>
>>>>>>
>>>>>> On Mon, Oct 22, 2018 at 5:21 PM Kevin Keane <kke...@sandiego.edu>
>>>>>> wrote:
>>>>>>
>>>>>>> I might also try "show grants for xcatadmin@10.4.0.2" - the show
>>>>>>> grants you used will only tell you what permissions the MN has, but you
>>>>>>> are
>>>>>>> interested in the service node.
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________________________________
>>>>>>> Kevin Keane | Systems Architect | University of San Diego ITS |
>>>>>>> kke...@sandiego.edu
>>>>>>> Maher Hall, 192 |5998 Alcalá Park | San Diego, CA 92110-2492 |
>>>>>>> 619.260.6859
>>>>>>>
>>>>>>> *REMEMBER! **No one from IT at USD will ever ask to confirm or
>>>>>>> supply your password*.
>>>>>>> These messages are an attempt to steal your username and password.
>>>>>>> Please do not reply to, click the links within, or open the attachments
>>>>>>> of
>>>>>>> these messages. Delete them!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 22, 2018 at 3:06 PM Brian Joiner <
>>>>>>> martinitime1...@gmail.com> wrote:
>>>>>>>
>>>>>>>> It looks like everything is setup correctly:
>>>>>>>>
>>>>>>>>
>>>>>>>> [root@service01 ~]# mysql -h 10.4.0.1 -u xcatadmin -D xcatdb -p
>>>>>>>> Enter password:
>>>>>>>> Reading table information for completion of table and column names
>>>>>>>> You can turn off this feature to get a quicker startup with -A
>>>>>>>>
>>>>>>>> Welcome to the MariaDB monitor. Commands end with ; or \g.
>>>>>>>> Your MariaDB connection id is 5
>>>>>>>> Server version: 5.5.56-MariaDB MariaDB Server
>>>>>>>>
>>>>>>>> Copyright (c) 2000, 2017, Oracle, MariaDB Corporation Ab and others.
>>>>>>>>
>>>>>>>> Type 'help;' or '\h' for help. Type '\c' to clear the current input
>>>>>>>> statement.
>>>>>>>>
>>>>>>>> MariaDB [xcatdb]> show grants for xcatadmin@10.4.0.1
>>>>>>>> -> ;
>>>>>>>>
>>>>>>>> +-----------------------------------------------------------------------------------------------------------------+
>>>>>>>> | Grants for xcatadmin@10.4.0.1
>>>>>>>> |
>>>>>>>>
>>>>>>>> +-----------------------------------------------------------------------------------------------------------------+
>>>>>>>> | GRANT USAGE ON *.* TO 'xcatadmin'@'10.4.0.1' IDENTIFIED BY
>>>>>>>> PASSWORD '*2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19' |
>>>>>>>> | GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.4.0.1'
>>>>>>>> |
>>>>>>>>
>>>>>>>> +-----------------------------------------------------------------------------------------------------------------+
>>>>>>>> 2 rows in set (0.00 sec)
>>>>>>>>
>>>>>>>> MariaDB [xcatdb]>
>>>>>>>>
>>>>>>>> On Sun, Oct 21, 2018 at 10:40 PM Bin XA Xu <bx...@cn.ibm.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Okay, it means you cannot access your mariadb server on MN from
>>>>>>>>> this service node.
>>>>>>>>> You can use `mysql -h 10.3.5.20 -u xcatadmin -D xcatdb -p` to
>>>>>>>>> verify it.
>>>>>>>>>
>>>>>>>>> Are you configure your mariadb well to GRANT your service node
>>>>>>>>> IPs? (
>>>>>>>>> https://xcat-docs.readthedocs.io/en/stable/advanced/hierarchy/databases/mysql_configure.html?highlight=mysqlsetup
>>>>>>>>> )
>>>>>>>>>
>>>>>>>>> You can use below command in mariadb shell to query the GRANT:
>>>>>>>>> ```
>>>>>>>>> show grants for xcatadmin@10.3.5.20;
>>>>>>>>>
>>>>>>>>> +------------------------------------------------------------------------------------------------------------------+
>>>>>>>>> | Grants for xcatadmin@10.3.5.20
>>>>>>>>> |
>>>>>>>>>
>>>>>>>>> +------------------------------------------------------------------------------------------------------------------+
>>>>>>>>> | GRANT USAGE ON *.* TO 'xcatadmin'@'10.3.5.20' IDENTIFIED BY
>>>>>>>>> PASSWORD '*359860933C260AB0F1316209DC2846250BAD59E3' |
>>>>>>>>> | GRANT ALL PRIVILEGES ON `xcatdb`.* TO 'xcatadmin'@'10.3.5.20'
>>>>>>>>> |
>>>>>>>>>
>>>>>>>>> +------------------------------------------------------------------------------------------------------------------+
>>>>>>>>> 2 rows in set (0.00 sec)
>>>>>>>>> ```
>>>>>>>>>
>>>>>>>>> You can use `mysqlsetup -u -f <clientip_list filename>` to add
>>>>>>>>> your service nodes, you just follow the mysql steps to do that.
>>>>>>>>>
>>>>>>>>> Bin Xu
>>>>>>>>> HPC Software Development
>>>>>>>>> Software Defined Infrastructure, IBM Systems
>>>>>>>>> Phone: 86-010-82454067
>>>>>>>>> E-mail: bx...@cn.ibm.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original message -----
>>>>>>>>> From: Brian Joiner <martinitime1...@gmail.com>
>>>>>>>>> To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
>>>>>>>>> Cc:
>>>>>>>>> Subject: Re: [xcat-user] New Service node cannot connect to
>>>>>>>>> Mariadb when starting xcatd
>>>>>>>>> Date: Fri, Oct 19, 2018 11:27 PM
>>>>>>>>>
>>>>>>>>> Bin, heres the output:
>>>>>>>>>
>>>>>>>>> [root@service01 ~]# XCATBYPASS=1 lsxcatd -a
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> Could not read the site table
>>>>>>>>>
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> Could not read the site table
>>>>>>>>>
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> Could not read the site table
>>>>>>>>>
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> Could not read the site table
>>>>>>>>>
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> Could not read the site table
>>>>>>>>>
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> Could not read the site table
>>>>>>>>>
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> Could not read the site table
>>>>>>>>>
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> Could not read the site table
>>>>>>>>>
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> Could not read the site table
>>>>>>>>>
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> Error loading module /opt/xcat/lib/perl/xCAT_plugin/
>>>>>>>>> zvmdiscovery.pm ...skipping
>>>>>>>>> Could not connect to the database. Database handle not defined.
>>>>>>>>> ERROR: Unable to open basic site table for configuration
>>>>>>>>> Version 2.14.1 (git commit
>>>>>>>>> 70d6e7f93cc9714a127c22df2e7ca53d4996a34c, built Fri Jun 1 03:00:53
>>>>>>>>> EDT
>>>>>>>>> 2018)
>>>>>>>>> This is a Service Node
>>>>>>>>> cfgloc=mysql:dbname=xcatdb;host=10.4.0.1|xcatadmin
>>>>>>>>> dbengine=mysql
>>>>>>>>> dbname=xcatdb
>>>>>>>>> dbhost=10.4.0.1
>>>>>>>>> dbadmin=xcatadmin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Oct 18, 2018 at 9:34 PM Bin XA Xu <bx...@cn.ibm.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> could you run `XCATBYPASS=1 lsxcatd -a` on your service node and
>>>>>>>>> paste the output here?
>>>>>>>>>
>>>>>>>>> Bin Xu
>>>>>>>>> HPC Software Development
>>>>>>>>> Software Defined Infrastructure, IBM Systems
>>>>>>>>> Phone: 86-010-82454067
>>>>>>>>> E-mail: bx...@cn.ibm.com
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ----- Original message -----
>>>>>>>>> From: Brian Joiner <martinitime1...@gmail.com>
>>>>>>>>> To: xCAT Users Mailing list <xcat-user@lists.sourceforge.net>
>>>>>>>>> Cc:
>>>>>>>>> Subject: [xcat-user] New Service node cannot connect to Mariadb
>>>>>>>>> when starting xcatd
>>>>>>>>> Date: Fri, Oct 19, 2018 5:45 AM
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I have deployed a service node, to the best of my knowledge, using
>>>>>>>>> the online documentation. I've decided to rsync the install and
>>>>>>>>> tftpboot
>>>>>>>>> folders to deploy nodes directly from it. rpm -qa shows the service
>>>>>>>>> node
>>>>>>>>> rpm was applied
>>>>>>>>>
>>>>>>>>> Attempting to start the xcatd daemon on the service node hangs,
>>>>>>>>> the errors being
>>>>>>>>>
>>>>>>>>> "Oct 18 16:22:18 service01 xcat[7109]: Could not connect to the
>>>>>>>>> database. Database handle not defined."
>>>>>>>>>
>>>>>>>>> headnode hostname: xcat (I know, super original) 10.4.0.1/24
>>>>>>>>> service node hostname: service01 10.4.0.2/24
>>>>>>>>>
>>>>>>>>> I have verified:
>>>>>>>>> Mariadb table setup is correct
>>>>>>>>> I can connect manually to the database from the service node and
>>>>>>>>> update tables (using xcatadmin account)
>>>>>>>>> DNS works
>>>>>>>>> /etc/xcat/cfgloc file exactly the same as head node
>>>>>>>>> tcpdump on port 3006 (on head node) shows no activity when
>>>>>>>>> starting xcatd on service node, but does show activity on above
>>>>>>>>> mentioned
>>>>>>>>> manual test
>>>>>>>>> /etc/xCATSN exists
>>>>>>>>>
>>>>>>>>> [root@xcat ~]# lsxcatd -d
>>>>>>>>> cfgloc=mysql:dbname=xcatdb;host=10.4.0.1|xcatadmin
>>>>>>>>> dbengine=mysql
>>>>>>>>> dbname=xcatdb
>>>>>>>>> dbhost=10.4.0.1
>>>>>>>>> dbadmin=xcatadmin
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> What am I missing? Let me know if you need to see my tables.
>>>>>>>>> --
>>>>>>>>> Brian Joiner
>>>>>>>>> _______________________________________________
>>>>>>>>> xCAT-user mailing list
>>>>>>>>> xCAT-user@lists.sourceforge.net
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> xCAT-user mailing list
>>>>>>>>> xCAT-user@lists.sourceforge.net
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Brian Joiner
>>>>>>>>> _______________________________________________
>>>>>>>>> xCAT-user mailing list
>>>>>>>>> xCAT-user@lists.sourceforge.net
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> xCAT-user mailing list
>>>>>>>>> xCAT-user@lists.sourceforge.net
>>>>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Brian Joiner
>>>>>>>> _______________________________________________
>>>>>>>> xCAT-user mailing list
>>>>>>>> xCAT-user@lists.sourceforge.net
>>>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> xCAT-user mailing list
>>>>>>> xCAT-user@lists.sourceforge.net
>>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Brian Joiner
>>>>>> _______________________________________________
>>>>>> xCAT-user mailing list
>>>>>> xCAT-user@lists.sourceforge.net
>>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>>>>
>>>>> _______________________________________________
>>>>> xCAT-user mailing list
>>>>> xCAT-user@lists.sourceforge.net
>>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>>>
>>>>
>>>>
>>>> --
>>>> Brian Joiner
>>>> _______________________________________________
>>>> xCAT-user mailing list
>>>> xCAT-user@lists.sourceforge.net
>>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>>
>>> _______________________________________________
>>> xCAT-user mailing list
>>> xCAT-user@lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>>
>>
>>
>> --
>> Brian Joiner
>> _______________________________________________
>> xCAT-user mailing list
>> xCAT-user@lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/xcat-user
>>
> _______________________________________________
> xCAT-user mailing list
> xCAT-user@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/xcat-user
>
--
Brian Joiner
_______________________________________________
xCAT-user mailing list
xCAT-user@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/xcat-user
