RPC connection failed:
info = yield self._fetch_rpc_info(info_url)
twisted.web.error.Error: 500 INTERNAL SERVER ERROR
2017-01-19 12:15:59 -: [info] Received SIGTERM, shutting down.
2017-01-19 12:17:20 -: [info] twistd 16.0.0 (/usr/bin/python3 3.5.2) starting
up.
2017-01-19 12:17:20 -: [info] reactor class:
twisted.internet.epollreactor.EPollReactor.
2017-01-19 12:40:55 -: [info] Received SIGTERM, shutting down.
2017-01-19 12:42:27 -: [info] twistd 16.0.0 (/usr/bin/python3 3.5.2) starting
up.
2017-01-19 12:42:27 -: [info] reactor class:
twisted.internet.epollreactor.EPollReactor.
2017-01-19 12:42:28 provisioningserver.rpc.clusterservice: [info] Region not
available: Connection was refused by other side: 111: Connection refused.
(While requesting RPC info at b'http://[::ffff:192.168.199.2]:5240/MAAS/rpc/').
2017-01-19 12:42:29 provisioningserver.rpc.clusterservice: [info] Region not
available: Connection was refused by other side: 111: Connection refused.
(While requesting RPC info at b'http://[::ffff:192.168.199.2]:5240/MAAS/rpc/').
Port is alive:
root@juju-rack2:~# netstat -untap | grep 5240
tcp6 0 0 :::5240 :::* LISTEN
1235/python3
tcp6 0 0 :::5240 :::* LISTEN
1217/python3
tcp6 0 0 :::5240 :::* LISTEN
1203/python3
tcp6 0 0 :::5240 :::* LISTEN
1211/python3
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39136 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39184 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39172 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39308 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39234 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39246 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39010 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39096 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39210 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39222 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:38986 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39148 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39060 TIME_WAIT
-
tcp6 0 0 192.168.199.2:5240 192.168.199.100:53545 ESTABLISHED
1211/python3
tcp6 0 0 192.168.199.2:5240 192.168.199.2:39022 TIME_WAIT
-
...
MaaS RPC service appears to be down according to /var/log/maas/maas.log:
Jan 19 11:54:32 juju-rack2 maas.bootsources: [info] Updated boot sources cache.
Jan 19 11:54:32 juju-rack2 maas.bootresources: [info] Started importing of boot
images from 1 source(s).
Jan 19 11:54:32 juju-rack2 maas.import-images: [info] Downloading image
descriptions from http://images.maas.io/ephemeral-v3/daily/
Jan 19 11:54:43 juju-rack2 maas.bootresources: [info] Importing images from
source: http://images.maas.io/ephemeral-v3/daily/
Jan 19 11:55:26 juju-rack2 maas.service_monitor_service: [error] Can't update
service statuses, no RPC connection to region.
Checking the services:
root@juju-rack2:/var/log/maas# service maas-rackd status
● maas-rackd.service - MAAS Rack Controller
Loaded: loaded (/lib/systemd/system/maas-rackd.service; enabled; vendor
preset: enabled)
Active: active (running) since Thu 2017-01-19 12:41:56 EST; 33min ago
Docs: https://maas.io/
Process: 1249 ExecStartPre=/bin/rm -f /var/lib/maas/dhcpd6.conf (code=exited,
status=0/SUCCESS)
Process: 1243 ExecStartPre=/bin/rm -f /var/lib/maas/dhcpd.conf (code=exited,
status=0/SUCCESS)
Process: 1233 ExecStartPre=/bin/rm -f /var/lib/maas/dhcpd.sock (code=exited,
status=0/SUCCESS)
Main PID: 1255 (sh)
CGroup: /system.slice/maas-rackd.service
├─1255 /bin/sh -c exec /usr/bin/authbind --deep /usr/bin/twistd3
--nodaemon --pidfile= --logger=provisioningserver
├─1257 /usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-rackd
└─1258 tee -a /var/log/maas/rackd.log
Jan 19 13:14:58 juju-rack2.home.spyderdyne.net
maas.service_monitor_service[1257]: [error] Can't update service statuses, no
RPC conn
Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9120]: maas :
TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/systemctl status
Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9121]: maas :
TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/systemctl status
Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9120]:
pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9121]:
pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9122]: maas :
TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/bin/systemctl status
Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9122]:
pam_unix(sudo:session): session opened for user root by (uid=0)
Jan 19 13:15:27 juju-rack2.home.spyderdyne.net sudo[9121]:
pam_unix(sudo:session): session closed for user root
Jan 19 13:15:28 juju-rack2.home.spyderdyne.net sudo[9122]:
pam_unix(sudo:session): session closed for user root
Jan 19 13:15:28 juju-rack2.home.spyderdyne.net sudo[9120]:
pam_unix(sudo:session): session closed for user root
root@juju-rack2:/var/log/maas# ps aux | grep maas
maas 1200 0.0 0.0 1548 340 ? Ss 12:42 0:00 /bin/sh -c
exec twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-regiond 2>&1 | tee -a
$LOGFILE
maas 1202 0.0 0.0 1548 336 ? Ss 12:42 0:00 /bin/sh -c
exec twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-regiond 2>&1 | tee -a
$LOGFILE
maas 1203 7.9 7.4 299712 70540 ? Sl 12:42 2:53
/usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-regiond
maas 1204 0.0 0.0 3508 184 ? S 12:42 0:00 tee -a
/var/log/maas/regiond.log
maas 1205 0.0 0.0 1548 328 ? Ss 12:42 0:00 /bin/sh -c
exec twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-regiond 2>&1 | tee -a
$LOGFILE
maas 1211 13.9 7.9 205604 75444 ? Sl 12:42 5:03
/usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-regiond
maas 1212 0.0 0.0 3508 116 ? S 12:42 0:00 tee -a
/var/log/maas/regiond.log
maas 1217 6.3 6.7 135016 63520 ? Sl 12:42 2:19
/usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-regiond
maas 1218 0.0 0.0 3508 188 ? S 12:42 0:00 tee -a
/var/log/maas/regiond.log
maas 1229 0.0 0.0 1548 340 ? Ss 12:42 0:00 /bin/sh -c
exec twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-regiond 2>&1 | tee -a
$LOGFILE
maas 1235 6.2 6.7 134932 63724 ? Sl 12:42 2:16
/usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-regiond
maas 1236 0.0 0.0 3508 116 ? S 12:42 0:00 tee -a
/var/log/maas/regiond.log
maas 1255 0.0 0.0 1548 328 ? Ss 12:42 0:00 /bin/sh -c
exec /usr/bin/authbind --deep /usr/bin/twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-rackd 2>&1 | tee -a
$LOGFILE
maas 1257 3.3 5.1 97312 48692 ? Sl 12:42 1:14
/usr/bin/python3 /usr/bin/twistd3 --nodaemon --pidfile=
--logger=provisioningserver.logger.EventLogger maas-rackd
maas 1258 0.0 0.0 3508 124 ? S 12:42 0:00 tee -a
/var/log/maas/rackd.log
root 1297 0.0 0.3 41144 2860 ? Ss 12:42 0:00
/usr/sbin/squid -Y -f /var/lib/maas/maas-proxy.conf
proxy 1299 0.0 0.9 46508 9240 ? S 12:42 0:01 (squid-1) -Y
-f /var/lib/maas/maas-proxy.conf
postgres 1445 0.0 0.7 190732 7468 ? Ss 12:42 0:00 postgres: maas
maasdb 127.0.0.1(55174) idle
postgres 1451 0.0 0.7 190732 7540 ? Ss 12:42 0:00 postgres: maas
maasdb 127.0.0.1(55180) idle
postgres 1458 0.0 0.7 190732 7540 ? Ss 12:42 0:00 postgres: maas
maasdb 127.0.0.1(55186) idle
postgres 1560 0.0 0.7 190732 7560 ? Ss 12:42 0:00 postgres: maas
maasdb 127.0.0.1(55236) idle
root 9887 0.0 0.0 4244 564 pts/0 S+ 13:18 0:00 grep
--color=auto maas
postgres 9888 0.0 1.0 190912 10164 ? Ss 13:18 0:00 postgres: maas
maasdb 127.0.0.1(40692) idle in transaction
postgres 9889 0.0 1.0 190896 9708 ? Ss 13:18 0:00 postgres: maas
maasdb 127.0.0.1(40694) idle in transaction
I found a hostname mismatch in /var/log/maas/regiond.log:
django.core.exceptions.ValidationError: {'hostname': ['Nonexistant domain.']}
2017-01-19 11:47:10 twisted.python.log: [info] ::ffff:192.168.199.2 - -
[19/Jan/2017:16:47:10 +0000] "GET /MAAS/static/assets/images/logos/favicon.ico
HTTP/1.1" 200 15663
"http://192.168.199.2:5240/MAAS/accounts/login/?next=%2FMAAS%2F"; "Mozilla/5.0
(X11; Linux armv7l) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu
Chromium/55.0.2883.87 Chrome/55.0.2883.87 Safari/537.36"
2017-01-19 11:47:10 maasserver.rpc.regionservice: [critical] Failed to update
regiond's process and endpoints; juju-rack2.home.spyderdyne.net:pid=15414
record's may be out of date
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 1203, in
mainLoop
self.runUntilCurrent()
File "/usr/lib/python3/dist-packages/twisted/internet/base.py", line 798, in
runUntilCurrent
f(*a, **kw)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 434, in
errback
self._startRunCallbacks(fail)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in
_startRunCallbacks
self._runCallbacks()
--- <exception caught here> ---
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in
_runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/lib/python3/dist-packages/maasserver/rpc/regionservice.py", line
1021, in ignoreCancellation
failure.trap(defer.CancelledError)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 342, in
trap
self.raiseException()
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 368, in
raiseException
raise self.value.with_traceback(self.tb)
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246,
in inContext
result = inContext.theWork()
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262,
in <lambda>
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in
callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in
callWithContext
return func(*args,**kw)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py",
line 857, in callInContext
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py",
line 225, in wrapper
result = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/__init__.py", line 162,
in call_with_lock
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 603, in
call_within_transaction
return func_outside_txn(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/utils/orm.py", line 422, in
retrier
return func(*args, **kwargs)
File "/usr/lib/python3.5/contextlib.py", line 30, in inner
return func(*args, **kwds)
File "/usr/lib/python3/dist-packages/maasserver/rpc/regionservice.py", line
1306, in update
region_obj.save(update_fields=update_fields)
File "/usr/lib/python3/dist-packages/maasserver/models/node.py", line 1485,
in save
super(Node, self).save(*args, **kwargs)
File "/usr/lib/python3/dist-packages/maasserver/models/cleansave.py", line
28, in save
self.full_clean()
File "/usr/lib/python3/dist-packages/django/db/models/base.py", line 1171, in
full_clean
raise ValidationError(errors)
I attempted to rejoin the rack controller to the region:
root@juju-rack2:/var/log/maas# dpkg-reconfigure maas-rack-controller
root@juju-rack2:/var/log/maas# maas-rack register --url
http://192.168.199.2:5240/MAAS --secret <munged>
It still didn't show up, so I deleted it and attempted to rejoin again.
No luck. I believe that this issue may have been caused by renaming the
instance AFTER installing MAAS initially. Purging and reinstalling to
see if I can recreate the issue a second time without injecting any more
variables into the scenario.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1560971
Title:
after default installation of maas the rack and region controller are
not connected
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/maas/+bug/1560971/+subscriptions
--
ubuntu-bugs mailing list
[email protected]
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
