[Bug 1389811] Re: tgtadm: out of memory crash
[Expired for tgt (Ubuntu) because there has been no activity for 60 days.] ** Changed in: tgt (Ubuntu) Status: Incomplete => Expired -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
We're catching up on old bugs. I see that this one is marked Invalid for MAAS, but I don't see any steps to reproduce specifically for tgt on its own. Does anyone know if this bug still exists in tgt in Jammy, and if so, please could you provide steps to reproduce? Marking Incomplete as it looks like we never got an answer to whether the bug existed in the latest release at the time, and we don't have specific steps to reproduce. If you can supply either of those answers, please comment and then change the bug status back to New. ** Changed in: tgt (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
Hi Andres, My full sentence was "test on zesty OR grab the package" I'm clear on the linking issues and that grabbing this implies way more messy changes - but in this case "grabbing" the change would have been like adding two lines to the service file of tgt. Anyway of course I'm ok with building something for Xenial to test. The fix there was about task-max (number, not size) so it wasn't sure if this will fix this issue here as well. That was the reason why we asked for a pre-check. The assumption is that this will fix the issue to be able to spawn more threads (when serving many images) but not the generic memory consumption (yet that might be avoided with swap). Yet IMHO that is not what -proposed is for (trial and error if it fixes the issues), but instead to ensure we have no regressions - well you know all that anyway. So if there is any chance you can test off a Xenial-ppa that would be much preferred to x-proposed. Only if there would be no way to do a xenial-paa I'd think it is ok to test directly in proposed. A ppa to test for your case is in [1]. [1]: https://launchpad.net/~ci-train-ppa-service/+archive/ubuntu/2991 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
@Christian, We cannot test the zesty package nor grab the package and install it in xenial because it needs to be built against the xenial archive for the library linking. As such, I would recommend this patch is pushed to -proposed where we can test it against the libraries of the archive. That would allow us to test it just once and be pushed into -xenial, and not have to test it in the PPA and then again test it in -proposed. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
** Changed in: maas Status: Won't Fix => Invalid -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
** Changed in: maas Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
Now that I am on the dev branch, the TGT service works great. Unfortunately now that I am on the dev branch, it is unable to determine the correct IP address for the controller during node discovery, and if you set up a new machine entry in MaaS for it using its MAC address, it ignores the ACTUAL MaaS IP address on commissioning and attemtps to hit the MaaS APIs on the default gateway instead. Is there currently or has there ever been an actual working version of MaaS somewhere? Does anyone know where to find one? Opening another new bug report... -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
I rolled both devices to the dev branch and this issue seems to be resolved. RPi 3B region: MAAS Version 2.2.0 (beta2+bzr5717) NUC rack: MAAS Version 2.2.0 (beta2+bzr5717) While I am ecstatic that this is also working now, I am deeply concerned that none of the Ubuntu "stable" stack is currently functional in a MaaS + Juju deployment due to various bugs, and only the devel PPAs seem to function right now with LXD and a network bridge to MaaS. (https://bugs.launchpad.net/juju/+bug/1633788) FWIW ** Attachment added: "region-status-page" https://bugs.launchpad.net/maas/+bug/1389811/+attachment/4819807/+files/Screen%20Shot%202017-02-15%20at%204.49.05%20PM.png -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
Same problem when running maas-rack controller on a second machine:
Intel NUC 5i5MYHE RACK CONTROLLER: /var/lib/maas/maas.log
Feb 15 19:58:13 rack2-maas-rack0 maas.import-images: [info] Updating boot image
iSCSI targets.
Feb 15 19:58:13 rack2-maas-rack0 maas.boot_image_download_service: [error]
Failed to download images: Command `sudo /usr/sbin/tgt-admin --conf
/var/lib/maas/boot-resources/current/maas.tgt --update ALL` returned non-zero
exit status 2:#012Config file /var/lib/maas/boot-resources/current/maas.tgt not
found. Exiting...
Feb 15 19:58:17 rack2-maas-rack0 maas.dhcp.probe: [error] Unable to probe for
DHCP servers: Connection was closed cleanly.
tgtd dies, rendering the rack controller unusable since it cannot
complete the maas-region import.
Raspberry Pi 3B REGION CONTROLLER: /var/lib/rediogd.log
2017-02-15 15:06:02 twisted.python.log: [info] :::192.168.199.6 - -
[15/Feb/2017:20:06:01 +] "GET /MAAS/rpc/ HTTP/1.0" 200 316 "-"
"provisioningserver.rpc.clusterservice.ClusterClientService"
sudo2017-02-15 15:06:28 provisioningserver.utils.services: [info] Neighbour
observation process for enxb827eb208904 started.
: a password is required
2017-02-15 15:06:26 maasserver.regiondservices.ntp: [critical] Failed to update
NTP configuration.
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py",
line 824, in worker
return target()
File "/usr/lib/python3/dist-packages/twisted/_threads/_threadworker.py", line
46, in work
task()
File "/usr/lib/python3/dist-packages/twisted/_threads/_team.py", line 190, in
doWork
task()
--- ---
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 246,
in inContext
result = inContext.theWork()
File "/usr/lib/python3/dist-packages/twisted/python/threadpool.py", line 262,
in
inContext.theWork = lambda: context.call(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 118, in
callWithContext
return self.currentContext().callWithContext(ctx, func, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/python/context.py", line 81, in
callWithContext
return func(*args,**kw)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py",
line 857, in callInContext
return func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/twisted.py",
line 225, in wrapper
result = func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/provisioningserver/ntp/config.py", line
49, in configure
mode=0o644)
File "/usr/lib/python3/dist-packages/provisioningserver/utils/fs.py", line
281, in sudo_write_file
raise ExternalProcessError(proc.returncode, command, stderr)
provisioningserver.utils.shell.ExternalProcessError: Command `maas-rack
atomic-write --filename /etc/ntp/maas.conf --mode 0644` returned non-zero exit
status 1:
/var/lib/rackd.log
2017-02-10 15:42:28 sstreams: [info]
maas:v2:download/maas:boot:ubuntu:amd64:ga-16.04:xenial: to_add=['20170207']
to_remove=[]
2017-02-10 15:42:31 provisioningserver.rackdservices.image_download_service:
[critical] Downloading images failed.
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 434, in
errback
self._startRunCallbacks(fail)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 501, in
_startRunCallbacks
self._runCallbacks()
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 588, in
_runCallbacks
current.result = callback(current.result, *args, **kw)
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1184,
in gotResult
_inlineCallbacks(r, g, deferred)
--- ---
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126,
in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in
throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File
"/usr/lib/python3/dist-packages/provisioningserver/rackdservices/image_download_service.py",
line 128, in maybe_start_download
yield self._start_download()
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126,
in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 389, in
throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File
"/usr/lib/python3/dist-packages/provisioningserver/rackdservices/image_download_service.py",
line 115, in _start_download
get_proxy_url("https"))
File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1126,
in _inlineCallbacks
result = result.throwExceptionIntoGenerator(g)
File
[Bug 1389811] Re: tgtadm: out of memory crash
Same issue, but on RPi3B Not sure if it should even be able to run or if this qualifies as a bug since memory is so scarce on these: tgtadm: out of memory root@juju-rack2:/var/log/maas# cat /etc/issue Ubuntu 16.04.1 LTS \n \l root@juju-rack2:/var/log/maas# uname -a Linux juju-rack2.home.spyderdyne.net 4.4.43-v7+ #948 SMP Sun Jan 15 22:20:07 GMT 2017 armv7l armv7l armv7l GNU/Linux root@juju-rack2:/var/log/maas# free -m totalusedfree shared buff/cache available Mem:925 521 25 36 379 341 Swap: 0 0 0 MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1) Attempted overclocking after noticing that python-twist was using so many resources and it is much faster now, but the iSCSI daemon still dies on startup. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
Hi Blake - I thought so, but could you try (on zesty or by grabbing the package from zesty into yours) if the fix for 1559088 actually fixes your issue on Xenial? Let me know if you would need a ppa. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
Yes this fix should be SRU to Xenial as anyone with lots of MAAS images comes across this issue. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
Hi, as planned I fixed bug 1559088 in the merge I made into zesty. If you could take a look and check if with tgt (1:1.0.67-1ubuntu1) this issue here would be solved for you (or at least less broken - as this is not fixing why tgt spawns so many threads, but it removes the systemd service limit on them) we could maybe consider an SRU for Xenial? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
I agree to Kiko's assumption and am currently fixing bug 1559088 along the next tgt merge. I'll ping you here once there is something out there. If that helps your case we can make it a dup and think on SRUs - if not - well - we at least would have lost nothing by trying. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
See also the Xenial issue, which may give a hint: https://bugs.launchpad.net/maas/+bug/1559088 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
I wonder if this is the issue caused by https://bugs.launchpad.net/ubuntu/+source/tgt/+bug/1547060 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
This is a huge problem affecting many people. If a users has lots of images imported into MAAS, tgt just fails to run. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
** Changed in: tgt (Ubuntu) Importance: Undecided => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
** Tags added: xenial -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
** Tags added: trusty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1389811] Re: tgtadm: out of memory crash
** Also affects: tgt (Ubuntu) Importance: Undecided Status: New ** Changed in: tgt (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1389811 Title: tgtadm: out of memory crash To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1389811/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
