[Bug 1559088] Re: [Xenial 2.0] tgt fails to start with tgtadm out of memory error
Hi Spyderdyne, trying to keep things together. This here was about the number of files (and thereby threads) being allowed to be served. The xenial/oom portion will be continued in "bug 1389811" until we are sure (or not) that it is the same issue where we then could dup it. I see you are active there as well, so here just a ping as FYI. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1559088 Title: [Xenial 2.0] tgt fails to start with tgtadm out of memory error To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1559088/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1559088] Re: [Xenial 2.0] tgt fails to start with tgtadm out of memory error
It looks like Python Twisted is actually taking up most of the resources. Wondering if there is a resource cap that I can force Python to stay under to keep from killing off other services. I suspected Postgres at first but it's definitely Python. ** Attachment removed: "cloud-in-a-box.mp4" https://bugs.launchpad.net/maas/+bug/1559088/+attachment/4815671/+files/cloud-in-a-box.mp4 ** Attachment added: "Top output" https://bugs.launchpad.net/maas/+bug/1559088/+attachment/4815689/+files/Screen%20Shot%202017-02-09%20at%2012.06.56%20AM.png -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1559088 Title: [Xenial 2.0] tgt fails to start with tgtadm out of memory error To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1559088/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1559088] Re: [Xenial 2.0] tgt fails to start with tgtadm out of memory error
Raspberry Pi 3B MAAS Version 2.1.3+bzr5573-0ubuntu1 (16.04.1) spyderdyne@juju-rack2:~$ uname -r 4.4.43-v7+ spyderdyne@juju-rack2:~$ cat /etc/issue Ubuntu 16.04.1 LTS \n \l spyderdyne@juju-rack2:~$ dpkg -l tgt Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name VersionArchitecture Description +++-==-==-==-= ii tgt1:1.0.63-1ubuntu1. armhf Linux SCSI target user-space daemon and tools spyderdyne@juju-rack2:~$ service tgt status ● tgt.service - (i)SCSI target daemon Loaded: loaded (/lib/systemd/system/tgt.service; enabled; vendor preset: enabled) Active: deactivating (stop-sigterm) (Result: exit-code) Docs: man:tgtd(8) Process: 5557 ExecStartPost=/usr/sbin/tgt-admin -e -c /etc/tgt/targets.conf (code=exited, status=22) Process: 5554 ExecStartPost=/usr/sbin/tgtadm --op update --mode sys --name State -v offline (code=exited, status=0/SUCCESS) Main PID: 5552 (tgtd) Status: "Starting event loop..." CGroup: /system.slice/tgt.service └─5552 /usr/sbin/tgtd -f Feb 08 22:25:24 juju-rack2.home.spyderdyne.net tgtd[5552]: tgtd: bs_thread_open(437) stopped the worker thread 4 Feb 08 22:25:24 juju-rack2.home.spyderdyne.net tgtd[5552]: tgtd: bs_thread_open(437) stopped the worker thread 3 Feb 08 22:25:24 juju-rack2.home.spyderdyne.net tgtd[5552]: tgtd: bs_thread_open(437) stopped the worker thread 2 Feb 08 22:25:24 juju-rack2.home.spyderdyne.net tgtd[5552]: tgtd: bs_thread_open(437) stopped the worker thread 1 Feb 08 22:25:24 juju-rack2.home.spyderdyne.net tgtd[5552]: tgtd: bs_thread_open(437) stopped the worker thread 0 Feb 08 22:25:24 juju-rack2.home.spyderdyne.net tgt-admin[5557]: tgtadm: out of memory Feb 08 22:25:24 juju-rack2.home.spyderdyne.net tgt-admin[5557]: Command: Feb 08 22:25:24 juju-rack2.home.spyderdyne.net tgt-admin[5557]: tgtadm -C 0 --lld iscsi --op new --mode logicalunit --ti Feb 08 22:25:24 juju-rack2.home.spyderdyne.net tgt-admin[5557]: exited with code: 22. Feb 08 22:25:24 juju-rack2.home.spyderdyne.net systemd[1]: tgt.service: Control process exited, code=exited status=22 spyderdyne@juju-rack2:~$ free -m totalusedfree shared buff/cache available Mem:925 557 65 24 302 316 Swap: 0 0 0 I cannot actually say this is a bug. I do not think that the iSCSI daemon is able to run with this tiny amount of memory. It would be a lot cooler if it could though. I can see it becoming popular to be able to spend $50 on a MaaS rack controller for each rack with a quad core computer, some sticky velcro, and a patch cable. We are already embedding mini-computers inside our servers for IPMI and management anyway. Unfortunately this might render my RPi MaaS rack controller idea unusable. Will attempt to set up a swap space on these 64GB Class10 SD cards and see if that works. I already gave it it's own power switch after all... ** Attachment added: "cloud-in-a-box.mp4" https://bugs.launchpad.net/maas/+bug/1559088/+attachment/4815671/+files/cloud-in-a-box.mp4 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1559088 Title: [Xenial 2.0] tgt fails to start with tgtadm out of memory error To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1559088/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1559088] Re: [Xenial 2.0] tgt fails to start with tgtadm out of memory error
This bug was fixed in the package tgt - 1:1.0.67-1ubuntu1
---
tgt (1:1.0.67-1ubuntu1) zesty; urgency=medium
* Merge from Debian. Remaining changes:
- Drop glusterfs support, package not in main.
- debian/control
- debian/rules
- debian/tests/{control, storage}
- debian/tgt-glusterfs.install (Deleted)
* Drop changes:
- debian/patches/util_strtoull_errno.patch by Stas Sergeev (now upstream)
- Disable AIO backend support. This was done to reduce risk in a previous
feature freeze; Now that we are before Zesty's feature freeze, we
can introduce it this cycle.
- changing file modes of debian/tests/{admin, daemon} (missing in
changelog, not needed)
* Add changes:
- fix tgt being killed when serving many targets (LP: #1559088)
- d/t/localtgt, d/t/control: add dep8 test that sets up targets and luns via
iscsi using rdwr/aio backends and runs fio read/write/verify (LP: #1640785)
-- Christian Ehrhardt Thu, 10 Nov
2016 10:35:15 +0100
** Changed in: tgt (Ubuntu)
Status: In Progress => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1559088
Title:
[Xenial 2.0] tgt fails to start with tgtadm out of memory error
To manage notifications about this bug go to:
https://bugs.launchpad.net/maas/+bug/1559088/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1559088] Re: [Xenial 2.0] tgt fails to start with tgtadm out of memory error
** Changed in: tgt (Ubuntu) Status: Triaged => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1559088 Title: [Xenial 2.0] tgt fails to start with tgtadm out of memory error To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1559088/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1559088] Re: [Xenial 2.0] tgt fails to start with tgtadm out of memory error
** Merge proposal linked: https://code.launchpad.net/~paelzer/ubuntu/+source/tgt/+git/tgt/+merge/310540 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1559088 Title: [Xenial 2.0] tgt fails to start with tgtadm out of memory error To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1559088/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1559088] Re: [Xenial 2.0] tgt fails to start with tgtadm out of memory error
Hi, the "workaround" is IMHO a valid fix and in use my lxd, libvirt, docker and others that know to spawn many threads. TGT is just another case - I guess we need no SRU since you closed it, but I'd add it to Zesty and push to Debian if all works out. ** Also affects: tgt (Ubuntu) Importance: Undecided Status: New ** Changed in: tgt (Ubuntu) Status: New => Triaged ** Changed in: tgt (Ubuntu) Importance: Undecided => Medium ** Changed in: tgt (Ubuntu) Assignee: (unassigned) => ChristianEhrhardt (paelzer) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1559088 Title: [Xenial 2.0] tgt fails to start with tgtadm out of memory error To manage notifications about this bug go to: https://bugs.launchpad.net/maas/+bug/1559088/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
