** Changed in: snapd (Ubuntu)
Status: Fix Committed => Fix Released
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge over time
To manage
The fix for this has landed, we now have an upper bound for the amount
of changes we keep.
** Changed in: snapd (Ubuntu)
Status: Triaged => Fix Committed
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
When running snapd with the following branches:
- https://github.com/snapcore/snapd/pull/2546
- https://github.com/snapcore/snapd/pull/2545
under:
`while true; do sudo smemstat -p snapd ; sleep 60; done`
and with constantly installing/removing a snap:
`while true; do sudo snap remove hello;
The latest snapd 2.20 does no longer use cgo, so the risk of memory
leaks is greatly reduced this way.
There is still the problem that changes accumulate over time and that
can eat quite a bit of memory. The following branch addresses this:
https://github.com/snapcore/snapd/pull/2545
With the
** Changed in: snapd (Ubuntu)
Status: New => In Progress
** Changed in: snapd (Ubuntu)
Status: In Progress => Triaged
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
I am having the same issue. The /usr/lib/snapd/snapd memory usage grows
very large and never releases. Here is what my application does:
1) In runtime attach volumes to the ubuntu server using the OpenStack cinder
2) run FIO test on each volume every 30 seconds
3) randomly detach volumes
the
On 18 November 2016 at 02:26, Colin Ian King <1642...@bugs.launchpad.net>
wrote:
> @gustavo, I was just reporting some incidental data that I observed with
> valgrind as I thought it may be pertinent.
FWIW, I wouldn't in general expect valgrind to be useful with a go program.
The Go runtime
If it never stops growing, it's obviously a leak.. somewhere! If it
stabilizes after a while, then it's a side effect of Go being a non-
deterministic garbage collected language. Again, data.
If the goal is testing this particular code path, it'd be best to have a
tiny test case exercising it,
@Colin, Sorry if it sounds different, but this is not a wish of mine.
It's just about respecting the data we have. Not even Valgrind itself is
reporting this as a leak. It's saying "block possibly lost". Unless we
have either very basic empirical evidence of a leak (e.g. offending code
alone in a
For example, doing `snap find` in a loop grows snapd. I struggle to not
call that a leak.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge over time
To
@Gustavo, OK, I'll defer to your wishes. Just not 100% how valgrind
could spot a leak which is not there.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge
Once more, unless we have evidence showing otherwise, there were no
leaks before either. Let's please stop pretending otherwise.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd
Could you try again with this one? This one is dynamically linked but
has an Environment line in /lib/systemd/system/snapd.service to force it
to use the Go resolver.
And, lastly, if you can remove that line from the service file and try
again, then it should use the cgo resolver.
** Attachment
@John, I'm not seeing memory leaks on the thread create paths now. I
guess that was expected.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge over time
attached
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge over time
To manage notifications about this bug go to:
(that one has a static binary; the only caveat is because it's master
it'll bump your state.json to the latest thing, which might then not let
you go back to whatever you were using before)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to
** Attachment added: "snapd.deb"
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1642068/+attachment/4778656/+files/snapd.deb
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
@Colin, That's what I understood previously, but thanks for confirming
it. The question was whether you had digged further to see whether that
warning made any sense, or whether you had any further data confirming
the leak to be real. I'll take your response as a no, which is fine.
--
You
@Gustavo, sorry for the confusion, my response is no.
@John, building this on my Zesty box:
$ dpkg-buildpackage
...
debian/rules:67: recipe for target 'override_dh_auto_build' failed
make[1]: *** [override_dh_auto_build] Error 1
make[1]: Leaving directory '/home/king/tmp/snappy'
debian/rules:50:
I've put a branch up that drops a few things so that building snapd
without cgo is possible. Could you take it from
https://github.com/chipaca/snappy/tree/no-c and either build it, and
repeat the memory tests?
You can either build it entirely by hand setting CGO_ENABLED=0 to force
it to build a
(if you'd rather I put the binary somewhere for you, do let me know)
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge over time
To manage notifications
@gustavo, I was just reporting some incidental data that I observed with
valgrind as I thought it may be pertinent. It is just an observation
that made me dig a bit deeper as it didn't explain the larger heap
allocations. I just wanted to be thorough and report my findings as it
may be useful if
Did you did into that problem? I don't see the potential leak looking at
the function. All of the arguments other than *ThreadStart are allocated
in the stack, and ts is freed on threadentry which is the function
provided as the start_routine parameter of pthread_create.
What is the actual issue
Sure. The other issue was the potential leak of ~288 bytes per thread
creation at the C/Go interface layer, as mentioned in comment #5. I
suspect that's an issue that needs fixing with the Go layering glue
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is
@Colin, Sure it definitely needs to be improved. As I said above, this
is just a cheap way to get us going, and the proper fix is a
transactional storage on disk.
@Mark, Yes, we can prune based on number of changes as well. It's not
expensive. That's a good trick to get us going farther.
@John,
Unless I'm misreading something, it's growing at about 2MB every 60 sec.
180 changes should use a *lot* less than 2MB. So there probably is
something else going on...
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Is pruning expensive? Could we set a bound on the number of unsuccessful
transactions in the system and prune when that is reached? So the number
is bounded in time (3 days) and in length?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to
@John Lenton: about 3 per sec
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge over time
To manage notifications about this bug go to:
I still think a daemon that can gobble up a load of memory through
misuse needs fixing. Maybe I'm old school when I believe that small
footprint daemons that function correctly when abused is the correct
behaviour.
--
You received this bug notification because you are a member of Ubuntu
Bugs,
As mentioned above, unsuccessful changes are aborted after a few days,
and pruned thereafter.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge over time
Out of curiosity, how many changes per second were you doing?
On 16 Nov 2016 23:20, "Colin Ian King" <1642...@bugs.launchpad.net>
wrote:
> Well, until it's fixed, does that mean that unsuccessful transactions
> will cause snapd to eat memory - and if so, that seems like a DoS vector
> to me.
>
>
Well, until it's fixed, does that mean that unsuccessful transactions
will cause snapd to eat memory - and if so, that seems like a DoS vector
to me.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
Pruning based on a time interval perhaps should be complimented with a
memory threshold check too - on small devices we need to be sure we
constrain memory footprint, especially on non-swap based devices
otherwise processes will be OOM'd.
--
You received this bug notification because you are a
Btw, there's a "snap changes" command you can see which changes
currently exist.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge over time
To manage
If you install/remove on a loop, this will create changes which are kept
in memory until they are pruned. This happens in the period of three
days for successful ones, and a week for unsuccessful ones (which are
aborted, and later pruned). Eventually we'll need to move to a model
where this data
And running health-check on snapd shows the same king of arena
expansions with mmap:
Memory Change via mmap() and munmap():
PID mmaps munmaps Change (K) Rate (K/Sec)
21627 snapd 361 87124 1452.06
(growing fast)
21637 snapd
gdb is showing that go is expanding the arena via anonymous mmap's, e.g:
128MB, 64MB etc
__mmap (addr=0x0, len=134217728, prot=0, flags=16418, fd=-1, offset=0)
__mmap (addr=0x7ff77400, len=67108864, prot=0, flags=16418, fd=-1, offset=0)
etc..
so something is causing go to chomp up large
valgrind is finding ~288 bytes per thread create being leaked:
==12683== Thread 1:
==12683== 288 bytes in 1 blocks are possibly lost in loss record 1 of 7
==12683==at 0x4C2EB45: calloc (in
/usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==12683==by 0x40139AA: allocate_dtv
I don't think so, this is just snapd itself growing as it eats up heap.
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge over time
To manage notifications
could this be related to Bug 1636847 ?
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1642068
Title:
snapd memory sizes grows huge over time
To manage notifications about this bug go to:
** Attachment added: "Data in LibreOffice spreadsheet + graph"
https://bugs.launchpad.net/ubuntu/+source/snapd/+bug/1642068/+attachment/4777860/+files/snapd-memory-footprint.ods
** Changed in: snapd (Ubuntu)
Importance: Undecided => High
--
You received this bug notification because you
41 matches
Mail list logo
