Cannot reproduce this on any recent MAAS.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to
kernel panic
** Changed in: maas
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to
Setting this to incomplete for maas as weel, since we couldn't reproduce
the issue with maas 2.7.
Please reopen if it happens again, with steps on how to reproduce
** Changed in: maas
Status: Confirmed => Incomplete
--
You received this bug notification because you are a member of
As this is subscribed to field-high and field-medium I am unsubscribing
field-high. There has been no update for 9 months to this bug. If folks
are still seeing this then additional logs and help reproducing would be
much appreciated.
--
You received this bug notification because you are a
It was already clear on the bug that for qemu I can't do much more as-is.
So I had a discussion with the Maas Team today.
To core issue for me stays that it seems it is unreproducible outside of Maas.
We agree that:
- they try to recreate it in a MAAS
- Once they can we sync and find day(s) to
** Tags added: cscc
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to
kernel panic
Status in MAAS:
Thanks Kellen,
I took your kernel/initrd and IPXE booted it fine via http from qemu's IPXE on
a 2G guest.
In the past we tried more (use squash, use tftp instead of http, use the same
parmlines as maas, ...), but it all comes down to the same thing every time -
only occurs inside Maas.
** Attachment added: "squashfs"
https://bugs.launchpad.net/maas/+bug/1797581/+attachment/5272424/+files/squashfs
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
** Attachment added: "boot-initrd"
https://bugs.launchpad.net/maas/+bug/1797581/+attachment/5272416/+files/boot-initrd
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
** Attachment added: "host1_dpkg.txt"
https://bugs.launchpad.net/maas/+bug/1797581/+attachment/5272418/+files/host1_dpkg.txt
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
** Attachment added: "host2_dpkg.txt"
https://bugs.launchpad.net/maas/+bug/1797581/+attachment/5272419/+files/host2_dpkg.txt
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
** Attachment added: "vm2.xml"
https://bugs.launchpad.net/maas/+bug/1797581/+attachment/5272422/+files/vm2.xml
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
** Attachment added: "vm1_console.log"
https://bugs.launchpad.net/maas/+bug/1797581/+attachment/5272421/+files/vm1_console.log
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
** Attachment added: "vm2_console.log"
https://bugs.launchpad.net/maas/+bug/1797581/+attachment/5272423/+files/vm2_console.log
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
** Attachment added: "vm1.xml"
https://bugs.launchpad.net/maas/+bug/1797581/+attachment/5272420/+files/vm1.xml
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
** Attachment added: "boot-kernel"
https://bugs.launchpad.net/maas/+bug/1797581/+attachment/5272417/+files/boot-kernel
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Ran into this today, attaching the information requested in #51.
Unable to commission/install 18.04 with "No minimum kernel"
MAAS version: maas 2.5.3-7533-g65952b418-0ubuntu1~18.04.1 all
Guests kernel paniced on two hosts (host1 and host2), dpkg -l attached for each
host.
Console output from
Yeah, it seems to only occur with the TFTP setup that maas creates - and in all
former cases it resolved a few days later - most likely by size of images
changing.
That is the reason we need the files from an affected case.
@Reto - please see in the bug description above, there is a section "If
I can also confirm the reported issue. I've tried with Juju and plain
MAAS to create a VM with just 2048MB and it fails in both cases.
Creating, registering and commissioning the machine works, but it cannot
be deployed.
System information: 18.04.2 LTS up-to-date
MAAS version: 2.5.3
Added to the description what we need from anyone affected.
@Maas Team see former comment for the requested guiding of users how to get
this data.
The Maas task is back from incomplete to confirmed for providing these howto's
** Changed in: maas
Status: Incomplete => Confirmed
--
You
@sfeole Thanks for the Dup hit that as well now - as all others I ask
you to please help to catch the data needed to finally recreate and
debug this.
>From IRC:
[07:00] sfeole: please attach your guest XML, and the used initrd
and kernel to the bug
[07:02] best would be also the pxeconfig that
After a longer session on IRC this is now also for Dmitrii un-reproducible.
Lets summarize the current status for the next oen coming by:
Current ideas still are:
a) at 2G the placement of kernel/initrd is too close (as placed by pxelinux),
then when the kernel unpacks it overwrites the initrd
Tried using direct kernel boot with QEMU and couldn't reproduce it:
sha256sum
/mnt/libvirt-images/boot-resources-20190419-115735/snapshot-20190419-115735/ubuntu/amd64/generic/bionic/daily/boot-initrd
69ca457a119fe309d315972ca2756a17bd9bc55bc98f2bea5542566a7f41b08f
The bionic/ga files from #63 need to be placed into both dirs:
1) /var/lib/maas/boot-resources/current/ubuntu/amd64/ga-18.04/bionic/daily
2) /var/lib/maas/boot-resources/current/ubuntu/amd64/generic/bionic/daily
The sha256 of the *initrd file* that triggers the issue is
tar -czvf boot-resources-20190419-115735-amd64-generic-bionic.tar.gz
boot-resources/snapshot-20190419-115735/ubuntu/amd64/generic/bionic/
boot-resources/snapshot-20190419-115735/ubuntu/amd64/generic/bionic/
boot-resources/snapshot-20190419-115735/ubuntu/amd64/generic/bionic/daily/
Fro experimenting with sizes things can easily be padded with zeros:
$ truncate -s 8000 bug-1797581-bad-initrd-extended
$ truncate -s 900 bug-1797581-xenial-base-extended
Boots just as well afterwards.
--
You received this bug notification because you are a member of Kernel
Packages,
Ran with:
- the cloud-image of Xenial of 20190419 [1]
- XML with a memory definition of:
2097152
2097152
- kernel are from the host
/var/lib/libvirt/images/bug-1797581-bionic-base
/var/lib/libvirt/images/bug-1797581-bad-initrd
- The attached broken initrd of comment #56
- Otherwise it is
@Dmitrii - we were at the "junk in ..." in comment #21 already.
The "broken padding" is interesting but might be the same issue with a
different message as the newer kernel understands slightly more about it.
That it does only occur with Xenial-HWE but on the same initrd not with
the base Xenial
** Changed in: qemu (Ubuntu)
Status: Confirmed => Incomplete
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048 MB RAM causes
Tested bionic-hwe - the issue does not occur with 2048 MiB.
The closest issue filed upstream I found is this:
https://bugzilla.kernel.org/show_bug.cgi?id=199845
** Bug watch added: Linux Kernel Bug Tracker #199845
https://bugzilla.kernel.org/show_bug.cgi?id=199845
--
You received this bug
Found something interesting.
Bionic + 2048 MiB of RAM (bad):
[1.520243] Unpacking initramfs...
[ 14.712821] Initramfs unpacking failed: broken padding
[ 14.723088] Freeing initrd memory: 56636K
Bionic + 2049 MiB of RAM (good):
[0.752624] Unpacking initramfs...
[5.572407]
I cannot reproduce the same with a xenial (GA kernel) image with 2048
MiB of RAM allocated to a VM.
So it seems to me that this is a kernel issue.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
The kernel code path mentioned in #55 is only executed if there is no
"early userspace init" - in other words, if there is no /init on initrd:
https://git.launchpad.net/~ubuntu-
kernel/ubuntu/+source/linux/+git/bionic/tree/init/main.c?id=Ubuntu-4.15.0-47.50#n1087
/*
* check if
echo -n
'BOOT_IMAGE=http://10.10.101.2:5248/images/ubuntu/amd64/generic/bionic/daily/boot-kernel
nomodeset ro
root=squash:http://10.10.101.2:5248/images/ubuntu/amd64/generic/bionic/daily/squashfs
ip=maas-vhost6:BOOTIF ip6=off overlayroot=tmpfs
overlayroot_cfgdisk=disabled
Just reproduced it on my env (where things used to work) after updating
from MAAS 2.5.0~rc2 to 2.5.2.
https://paste.ubuntu.com/p/CY834ZGnvS/
[ 15.458594] VFS: Cannot open root device
"squash:http://10.10.101.2:5248/images/ubuntu/amd64/generic/bion; or
unknown-block(0,0): error -6
tree
Ok, thanks Jason.
That is great info, but about as much as we had so far :-/
Can you set your deployment up in a way that if that happens again you
can collect these binaries?
If it right now still triggers if you (re)deploy a 2G guest use the
chance to finally give us the bits that we will need
@Christian
- release: bionic
- seabios: 1.10.2-1ubuntu1
- qemu: 1:2.11+dfsg-1ubuntu7.10
- libvirt: 4.0.0-1ubuntu8.8
- ovmf - this is a uefi thing right? we're not using it.
- kernel 2019-03-18T12:17:11+00:00 elastic-2 kernel: [0.00]
Linux version 4.15.0-46-generic
Jason,
thanks for chiming in - so far this was non-reproducible every time we looked
at it. We had plenty of approaches with the MAAS team to this and later on with
Thiago, but still miss the right data to continue.
Last time I asked - I think it was still Mike - that I'd really like to
get the
Bumped to field-high as we ran into this again in testing.
We have a workaround, but it's to not use 2G VM's, which is really silly
and hard to remember when we go and add new deployments, especially
because the failure mode is not obvious at all.
--
You received this bug notification because
** Changed in: maas
Importance: Critical => Undecided
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to
Ryan,
If I remove the UEFI line (back to BIOS), the machine enters in Kernel
Panic during the boot / commissioning.
That was actually, the very test that I executed in first place (add
UEFI to see)! Then, I came with this UEFI workaround, which is even
better for me.
Cheers!
Thiago
--
You
On Mon, Jan 14, 2019 at 1:55 PM Thiago Martins
wrote:
> @Ryan,
>
> The working XML file is attached here, with 2048 MB of RAM.
>
> NOTE: This XML was created using Virt-Manager, then, MaaS took it over
> after being "refreshed".
>
Thanks!
If you drop the section (which is what tells libvirt
@Ryan,
The working XML file is attached here, with 2048 MB of RAM.
NOTE: This XML was created using Virt-Manager, then, MaaS took it over
after being "refreshed".
Cheers!
Thiago
** Attachment added: "vunft-1.xml"
@Thiago,
Can you attach your guest XML that's working successfully with 2048MB?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048
I'm also facing this problem.
My workaround is to compose VMs using Virt-Manager, Firmware = UEFI for
the VM and then, refreshing the MaaS Pod.
There is a need to install ovmf
sudo apt install ovmf
On MaaS Pod.
Then, no more Kernel Panic!
Details:
Reproduced today on MAAS 2.5 latest bionic
It is not reproduced if switch from ga-18.04 kernel to ga--18.04-low-
latency kernel.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
@Ryan - the resource and other bits being different is because in the
bad case the domain was up and in the good case down. That in itself is
not a problem.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
One other oddity in the xml is the cgroup construction in the "bad"
case.
/machine
>
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/maas/+bug/1797581/+subscriptions
>
--
You received this bug notification because you are a member of Kernel
Packages,
Again it seems to break on initramfs being bad:
[0.840153] Initramfs unpacking failed: no cpio magic
@Mike/Andres - can you reproduce it on your own test env this time?
@Vladimir - could you internally share exactly your kernel and initrd
that is tried to be booted in this case? Last time we
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: qemu (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Status changed to 'Confirmed' because the bug affects multiple users.
** Changed in: linux (Ubuntu)
Status: New => Confirmed
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Setting this back to incomplete for MAAS because there is not enough
information to determine this is a MAAS Bug. That said, given that this
work with any VM other than one with 2048 of RAM, the issue may appear
to be the kernel or libvirt.
--
You received this bug notification because you are a
Subscribing field-medium as workaround (e.g not to use 2048mb ram VMs)
exists, but solution still need to be found.
Also, before yesterday (e.g on libvirt 1:2.11+dfsg-1ubuntu7.8 + maas
2.5rc2) worked fine.
--
You received this bug notification because you are a member of Kernel
Packages, which
** Changed in: maas
Status: Confirmed => Incomplete
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to
Reproduced today on MAAS 2.5 / latest Bionic.
qemu/maas versions: https://paste.ubuntu.com/p/kDk3c9tVFX/
good vm dump: http://paste.ubuntu.com/p/zTW2p6JGQJ/
good vm log: http://paste.ubuntu.com/p/V7GP9KgP8S/
good vm serial output: http://paste.ubuntu.com/p/FDymQcH8qD/
bad vm dump:
Hi,
thanks for your feedback.
We are at least much closer to what happened and thereby should be faster if it
reoccurs.
I don't think it is an interaction of the Kernel and Squashfs as we found that
already initramfs unpack was broken. That is before Squash comes into play.
I'll keep my test
Looking again at the date stamps, I don't see any squashfs filesystems
older than October 15th. The kernels are all from ~September 25th. So I
feel like this must have been an interaction between the kernel
September 25th and whatever the previous squashfs was.
--
You received this bug
After triaging this again on a call with Andres (who originally reported
this) this morning, we determined that this issue is no longer
reproducible with MAAS. The only way for me to explain it at this point
was that there was something wrong with the daily image MAAS was using
last week, and a
** Changed in: linux (Ubuntu)
Status: Confirmed => Incomplete
** Changed in: maas
Status: Invalid => Incomplete
** Changed in: qemu (Ubuntu)
Status: Invalid => Incomplete
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed
There actually is no need to debug further on my non-maas PXE setup.
As identified before - your setup breaks due to
[ 20.690329] Initramfs unpacking failed: junk in compressed archive
Everything else is a follow on issue, with eventually:
[ 22.524541] VFS: Cannot open root device
Taking the x86 pxelinux.0 from /usr/lib/PXELINUX/pxelinux.0 and
otherwise doing the same tftp setup as in [1] and switching from the
"maas" to the "default" network where I have set up dhcp to netboot by
libvirt as in [1]. I can see netboot happening.
With that instead of pushing things from
I didn't let me calm down - I found in the doc [1] that the switch might
be on the VLAN and not the subnet (Hrm why?)
It was on the vlan
http://horsea:5240/MAAS/#/vlan/5021
Not on the subnet
http://horsea:5240/MAAS/#/subnet/11
There I found the "provide dhcp" entry \o/
That unlocked some
I didn't see any guests crashing, instead they all just hang finding
nothing to boot.
I reduced this to just qemu (for debugging later on) but it reliably
breaks finding nothing from PXE.
Guest Console:
iPXE (PCI 00:03.0) starting execution...ok
iPXE initialising devices...ok
iPXE
I must admit, compared to you just sharing your kernel/initrd/squash asking to
install an own dev maas in comment #23 consumes quite some time :-/
Waiting for a sync here, setting up users there, sorting out how to make us
provide PXE and such on the "maas" virtual network, ... - it just isn't
... it's interesting that [practically?] the same kernel version that
fails consistently with Bionic works just fine with Xenial.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
I just tried Xenial with the HWE kernel - same result (success). FYI.
https://paste.ubuntu.com/p/WpqHnzbsS7/
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a
Good idea @rharper. It's easy for MAAS to attempt commissioning on
Xenial or Bionic, so I gave Xenial a try. It works fine![1]
[1]: Console log -
https://paste.ubuntu.com/p/58hfBX6BhY/
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to
Does this fail with other releases? like trusty? I was wondering if
initrd size plays a factor here:
precise/hwe-t: 25M
trusty/hwe-x: 35M
xenial/ga: 39M
xenial/hwe:53M
xenial/edge: 53M
bionic/ga: 55M
cosmic/ga: 57M
That might be faster for you to test than for us to replicate
** Changed in: qemu (Ubuntu)
Status: Incomplete => Invalid
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048 MB RAM causes
I agree that this doesn't look like a QEMU issue, and I agree that it
doesn't look like a [general] networking hiccup.
@paelzer, the easiest way to get the exact configs you need would be to
install MAAS similar to how I've described on our discourse forum[1].
The PXE config will be /similar/ to
** Changed in: maas
Milestone: 2.5.0rc1 => None
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to
Since it is reproducible maybe not a networking hiccup.
But maybe a hiccup of the PXE setup (thats why I asked for it).
Or a hiccup of the quite complex shim loading if signed kernels are used.
@Mike - in addition to my questions above could you use a non
signed/shim load pattern as well to check
[0.943808] Unpacking initramfs...
[ 20.690329] Initramfs unpacking failed: junk in compressed archive
[ 20.703673] Freeing initrd memory: 56612K
Looks like the initrd was compromised, possibly a networking hiccup?
Can you confirm the checksums on the source and attempt to download the
URL
Working on this I found by accident that I actually can reproduce:
VFS: Cannot open root device "squash:http://192.168.122.1:80/squashfs; or
unknown-block(0,0): error -6
But the way I got there lets assume some more potential reasons.
I got there by breaking my initramfs :-)
After realizing
After DHCP is up it works just fine.
(initramfs) wget http://192.168.122.1:80/squashfs
Connecting to 192.168.122.1:80 (192.168.122.1:80)
squashfs 100% |***| 174M 0:00:00 ETA
(initramfs) mount -t squashfs squashfs /root
(initramfs) mount
rootfs on / type
As Ryan I can not reproduce locally - hrm.
The crash in your log is the root-fs mount.
[ 22.524541] VFS: Cannot open root device
"squash:http://172.16.99.2:5248/images/ubuntu/amd64/generic/bion; or
unknown-block(0,0): error -6
[ 22.575588] Please append a correct "root=" boot option; here
Here's a full console log from the failure.
https://paste.ubuntu.com/p/zmS7CP7NKr/
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly
It's an empty image - MAAS PXE boots the VM.
Could you give it a try with MAAS? I can help you with the setup if
needed - just ping me on IRC.
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
I'm unable to recreate with a daily bionic cloud-image on a bionic host
with the same versions.
% sudo apt install uvtool libvirt
% uvt-simplestreams-libvirt -vv sync --source
http://cloud-images.ubuntu.com/daily 'supported=True' arch=amd64 release=bionic
% uvt-kvm create --memory 2048 --cpu 1
The backing image:
/var/lib/libvirt/maas-images/e5d185a9-8ccb-4ca6-959a-bd8eff0ee184
What boot image is that? Can I get a copy of that from maas-images? or
how is it created?
On the node with the vm that fails, can you:
virsh start --console
Assuming it's a normal ubuntu image which has
Here's the log from the failing VM. Doesn't look too unusual to me...
https://paste.ubuntu.com/p/JpmSxmwjfM/
** Description changed:
Using latest MAAS master, I'm unable to compose a VM over the UI
- successfully. BY this it means that the VM is created, but it fails with
- a kernel panic.
-
And /var/log/libvirt/qemu/.log ?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to
kernel panic
Status in
Here's an example of working XML (with 2047 MB RAM) that MAAS generated:
https://paste.ubuntu.com/p/mkF6Kp4hx8/
And here's an example of non-working XML (with 2048 MB RAM) that MAAS
generated:
https://paste.ubuntu.com/p/27HHDrzwm8/
** Changed in: linux (Ubuntu)
Status: Incomplete =>
Here's the version information.
libvirt-bin:
Installed: 4.0.0-1ubuntu8.5
Candidate: 4.0.0-1ubuntu8.5
Version table:
*** 4.0.0-1ubuntu8.5 500
500 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages
100 /var/lib/dpkg/status
4.0.0-1ubuntu8.2 500
Can you attach the guest xml and host kernel/qemu/libvirt packages?
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
https://bugs.launchpad.net/bugs/1797581
Title:
Composing a VM in MAAS with exactly 2048 MB RAM causes
** Summary changed:
- Composing a VM in MAAS with exactly 2048 MB RAM causes kernel panic
+ Composing a VM in MAAS with exactly 2048 MB RAM causes the VM to kernel panic
--
You received this bug notification because you are a member of Kernel
Packages, which is subscribed to linux in Ubuntu.
86 matches
Mail list logo
