Re: [CentOS-virt] LZ4 Kernel Decompression not supported
On 10/12/20 3:24 AM, Dr Francis Greaves wrote: I am running XEN on CentOS Linux release 7.8.2003 I have a Debian 7, and two Gentoo DomU Guests running fine. I am trying to get an Ubuntu 20.04 DomU Guest running but I get this error when trying to run the netboot kernel image xc_dom_probe_bzimage_kernel unable to LZ4 decompress kernel I understand that CentOS Xen does not support LZ4 decompression. Is there a workaround for this? Use HVM and a bootloader. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] CentOS 8 Install as DOMU in PV Environment
On 9/27/20 1:23 PM, 9f9dcad3f78905b03201--- via CentOS-virt wrote: Christoph, I understand this is the better option if HVM is available.. Im not sure how to make use of this kernel in a non-HVM, PV environment. I'm a little disturbed by the fact that there isnt PV support in CentOS 8, I hope it comes later so at least PVH becomes an option. It's existentially possible for there to be PVH support without PV support. I don't know about the current implementation. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] xen hvm domU and > 3 nic's... bug?
On 2/27/20 5:12 AM, Christoph wrote: It seems to be a problem with kernel... I use the kernel-ml there (5.5.6)... with kernel from centos-virt repo (4.9.*) it works without this problem... Have you made some pass to look for related problems on the xen-users or xen-devel mailing list or in the git commit history of mainline since 5.5.6 was released? If you can't find any evidence the bug has been reported or fixed already, I would encourage you to do so on the xen-devel mailing list since Linux 4.9 won't be supported forever. But in addition to the example configuration file, you may need to provide details about exactly what happens with 4 NICs to get a useful response. The output from running the xl create command, the kernel log, or the xen-hotplug log would likely contain more information. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] centos-virt CPU microcode updates?
On 5/16/19 2:12 AM, Karel Hendrych wrote: > Hi, > > is there any guide for CPU microcode updates on CentOS6, Xen 4.10, kernel 4.9 > ? I'm not sure off the top of my head. You can add ucode=scan to the xen command line, add a file /path/to/microcode_file as the last item in your boot list, and generate that microcode file using something like: iucode_tool --write-earlyfw "${OUT_BLOB_FILE}" \ "${TEMP_DIR_NAME}/intel-ucode" \ "${TEMP_DIR_NAME}/intel-ucode-with-caveats" Where the last two items have been extracted from the intel microcode tarball. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Xen-kernel: Update to 4.14 or 4.19?
On 3/7/19 10:30 AM, Akemi Yagi wrote: > On Thu, Mar 7, 2019 at 9:42 AM George Dunlap wrote: >> >> Hey all, >> >> We've been on 4.9 for some time now, and while it's still supported, I >> think it's time to start thinking about upgrading, and I'd like input >> from the community about which version to move up to. >> >> 4.19 has been out for almost 5 months now. It will include PVH domU >> support, and PVH dom0 support in what _is believed_ to be the final >> form; so when the Virt SIG moves to a version of Xen that supports PVH >> dom0, the kernel will already be in place with no need to upgrade. >> >> The other option would be to move to 4.14: Probably more stable (as >> it's been out for over a year now), but doesn't have either PVH domU >> or PVH dom0 support. >> >> I'd suggest 4.19. Any other opinions? >> >> -George > > You may also want to consider each version's EOL: > > 4.9 Jan, 2023 > 4.14 Jan, 2020 > 4.19 Dec, 2020 Regardless of EOL date, I think it's worth trying to upgrade when Xen has stable PVH dom0 support. I am pretty sure historically that there have been difficulties backporting some of the side channel mitigations as they can be quite invasive. That may be another reason to upgrade sooner rather than later. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] how to update ucode with xen
On 09/18/2018 11:55 PM, Christoph wrote: > > Hi > > can someone say me how to update the µcode of the cpu with xen? > > I have added the ucode=scan parameter to xen but it does not seem to work... > > the µcode version of my xeon is really old :/ > > model name : Intel(R) Xeon(R) CPU E3-1271 v3 @ 3.60GHz > microcode : 0x10 > We use ucode=-1 and provide it as a module for grub. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] ANNOUNCE: centos-release-xen switching to 4.8 next week
On 08/02/2018 03:58 AM, T.Weyergraf wrote: > Hi > > Thanks for providing updated Packages, they are much appreciated. At work, we > are currently running an entire production infrastructure on Xen4CentOS, > with quite some success. > > We are looking into a refresh towards CentOS 7 along with newer Xen and Dom0 > Kernel packages. However, even the updated packages are quite old. Xen > 4.8 is out of active support since June and will see the end of security > support in less than a year. https://wiki.xenproject.org/wiki/Xen_Project_Release_Features says December 2019, so about a year and 4 months. > Likewise, a newer LTS kernel (4.14) exists for > quite some time, while the Xen4CentOS effort currently uses 4.9. The end support date for that is much sooner: January 2019. Based on https://www.kernel.org/category/releases.html I'm not sure if it makes sense to move until a new LTS is available, as I suspect that will happen before 4.9 support ends. If you wanted to contribute an experimental 4.14 kernel it might be accepted. > Are there any short to mid-term plans to bump both versions to more current > ones (i.e.: 4.10 and 4.14)? Anthony's original email said "that update will come with a new package centos-release-xen-410 to install Xen 4.10." So yes on Xen. > As a side note, is there anything reasonable, people like me could to, to > support the speed-up of that process? I would consider testing to be > important, but are there any regression test-suites, one could use? I am > aware, there are such tests, but I have not found something to actually try > in our test-infrastructure. My understanding is that would be most welcome if you have anything to contribute. There are occasional meetings on freenode in #centos-devel for the virt-sig but you can also try the #centos-virt irc room. > Finally a big shout-out and kudos to the Xen community and Xen4CentOS. Your > work is used and much appreciated. +1, we haven't given back enough ourselves. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] custom Xen on custom kernel on CentOS 7
On 02/27/2018 07:50 AM, John Vetter wrote: > Hi, > I'm trying to run an arbitrary Xen version (4.7.x) on a recent kernel (say, > 4.13.x) on CentOS 7. > What is the recommended way for doing this? (I am new to Xen and > virtualization). > I tried the following: > 1. installed xen4centos. > 2. built linux kernel 4.13.x and installed it (using make install) > 3. built xen 4.7.x and installed it (using make install). > grub2-mkconfig and grub-bootxen.sh don't seem to be picking up the > combination > of new kernel and new xen and making an entry in the grub.cfg file. > It did make an entry for the new Xen with the kernel installed with > xen4centos. > I looked at the grub-bootxen.sh script but was unable to figure a way out. > How > do I get the installation scripts to make an entry for my new kernel and new > Xen? I would suggest one of 1. Run the scripts with -x to figure out where they're quitting early 2. Make an rpm for the kernel you're building, or 3. Manually edit the grub files and run dracut. It's not hard to do what those scripts do by hand. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Xen 4.4 Immediate EOL
On 01/18/2018 09:56 AM, Kevin Stange wrote: > Apparently I failed to do proper due diligence before making this > recommendation. The Xen 4.4 repo does not have vixen build because of a > dependency upon grub2 which isn't available under CentOS 6. Your best > bet would be to use Vixen for PV domains, so if you think that's > something you want to do, we need some volunteers to help with packaging > and testing. Otherwise, use HVM domains or upgrade to a newer version > of Xen. Sorry for this error on my part. > We have a SPEC file available for grub2: https://github.com/prgmrcom/grub2 you will need epel installed. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Centos 6 2.6.32-696.18.7.el6.x86_64 does not boot in Xen PV mode
On 01/05/2018 05:26 PM, Shaun Reitan wrote: > I can confirm the issue with 2.6.32-696.18.7.el6.x86_64, but mine looks > alittle different... Maybe because i'm using pvgrub. > > = Init TPM Front > Tpmfront:Error Unable to read device/vtpm/0/backend-id during tpmfront > initialization! error = ENOENT > Tpmfront:Info Shutting down tpmfront > close blk: backend=/local/domain/0/backend/vbd/14/51712 node=device/vbd/51712 > close blk: backend=/local/domain/0/backend/vbd/14/51728 node=device/vbd/51728 > [root@devhost1]# > > -- > Shaun Reitan > NDCHost.com You need to add earlyprintk=xen. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] Centos 6 2.6.32-696.18.7.el6.x86_64 does not boot in Xen PV mode
Problems start before any of the kaiser code executes, though it could still be related to CONFIG_KAISER since that has effects beyond kaiser.c. --- (early) Initializing cgroup subsys cpuset (early) Initializing cgroup subsys cpu (early) Linux version 2.6.32-696.18.7.el6.x86_64 (mockbu...@c1bl.rdu2.centos.org) (gcc version 4.4.7 20120313 (Red Hat 4.4.7-18) (GCC) ) #1 SMP Thu Jan 4 17:31:22 UTC 2018 (early) Command line: ro console=hvc0 rootflags=barrier=0 crashkernel=auto SYSFONT=latarcyrheb-sun16 LANG=en_US.UTF-8 KEYTABLE=us earlyprintk=xen (early) KERNEL supported cpus: (early) Intel GenuineIntel (early) AMD AuthenticAMD (early) Centaur CentaurHauls (early) 1 multicall(s) failed: cpu 0 (early) Pid: 0, comm: swapper Not tainted 2.6.32-696.18.7.el6.x86_64 #1 (early) Call Trace: (early) [] ? xen_mc_flush+0x1c3/0x250 (early) [] ? xen_extend_mmu_update+0xde/0x1b0 (early) [] ? xen_set_pmd_hyper+0x9d/0xc0 (early) [] ? early_ioremap_init+0x98/0x133 (early) [] ? setup_arch+0x40/0xca6 (early) [] ? vprintk_default+0xe/0x10 (early) [] ? printk+0x4f/0x52 (early) [] ? start_kernel+0xdc/0x43b (early) [] ? reserve_early+0x30/0x39 (early) [] ? x86_64_start_reservations+0x125/0x129 (early) [] ? xen_start_kernel+0x4fe/0x505 (early) [ cut here ] (early) WARNING: at arch/x86/xen/multicalls.c:182 xen_mc_flush+0x21f/0x250() (Not tainted) (early) Modules linked in: (early) Pid: 0, comm: swapper Not tainted 2.6.32-696.18.7.el6.x86_64 #1 (early) Call Trace: (early) [] ? warn_slowpath_common+0x91/0xe0 (early) [] ? __raw_callee_save_xen_restore_fl+0x11/0x1e (early) [] ? warn_slowpath_null+0x1a/0x20 (early) [] ? xen_mc_flush+0x21f/0x250 (early) [] ? xen_extend_mmu_update+0xde/0x1b0 (early) [] ? xen_extend_mmu_update+0xde/0x1b0 (early) [] ? xen_set_pmd_hyper+0x9d/0xc0 (early) [] ? early_ioremap_init+0x98/0x133 (early) [] ? setup_arch+0x40/0xca6 (early) [] ? vprintk_default+0xe/0x10 (early) [] ? printk+0x4f/0x52 (early) [] ? start_kernel+0xdc/0x43b (early) [] ? reserve_early+0x30/0x39 (early) [] ? x86_64_start_reservations+0x125/0x129 (early) [] ? xen_start_kernel+0x4fe/0x505 (early) ---[ end trace a7919e7f17c0a725 ]--- (early) ACPI in unprivileged domain disabled (early) released 0 pages of unused memory (early) BIOS-provided physical RAM map: (early) Xen: - 000a (usable) (early) Xen: 000a - 0010 (reserved) (early) Xen: 0010 - 2000 (usable) (early) bootconsole [xenboot0] enabled ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] CentOS-virt - Kernel Side-Channel Attacks
On 01/04/2018 10:49 AM, Akemi Yagi wrote: > On Thu, Jan 4, 2018 at 9:51 AM, wrote: > >> Please patch the CentOS-virt Kernel to fix the >> Kernel Side-Channel Attacks vulnerabilities. >> >> The latest CentOS-virt kernel was released in November, as seen below. >> >> kernel-4.9.63-29.el7.x86_64.rpm 2017-11-21 13:30 >> >> https://access.redhat.com/security/vulnerabilities/speculativeexecution >> http://mirror.centos.org/centos/7/virt/x86_64/xen/ >> > > As far as I can see, the patches for > KAISER (Kernel Address > Isolation to have Side-channels Efficiently Removed) will appear in > kernel 4.9.75. Looks like it will be released soon upstream (kernel.org). > To my best knowledge KAISER doesn't matter for Xen Dom0's given they run in PV mode, and KAISER isn't enabled for PV guests. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Live migration haswell, broadwell
On 11/14/2017 03:13 AM, T.Weyergraf wrote: > Hi > > I wonder, if live migration (back and forth) is possible on mixed Haswell > (Xeon V3) and Broadwell (Xeon V4) installations. The only notable difference > between the two is apparently a working TSX implementation on V4, which got > disabled on V3 due to bugs. The rest (VMCS-shadowing, posted interrupts) > should not apply to our environment, as we do not run nested-vmx nor > device-passthrough on our Xen servers > > Now, I found no sane way to disable TSX on a given system but I cannot rule > out, that some (Linux-)software, such as Postgres will use them > eventually. Also, I have a hard time trying to assess, if TSX can be disabled > on V4 to enable seamless migration. > > Any hint would be greately appreciated. Have you looked at CPUID masking? I haven't tried this but someone wrote a handy calculator for it. https://github.com/twizted/xen_maskcalc ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] Crash in network stack under Xen
Hi, We had a potentially network related crash on a dom0 with Linux 4.9.39 / Xen 4.8 and as of today I can't find any fixes in stable/linux-4.9.y, xen/staging-4.8, or CPU microcode updates that look like a smoking gun. I can't rule out that it's Xen related. The backtraces are: [ cut here ] WARNING: CPU: 0 PID: 0 at net/ipv4/af_inet.c:1473 inet_gro_complete+0xbb/0xd0 Call Trace: dump_stack+0x63/0x8e __warn+0xd1/0xf0 warn_slowpath_null+0x1d/0x20 inet_gro_complete+0xbb/0xd0 napi_gro_complete+0x73/0xa0 napi_gro_flush+0x5f/0x80 napi_complete_done+0x6a/0xb0 igb_poll+0x38d/0x720 [igb] ? igb_msix_ring+0x2e/0x40 [igb] ? __handle_irq_event_percpu+0x4b/0x1a0 net_rx_action+0x158/0x360 __do_softirq+0xd1/0x283 irq_exit+0xe9/0x100 xen_evtchn_do_upcall+0x35/0x50 xen_do_hypervisor_callback+0x1e/0x40 ? xen_hypercall_sched_op+0xa/0x20 ? xen_hypercall_sched_op+0xa/0x20 ? xen_safe_halt+0x10/0x20 ? default_idle+0x1e/0xd0 ? arch_cpu_idle+0xf/0x20 ? default_idle_call+0x2c/0x40 ? cpu_startup_entry+0x1ac/0x240 ? rest_init+0x77/0x80 ? start_kernel+0x4a7/0x4b4 ? set_init_arg+0x55/0x55 ? x86_64_start_reservations+0x24/0x26 ? xen_start_kernel+0x555/0x561 general protection fault: [#1] SMP Call Trace: ? napi_gro_complete+0x5e/0xa0 skb_release_all+0x24/0x30 kfree_skb+0x32/0x90 napi_gro_complete+0x5e/0xa0 napi_gro_flush+0x5f/0x80 napi_complete_done+0x6a/0xb0 igb_poll+0x38d/0x720 [igb] ? igb_msix_ring+0x2e/0x40 [igb] ? __handle_irq_event_percpu+0x4b/0x1a0 net_rx_action+0x158/0x360 __do_softirq+0xd1/0x283 irq_exit+0xe9/0x100 xen_evtchn_do_upcall+0x35/0x50 xen_do_hypervisor_callback+0x1e/0x40 ? xen_hypercall_sched_op+0xa/0x20 ? xen_hypercall_sched_op+0xa/0x20 ? xen_safe_halt+0x10/0x20 ? default_idle+0x1e/0xd0 ? arch_cpu_idle+0xf/0x20 ? default_idle_call+0x2c/0x40 ? cpu_startup_entry+0x1ac/0x240 ? rest_init+0x77/0x80 ? start_kernel+0x4a7/0x4b4 ? set_init_arg+0x55/0x55 ? x86_64_start_reservations+0x24/0x26 ? xen_start_kernel+0x555/0x561 RIP skb_release_data+0x73/0xf0 Kernel panic - not syncing: Fatal exception in interrupt Kernel Offset: disabled (XEN) Hardware Dom0 crashed: rebooting machine in 5 seconds. If anyone has had a similar backtrace or knows of a potential fix please respond. This server has ECC and there were no ECC or other errors in the BIOS event log, nor were there any indications of any problems in the serial console log leading up to the warning. This particular server had an uptime of about a month and a half, and so far we've had this error exactly once across all our servers since switching to 4.9.39 in August, so I don't think it's going to be easy to reproduce. --- It looks to me like in the first backtrace, this check from inet_gro_complete failed: ops = rcu_dereference(inet_offloads[proto]); Which I'm guessing means the packet didn't have a valid layer 4 protocol definition, or we don't have that protocol enabled. Then when attempting to handle that failure there was a GPF, I believe by accessing invalid data in shinfo->frag_list . "skb_release_data+0x73" is in __read_once_size, which I think is generated by "kfree_skb: if (likely(atomic_read(&skb->users) == 1))" . --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Stability issues since moving to 4.6 - Kernel paging request bug + VM left in null state
On 11/07/2017 04:57 PM, Sarah Newman wrote: > On 11/07/2017 03:12 PM, Nathan March wrote: >> Since moving from 4.4 to 4.6, I've been seeing an increasing number of >> stability issues on our hypervisors. I'm not clear if there's a singular >> root cause here, or if I'm dealing with multiple bugs. >> >> >> >> One of the more common ones I've seen, is a VM on shutdown will remain in >> the null state and a kernel bug is thrown: >> >> >> >> xen001 log # xl list >> >> NameID Mem VCPUs State >> Time(s) >> >> Domain-0 0 614424 r- >> 6639.7 >> >> (null) 3 0 1 --pscd >> 36.3 >> >> >> >> [89920.839074] BUG: unable to handle kernel paging request at >> 88020ee9a000 >> > > >> This is on xen 4.6.6-4.el6 with 4.9.58-29.el6.x86_64. I see these issues >> across a wide number of systems with from both Dell and Supermicro, although >> we run the same Intel x540 10gb nic's in each system with the same netapp >> nfs backend storage. > > We don't use NFS and have not seen the exact same issue. Additionally we aren't using xen 4.6 any more, we're using 4.8, but we didn't see issues like that when we were using xen 4.6. We're also still on 4.9.39. You might try an older kernel or a newer version of xen in addition to looking for nfs specific issues. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Stability issues since moving to 4.6 - Kernel paging request bug + VM left in null state
On 11/07/2017 03:12 PM, Nathan March wrote: > Since moving from 4.4 to 4.6, I've been seeing an increasing number of > stability issues on our hypervisors. I'm not clear if there's a singular > root cause here, or if I'm dealing with multiple bugs. > > > > One of the more common ones I've seen, is a VM on shutdown will remain in > the null state and a kernel bug is thrown: > > > > xen001 log # xl list > > NameID Mem VCPUs State > Time(s) > > Domain-0 0 614424 r- > 6639.7 > > (null) 3 0 1 --pscd > 36.3 > > > > [89920.839074] BUG: unable to handle kernel paging request at > 88020ee9a000 > > This is on xen 4.6.6-4.el6 with 4.9.58-29.el6.x86_64. I see these issues > across a wide number of systems with from both Dell and Supermicro, although > we run the same Intel x540 10gb nic's in each system with the same netapp > nfs backend storage. We don't use NFS and have not seen the exact same issue. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Xen 4.6.3-15 packages, including XSAs 216-219, 221-225 on their way through the build system
On 06/20/2017 05:06 AM, George Dunlap wrote: > Xen 4.6.3-15 packages for CentOS 6 and CentOS 7 are on their way > through the build system. They should show up in centos-virt-testing > in a few hours, and in the main mirrors tomorrow morning (God > willing). > > These contain several critical updates; users are encouraged to update > as soon as possible. > There are also Linux kernel changes in XSA-216. They apply cleanly to 4.9.31 and presumably also 4.9.25. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] xen4centos kernel version / debuginfo
On 06/12/2017 03:25 PM, Kevin Stange wrote: > On 06/12/2017 05:17 PM, Sarah Newman wrote: >> Is there any problem moving to 4.9.31? This contains upstream commits >> f2e767bb5d6ee0d9 for mpt3sas and >> 69861e0a52f87333 for dom0 memory mappings. > > 4.9.31-27 is tagged for testing presently along with updated firmware. > I have it running on one hypervisor on Xen-44 and it has been stable for > 3 days so far. We could probably use some more testing before tagging > it stable. > > I have been having some dom0 page allocation failures on 4.9.25... does > 69861e0a52f87333 address that issue? I was having trouble determining > if it was related. I don't know if it addresses it. I have also been having similar problems. > > https://buildlogs.centos.org/centos/6/virt/x86_64/xen/ > >> I am also trying to build a debuginfo package but it's not correct. >> Specifically I'm able to get vmlinux but the .debug files for modules are >> stripped, which isn't right. Does anyone know how to fix this? > > How are you building? CentOS builds with mock, but I'm not super > familiar with the environment. rpmbuild -b doesn't seem to always work > that well. I am building with mock. There are currently no debuginfo packages, this is a capability I'm trying to add. >> I'm also not sure how you're supposed to build the kernel-firmware package >> with the current spec file definition. > > I'm also not sure about how the firmware builds work but here's the Koji > builds from the last update: > > http://cbs.centos.org/koji/buildinfo?buildID=17333 > http://cbs.centos.org/koji/buildinfo?buildID=17332 > Not from the kernel spec file, apparently, even though there's tons of references in there. I wonder why this changed. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] xen4centos kernel version / debuginfo
lude/generated/uapi/linux/version.h +# +# save the vmlinux file for kernel debugging into the kernel-debuginfo rpm +# +%if %{with_debuginfo} +mkdir -p $RPM_BUILD_ROOT%{debuginfodir}/lib/modules/%{KVRFA} +cp vmlinux $RPM_BUILD_ROOT%{debuginfodir}/lib/modules/%{KVRFA} +%endif + # Remove any 'left-over' .cmd files. /usr/bin/find $RPM_BUILD_ROOT/lib/modules/%{KVRFA}/build/ -type f -name "*.cmd" | xargs --no-run-if-empty %{__rm} -f @@ -553,6 +595,20 @@ find Documentation -type d | xargs %{__chmod} u+w popd > /dev/null +%if %{with_debuginfo} + +%define __debug_install_post \ + /usr/lib/rpm/find-debuginfo.sh --strict-build-id %{_builddir}/%{?buildsubdir}\ +%{nil} + +%ifnarch noarch +%global __debug_package 1 +%files -f debugfiles.list debuginfo +%defattr(-,root,root) +%endif + +%endif + %install pushd linux-%{version}-%{release}.%{_target_cpu} > /dev/null @@ -825,6 +881,12 @@ fi %endif %changelog +* Sun Jun 11 2017 Sarah Newman 4.9.31-28 +- Upgraded to LTS 4.9.31, removed patch 10003 as it is now + in the upstream kernel. +- Enable livepatch +- Add debuginfo + * Wed May 3 2017 Johnny Hughes 4.9.25-27 - Upgraded to LTS 4.9.25, removed patch 10002 as it is now in the upstream kernel. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] Crash in CentOS 7 kernel-3.10.0-514.16.1.el7.x86_64 in Xen PV mode
I experienced a bug that is likely the same as https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1350373 . Commit b7dd0e350e0bd4c0fddcc9b8958342700b00b168 , which is supposed to fix it, doesn't appear in this kernel and doesn't apply cleanly either. Is there any point in trying to backport the patch? The backtrace is as follows: [ 32.304666] [ cut here ] [ 32.304679] kernel BUG at arch/x86/kernel/paravirt.c:252! [ 32.304683] invalid opcode: [#1] SMP [ 32.304687] Modules linked in: ip6t_rpfilter ipt_REJECT nf_reject_ipv4 ip6t_REJECT nf_reject_ipv6 xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter intel_powerclamp coretemp pcspkr ip_tables ext4 mbcache jbd2 xen_netfront xen_blkfront crc32c_intel [ 32.304734] CPU: 0 PID: 3901 Comm: dracut Not tainted 3.10.0-514.16.1.el7.x86_64 #1 [ 32.304739] task: 880002598000 ti: 88001b728000 task.ti: 88001b728000 [ 32.304743] RIP: e030:[] [] enter_lazy.part.0+0x4/0x6 [ 32.304755] RSP: e02b:88001f803aa8 EFLAGS: 00010002 [ 32.304758] RAX: 0001 RBX: 88001eacd640 RCX: 3000 [ 32.304761] RDX: 88000640 RSI: c90c8000 RDI: 0001 [ 32.304765] RBP: 88001f803aa8 R08: 88001f803b78 R09: 813d50f9 [ 32.304771] R10: 88001e801e00 R11: ea093dc0 R12: c90c9000 [ 32.304777] R13: c90c8000 R14: R15: 88001d150340 [ 32.304787] FS: 7f64425b0740() GS:88001f80() knlGS: [ 32.304796] CS: e033 DS: ES: CR0: 8005003b [ 32.304801] CR2: 006de2c8 CR3: 1b405000 CR4: 2660 [ 32.304807] DR0: DR1: DR2: [ 32.304813] DR3: DR6: 0ff0 DR7: 0400 [ 32.304818] Stack: [ 32.304823] 88001f803ab8 81061857 88001f803b60 811b1fe2 [ 32.304833] c90c8fff c90c9000 819bac90 c90c8fff [ 32.304843] c90c9000 88001eacb000 810206b0 [ 32.304854] Call Trace: [ 32.304858] [ 32.304861] [] paravirt_enter_lazy_mmu+0x27/0x30 [ 32.304879] [] apply_to_page_range+0x282/0x460 [ 32.304888] [] ? map_pte_fn+0x60/0x60 [ 32.304894] [] arch_gnttab_map_status+0x3b/0x70 [ 32.304904] [] gnttab_map_frames_v2+0xd6/0x150 [ 32.304910] [] gnttab_map+0xa1/0x140 [ 32.304917] [] get_free_entries+0x100/0x2e0 [ 32.304923] [] gnttab_alloc_grant_references+0x15/0x30 [ 32.304933] [] do_blkif_request+0x6bf/0x8a0 [xen_blkfront] [ 32.304945] [] ? __freed_request+0x92/0xa0 [ 32.304951] [] __blk_run_queue+0x33/0x40 [ 32.304957] [] blk_start_queue+0x29/0x40 [ 32.304964] [] kick_pending_request_queues+0x21/0x30 [xen_blkfront] [ 32.304975] [] blkif_interrupt+0x76e/0x820 [xen_blkfront] [ 32.304986] [] ? kmem_cache_free+0x1bb/0x1f0 [ 32.304995] [] handle_irq_event_percpu+0x3e/0x1e0 [ 32.305003] [] handle_irq_event+0x3d/0x60 [ 32.305004] [] handle_edge_irq+0x77/0x130 [ 32.305004] [] __xen_evtchn_do_upcall+0x227/0x350 [ 32.305004] [] xen_evtchn_do_upcall+0x33/0x50 [ 32.305004] [] xen_do_hypervisor_callback+0x1e/0x30 [ 32.305004] [ 32.305004] [] ? copy_pte_range+0x2b6/0x5a0 [ 32.305004] [] ? copy_pte_range+0x286/0x5a0 [ 32.305004] [] ? copy_page_range+0x312/0x490 [ 32.305004] [] ? dup_mm+0x362/0x680 [ 32.305004] [] ? copy_process+0x144e/0x1960 [ 32.305004] [] ? do_fork+0x91/0x2c0 [ 32.305004] [] ? SyS_clone+0x16/0x20 [ 32.305004] [] ? stub_clone+0x69/0x90 [ 32.305004] [] ? system_call_fastpath+0x16/0x1b [ 32.305004] Code: 20 e9 2f ff ff ff 44 89 fa 44 89 ee 48 c7 c7 10 45 8c 81 31 c0 e8 9d 14 00 00 58 5a 5b 41 5c 41 5d 41 5e 41 5f 5d c3 55 48 89 e5 <0f> 0b 66 66 66 66 90 55 48 89 e5 0f 0b 55 48 89 e5 0f 0b 55 48 [ 32.305004] RIP [] enter_lazy.part.0+0x4/0x6 [ 32.305004] RSP [ 32.305004] ---[ end trace 49f67f0d85e1ac69 ]--- [ 32.305004] Kernel panic - not syncing: Fatal exception in interrupt Thanks, Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] 4.9 kernel fails to boot because it didn't have the mpt3sas module
On 05/17/2017 01:30 PM, Jerry wrote: > For some reason each of the 5, 6 & 7 releases have issues with the mpt2sas > driver (there isn't enough swiotlb allocated by default for the driver to > work properly, and the units for that parameter changed between 5 and 6). > It's like this driver has bad luck or something. To my best knowledge they shouldn't use swiotlb if you give the dom0 4GiB+ of RAM. I posted the exact same problem before and that was the recommendation I got. It works. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] 4.9 kernel fails to boot because it didn't have the mpt3sas module
On 05/16/2017 09:04 PM, Jerry wrote: > > Turns out dracut was unable to mount the root file system. So I went back > into the 3.10 kernel again to see if the mpt2sas or mpt3sas driver was in > its initramfs file... and it wasn't: > > > $ sudo lsinitrd -k 4.9.25-27.el7.x86_64 | grep mpt > -rw-r--r-- 1 root root0 May 16 12:39 etc/fstab.empty > -rw-r--r-- 1 root root 22 Nov 5 2016 > usr/lib/kbd/unimaps/empty.uni > > For comparison: > > $ sudo lsinitrd -k 3.10.0-514.16.1.el7.x86_64 | grep mpt > -rw-r--r-- 1 root root0 May 16 04:37 etc/fstab.empty > -rw-r--r-- 1 root root 22 Nov 5 2016 > usr/lib/kbd/unimaps/empty.uni > drwxr-xr-x 2 root root0 May 16 04:37 > usr/lib/modules/3.10.0-514.16.1.el7.x86_64/kernel/drivers/scsi/mpt3sas > -rw-r--r-- 1 root root 379021 Apr 12 08:51 > usr/lib/modules/3.10.0-514.16.1.el7.x86_64/kernel/drivers/scsi/mpt3sas/mpt2sas.ko > > > So I added it: > > > $ sudo dracut --force --add-drivers mpt3sas --kver=4.9.25-27.el7.x86_64 > $ sudo lsinitrd -k 4.9.25-27.el7.x86_64 | grep mpt > Arguments: --force --add-drivers 'mpt3sas' --kver '4.9.25-27.el7.x86_64' > -rw-r--r-- 1 root root0 May 16 12:57 etc/fstab.empty > -rw-r--r-- 1 root root 22 Nov 5 2016 > usr/lib/kbd/unimaps/empty.uni > drwxr-xr-x 2 root root0 May 16 12:57 > usr/lib/modules/4.9.25-27.el7.x86_64/kernel/drivers/scsi/mpt3sas > -rwxr--r-- 1 root root 374152 May 16 12:57 > usr/lib/modules/4.9.25-27.el7.x86_64/kernel/drivers/scsi/mpt3sas/mpt3sas.ko > > > After this I was able to get the 4.9 kernel to boot and Xen is now working. You should be able to make this change permanently from /etc/dracut.conf with the add_drivers+= line, or build a generic initrd with hostonly="no" . I don't know why it wasn't originally added. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] centos-virt IRC meetings
Hi, We were hoping to attend an IRC meeting this morning but it looks like that didn't happen. Has this been moved to once a month or was this a special week? Thanks, Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Xen C6 kernel 4.9.13 and testing 4.9.15 only reboots.
On 03/28/2017 02:55 PM, PJ Welsh wrote: > The mystery gets more interesting... I now have a CentOS 7.3 Dell R710 > server doing the exact same thing of rebooting immediately after the Xen > kernel load. Just to note this is a second system and not just the first > system with an update. I hope I'm not introducing something odd. They only > "interesting" thing I have done for historical reasons is to change the > following /etc/sysconfig/grub line: > GRUB_CMDLINE_XEN_DEFAULT="dom0_mem=6G,max:8G cpuinfo com1=115200,8n1 > console=com1,tty loglvl=all guest_loglvl=all" > But I've done that on other servers without issue. In fact I have a Dell > R710 that DOES work with CentOS 7 and the new kernel... so confused. I am having no similar issues with several Dell Proliant DL160p's and CentOS 6. They are either G5 or G6, I don't recall which. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Clocksource boot issues 4.9.13
On 04/02/2017 02:49 AM, Chris Elliott wrote: > Hi all > > I’ve got a few Intel Z87 chipset machines with Adaptec 5405 raid cards > (latest firmware), they work fine on 3.18 but during Dom0 boot using kernel > 4.9.13 it hangs at “Using clocksource tsc” and the aacraid driver keeps > trying to reset > > Has anyone seen anything like this? > > I’ve tried specifying clocksource=xen in grub instead of the default of tsc, > and that has the same issue. HPET is enabled and Xen is seeing it: > > (XEN) ACPI: HPET D9649CB0, 0038 (r1 ALASKAA M I 1072009 AMI.5) > (XEN) Platform timer is 14.318MHz HPET I saw a hang at a similar place in the boot process when trying to boot xen-on-xen for our test system. On a hunch I was going to to try recompiling without the PVHVM PCI related driver (pci-platform ? platform-pci ? ) before saying anything about it. Since you tried changing the clock source I'm wondering is that the boot issue is unrelated to the clock source, in which case you may get a better idea of what's hanging by comparing the boot logs from 3.18 to 4.9 and seeing what's present in 3.18 but not in 4.9. Presumably the messages in the 3.18 but not 4.9 logs are either removed from the kernel source or happen after whatever is hanging. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] NIC Stability Problems Under Xen 4.4 / CentOS 6 / Linux 3.18
On 03/16/2017 04:22 PM, Kevin Stange wrote: >> I still can't rest assured the NIC issue is fixed, but no 4.4 or 4.9 >> server has yet had a NIC issue, with some being up almost a full month. >> It looks promising! (I'm knocking on all the wood everywhere, though.) > > I'm ready to call this conclusive. The problems I was having across the > board seemed to be caused by something seriously broken in 3.18. Most > of my servers are now on 4.9.13 or newer and everything has been working > very well. > > I'm not going to post any further updates unless something breaks. > Thanks to everyone that provided tips and suggestions along the way. > Do you mind sharing what hardware have you been running the 4.9 kernel on other than "Supermicro X9DRT, Dual Xeon E5-2650, 2x I350, 2x 82571EB" and "Supermicro X9DRD-iF/LF, Dual Xeon E5-2630, 2x I350, 2x 82575EB" if any? Are you using any SATA/SAS controllers? Thanks, Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Xen C6 kernel 4.9.13 and testing 4.9.15 only reboots.
On 03/24/2017 11:35 AM, PJ Welsh wrote: > As a follow up I was able to test fresh install on Dell R710 and a Dell > R620 with success on CentOS 7.3 without issue on the new kernel. My new > plan will be to just move this C6 to one of the C7 I just created. That sounds like a compiler problem, since I think the C6 and C7 kernels are built from the same source. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] xen 4.7 or 4.8
On 02/18/2017 06:23 AM, Johnny Hughes wrote: > On 02/18/2017 02:07 AM, Christoph wrote: >> >> Hi >> >> are there somewhere pkgs with xen 4.7 or 4.8 for centos7? >> > > The SIG has agreed to maintain every other release (even numbered release) of > Xen. So there will be a 4.8 set of packages at some point. > > There are currently no released 4.8 packages. > > I do see that we have some testing packages here: > > http://cbs.centos.org/repos/virt7-xen-48-testing/x86_64/os/Packages/ > > George Dunlap can talk to the planned schedule for 4.8 and if those packages > are really ready for testing. Any updates or asks on this? Thanks, Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Selinux Problem
On 01/30/2017 03:22 AM, George Dunlap wrote: > > I think that comment may be a little old. I do try to support SELinux > -- the smoke tests I use before pushing changes have it enabled by > default, and they use both qemu-xen and blktap. > > But it's difficult to help debug problems when you haven't even said > what problem(s) you're having. :-) > > Please be sure to include the output of `dmesg`, `xl dmesg`, your > xl.cfg, and /var/log/audit/audit.log. > > Thanks, > -George George, I appreciate you try to keep SELinux working and thank you. If SELinux isn't appropriate for an environment, disabling it is easy. But if it is needed for whatever reason, adding support is hard. Looking through our ansible role, it turns out that for xenconsoled to be able to work with oxenstored I had to make a policy change. I hesitate to publish that policy as-is because I used audit2allow without taking enough time to tune it and the policy is probably too permissive. But running xenconsoled with oxenstored on CentOS 6 should allow you to duplicate. If you don't have time to duplicate, I should be able to do that and get you the original audit.log messages. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Selinux Problem
On 01/26/2017 08:45 AM, Sarah Newman wrote: > On 01/26/2017 08:06 AM, Günther J. Niederwimmer wrote: >> Hello, >> >> CentOS 7.(3) Xen 4.4, >> >> Can I find any Doc for selinux with XEN, I found many Problems with selinux >> on >> Dom0 ? >> >> Or have I to disable selinux when I install XEN. >> >> Thank's for a answer. >> > > What problems and what version of CentOS? > > We leave selinux enabled. Sorry I'm blind, should have had more coffee. I would like to know what problems you're having specifically. We aren't on CentOS 7 yet unfortunately. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Selinux Problem
On 01/26/2017 08:06 AM, Günther J. Niederwimmer wrote: > Hello, > > CentOS 7.(3) Xen 4.4, > > Can I find any Doc for selinux with XEN, I found many Problems with selinux > on > Dom0 ? > > Or have I to disable selinux when I install XEN. > > Thank's for a answer. > What problems and what version of CentOS? We leave selinux enabled. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] Xen4CentOS kernel-debuginfo
I think this may have been asked before, but what would it take to get debuginfo packages built for the Xen4CentOS kernels? If it's just a patch for kernel.spec file that nobody has gotten around to, what is the best starting place for making that patch? Thanks, Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] how to enable auto-start on newer versions of Xen ?
On 08/23/2016 01:10 PM, Sarah Newman wrote: > On 08/23/2016 12:59 PM, Craig Thompson wrote: > >> I was thinking that in iterations past, it was possible just to run the >> hypervisor, put a symlink in /etc/xen/auto, and VMs would be auto-started >> upon reboot. Maybe I'm wrong, and maybe those older boxes had libvirt >> installed on them, too. >> > > We haven't had issues on xen 4.6. We don't use libvirt. > > Are you sure xendomains is enabled? Maybe in the past it was enabled by > default but isn't now. > > If it is enabled, have you tried running the xendomains init script manually > with bash -x to see where it's going wrong? You need to add 'set -x' to > /usr/lib64/xen/bin/xendomains as well for full output. > Sorry, I missed the CentOS 7 bit. Looking at https://github.com/CentOS-virt7/xen/blob/xen-46/SPECS/xen.spec#L613 it looks like xendomains is not enabled by default. It looks like that happened in https://github.com/CentOS-virt7/xen/commit/01a31207ef50ec8c664fa0807ac85075dca3aa53. No clue as to why. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] how to enable auto-start on newer versions of Xen ?
On 08/23/2016 12:59 PM, Craig Thompson wrote: > I was thinking that in iterations past, it was possible just to run the > hypervisor, put a symlink in /etc/xen/auto, and VMs would be auto-started > upon reboot. Maybe I'm wrong, and maybe those older boxes had libvirt > installed on them, too. > We haven't had issues on xen 4.6. We don't use libvirt. Are you sure xendomains is enabled? Maybe in the past it was enabled by default but isn't now. If it is enabled, have you tried running the xendomains init script manually with bash -x to see where it's going wrong? You need to add 'set -x' to /usr/lib64/xen/bin/xendomains as well for full output. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Problems with scsi-target-utils when hosted on dom0 centos 7 xen box
On 04/12/2016 05:18 PM, Nathan Coulson wrote: > (Apologies for the earlier top post) > > Running the kernel natively, on 3.10 or 3.18 (kernel from virt sig) > * CentOS Linux (3.18.25-19.el7.x86_64) 7 (Core)) > * CentOS Linux (3.10.0-327.13.1.el7.x86_64) 7 (Core) > > It works as expected with no issues. > > > > But when booting as dom0 using the xen hypervisor > * CentOS Linux, with Xen hypervisor > > tested with dom0_mem=2048M,max:2048M (as well as 1024M) > > the problems occur as I have describe. > This probably isn't it, but there was a problem I was having where the workaround was to give the dom0 4096M+ so that 64 bit references were used instead of 32 bit. You could try that and see what happens. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Xen custom vifname
On 04/05/2016 01:42 PM, Peter Braun wrote: > Hi, > > Centos 7 > Xen4CentOS - 4.6.1 > Open vSwitch > > When i specify custom name of the domU NIC than it refues to start > > vif = [ 'vifname=vNIC1' ] > > > libxl: error: libxl_exec.c:118:libxl_report_child_exitstatus: > /etc/xen/scripts/vif-openvswitch online [10168] exited with error status 1 > libxl: error: libxl_device.c:1084:device_hotplug_child_death_cb: script: ip > link set vif38.0 name vNIC1 failed > libxl: error: libxl_create.c:1381:domcreate_attach_vtpms: unable to add nic > devices > libxl: error: libxl.c:1591:libxl__destroy_domid: non-existant domain 38 > libxl: error: libxl.c:1549:domain_destroy_callback: unable to destroy guest > with domid 38 > libxl: error: libxl.c:1476:domain_destroy_cb: destruction of domain 38 failed > > > When removed the vifname settings the domU starts just fine. > > vif = [ '' ] > > Same error when used wi classic Bridge for networking. > Can you add set -x to capture the output from the scripts? It's not possible to change the name while the interface is in use. Maybe something is making the interface active before there's a chance to change the name? Do you have network manager or anything like it running? I think there would be logs in /var/log/messages if it was doing something with the vif. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Soft lockups with Xen4CentOS 3.18.25-18.el6.x86_64
On 03/10/2016 12:05 AM, Sarah Newman wrote: > On 03/09/2016 08:15 PM, Sarah Newman wrote: >> I've been running 3.18.25-18.el6.x86_64 + our build of xen 4.4.3-9 on one >> host for the last couple of weeks and have gotten several soft lockups >> within the last 24 hours. I am posting here first in case anyone else has >> experienced the same issue. >> > > Here is mpstat from around the time of the issue: > > 0:08:56 PM CPU%usr %nice%sys %iowait%irq %soft %steal > %guest %idle > 10:09:10 PM all0.000.00 66.670.000.00 33.330.00 > 0.000.00 > 10:09:11 PM all2.170.005.43 32.610.00 58.701.09 > 0.000.00 > 10:09:12 PM all0.000.001.150.000.00 85.060.00 > 0.00 13.79 > 10:09:13 PM all0.000.001.080.000.00 83.870.00 > 0.00 15.05 > 10:09:14 PM all0.000.001.100.000.00 83.520.00 > 0.00 15.38 > 10:09:15 PM all1.090.001.090.000.00 85.870.00 > 0.00 11.96 > 10:09:51 PM all0.000.001.090.000.00 84.781.09 > 0.00 13.04 > Message from syslogd at Mar 9 22:09:51 ... > kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0] > 10:10:02 PM all0.000.00 33.33 50.000.00 16.670.00 > 0.000.00 > 10:10:03 PM all3.160.00 10.538.420.002.111.05 > 0.00 74.74 > 10:10:04 PM all0.000.003.23 38.710.001.081.08 > 0.00 55.91 > 10:10:05 PM all0.000.004.30 11.830.003.231.08 > 0.00 79.57 > > Typical load: > > 10:22:15 PM CPU%usr %nice%sys %iowait%irq %soft %steal > %guest %idle > 10:22:16 PM all0.000.001.020.000.001.020.00 > 0.00 97.96 > 10:22:17 PM all0.000.000.000.000.000.001.04 > 0.00 98.96 > 10:22:18 PM all0.000.000.000.000.001.011.01 > 0.00 97.98 > 10:22:19 PM all0.000.001.010.000.001.010.00 > 0.00 97.98 > 10:22:20 PM all0.000.000.000.000.000.001.02 > 0.00 98.98 > 10:22:21 PM all0.000.001.020.000.001.020.00 > 0.00 97.96 > 10:22:22 PM all0.000.000.000.000.001.011.01 > 0.00 97.98 > > > I reverted to an older kernel since the older kernel had run for a couple of > months without issues. This did not fix it. I isolated the issue to a vif rate limit of 100Mb/s being applied to one of the guests and am now able to reproduce on a different machine. I will look into whether this has been fixed already; if so I will submit a pull request for the Xen4CentOS kernel and if not I will take it up with the xen-devel list. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Soft lockups with Xen4CentOS 3.18.25-18.el6.x86_64
On 03/09/2016 08:15 PM, Sarah Newman wrote: > I've been running 3.18.25-18.el6.x86_64 + our build of xen 4.4.3-9 on one > host for the last couple of weeks and have gotten several soft lockups > within the last 24 hours. I am posting here first in case anyone else has > experienced the same issue. > Here is mpstat from around the time of the issue: 0:08:56 PM CPU%usr %nice%sys %iowait%irq %soft %steal %guest %idle 10:09:10 PM all0.000.00 66.670.000.00 33.330.00 0.000.00 10:09:11 PM all2.170.005.43 32.610.00 58.701.09 0.000.00 10:09:12 PM all0.000.001.150.000.00 85.060.00 0.00 13.79 10:09:13 PM all0.000.001.080.000.00 83.870.00 0.00 15.05 10:09:14 PM all0.000.001.100.000.00 83.520.00 0.00 15.38 10:09:15 PM all1.090.001.090.000.00 85.870.00 0.00 11.96 10:09:51 PM all0.000.001.090.000.00 84.781.09 0.00 13.04 Message from syslogd at Mar 9 22:09:51 ... kernel:NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0] 10:10:02 PM all0.000.00 33.33 50.000.00 16.670.00 0.000.00 10:10:03 PM all3.160.00 10.538.420.002.111.05 0.00 74.74 10:10:04 PM all0.000.003.23 38.710.001.081.08 0.00 55.91 10:10:05 PM all0.000.004.30 11.830.003.231.08 0.00 79.57 Typical load: 10:22:15 PM CPU%usr %nice%sys %iowait%irq %soft %steal %guest %idle 10:22:16 PM all0.000.001.020.000.001.020.00 0.00 97.96 10:22:17 PM all0.000.000.000.000.000.001.04 0.00 98.96 10:22:18 PM all0.000.000.000.000.001.011.01 0.00 97.98 10:22:19 PM all0.000.001.010.000.001.010.00 0.00 97.98 10:22:20 PM all0.000.000.000.000.000.001.02 0.00 98.98 10:22:21 PM all0.000.001.020.000.001.020.00 0.00 97.96 10:22:22 PM all0.000.000.000.000.001.011.01 0.00 97.98 I reverted to an older kernel since the older kernel had run for a couple of months without issues. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] Soft lockups with Xen4CentOS 3.18.25-18.el6.x86_64
I've been running 3.18.25-18.el6.x86_64 + our build of xen 4.4.3-9 on one host for the last couple of weeks and have gotten several soft lockups within the last 24 hours. I am posting here first in case anyone else has experienced the same issue. Here is the first instance: sched: RT throttling activated NMI watchdog: BUG: soft lockup - CPU#0 stuck for 22s! [swapper/0:0] Modules linked in: ebt_arp xen_pciback xen_gntalloc ebt_ip ebtable_filter ebtables ipt_REJECT nf_reject_ipv4 nf_conntrack_ipv4 nf_defrag_ipv4 iptable_filter ip_tables xt_physdev br_netfilter bridge stp llc ip6t_REJECT nf_reject_ipv6 nf_c onntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 xen_acpi_processor blktap xen_netback xen_blkback xen_gntdev xen_evtchn xenfs xen_privcmd joydev sg 8250_fintek serio_raw gpio_ich iTCO_wdt iTCO_vendor_su pport coretemp intel_powerclamp crct10dif_pclmul crc32_pclmul crc32c_intel pcspkr i2c_i801 lpc_ich igb ptp pps_core hwmon ioatdma dca i7core_edac edac_core shpchp ext3 jbd mbcache raid10 raid1 sd_mod mptsas mptscsih mptbase scsi_transpor t_sas aesni_intel ablk_helper cryptd lrw gf128mul glue_helper aes_x86_64 ahci libahci mgag200 ttm drm_kms_helper dm_mirror dm_region_hash dm_log dm_mod CPU: 0 PID: 0 Comm: swapper/0 Not tainted 3.18.25-18.el6.x86_64 #1 Hardware name: Supermicro X8DTN+-F/X8DTN+-F, BIOS 080016 08/03/2011 task: 81c1b4c0 ti: 81c0 task.ti: 81c0 RIP: e030:[] [] xenvif_tx_build_gops+0xa5/0x890 [xen_netback] RSP: e02b:88013f403ca8 EFLAGS: 0206 RAX: 003c RBX: c90012c28000 RCX: c90012c280d0 RDX: 00071ea1 RSI: 0040 RDI: c90012c28000 RBP: 88013f403e38 R08: R09: 0001 R10: R11: R12: 00071e65 R13: 88013f403e50 R14: 003c R15: 0032 FS: 7fe942ac7980() GS:88013f40() knlGS: CS: e033 DS: ES: CR0: 8005003b CR2: 807f6800 CR3: bcd4c000 CR4: 2660 Stack: 88013f403d30 880006de6800 8800adfee1c0 88013f403e54 00015786e75a 0040a0351878 c90012c2def0 00015786e73c c90012c280d0 c90012c2def0 a03516c0 8800bf29e000 Call Trace: [] ? br_handle_frame_finish+0x3f0/0x3f0 [bridge] [] ? __netif_receive_skb_core+0x1ee/0x640 [] ? __netif_receive_skb+0x27/0x70 [] ? netif_receive_skb_internal+0x2d/0x90 [] ? igb_alloc_rx_buffers+0x63/0xe0 [igb] [] xenvif_tx_action+0x4d/0xa0 [xen_netback] [] xenvif_poll+0x35/0x68 [xen_netback] [] net_rx_action+0x112/0x2a0 [] __do_softirq+0xfc/0x2b0 [] irq_exit+0xbd/0xd0 [] xen_evtchn_do_upcall+0x3c/0x50 [] xen_do_hypervisor_callback+0x1e/0x40 [] ? xen_hypercall_sched_op+0xa/0x20 [] ? xen_hypercall_sched_op+0xa/0x20 [] ? xen_safe_halt+0x10/0x20 [] ? default_idle+0x24/0xc0 [] ? arch_cpu_idle+0xf/0x20 [] ? cpuidle_idle_call+0xd6/0x1d0 [] ? __atomic_notifier_call_chain+0x12/0x20 [] ? cpu_idle_loop+0x135/0x1e0 [] ? cpu_startup_entry+0x1b/0x70 [] ? cpu_startup_entry+0x60/0x70 [] ? rest_init+0x77/0x80 [] ? start_kernel+0x441/0x448 [] ? set_init_arg+0x5d/0x5d [] ? x86_64_start_reservations+0x2a/0x2c [] ? xen_start_kernel+0x5ef/0x5f1 Code: 00 0f 87 06 07 00 00 44 8b b3 b8 00 00 00 44 03 b3 c0 00 00 00 45 29 e6 41 39 c6 44 0f 47 f0 45 85 f6 0f 84 8f 00 00 00 0f ae e8 <8b> 83 c0 00 00 00 83 e8 01 44 21 e0 48 8d 04 40 48 c1 e0 02 48 Of the remaining lockups, here is the common backtrace with the exception that there have been two instances of RIP being in net_rx_action: [] net_rx_action+0x112/0x2a0 [] __do_softirq+0xfc/0x2b0 [] irq_exit+0xbd/0xd0 [] xen_evtchn_do_upcall+0x3c/0x50 [] xen_do_hypervisor_callback+0x1e/0x40 [] ? xen_hypercall_sched_op+0xa/0x20 [] ? xen_hypercall_sched_op+0xa/0x20 [] ? xen_safe_halt+0x10/0x20 [] ? default_idle+0x24/0xc0 [] ? arch_cpu_idle+0xf/0x20 [] ? cpuidle_idle_call+0xd6/0x1d0 [] ? __atomic_notifier_call_chain+0x12/0x20 [] ? cpu_idle_loop+0x135/0x1e0 [] ? cpu_startup_entry+0x1b/0x70 [] ? cpu_startup_entry+0x60/0x70 [] ? rest_init+0x77/0x80 [] ? start_kernel+0x441/0x448 [] ? set_init_arg+0x5d/0x5d [] ? x86_64_start_reservations+0x2a/0x2c [] ? xen_start_kernel+0x5ef/0x5f1 I can post more complete backtraces if that information would be useful to someone. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Kernel 3.18.21-17 and 3.18.21-18 kernel panics
On 02/21/2016 08:02 PM, Shaun Reitan wrote: > I've seen this issue on about 15 different servers now. Anybody else seeing > this? > > Screenshot: http://imgur.com/cBcwr8l > > I also have a video of the boot process if that will be helpful. Are you missing the initrd line in /boot/grub/menu.lst? That's a known bug that I believe has been fixed with 3.18.25-something. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] XSAs 170 and 154, repository layouts, and centos-release-xen 8-1
On 02/17/2016 04:30 AM, George Dunlap wrote: > I have the following packages going through the CBS: > * A CentOS 7 xen-4.6.1-2, with XSAs 170 and 154 > * A CentOS 6 xen-4.6.1-2, with XSAs 170 and 154 > * A CentOS 6 xen-4.4.3-11, with XSAs 170 > > All these should show up in mirrors hopefully sometime later today. > As usual, please report any problems here. Domains using the distribution provided pvgrub won't boot after upgrade. Old location of pvgrub: /usr/lib/xen/boot/pv-grub-x86_32.gz /usr/lib/xen/boot/pv-grub-x86_64.gz New location of pvgrub: /usr/lib64/xen/boot/pv-grub-x86_32.gz /usr/lib64/xen/boot/pv-grub-x86_64.gz ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] CentOS 6 Virt SIG Xen 4.6 packages available in centos-virt-xen-testing
On 01/21/2016 04:32 AM, George Dunlap wrote: > I'm a developer, not a server admin, so I can't gauge how important > this issue is. Before making such a change, I'd like to hear opinions > from other people in the community about how important (or not) it is > to avoid breaking xm, given the ample warning (>1 year) users have > had. > > On the other hand, explicitly moving to a "xen${VER}" (both for C6 and > C7) would make it simpler for people to step up and maintain older > versions in parallel if anybody wanted to do so. My inclination is towards a naming scheme like xen46, xen48, etc + a meta package that always depends on the latest. It should be more obvious when there's a major upgrade, and those who can't afford a major upgrade can uninstall the meta package. For the record, we have no particular desire for xen 4.4 but haven't done enough testing to say xen 4.6 is good yet. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] Xen4CentOS and XSA-142
It looks like no XSA-142 patch, which is "libxl fails to honour readonly flag on disks with qemu-xen" has been applied to Xen4CentOS. I assume this was on purpose? If not, I can have someone try adding the original patch from http://xenbits.xen.org/xsa/advisory-142.html and some variant of the commit from ef6cb76026628e26e3d1ae53c50ccde1c3c78b1b http://xen.1045712.n5.nabble.com/xen-master-libxl-relax-readonly-check-introduced-by-XSA-142-fix-td5729704.html . Thanks, Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] 3.18.21-16 upgrade, kernel panic, unable to mount root fs
On 11/01/2015 12:07 PM, Karel Hendrych wrote: > Hi, just a heads-up: 3.18.21 didn't boot up on HP ML310e Gen8 v2, SATA drives > in AHCI mode, / on software raid 1, no LVM. It ended up in kernel panic > with unable to mount root fs. Attached. No difference in grub kernel/xen > settings. > > I didn't investigate the things deeper so far. 4.4.3-3/3.18.17 is booting > fine. > > Any similar experience? Check the module line for the initramfs is present in grub.conf. It's gone missing in all our recent upgrades, but I wasn't sure if it was an upstream problem or our own. ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] Changelog for xen-4.4.3-3
Just a heads up, the changelog for the xen package says * Thu Oct 22 2015 George Dunlap - 4.4.3-2.el6.centos - Import XSAs 149-153 The source RPM also has the patch for XSA 148, as shown in the git log. Regards, Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org https://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] How to kickstart a guest with Centos 6.6 & Xen 4.4
On 07/30/2015 05:29 AM, Elliot Fox wrote: > The TL;DR is: Everything I've found to kickstart a new vanilla rhel/centos > guest points to specifying a kernel & and initrd- for RHEL/Centos 5. But > where is the xen initrd for Centos 6? The Xen4Quickstart instructions are > awesome, but after install you are left with a kernel and an initramfs & no > initrd for a centos 6 guest in xen. > > So is there a better way to kickstart a fresh/vanilla VM post Xen4Quickstart > instructions? Shall I build an initrd from the initramfs (and how) or is > there another way? If you want do a kickstart install using the PXE initrd.img, use the matching PXE vmlinuz. Generally speaking, don't mix and match kernel and initramfs versions and assume it will work even if they're from the same major release. Where to find the kickstart file needs to be specified in the normal way https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Installation_Guide/s1-kickstart2-startinginstall.html ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] /var/lib/xenstored & Xen4CentOS
On 06/04/2015 08:32 AM, George Dunlap wrote: > for i in {1..80} ; do xl create null.cfg name=\"t$i\" > kernel=\"/boot/vmlinuz-3.18.12-11.el7.x86_64\" memory=\"40\" > on_crash=\"preserve\" ; done > > And then used the shell snippet from the bug report to generate load: > > while true; do xl list &> /dev/null; usleep 5000;done > > With xenstored normally the CPU load was about 30%. Mounted under > tmpfs, it was slightly smaller, around 20%, but swung around wildly. > With oxenstored it was about 15%. > > This is with xfs as the filesystem. > > That doesn't seem big enough to warrant a lot of effort: if you can > reproduce the 75% numbers from the bug report I'll pursue it further, > otherwise I'll probably just leave it. I'm actually more concerned about I/O wait than CPU usage. When shutting down or starting a lot of guests the wait time could be significantly increased. I only started to look into this because I was doing load testing and it was taking much longer to shut everything down then it seemed like it should. Running multiple versions of xl in parallel (especially xl destroy) can make many, many backups of tdb such that there's hundreds of MB of backup copies. --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
[CentOS-virt] /var/lib/xenstored & Xen4CentOS
By default, /var/lib/xenstored is mounted tmpfs in centos 5 but this doesn't appear to be true for Xen4CentOS. This can cause performance issues as mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=446089 I'm not sure if this should be part of xencommons, but if not, adding a tmpfs mount should probably be documented on the quick start wiki page. Thanks, Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] CentOS 6 VM image for paravirtualizaton on CentOS Xen server
On 03/15/2015 07:39 PM, Nico Kadel-Garcia wrote: > On Sun, Mar 15, 2015 at 8:26 PM, Sarah Newman wrote: >> On 03/15/2015 03:25 PM, Nico Kadel-Garcia wrote: >>> So I'm right back to my effectively unanswered original questions. So >>> please: I asked a very specific pair of questions, and they remain >>> unanswered. CentOS 5 Xen server (hypervisor, or Dom0, whatever we want >>> to call it this week): Does CentOS 6 work, paravirtualized, on such a >>> server? And given my deployment issues, does anyone have a base OS >>> image I can get a copy of? >> >> I'm not sure why you need tftp to do a net install assuming you control the >> guest configuration. >> >> How about >> >> kernel = file from >> https://mirrors.kernel.org/centos/6/os/i386/images/pxeboot/vmlinuz >> ramdisk = file from >> https://mirrors.kernel.org/centos/6/os/i386/images/pxeboot/initrd.img >> extra = "console=hvc0" >> memory = 512 or larger >> >> --Sarah > > I'll be happy to try this. Thank you for the pointer. Are you > confident that 'console=hvc0' is the right installation time message > for this? And will I be able to access a CD or DVD image for actual OS > installation with these options? It is net install IE you download packages over the network. No CD/DVD. console=hvc0 is the xen part, otherwise look at the standard installation instructions for centos 6. > > I'd really prefer to work from 'virsh' than from hand-writing xl > configuration files.When I last did this sort of thing, I worked from > a PXE environment that I controlled and could reserve DHCP settings > based on MAC addresses, and tune PXE to boot from disk by default but > allow users to select a clean re-install of the operating system they > wanted. virt-install with --location maybe? Never tried it but it looks like what you want. http://linux.die.net/man/1/virt-install ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] CentOS 6 VM image for paravirtualizaton on CentOS Xen server
On 03/15/2015 03:25 PM, Nico Kadel-Garcia wrote: > So I'm right back to my effectively unanswered original questions. So > please: I asked a very specific pair of questions, and they remain > unanswered. CentOS 5 Xen server (hypervisor, or Dom0, whatever we want > to call it this week): Does CentOS 6 work, paravirtualized, on such a > server? And given my deployment issues, does anyone have a base OS > image I can get a copy of? I'm not sure why you need tftp to do a net install assuming you control the guest configuration. How about kernel = file from https://mirrors.kernel.org/centos/6/os/i386/images/pxeboot/vmlinuz ramdisk = file from https://mirrors.kernel.org/centos/6/os/i386/images/pxeboot/initrd.img extra = "console=hvc0" memory = 512 or larger --Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Loading a FreeBSD 10 VM on QEMU-KVM..
Do you have to use KVM? IIRC it runs fine under virtualbox. I think 10.2+ should work on xen but haven't tried it yet. On 01/13/2014 07:52 PM, Howard Leadmon wrote: > > I was trying to load a FreeBSD 10 VM on my CentOS 6.4 machine, and it keeps > hanging and not completing the boot. I have FBSD 9.x VM's running just > fine, but if I try and load 10.x it's a no go. > > Attaching to the console using VNC, I see: > > > gPXE (http://etherboot.org) - 00:04.0 C980 PCI2.10 PnP BBS PMM7FC0@20 C980 > > Booting from DVD/CD... > CD Loader 1.2 > > Building the boot loader arguments > Looking up the /BOOT/LOADER... Found > Relocating the loader and the BTX > > > > That is it, at that point it just hangs. I have tried from 2G RAM to 6G > RAM for the VM, and from 1 to 4 CPU's, but no effect. The FreeBSD lists > said I should load a newer QEMU-KVM, that there are much newer releases but > everything I have seen claims that is not a good idea, that RH does version > numbering much differently. > > Has anyone run into this, or have any ideas on how to get past it, as I > would love to load up the newest FBSD and give it a run as well.. > > > > --- > Howard Leadmon > > > > > ___ > CentOS-virt mailing list > CentOS-virt@centos.org > http://lists.centos.org/mailman/listinfo/centos-virt > ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Xen on CentOS 6.4
On 05/26/2013 05:33 PM, Trey Dockendorf wrote: > > On May 26, 2013 6:52 PM, "Sarah Newman" <mailto:s...@prgmr.com>> wrote: > > > > On 05/26/2013 02:12 PM, John R. Dennison wrote: > > > http://dev.centos.org/centos/6/xen-c6 > > > > > > Still in development and testing but it has a lot of momentum behind it. > > > > I saw packages for libvirt but not xcp-xapi. Do you know if there are any > plans to support xapi? > > The lines between Xen and KVM may become less clear in the future as the > Linux Foundation has taken > up Xen [1]. > > As for Xen vs KVM now, I like the higher level tools built to use KVM, such > as oVirt/RHEV as well as > the integration of KVM management via Libvirt with tools like Foreman. I've > run some IO intense > programs using KVM without issue and am able to use all stock EL packages to > do so. It's true that libvirt also supports KVM. My question is about plans for the xcp-xapi toolstack, see http://wiki.xen.org/wiki/Choice_of_Toolstacks In some higher-level tools libvirt is used for kvm only, for xen they only support xcp-xapi. Regards, Sarah ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt
Re: [CentOS-virt] Xen on CentOS 6.4
On 05/26/2013 02:12 PM, John R. Dennison wrote: > http://dev.centos.org/centos/6/xen-c6 > > Still in development and testing but it has a lot of momentum behind it. I saw packages for libvirt but not xcp-xapi. Do you know if there are any plans to support xapi? ___ CentOS-virt mailing list CentOS-virt@centos.org http://lists.centos.org/mailman/listinfo/centos-virt