Hi Guys,

I am having rather odd issue where the system locks up, I have purchased a XEN based VPS and decided to go with FreeBSD as the OS. I frequently get cases where there seem to be noticeable pauses, ie: I could be SSH'd in and issue an "ls" command or "mc" to start midnight commander, as soon as i press enter to run the command, there is often a good pause for 20 seconds or so. Load averages are almost non existent because it's a new install, almost no network traffic hitting it. if i ping the IP address of the VPS during the pauses, there is abnormally high latency (instead of 200ms, 1500-2000ms). A traceroute during these times shows the high latency is at the final hop.

I noticed some oddities, here is a full DMESG and my kernel config:

--------------------------------------------------------------------

Copyright (c) 1992-2010 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.1-RELEASE-p2 #0: Sat Dec 18 02:49:02 EST 2010
    a...@svr.mydomain.net:/usr/obj/usr/src/sys/custom amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2398.80-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206c2 Family = 6 Model = 2c Stepping = 2 Features=0x1781fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x80982201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,POPCNT,<b31>>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  TSC: P-state invariant
real memory  = 1572864000 (1500 MB)
avail memory = 1508638720 (1438 MB)
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  2
 cpu2 (AP): APIC ID:  4
 cpu3 (AP): APIC ID:  6
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
kbd1 at kbdmux0
acpi0: <Xen> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x1f48-0x1f4b on acpi0
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc220-0xc22f at device 1.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pci0: <serial bus, USB> at device 1.2 (no driver attached)
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3000000-0xf3000fff at device 2.0 on pci0
pci0: <unknown> at device 3.0 (no driver attached)
re0: <RealTek 8139C+ 10/100BaseTX> port 0xc100-0xc1ff mem 0xf3001000-0xf30010ff irq 32 at device 4.0 on pci0
re0: Chip rev. 0x74800000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rlphy0: <RealTek internal media interface> PHY 0 on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
re0: Ethernet address: 00:16:3e:f0:8b:6c
re0: [FILTER]
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
orm0: <ISA Option ROM> at iomem 0xc9000-0xc97ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
ad0: 46080MB <QEMU HARDDISK 0.10.2> at ata0-master WDMA2
acd0: CDROM <QEMU DVD-ROM/0.10.2> at ata1-master WDMA2
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Trying to mount root from ufs:/dev/ad0s1a
calcru: runtime went backwards from 23875254 usec to 22130573 usec for pid 1414 (csh)
calcru: runtime went backwards from 5675 usec to 5054 usec for pid 1413 (su)
calcru: runtime went backwards from 19908 usec to 17728 usec for pid 1395 (bash) calcru: runtime went backwards from 74003 usec to 65901 usec for pid 1395 (bash) calcru: runtime went backwards from 27707 usec to 24673 usec for pid 1390 (sshd) calcru: runtime went backwards from 16882 usec to 15033 usec for pid 1390 (sshd) calcru: runtime went backwards from 2960 usec to 2636 usec for pid 1377 (getty) calcru: runtime went backwards from 4201 usec to 3741 usec for pid 1376 (getty) calcru: runtime went backwards from 2334 usec to 2078 usec for pid 1375 (getty) calcru: runtime went backwards from 3360 usec to 2992 usec for pid 1374 (getty) calcru: runtime went backwards from 4099 usec to 3650 usec for pid 1373 (getty) calcru: runtime went backwards from 3052 usec to 2718 usec for pid 1372 (getty) calcru: runtime went backwards from 3001 usec to 2673 usec for pid 1371 (getty) calcru: runtime went backwards from 3503 usec to 3120 usec for pid 1370 (getty) calcru: runtime went backwards from 2561 usec to 2280 usec for pid 1293 (sshd) calcru: runtime went backwards from 20635 usec to 18376 usec for pid 1171 (sh) calcru: runtime went backwards from 88696 usec to 78985 usec for pid 1171 (sh) calcru: runtime went backwards from 231 usec to 205 usec for pid 675 (dhclient) calcru: runtime went backwards from 2342 usec to 2086 usec for pid 659 (dhclient) calcru: runtime went backwards from 25186 usec to 22429 usec for pid 659 (dhclient) calcru: runtime went backwards from 464 usec to 413 usec for pid 110 (adjkerntz)
calcru: runtime went backwards from 10422 usec to 9861 usec for pid 1 (init)
calcru: runtime went backwards from 61599028 usec to 56035925 usec for pid 1906 (httpd) calcru: runtime went backwards from 2636 usec to 2063 usec for pid 1377 (getty) calcru: runtime went backwards from 3741 usec to 2928 usec for pid 1376 (getty) calcru: runtime went backwards from 2078 usec to 1626 usec for pid 1375 (getty) calcru: runtime went backwards from 2992 usec to 2341 usec for pid 1374 (getty) calcru: runtime went backwards from 3650 usec to 2857 usec for pid 1373 (getty) calcru: runtime went backwards from 2718 usec to 2127 usec for pid 1372 (getty) calcru: runtime went backwards from 2673 usec to 2092 usec for pid 1371 (getty) calcru: runtime went backwards from 3120 usec to 2441 usec for pid 1370 (getty) calcru: runtime went backwards from 2511 usec to 1965 usec for pid 1337 (inetd) calcru: runtime went backwards from 88691784 usec to 69414995 usec for pid 1337 (inetd) calcru: runtime went backwards from 96115 usec to 76173 usec for pid 1293 (sshd) calcru: runtime went backwards from 18376 usec to 14382 usec for pid 1171 (sh) calcru: runtime went backwards from 78985 usec to 61818 usec for pid 1171 (sh) calcru: runtime went backwards from 1542 usec to 1306 usec for pid 676 (devd) calcru: runtime went backwards from 2086 usec to 1632 usec for pid 659 (dhclient) calcru: runtime went backwards from 22429 usec to 17554 usec for pid 659 (dhclient) calcru: runtime went backwards from 413 usec to 323 usec for pid 110 (adjkerntz)
calcru: runtime went backwards from 10170 usec to 8126 usec for pid 1 (init)
calcru: runtime went backwards from 111215702 usec to 89125025 usec for pid 1 (init)

----------------------------------------------------------------------

[a...@svr ~]$ cat /usr/src/sys/amd64/conf/custom
#

cpu             HAMMER
ident           custom

# To statically compile in device wiring instead of /boot/device.hints
#hints "GENERIC.hints" # Default places to look for devices.

# Use the following to compile in values accessible to the kernel
# through getenv() (or kenv(1) in userland). The format of the file
# is 'variable=value', see kenv(1)
#
# env           "GENERIC.env"

makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols

options         SCHED_ULE               # ULE scheduler
options         PREEMPTION              # Enable kernel thread preemption
options         INET                    # InterNETworking
options         FFS                     # Berkeley Fast Filesystem
options         SOFTUPDATES             # Enable FFS soft updates support
options         UFS_ACL                 # Support for access control lists
options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling
options         MD_ROOT                 # MD is a potential root device
options         MSDOSFS                 # MSDOS Filesystem
options         CD9660                  # ISO 9660 Filesystem
options PROCFS # Process filesystem (requires PSEUDOFS)
options         PSEUDOFS                # Pseudo-filesystem framework
options         GEOM_PART_GPT           # GUID Partition Tables.
options         GEOM_LABEL              # Provides labelization
options         COMPAT_43TTY            # BSD 4.3 TTY compat (sgtty)
options         COMPAT_FREEBSD32        # Compatible with i386 binaries
options         COMPAT_FREEBSD4         # Compatible with FreeBSD4
options         COMPAT_FREEBSD5         # Compatible with FreeBSD5
options         COMPAT_FREEBSD6         # Compatible with FreeBSD6
options         COMPAT_FREEBSD7         # Compatible with FreeBSD7
options         SCSI_DELAY=5000         # Delay (in ms) before probing SCSI
options         KTRACE                  # ktrace(1) support
options         STACK                   # stack(9) support
options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores
options         P1003_1B_SEMAPHORES     # POSIX-style semaphores
options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed.
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4)
options         AUDIT                   # Security event auditing
options         MAC                     # TrustedBSD MAC Framework
options         FLOWTABLE               # per-cpu routing cache
#options        KDTRACE_FRAME           # Ensure frames are compiled in
#options        KDTRACE_HOOKS           # Kernel DTrace hooks
options         INCLUDE_CONFIG_FILE     # Include this file in kernel

# Make an SMP-capable kernel by default
options        SMP                     # Symmetric MultiProcessor Kernel

# CPU frequency control
device         cpufreq

# Bus support.
device          acpi
device          pci

# ATA and ATAPI devices
device          ata
device          atadisk         # ATA disk drives
device          atapicd         # ATAPI CDROM drives
device          atapist         # ATAPI tape drives
options         ATA_STATIC_ID   # Static device numbering

# SCSI peripherals
device          scbus           # SCSI bus (required for SCSI)
device          da              # Direct Access (disks)
device          cd              # CD
device          pass            # Passthrough device (direct SCSI access)

# atkbdc0 controls both the keyboard and the PS/2 mouse
device          atkbdc          # AT keyboard controller
device          atkbd           # AT keyboard

device          kbdmux          # keyboard multiplexer
device          vga             # VGA video card driver
device          splash          # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device          sc

# PCI Ethernet NICs that use the common MII bus controller code.
# NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
device          miibus          # MII bus support
device          re              # RealTek 8139C+/8169/8169S/8110S

# Pseudo devices.
device          loop            # Network loopback
device          random          # Entropy device
device          ether           # Ethernet support
device          vlan            # 802.1Q VLAN support
device          tun             # Packet tunnel.
device          pty             # BSD-style compatibility pseudo ttys
device          md              # Memory "disks"
device          firmware        # firmware assist module

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device          bpf             # Berkeley packet filter

device pf
device pflog
device pfsync

# Xen HVM support
#options         XENHVM
#device          xenpci

-------------------------------------------

The XENHVM options commented out causes the VM not to boot (seems to hang after saying it's found a generic block device attached to the xenpci bus (virtual hard disk) and that it's adding it as ad0. The calcru: runtime messages are rather puzzling, perhaps this is related to the hangs I am seeing regularly. Any ideas on where to start looking?

Thanks!
Alex.


_______________________________________________
freebsd-xen@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-xen
To unsubscribe, send any mail to "freebsd-xen-unsubscr...@freebsd.org"

Reply via email to