Re: openldap client GSSAPI authentication segfaults in fbsd8stablei386
Applying Benjamin's patch to RELENG_8 sources csupped yesterday stops the buildworld in last library stage: In file included from /usr/src/lib/libc/rpc/clnt_dg.c:55: /usr/src/lib/libc/../../include/rpc/rpcsec_gss.h:52: error: expected specifier-qualifier-list before 'gss_cred_id_t' /usr/src/lib/libc/../../include/rpc/rpcsec_gss.h:65: error: expected specifier-qualifier-list before 'gss_ctx_id_t' /usr/src/lib/libc/../../include/rpc/rpcsec_gss.h:119: error: expected declaration specifiers or '...' before 'gss_cred_id_t' /usr/src/lib/libc/../../include/rpc/rpcsec_gss.h:120: error: expected declaration specifiers or '...' before 'gss_ctx_id_t' /usr/src/lib/libc/../../include/rpc/rpcsec_gss.h:149: error: expected declaration specifiers or '...' before 'gss_OID' /usr/src/lib/libc/../../include/rpc/rpcsec_gss.h:150: error: expected ')' before 'oid' *** Error code 1 Stop in /usr/src/lib/libc. *** Error code 1 Tried to poke around a bit but I was unable to find declarations for the missing types. How to proceed if I want to test the patch? -Reko ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: openldap client GSSAPI authentication segfaults in fbsd8stablei386
After manually changing the gssapi header used in /usr/src/include/rpc/rpcsec_gss.h to somewhat klunky #include /usr/src/crypto/heimdal/lib/gssapi/gssapi/gssapi.h system csupped yesterday built okay and after rebuilding cyrus-sasl, saslauthd and cyrus I get the following failures in log: Jul 18 16:37:35 moria perl: GSSAPI Error: Miscellaneous failure (see text)^B (open(/tmp/krb5cc_0): No such file or directory) -This is expected behaviour as Kerberos was not running at the moment, but with Benjamin's patch Kerberos/GSSAPI spat out a meaningful error message After dusting off my old Kerberos setup, doing basic kinit and running cyradm localhost I got: Jul 18 16:39:00 moria perl: GSSAPI Error: Miscellaneous failure (see text) (Server (imap/localh...@xxx.domain.com) unknown) -Again expected as there is no imap trust relationship defined. So at least after cursory testing it looks like that with Benjamin's patch there is a working GSSAPI/Kerberos backend available, instead of something that chokes on passed parameters that are ok for every other tested gssapi implementation. Of course, more thorough testing in proper kerberised/LDAP environment needs to be done, which is something I haven't got time at the moment. -Reko ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: SIGEPIPE after update to 8.1-RC2
On Sun, Jul 18, 2010 at 12:11:50AM +0200, Jilles Tjoelker wrote:
On Sat, Jul 17, 2010 at 06:24:55PM +0300, Alex Kozlov wrote:
After updating my buildbox from 26 April 8-STABLE
to 8.1-RC2 I constantly getting SIGEPIPE
portsnap:
Fetching 4 metadata patches... done.
Applying metadata patches... done.
Fetching 0 metadata files... done.
Fetching 27 patches.1020... done.
Applying patches... done.
Fetching 3 new ports or files... done.
sort: write failed: standard output: Broken pipe
sort: write error
Removing old files and directories... done.
sudo make -C /usr/ports/converters/ascii2binary:
=== Patching for ascii2binary-2.13_2
=== Applying FreeBSD patches for ascii2binary-2.13_2
=== ascii2binary-2.13_2 depends on shared library: intlgrep: writing
output: Broken pipe
grep: writing output: Broken pipe
[snip repetition]
- found
=== Configuring for ascii2binary-2.13_2
Does anyone know something about this issue?
This looks more like the absence of SIGPIPE than an inappropriate
SIGPIPE. I can reproduce both of those error messages by running the
commands with SIGPIPE ignored. grep(1) seems to behave strangely on
write errors, not aborting, for example
yes | { trap '' PIPE; grep -v foo; echo $? 2; } | :
prints an endless stream of error messages.
Note that sh(1) silently ignores attempts to change the disposition of
signals that were ignored on entry to the shell, so a
trap - PIPE
is unlikely to help you.
Similarly, SIGPIPE may be blocked (masked). Few programs expect this.
The -i and -j options in procstat should be helpful in finding what
exactly is wrong with SIGPIPE. (These options are relatively new, but
should be in 8.1.)
Might be, but now I have a feel that something more strange happens
there. One of my workstations does not exhibit the behaviour, while
another one did.
I composed the following grep wrapper to catch the situation you
guessed:
#!/bin/sh
disp=$(procstat -i $$ | awk '/PIPE/{print $4}')
if expr -- $disp : I /dev/null ; then
echo grep: SIGPIPE ignored /dev/tty
kill -STOP $$
fi
exec /usr/bin/grep $@
Amazingly enough, the messages stopped spitting. Even more, I cannot
reproduce them on the machine without the wrapper.
Side note: despite ports/Mk/bsd.commands.mk defining GREP and EGREP,
there are still several instances of the direct grep invocation
among Mk/* files. Do port people consider this worth fixing ?
pgpQuV0FjJApq.pgp
Description: PGP signature
em(4) + ALTQ
I'm wondering what the status of this problem is.. Will the fixes be in 8.1-RELEASE? I've checked through everything since the initial reports in February, and haven't seen a definitive answer.. Thanks! ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: openldap client GSSAPI authentication segfaults in fbsd8stablei386
On 07/17/2010 03:37 PM, Reko Turja wrote: Can you try reproducing the issue on 8-STABLE? I recently submitted a Heimdal patch against 8.1-STABLE and 9.0-CURRENT that resolves some libgssapi-related issues: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/147454 The patch breaks ABI, so you'll have to rebuild libgssapi-dependent applications. When linking cyrus-sasl2 against gssapi library from either the 1.0.1 official port or the inofficial 1.2.1 patchset cyradm works as expected and it logs a message from gssapi/kerberos telling that no KDC's are available - which is to be expected on a system that isn't using gssapi/kerberos in authenticating. So the present behaviour in 8-RELEASE and 8-PRERELASE updated Monday the 5th is clearly some kind of regression as system gsslib doesn't seem to recognize the mech used or segfaults. Benjamin, can you clarify how to apply your patch against the source tree - I tried 'patch the_patchset.diff' in /usr/src but it just created a bunch of files in the /usr/src which I think isn't the intention. Hi Reko, It looks like you've already figured it out (based on your responses elsewhere in the thread), but for the record you can apply the patch by running: cd /usr/src patch -p1 -E foo.diff That patch is over a month old now and no longer applies cleanly. I'll port it forward when I get a chance. -- Benjamin Lee http://www.b1c1l1.com/ signature.asc Description: OpenPGP digital signature
Re: openldap client GSSAPI authentication segfaults in fbsd8stablei386
On 07/18/2010 06:52 AM, Reko Turja wrote: After manually changing the gssapi header used in /usr/src/include/rpc/rpcsec_gss.h to somewhat klunky #include /usr/src/crypto/heimdal/lib/gssapi/gssapi/gssapi.h system csupped yesterday built okay and after rebuilding cyrus-sasl, saslauthd and cyrus I get the following failures in log: Jul 18 16:37:35 moria perl: GSSAPI Error: Miscellaneous failure (see text)^B (open(/tmp/krb5cc_0): No such file or directory) -This is expected behaviour as Kerberos was not running at the moment, but with Benjamin's patch Kerberos/GSSAPI spat out a meaningful error message After dusting off my old Kerberos setup, doing basic kinit and running cyradm localhost I got: Jul 18 16:39:00 moria perl: GSSAPI Error: Miscellaneous failure (see text) (Server (imap/localh...@xxx.domain.com) unknown) -Again expected as there is no imap trust relationship defined. So at least after cursory testing it looks like that with Benjamin's patch there is a working GSSAPI/Kerberos backend available, instead of something that chokes on passed parameters that are ok for every other tested gssapi implementation. Of course, more thorough testing in proper kerberised/LDAP environment needs to be done, which is something I haven't got time at the moment. Thanks for your testing! Based on the lack of attention my PR has received it seems that not many people have noticed the regression in libgssapi, even though the breaking commit happened in -CURRENT way back in 2008. When you get a chance, please append your test results to PR kern/147454. That may be helpful in attracting some more attention to this issue. -- Benjamin Lee http://www.b1c1l1.com/ signature.asc Description: OpenPGP digital signature
Reporting Functional Server Models
I spent some time last week validating the 7, 8 and -CURRENT on different vendor hardware over here in my lab. Is there a current h/w compatibility list that folks are maintaining that I can update with my findings? Sean ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
deadlock or bad disk ? RELENG_8
On the serial console I see swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 and on a session I had open from before # killall -9 watchdogd just hangs, I guess because its having trouble reading from the disk. If I hit CTRL+t, I see load: 0.00 cmd: csh 73167 [vnread] 22.32r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 22.65r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 22.96r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 23.20r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 23.40r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 23.61r 0.00u 0.00s 0% 3232k Its RELENG_8 amd64 from July 13th and the swap is on an ARECA drive and I dont see any errors on any of the raidset members. I also have a large zfs spool and a small mount point on a 3ware controller but unfortunately, nothing in the logs post reboot and nothing from smartctl cat /var/run/dmesg.boot Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-PRERELEASE #0: Tue Jul 13 09:55:48 EDT 2010 mdtan...@backup3.sentex.ca:/usr/obj/usr/src/sys/backup amd64 Timecounter i8254 frequency 1193182 Hz quality 0 CPU: Intel(R) Core(TM)2 Quad CPUQ6600 @ 2.40GHz (2400.10-MHz K8-class CPU) Origin = GenuineIntel Id = 0x6fb Family = 6 Model = f Stepping = 11 Features=0xbfebfbffFPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE Features2=0xe3bdSSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM AMD Features=0x20100800SYSCALL,NX,LM AMD Features2=0x1LAHF TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8267673600 (7884 MB) ACPI APIC Table: A_M_I_ OEMAPIC FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0 Version 2.0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: A_M_I_ OEMXSDT on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: reservation of fed08000, 1000 (3) failed acpi0: reservation of fed1c000, 4000 (3) failed acpi0: reservation of fed2, 2 (3) failed acpi0: reservation of fed5, 4 (3) failed acpi0: reservation of ffc0, 30 (3) failed acpi0: reservation of fec0, 1000 (3) failed acpi0: reservation of fee0, 1000 (3) failed acpi0: reservation of e000, 1000 (3) failed acpi0: reservation of 0, a (3) failed acpi0: reservation of 10, dff0 (3) failed Timecounter ACPI-fast frequency 3579545 Hz quality 1000 acpi_timer0: 24-bit timer at 3.579545MHz port 0x808-0x80b on acpi0 cpu0: ACPI CPU on acpi0 ACPI Warning: Incorrect checksum in table [OEMB] - 0xD1, should be 0xD0 (20100331/tbutils-354) cpu1: ACPI CPU on acpi0 cpu2: ACPI CPU on acpi0 cpu3: ACPI CPU on acpi0 acpi_hpet0: High Precision Event Timer iomem 0xfed0-0xfed003ff on acpi0 Timecounter HPET frequency 14318180 Hz quality 900 pcib0: ACPI Host-PCI bridge port 0xcf8-0xcff on acpi0 pci0: ACPI PCI bus on pcib0 pcib1: ACPI PCI-PCI bridge irq 16 at device 1.0 on pci0 pci1: ACPI PCI bus on pcib1 pcib2: PCI-PCI bridge at device 0.0 on pci1 pci3: PCI bus on pcib2 arcmsr0: Areca SATA Host Adapter RAID Controller mem 0xfc9ff000-0xfc9f irq 18 at device 14.0 on pci3 ARECA RAID ADAPTER0: Driver Version 1.20.00.16 2009-10-10 ARECA RAID ADAPTER0: FIRMWARE VERSION V1.44 2008-2-1 arcmsr0: [ITHREAD] pcib3: PCI-PCI bridge at device 0.2 on pci1 pci2: PCI bus on pcib3 uhci0: Intel 82801JI (ICH10) USB controller USB-D port 0x7800-0x781f irq 16 at device 26.0 on pci0 uhci0: [ITHREAD] usbus0: Intel 82801JI (ICH10) USB controller USB-D on uhci0 uhci1: Intel 82801JI (ICH10) USB controller USB-E port 0x7880-0x789f irq 21 at device 26.1 on pci0 uhci1: [ITHREAD] usbus1: Intel 82801JI (ICH10) USB controller USB-E on uhci1
Re: deadlock or bad disk ? RELENG_8
On Sun, Jul 18, 2010 at 05:08:09PM -0400, Mike Tancsa wrote: On the serial console I see swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 69, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 6, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 74, size: 4096 swap_pager: indefinite wait buffer: bufobj: 0, blkno: 128, size: 20480 [...] load: 0.00 cmd: csh 73167 [vnread] 22.32r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 22.65r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 22.96r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 23.20r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 23.40r 0.00u 0.00s 0% 3232k load: 0.00 cmd: csh 73167 [vnread] 23.61r 0.00u 0.00s 0% 3232k Where exactly is your swap partition? I ask because of the below paragraph starting with Its RELENG_8 amd64 from July 13th Its RELENG_8 amd64 from July 13th and the swap is on an ARECA drive and I dont see any errors on any of the raidset members. I also have a large zfs spool and a small mount point on a 3ware controller but unfortunately, nothing in the logs post reboot and nothing from smartctl If you Google for swap_pager: indefinite wait buffer: bufobj you'll find this is a pretty well-established problem, but the situation varies per person. A common one is here (read the entire thread): http://www.mail-archive.com/freebsd-questi...@freebsd.org/msg192481.html I have no advice as far as how to solve this problem. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: deadlock or bad disk ? RELENG_8
At 05:14 PM 7/18/2010, Jeremy Chadwick wrote: Where exactly is your swap partition? On one of the areca raidsets. # swapctl -l Device: 1024-blocks Used: /dev/da0s1b10485760 108 If you Google for swap_pager: indefinite wait buffer: bufobj you'll find this is a pretty well-established problem, but the situation varies per person. A common one is here (read the entire thread): http://www.mail-archive.com/freebsd-questi...@freebsd.org/msg192481.html I have no advice as far as how to solve this problem. If feels like a disk issue, but SMART values all seem ok eg CLI disk smart drv=1 S.M.A.R.T Information For Drive[#01] # Attribute Items Flag Value Thres State === 1 Raw Read Error Rate 0x0f 108 6 OK 3 Spin Up Time 0x03 91 0 OK 4 Start/Stop Count 0x32 100 20 OK 5 Reallocated Sector Count 0x33 100 36 OK 7 Seek Error Rate 0x0f 81 30 OK 9 Power-on Hours Count 0x32 79 0 OK 10 Spin Retry Count 0x13 100 97 OK 12 Device Power Cycle Count 0x32 100 20 OK 194 Temperature 0x22 30 0 OK 197 Current Pending Sector Count 0x12 100 0 OK 198 Off-line Scan Uncorrectable Sector Count 0x10 100 0 OK 199 Ultra DMA CRC Error Count 0x3e 200 0 OK smartctl -a -d 3ware,1 /dev/twa0 smartctl 5.39.1 2010-01-28 r3054 [FreeBSD 8.1-PRERELEASE amd64] (local build) Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Western Digital Raptor family Device Model: WDC WD740ADFD-00NLR1 Serial Number:WD-WMANS1051760 Firmware Version: 20.07P20 User Capacity:74,355,769,344 bytes Device is:In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: ATA/ATAPI-7 published, ANSI INCITS 397-2005 Local Time is:Sun Jul 18 17:41:36 2010 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x82) Offline data collection activity was completed without error. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (2391) seconds. Offline data collection capabilities:(0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities:(0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability:(0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time:( 2) minutes. Extended self-test routine recommended polling time:( 39) minutes. Conveyance self-test routine recommended polling time:( 5) minutes. SCT capabilities: (0x103f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 200 200 051Pre-fail Always - 0 3 Spin_Up_Time0x0007 170 170 021Pre-fail Always - 2508 4 Start_Stop_Count0x0032 100 100 040Old_age Always - 45 5 Reallocated_Sector_Ct 0x0033 200 200 140Pre-fail Always - 0 7 Seek_Error_Rate 0x000a 200 200 051Old_age Always - 0
Re: deadlock or bad disk ? RELENG_8
On Sun, Jul 18, 2010 at 05:42:14PM -0400, Mike Tancsa wrote: At 05:14 PM 7/18/2010, Jeremy Chadwick wrote: Where exactly is your swap partition? On one of the areca raidsets. # swapctl -l Device: 1024-blocks Used: /dev/da0s1b10485760 108 So is da0 actually a RAID volume behind the scenes on the Areca controller? How many disks are involved in that set? If you Google for swap_pager: indefinite wait buffer: bufobj you'll find this is a pretty well-established problem, but the situation varies per person. A common one is here (read the entire thread): http://www.mail-archive.com/freebsd-questi...@freebsd.org/msg192481.html I have no advice as far as how to solve this problem. If feels like a disk issue, but SMART values all seem ok Well, the thread I linked you stated that the problem has to do with a controller or disk taking too long. I have no idea what the threshold is. I suppose it could also indicate that your system is (possibly) running low on resources (RAM); I would imagine swap_pager would get called if a processes needed to be offloaded to swap. So maybe this is a system tuning thing more than a hardware thing. You should probably set up a series of monitoring scripts that monitor things like interrupt rate on devices, I/O statistics, and some general memory statistics to determine if processes are being swapped out excessively. vmstat and iostat would help here; see man page for relevant options (for swap stuff, vmstat -s). There's also systat with the -vmstat flag. CLI disk smart drv=1 [...] Unrelated to the problem, but important to note: The SMART output from the Areca CLI is hardly useful (bordering on worthless); it only shows the adjusted/calculated values and not the actual raw values. Even if the CLI lets you print this information, I would still strongly suggest using smartctl. There's no indication the Areca CLI has a quirks database for each drive model/type. I'm also not sure if the Areca CLI can provide the SMART error log, self-test log, or the selective self-test log. smartctl -a -d 3ware,1 /dev/twa0 Now I'm confused -- this indicates twa(4) is involved, not arcmsr(4). Can you please provide a verbose explanation of the configuration of the disks and controllers in this machine, including device and disk names and what they're associated with, plus if they're RAIDed in any way? Thanks. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: deadlock or bad disk ? RELENG_8
On Sun, Jul 18, 2010 at 07:34:19PM -0700, Jeremy Chadwick wrote: Now I'm confused -- this indicates twa(4) is involved, not arcmsr(4). Can you please provide a verbose explanation of the configuration of the disks and controllers in this machine, including device and disk names and what they're associated with, plus if they're RAIDed in any way? Thanks. I re-worked this out myself based on the OP's dmesg. It's confusing because there's literally 6 different storage controllers on a single machine: * arcmsr0 -- irq 18 -- Areca SATA Host Adapter RAID Controller siis0 -- irq 17 -- SiI3132 SATA controller * twa0-- irq 18 -- 3ware 9000 series Storage Controller ahci0 -- irq 16 -- JMicron JMB361 AHCI SATA controller atapci0 -- irq 17 -- JMicron JMB361 ATA controller * ahci1 -- irq 19 -- Intel ICH10 AHCI SATA controller Controllers marked with asterisk (*) are in use/involved. Others don't appear to have anything connected to them. Channels and what above controllers they're connected to. Again, same with the asterisk: ahcich0 -- ahci0 ahcich1 -- ahci0 ata2-- atapci0 * ahcich2 -- ahci1 * ahcich3 -- ahci1 * ahcich4 -- ahci1 * ahcich5 -- ahci1 ahcich6 -- ahci1 ahcich7 -- ahci1 The dmesg output also shows this. I have no idea what it means: (probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step Now we get into the disks. The kernel interspersed output within drivers so I had to work this out myself. da0 -- arcmsr0 -- Areca usrvar (RAID volume) da1 -- arcmsr0 -- Areca backup1 (RAID volume) da2 -- twa0-- No idea, but looks like a RAID volume ada0 -- ahcich2 -- ST31000340AS (disk) ada1 -- ahcich3 -- ST31000340AS (disk) ada2 -- ahcich4 -- ST31000333AS (disk) ada3 -- ahcich5 -- ST31000528AS (disk) So one thing of interest is that the Areca and 3ware controllers are sharing an IRQ. If you do extensive bidirectional I/O between disks on the arcmsr0 and twa0 controllers at the same time (e.g. read from arcmsr0 which writes to twa0, and read from twa0 which writes to arcmsr0), do you see this problem? vmstat -i output would help here, except that it's going to show the rate as a total (for both controllers). I don't know if a way to get more granular output. pciconf -lvc output might also help (to see if the controllers are using MSI or not); only interested in the arcmsr0, twa0, and ahci1 entries. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: deadlock or bad disk ? RELENG_8
At 10:34 PM 7/18/2010, Jeremy Chadwick wrote: On Sun, Jul 18, 2010 at 05:42:14PM -0400, Mike Tancsa wrote: At 05:14 PM 7/18/2010, Jeremy Chadwick wrote: Where exactly is your swap partition? On one of the areca raidsets. # swapctl -l Device: 1024-blocks Used: /dev/da0s1b10485760 108 So is da0 actually a RAID volume behind the scenes on the Areca controller? How many disks are involved in that set? yes, da0 is a RAID volume with 4 disks behind the scenes. Well, the thread I linked you stated that the problem has to do with a controller or disk taking too long. I have no idea what the threshold is. I suppose it could also indicate that your system is (possibly) running low on resources (RAM); I would imagine swap_pager would get called if a processes needed to be offloaded to swap. So maybe this is a system tuning thing more than a hardware thing. Prior to someone rebooting it, it had been stuck in this state for a good 90min. Apart from upgrading to a later RELENG_8 to get the security patches, the machine had been running a few versions of RELENG_8 doing the same workloads every week without issue. /boot/loader.conf has ahci_load=YES siis_load=YES sysctl.conf has net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=131072 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.sendspace=32768 net.inet.udp.recvspace=65536 kern.ipc.somaxconn=1024 kern.ipc.maxsockbuf=4194304 net.inet.ip.redirect=0 net.inet.ip.intr_queue_maxlen=4096 net.route.netisr_maxqlen=1024 kern.ipc.nmbclusters=131072 I do track some basic mem stats via rrd. Looking at the graphs upto that period, nothing unusual was happening CPU: 16.6% user, 0.0% nice, 4.3% system, 0.2% interrupt, 78.8% idle Mem: 443M Active, 5707M Inact, 1462M Wired, 147M Cache, 828M Buf, 166M Free Swap: 10G Total, 124K Used, 10G Free smartctl -a -d 3ware,1 /dev/twa0 Now I'm confused -- this indicates twa(4) is involved, not arcmsr(4). The other controllers (3ware and onboard ich in ahci mode) provider other storage on the same box. I only noted them in that I checked all their disks for errors of which there were none either. The dmesg from the original post enumerates all the devices on the box. ---Mike Mike Tancsa, tel +1 519 651 3400 Sentex Communications,m...@sentex.net Providing Internet since 1994www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: deadlock or bad disk ? RELENG_8
At 10:58 PM 7/18/2010, Jeremy Chadwick wrote: I re-worked this out myself based on the OP's dmesg. It's confusing because there's literally 6 different storage controllers on a single machine: Its a big storage server. Some files dont require fast or frequent access, others do. The disks on the sata controllers are used with zfs for the large files that require infrequent access for example. (probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step Thats been a normal message for Areca controllers for some time on AMD64 So one thing of interest is that the Areca and 3ware controllers are sharing an IRQ. If you do extensive bidirectional I/O between disks on the arcmsr0 and twa0 controllers at the same time (e.g. read from arcmsr0 which writes to twa0, and read from twa0 which writes to arcmsr0), do you see this problem? Its never been an issue in the past 2yrs. The same box was RELENG_7 for some time and then in the past 3 months updated to RELENG_8. vmstat -i output would help here, except that it's going to show the rate as a total (for both controllers). I don't know if a way to get more granular output. pciconf -lvc output might also help (to see if the controllers are using MSI or not); only interested in the arcmsr0, twa0, and ahci1 entries. interrupt total rate irq1: atkbd0 6 0 irq4: uart0 1049 0 irq18: arcmsr0 twa* 5221148151 irq19: fwohci0+ 151346 4 irq23: uhci3 ehci1 2 0 cpu0: timer 67544881 1962 irq256: em0 57430641 1668 irq257: ale0 3262365 94 irq258: ahci1 10406081302 cpu1: timer 67539701 1962 cpu2: timer 67168885 1951 cpu3: timer 67169530 1951 Total 345895635 10049 arcm...@pci0:3:14:0:class=0x010400 card=0x121017d3 chip=0x121017d3 rev=0x00 hdr=0x00 vendor = 'Areca Technology Corporation' device = 'ARC-1210 4-Port PCIe to SATA RAID Controller' class = mass storage subclass = RAID cap 01[c0] = powerspec 2 supports D0 D1 D3 current D0 cap 05[d0] = MSI supports 2 messages, 64 bit cap 07[e0] = PCI-X 64-bit supports 133MHz, 1024 burst read, 4 split transactions si...@pci0:8:0:0: class=0x010400 card=0x71321095 chip=0x31321095 rev=0x01 hdr=0x00 vendor = 'Silicon Image Inc (Was: CMD Technology Inc)' device = 'PCI Express (1x) to 2 Port SATA300 (SiI 3132)' class = mass storage subclass = RAID cap 01[54] = powerspec 2 supports D0 D1 D2 D3 current D0 cap 05[5c] = MSI supports 1 message, 64 bit cap 10[70] = PCI-Express 1 legacy endpoint max data 128(1024) link x1(x1) t...@pci0:7:0:0:class=0x010400 card=0x100413c1 chip=0x100413c1 rev=0x01 hdr=0x00 vendor = '3ware Inc' device = 'PCI-Express SATA2 Raid Controller (9650SE Series)' class = mass storage subclass = RAID cap 01[40] = powerspec 2 supports D0 D1 D2 D3 current D0 cap 05[50] = MSI supports 32 messages, 64 bit cap 10[70] = PCI-Express 1 legacy endpoint max data 128(512) link x1(x8) ah...@pci0:5:0:0: class=0x010601 card=0x824f1043 chip=0x2361197b rev=0x02 hdr=0x00 vendor = 'JMicron Technology Corp.' device = 'PCI Express to SATA II and PATA Host Controller (JMB363)' class = mass storage subclass = SATA cap 01[68] = powerspec 2 supports D0 D3 current D0 cap 10[50] = PCI-Express 1 legacy endpoint IRQ 2 max data 128(128) link x1(x1) ah...@pci0:0:31:2: class=0x010601 card=0x82d41043 chip=0x3a228086 rev=0x00 hdr=0x00 vendor = 'Intel Corporation' device = '6 port SATA AHCI Controller' class = mass storage subclass = SATA cap 05[80] = MSI supports 16 messages enabled with 1 message cap 01[70] = powerspec 3 supports D0 D3 current D0 cap 12[a8] = SATA Index-Data Pair cap 09[b0] = vendor (length 6) Intel cap 2 version 0 -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB | Mike Tancsa, tel +1 519 651 3400 Sentex Communications,m...@sentex.net Providing Internet since 1994www.sentex.net Cambridge, Ontario Canada www.sentex.net/mike ___ freebsd-stable@freebsd.org mailing list
Re: deadlock or bad disk ? RELENG_8
On Sun, Jul 18, 2010 at 11:01:03PM -0400, Mike Tancsa wrote: At 10:34 PM 7/18/2010, Jeremy Chadwick wrote: On Sun, Jul 18, 2010 at 05:42:14PM -0400, Mike Tancsa wrote: At 05:14 PM 7/18/2010, Jeremy Chadwick wrote: Where exactly is your swap partition? On one of the areca raidsets. # swapctl -l Device: 1024-blocks Used: /dev/da0s1b10485760 108 So is da0 actually a RAID volume behind the scenes on the Areca controller? How many disks are involved in that set? yes, da0 is a RAID volume with 4 disks behind the scenes. Okay, so can you get full SMART statistics for all 4 of those disks? The adjusted/calculated values for SMART thresholds won't be helpful here, one will need the actual raw SMART data. I hope the Areca CLI can provide that. Also, I'm willing to bet that the da0 volume and the da1 volume actually share the same physical disks on the Areca controller. Is that correct? If so, think about what would happen if heavy I/O happened on both da0 and da1 at the same time. I talk about this a bit more below. Well, the thread I linked you stated that the problem has to do with a controller or disk taking too long. I have no idea what the threshold is. I suppose it could also indicate that your system is (possibly) running low on resources (RAM); I would imagine swap_pager would get called if a processes needed to be offloaded to swap. So maybe this is a system tuning thing more than a hardware thing. Prior to someone rebooting it, it had been stuck in this state for a good 90min. Apart from upgrading to a later RELENG_8 to get the security patches, the machine had been running a few versions of RELENG_8 doing the same workloads every week without issue. Then I would say you'd need to roll back kernel+world to a previous date and try to figure out when the issue began, if that is indeed the case. /boot/loader.conf has ahci_load=YES siis_load=YES sysctl.conf has net.inet.tcp.recvbuf_max=16777216 net.inet.tcp.recvspace=131072 net.inet.tcp.sendbuf_max=16777216 net.inet.tcp.sendspace=32768 net.inet.udp.recvspace=65536 kern.ipc.somaxconn=1024 kern.ipc.maxsockbuf=4194304 net.inet.ip.redirect=0 net.inet.ip.intr_queue_maxlen=4096 net.route.netisr_maxqlen=1024 kern.ipc.nmbclusters=131072 None of these, to my knowledge, would affect what you're seeing; they're all network-related. I do track some basic mem stats via rrd. Looking at the graphs upto that period, nothing unusual was happening sysctl vm.stats.vm | grep swap Here's another post basically reiterating the same thing: that the controller the swap slice is on (in your case a 4-disk RAID array) is basically taking too long to respond. http://groups.google.com/group/mailing.freebsd.stable/browse_thread/thread/2e7faeeaca719c52/cdcd4601ce1b90c5 I have no idea where the timeout values are in the kernel. I do see these two entries in sysctl that look to be of interest though. You might try adjusting these (not sure if they're sysctls or loader.conf tunables only): vm.swap_idle_threshold2: 10 vm.swap_idle_threshold1: 2 Descriptions: vm.swap_idle_threshold2: Time before a process will be swapped out vm.swap_idle_threshold1: Guaranteed swapped in time for a process I want to point out that the actual amount of data being swapped out is fairly small -- note the size fields the swap_pager kernel messages. There doesn't necessarily have to be a shortage of memory to cause a swapout (case in point, see above). It would also help if you could provide timestamps of those messages; are they all happening at once, or gradual over time? If over time, do they all happen around the same time every day, etc.? You see where I'm going with this. Workaround recommendation: put swap directly on a device and not as part of a 4-disk RAID volume (regardless of what type of RAID) and see if the problem goes away. I realise that probably isn't plausible in your situation (since you'd then be dedicating an entire disk to just swap). Others may have other advice. You mention in a later mail that the ada[0-3] disks make up a ZFS pool of some sort. You might try splitting ada0 into two slices, one for swap and the other used as a pool member. Again: I don't think this is necessarily a bad disk problem. The only way you'd be able to determine that would be to monitor on a per-disk basis the I/O response time of each disk member on the Areca. If the CLI tools provide this, awesome. Otherwise you'll probably need to involve Areca Support. Remember: CAM thinks da0 and da1 are actually individual disks, and lacks knowledge of them being associated with a 4-disk RAID volume behind the scenes. -- | Jeremy Chadwick j...@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
Re: deadlock or bad disk ? RELENG_8
On Sun, Jul 18, 2010 at 08:34:24PM -0700, Jeremy Chadwick wrote:
On Sun, Jul 18, 2010 at 11:01:03PM -0400, Mike Tancsa wrote:
I do track some basic mem stats via rrd. Looking at the graphs upto
that period, nothing unusual was happening
sysctl vm.stats.vm | grep swap
Here's another post basically reiterating the same thing: that the
controller the swap slice is on (in your case a 4-disk RAID array) is
basically taking too long to respond.
http://groups.google.com/group/mailing.freebsd.stable/browse_thread/thread/2e7faeeaca719c52/cdcd4601ce1b90c5
I have no idea where the timeout values are in the kernel. I do see
these two entries in sysctl that look to be of interest though. You
might try adjusting these (not sure if they're sysctls or loader.conf
tunables only):
vm.swap_idle_threshold2: 10
vm.swap_idle_threshold1: 2
Descriptions:
vm.swap_idle_threshold2: Time before a process will be swapped out
vm.swap_idle_threshold1: Guaranteed swapped in time for a process
I want to point out that the actual amount of data being swapped out is
fairly small -- note the size fields the swap_pager kernel messages.
There doesn't necessarily have to be a shortage of memory to cause a
swapout (case in point, see above).
I took a look at the RELENG_8 code responsible for printing this
message: src/sys/vm/swap_pager.c
1067 /*
1068 * SWAP_PAGER_GETPAGES() - bring pages in from swap
1069 *
1070 * Attempt to retrieve (m, count) pages from backing store, but make
1071 * sure we retrieve at least m[reqpage]. We try to load in as large
1072 * a chunk surrounding m[reqpage] as is contiguous in swap and which
1073 * belongs to the same object.
1074 *
1075 * The code is designed for asynchronous operation and
1076 * immediate-notification of 'reqpage' but tends not to be
1077 * used that way. Please do not optimize-out this algorithmic
1078 * feature, I intend to improve on it in the future.
1079 *
1080 * The parent has a single vm_object_pip_add() reference prior to
1081 * calling us and we should return with the same.
1082 *
1083 * The parent has BUSY'd the pages. We should return with 'm'
1084 * left busy, but the others adjusted.
1085 */
1086 static int
1087 swap_pager_getpages(vm_object_t object, vm_page_t *m, int count, int
reqpage)
1088 {
1210 /*
1211 * wait for the page we want to complete. VPO_SWAPINPROG is always
1212 * cleared on completion. If an I/O error occurs, SWAPBLK_NONE
1213 * is set in the meta-data.
1214 */
1215 VM_OBJECT_LOCK(object);
1216 while ((mreq-oflags VPO_SWAPINPROG) != 0) {
1217 mreq-oflags |= VPO_WANTED;
1218 PCPU_INC(cnt.v_intrans);
1219 if (msleep(mreq, VM_OBJECT_MTX(object), PSWP, swread,
hz*20)) {
1220 printf(
1221 swap_pager: indefinite wait buffer: bufobj: %p, blkno: %jd, size: %ld\n,
1222 bp-b_bufobj, (intmax_t)bp-b_blkno,
bp-b_bcount);
1223 }
1224 }
So I believe this indicates the message only gets printed during swapin,
not swapout. Meaning it's happening during an I/O read from da0.
Reading msleep(9) provides us some details about what swread
correlates with (now I know where that column in ps/top comes from), and
the timeout value (hz*20):
The parameter wmesg is a string describing the sleep condition for tools
like ps(1). Due to the limited space of those programs to display arbi‐
trary strings, this message should not be longer than 6 characters.
The parameter timo specifies a timeout for the sleep. If timo is not 0,
then the thread will sleep for at most timo / hz seconds. If the timeout
expires, then the sleep function will return EWOULDBLOCK.
So what's hz? Well, I want to assume it's kern.hz, which defaults to
1000. 1000*20 = 2, so the timeout would be 2/1000 = 20 seconds.
That's a pretty long time to be waiting for an I/O read to return.
So does vm.swap_idle_threshold1 play a role? I doubt it. The code is
in src/sys/vm/vm_glue.c, but I don't understand it (especially since
it's used in a function called swapout_procs()). I just wish I knew why
the description was Guaranteed swapped in time for a process when it
looks more like it's guaranteed swapped out time?
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
Re: deadlock or bad disk ? RELENG_8
On Sun, Jul 18, 2010 at 08:58:44PM -0700, Jeremy Chadwick wrote:
I took a look at the RELENG_8 code responsible for printing this
message: src/sys/vm/swap_pager.c
[...]
1086 static int
1087 swap_pager_getpages(vm_object_t object, vm_page_t *m, int count, int
reqpage)
1088 {
[...]
There was a change to this piece of code on May 13th. See commit
1.312.2.4 below:
http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/vm/swap_pager.c
Before I pull in kib@ or alc@ to help determine if what was introduced
could cause what you're seeing, can you clarify this point?
FreeBSD 8.1-PRERELEASE #0: Tue Jul 13 09:55:48 EDT 2010
mdtancsa at backup3.sentex.ca:/usr/obj/usr/src/sys/backup amd64
[...]
Its never been an issue in the past 2yrs. The same box was RELENG_7
for some time and then in the past 3 months updated to RELENG_8.
Has this system run a previous version of RELENG_8 within the past 3
months, or are you seeing this issue as a result of upgrading from
RELENG_7 to RELENG_8?
--
| Jeremy Chadwick j...@parodius.com |
| Parodius Networking http://www.parodius.com/ |
| UNIX Systems Administrator Mountain View, CA, USA |
| Making life hard for others since 1977. PGP: 4BD6C0CB |
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to freebsd-stable-unsubscr...@freebsd.org
