Re: debugging frequent kernel panics on 8.2-RELEASE
on 18/08/2011 02:15 Steven Hartland said the following: - Original Message - From: Andriy Gapon a...@freebsd.org Thanks to the debug that Steven provided and to the help that I received from Kostik, I think that now I understand the basic mechanics of this panic, but, unfortunately, not the details of its root cause. It seems like everything starts with some kind of a race between terminating processes in a jail and termination of the jail itself. This is where the details are very thin so far. What we see is that a process (http) is in exit(2) syscall, in exit1() function actually, and past the place where P_WEXIT flag is set and even past the place where p_limit is freed and reset to NULL. At that place the thread calls prison_proc_free(), which calls prison_deref(). Then, we see that in prison_deref() the thread gets a page fault because of what seems like a NULL pointer dereference. That's just the start of the problem and its root cause. Thats interesting, are you using http as an example or is that something thats been gleaned from the debugging of our output? I ask as there's only one process running in each of our jails and thats a single java process. It's from the debug data: p_comm = httpd I also would like to ask you to revert the last patch that I sent you (with tf_rip comparisons) and try the patch from Kostik instead. Given what we suspect about the problem, can please also try to provoke the problem by e.g. doing frequent jail restarts or something else that supposedly should hit the bug. -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: debugging frequent kernel panics on 8.2-RELEASE
- Original Message - From: Andriy Gapon a...@freebsd.org Thats interesting, are you using http as an example or is that something thats been gleaned from the debugging of our output? I ask as there's only one process running in each of our jails and thats a single java process. It's from the debug data: p_comm = httpd Hmm, there's only one httpd thats ever run on the machine and thats not in the jail its on the raw machine. I also would like to ask you to revert the last patch that I sent you (with tf_rip comparisons) and try the patch from Kostik instead. Sure. Given what we suspect about the problem, can please also try to provoke the problem by e.g. doing frequent jail restarts or something else that supposedly should hit the bug. I've tried doing this for quite some days on the test machine, but I've been unable to provoke it, will continue to try. Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [nvi-iconv]Call for test
Hi! I just tried you patch on latest current with clang. [root@current64 /usr/src]# uname -a FreeBSD current64 9.0-BETA1 FreeBSD 9.0-BETA1 #0: Thu Aug 18 03:56:45 MSK 2011 mox@current64:/usr/obj/usr/src/sys/GENERIC amd64 [root@current64 /usr/src]# patch ~/nvi2-freebsd-2011-08-17.diff [root@current64 /usr/src]# make WITH_ICONV=1 buildworld lng time. .. === usr.bin/vgrind (depend) rm -f .depend CC='clang' mkdep -f .depend -a /usr/src/usr.bin/vgrind/regexp.c /usr/src/usr .bin/vgrind/vfontedpr.c echo vfontedpr: /usr/obj/usr/src/tmp/usr/lib/libc.a .depend === usr.bin/vi (depend) make: don't know how to make cl_bsd.c. Stop *** Error code 2 Stop in /usr/src/usr.bin. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. Maybe do I something wrong? -- View this message in context: http://freebsd.1045724.n5.nabble.com/nvi-iconv-Call-for-test-tp4698373p4711635.html Sent from the freebsd-hackers mailing list archive at Nabble.com. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: debugging frequent kernel panics on 8.2-RELEASE
on 18/08/2011 13:35 Steven Hartland said the following: - Original Message - From: Andriy Gapon a...@freebsd.org Thats interesting, are you using http as an example or is that something thats been gleaned from the debugging of our output? I ask as there's only one process running in each of our jails and thats a single java process. It's from the debug data: p_comm = httpd Hmm, there's only one httpd thats ever run on the machine and thats not in the jail its on the raw machine. Probably I have mistakenly assumed that the 'prison' in prison_derefer() has something to do with an actual jail, while it could have been just prison0 where all non-jailed processes belong. -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: debugging frequent kernel panics on 8.2-RELEASE
- Original Message - From: Andriy Gapon a...@freebsd.org Probably I have mistakenly assumed that the 'prison' in prison_derefer() has something to do with an actual jail, while it could have been just prison0 where all non-jailed processes belong. That makes sense as this particular panic was caused by a machine reboot, which is slightly different from the more common jail panic we're seeing. Doesn't help with our reproduction scenario though unfortunately. If we don't have any joy reproducing on our single test machine I'll have this kernel rolled out across a portion of the farm, which should mean we see the panic results in a few days time. I understand there's a risk involved in this but, its important for us to determine the cause and get a confirmed fix, as well as being able to prove that the panic fix works which will help everyone in the long run. Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: debugging frequent kernel panics on 8.2-RELEASE
on 18/08/2011 14:11 Andriy Gapon said the following:
Probably I have mistakenly assumed that the 'prison' in prison_derefer() has
something to do with an actual jail, while it could have been just prison0
where
all non-jailed processes belong.
So, indeed:
(kgdb) p $2-p_ucred-cr_prison
$10 = (struct prison *) 0x807d5080
(kgdb) p prison0
$11 = (struct prison *) 0x807d5080
(kgdb) p *$2-p_ucred-cr_prison
$12 = {pr_list = {tqe_next = 0x0, tqe_prev = 0x0}, pr_id = 0, pr_ref = 398,
pr_uref = 0, pr_flags = 386, pr_children = {lh_first = 0x0}, pr_sibling =
{le_next
= 0x0, le_prev = 0x0}, pr_parent = 0x0,
pr_mtx = {lock_object = {lo_name = 0x8063007c jail mutex, lo_flags =
16973824, lo_data = 0, lo_witness = 0x0}, mtx_lock = 4}, pr_task = {ta_link =
{stqe_next = 0x0}, ta_pending = 0,
ta_priority = 0, ta_func = 0, ta_context = 0x0}, pr_osd = {osd_nslots = 0,
osd_slots = 0x0, osd_next = {le_next = 0x0, le_prev = 0x0}}, pr_cpuset =
0xff0012d65dc8, pr_vnet = 0x0,
pr_root = 0xff00166ebce8, pr_ip4s = 0, pr_ip6s = 0, pr_ip4 = 0x0, pr_ip6 =
0x0, pr_sparep = {0x0, 0x0, 0x0, 0x0}, pr_childcount = 0, pr_childmax = 99,
pr_allow = 127, pr_securelevel = -1,
pr_enforce_statfs = 0, pr_spare = {0, 0, 0, 0, 0}, pr_hostid = 3251597242,
pr_name = 0, '\0' repeats 254 times, pr_path = /, '\0' repeats 1022
times,
pr_hostname = censored, '\0' repeats 231 times, pr_domainname = '\0'
repeats 255 times, pr_hostuuid = 54443842-0054-2500-902c-0025902c3cb0, '\0'
repeats 27 times}
Also, let's consider this code:
if (flags PD_DEUREF) {
for (tpr = pr;; tpr = tpr-pr_parent) {
if (tpr != pr)
mtx_lock(tpr-pr_mtx);
if (--tpr-pr_uref 0)
break;
KASSERT(tpr != prison0, (prison0 pr_uref=0));
mtx_unlock(tpr-pr_mtx);
}
/* Done if there were only user references to remove. */
if (!(flags PD_DEREF)) {
mtx_unlock(tpr-pr_mtx);
if (flags PD_LIST_SLOCKED)
sx_sunlock(allprison_lock);
else if (flags PD_LIST_XLOCKED)
sx_xunlock(allprison_lock);
return;
}
if (tpr != pr) {
mtx_unlock(tpr-pr_mtx);
mtx_lock(pr-pr_mtx);
}
}
The most suspicious thing is that pr_uref is zero in the debug data.
With INVARIANTS we would hit the prison0 pr_uref=0 KASSERT.
Then, because this is prison0 and because pr_uref reached zero, tpr gets
assigned
to NULL. And then because tpr != pr we try to execute mtx_unlock(tpr-pr_mtx).
That's where the NULL pointer deref happens.
So, now the big question is how/why we reached pr_uref == 0.
--
Andriy Gapon
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: building my own release images from -CURRENT
On Tue, 2011-08-16 at 15:15 -0700, Test Rat wrote: Have you tried the patch in misc/159666 ? Committed to -current svn R 224978. thanks for the patch! Sean ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
ZFS installs on HD with 4k physical blocks without any warning as on 512 block size device
Some latest hard drives have logical sectors of 512 byte when they actually have 4k physical sectors. Here is the document describing what to do in such case: http://ivoras.net/blog/tree/2011-01-01.freebsd-on-4k-sector-drives.html . For UFS: newfs -U -f 4096 /dev/da0 For ZFS: gnop create -S 4096 /dev/da0 zpool create data /dev/da0.nop I am sure most people just install such hard drive without doing this and potentially get suboptimal performance since they aren't aware about this. Shouldn't UFS and ZFS drivers be able to either read the right sector size from the underlying device or at least issue a warning? Yuri ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [nvi-iconv]Call for test
Em... I can't reproduce this. Can you post your make.conf and src.conf? On Thu, Aug 18, 2011 at 5:30 AM, timp tim...@gmail.com wrote: Hi! I just tried you patch on latest current with clang. [root@current64 /usr/src]# uname -a FreeBSD current64 9.0-BETA1 FreeBSD 9.0-BETA1 #0: Thu Aug 18 03:56:45 MSK 2011 mox@current64:/usr/obj/usr/src/sys/GENERIC amd64 [root@current64 /usr/src]# patch ~/nvi2-freebsd-2011-08-17.diff [root@current64 /usr/src]# make WITH_ICONV=1 buildworld lng time. .. === usr.bin/vgrind (depend) rm -f .depend CC='clang' mkdep -f .depend -a /usr/src/usr.bin/vgrind/regexp.c /usr/src/usr .bin/vgrind/vfontedpr.c echo vfontedpr: /usr/obj/usr/src/tmp/usr/lib/libc.a .depend === usr.bin/vi (depend) make: don't know how to make cl_bsd.c. Stop *** Error code 2 Stop in /usr/src/usr.bin. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. *** Error code 1 Stop in /usr/src. Maybe do I something wrong? -- View this message in context: http://freebsd.1045724.n5.nabble.com/nvi-iconv-Call-for-test-tp4698373p4711635.html Sent from the freebsd-hackers mailing list archive at Nabble.com. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org -- Zhihao Yuan, nickname lichray The best way to predict the future is to invent it. ___ 4BSD -- http://4bsd.biz/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: debugging frequent kernel panics on 8.2-RELEASE
on 17/08/2011 23:21 Andriy Gapon said the following: It seems like everything starts with some kind of a race between terminating processes in a jail and termination of the jail itself. This is where the details are very thin so far. What we see is that a process (http) is in exit(2) syscall, in exit1() function actually, and past the place where P_WEXIT flag is set and even past the place where p_limit is freed and reset to NULL. At that place the thread calls prison_proc_free(), which calls prison_deref(). Then, we see that in prison_deref() the thread gets a page fault because of what seems like a NULL pointer dereference. That's just the start of the problem and its root cause. Then, trap_pfault() gets invoked and, because addresses close to NULL look like userspace addresses, vm_fault/vm_fault_hold gets called, which in its turn goes on to call vm_map_growstack. First thing that vm_map_growstack does is a call to lim_cur(), but because p_limit is already NULL, that call results in a NULL pointer dereference and a page fault. Goto the beginning of this paragraph. So we get this recursion of sorts, which only ends when a stack is exhausted and a CPU generates a double-fault. BTW, does anyone has an idea why the thread in question would disappear from the kgdb's point of view? (kgdb) p cpuid_to_pcpu[2]-pc_curthread-td_tid $3 = 102057 (kgdb) tid 102057 invalid tid info threads also doesn't list the thread. Is it because the panic happened while the thread was somewhere in exit1()? is there an easy way to examine its stack in this case? -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: debugging frequent kernel panics on 8.2-RELEASE
2011/8/18 Andriy Gapon a...@freebsd.org: on 17/08/2011 23:21 Andriy Gapon said the following: It seems like everything starts with some kind of a race between terminating processes in a jail and termination of the jail itself. This is where the details are very thin so far. What we see is that a process (http) is in exit(2) syscall, in exit1() function actually, and past the place where P_WEXIT flag is set and even past the place where p_limit is freed and reset to NULL. At that place the thread calls prison_proc_free(), which calls prison_deref(). Then, we see that in prison_deref() the thread gets a page fault because of what seems like a NULL pointer dereference. That's just the start of the problem and its root cause. Then, trap_pfault() gets invoked and, because addresses close to NULL look like userspace addresses, vm_fault/vm_fault_hold gets called, which in its turn goes on to call vm_map_growstack. First thing that vm_map_growstack does is a call to lim_cur(), but because p_limit is already NULL, that call results in a NULL pointer dereference and a page fault. Goto the beginning of this paragraph. So we get this recursion of sorts, which only ends when a stack is exhausted and a CPU generates a double-fault. BTW, does anyone has an idea why the thread in question would disappear from the kgdb's point of view? (kgdb) p cpuid_to_pcpu[2]-pc_curthread-td_tid $3 = 102057 (kgdb) tid 102057 invalid tid info threads also doesn't list the thread. Is it because the panic happened while the thread was somewhere in exit1()? is there an easy way to examine its stack in this case? Yes it is likely it. 'tid' command should lookup the tid_to_thread() table (or similar name) which returns NULL, which means the thread has past beyond the point it was in the lookup table. Attilio -- Peace can only be achieved by understanding - A. Einstein ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Alternate source trees
Tue, Aug 16, 2011 at 01:09:32PM +0100, Matt Burke wrote:
How does the build process know about the non-symlinked path anyway?
I can't see where (or understand why) it uses pwd -P
Make(1)'s .OBJDIR is used:
{{{
.OBJDIR A path to the directory where the targets are built. At
startup, make searches for an alternate directory to
place target files. It will attempt to change into this
special directory and will search this directory for
makefiles not found in the current directory. The fol-
lowing directories are tried in order:
1. ${MAKEOBJDIRPREFIX}/`pwd`
2. ${MAKEOBJDIR}
3. obj.${MACHINE}
4. obj
5. /usr/obj/`pwd`
The first directory that make successfully changes into
is used. If either MAKEOBJDIRPREFIX or MAKEOBJDIR is set
in the environment but make is unable to change into the
corresponding directory, then the current directory is
used without checking the remainder of the list. If they
are undefined and make is unable to change into any of
the remaining three directories, then the current direc-
tory is used. Note, that MAKEOBJDIRPREFIX and MAKEOBJDIR
must be environment variables and should not be set on
make's command line.
The make utility sets .OBJDIR to the canonical path given
by getcwd(3).
}}}
getcwd(3) always fully resolves current path, so symlinking
won't be taken into account.
I'm trying to setup a box to do automated FreeBSD builds for other hosts
from multiple source trees.
I have a couple of source trees mounted - for legibility's sake let's say
/build/stable and /build/current. I also have a few obj dirs for different
targets. The current obj tree is symlinked to /usr/obj, and this works fine.
The problem comes when I symlink /usr/src: when I buildworld, I get
/usr/obj/build/current/[...] instead of the desired /usr/obj/usr/src/[...]
This is presumably fine when installing on the same machine, but it breaks
when using it on another host with /usr/src and /usr/obj mounted over nfs.
Hmm, current Makefile system for src sets MAKEOBJDIRPREFIX via ?=, so
you're out of luck with symlinking on the build machine. May be you
can mount /build/stable or /build/current on the target machine via
NFS (and, of course, /usr/obj or other directory that you can choose
with the MAKEOBJPREFIX, this leaves room for multi-architecture object
trees on the same build host) and invoke make from the relevant
directory?
Another possibility, if for some reason you want /usr/src to point to
the correct place at your destination machines, to always mount
/build across hosts where you're installing the stuff and to symlink
/usr/src to the proper location inside the /build.
--
Eygene Ryabinkin,,,^..^,,,
[ Life's unfair - but root password helps! | codelabs.ru ]
[ 82FE 06BC D497 C0DE 49EC 4FF0 16AF 9EAE 8152 ECFB | freebsd.org ]
pgp1Ikwq7VChK.pgp
Description: PGP signature
Re: [nvi-iconv]Call for test
timp tim...@gmail.com writes: Hi! I just tried you patch on latest current with clang. [root@current64 /usr/src]# uname -a FreeBSD current64 9.0-BETA1 FreeBSD 9.0-BETA1 #0: Thu Aug 18 03:56:45 MSK 2011 mox@current64:/usr/obj/usr/src/sys/GENERIC amd64 [root@current64 /usr/src]# patch ~/nvi2-freebsd-2011-08-17.diff [...] === usr.bin/vi (depend) make: don't know how to make cl_bsd.c. Stop *** Error code 2 Use `-p0' otherwise new directories won't be created. This is documented in patch(1). And cl_bsd.c ended up in current directory (/usr/src) $ diffstat ~/nvi2-freebsd-2011-08-17.diff.gz | fgrep cl_bsd.c contrib/nvi2/cl/cl_bsd.c| 346 +++ Zhihao Yuan lich...@gmail.com writes: The patch will create contrib/nvi2, and it will not remove the unused contrib/nvi (patch(1) can not really remove files anyway). patch(1) can remove *empty* files with `-E', e.g. $ svn rm UPDATING $ svn di UPDATING | patch -E -d /usr/src ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: [nvi-iconv]Call for test
On Thu, Aug 18, 2011 at 9:26 PM, Test Rat ttse...@gmail.com wrote: timp tim...@gmail.com writes: Hi! I just tried you patch on latest current with clang. [root@current64 /usr/src]# uname -a FreeBSD current64 9.0-BETA1 FreeBSD 9.0-BETA1 #0: Thu Aug 18 03:56:45 MSK 2011 mox@current64:/usr/obj/usr/src/sys/GENERIC amd64 [root@current64 /usr/src]# patch ~/nvi2-freebsd-2011-08-17.diff [...] === usr.bin/vi (depend) make: don't know how to make cl_bsd.c. Stop *** Error code 2 Use `-p0' otherwise new directories won't be created. This is documented in patch(1). And cl_bsd.c ended up in current directory (/usr/src) $ diffstat ~/nvi2-freebsd-2011-08-17.diff.gz | fgrep cl_bsd.c contrib/nvi2/cl/cl_bsd.c | 346 +++ zzz... I always use -p0 but I did not know what it does... Zhihao Yuan lich...@gmail.com writes: The patch will create contrib/nvi2, and it will not remove the unused contrib/nvi (patch(1) can not really remove files anyway). patch(1) can remove *empty* files with `-E', e.g. $ svn rm UPDATING $ svn di UPDATING | patch -E -d /usr/src Got it. But removing contrib/nvi with patch will just double the patch size anyway. A svn rm will do it if some day the patch got committed. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org -- Zhihao Yuan, nickname lichray The best way to predict the future is to invent it. ___ 4BSD -- http://4bsd.biz/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
