Re: dnsperf and BIND memory consumption
At Thu, 01 Jan 2009 00:04:49 -0500, Danny Mayer ma...@gis.net wrote: Personally, I'm not convinced that it will make a difference outside of Windows. The fix is to make sure a lock gets destroyed when done and the function exits. On Windows the lock gets created and memory is allocated for it outside of the function using it and needs to be explicitly destroyed, but my understanding of pthreads was that this worked differently on Unix and the lock structure would be automatically destroyed when the function is exited since the lock was local to the function. The pthread implementation of FreeBSD I tested allocated memory for a lock. That was the reason for the leak, and that was why the Windows-specific fix also helps FreeBSD (that happens to use pthread locks instead of atomic operations). --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Thu, 1 Jan 2009 00:47:10 -0500, Vinny Abello vi...@tellurian.com wrote: I just loaded up the BIND 9.5.1 port on FreeBSD 7.0 AMD64 with threads. I don't see the prominent memory leak present on my system any longer. I lost track of this thread and think two different changes might have been made, however. One via the port and one in BIND itself, so I don't know which fix is actually in effect here, but the results seem good so far. Maybe Doug and Jinmei and comment on their respective changes just for the record. The memory leak only happens when autoconf fails to detect the availability of hardware-dependent atomic operations. The patch for the FreeBSD port fixes the autoconf problem. The patch I mentioned fixes the memory leak itself. The result is the same either way; the leak will disappear. --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Mon, 5 Jan 2009 18:58:07 -0500, Vinny Abello vi...@tellurian.com wrote: My basic question is: Is there any advantage to compiling BIND in the previous manner now that there is a fix in the BIND source code? Do you mean compiling BIND with the memory leak fix and without the FreeBSD port change by in the previous manner? For this particular memory leak problem, there's nothing wrong with it. But then you won't benefit from the other (main) advantage of atomic operations: performance improvement. So, you'll still be happier with applying two patches. --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Doug Barton wrote: Vinny Abello wrote: Just for clarification, is there any downside to this autoconf fix vs. how it was previously working? It was not working correctly previously, so no. Does autoconf still not understand AMD64 on FreeBSD You're confusing autoconf and the configure script provided in the sources for BIND = 9.5.x. There is also a variable of what the FreeBSD ports provide to configure by default that I have deliberately omitted for the sake of clarity. and is your fix just defaulting to something lesser? No. It appeared that the switches being run were for 32 bit after this fix, but I'm no expert on compilers by far. You'd have to dig into the source and really understand what's happening now vs. what was happening before in order for me to answer this question, and by the time you had done that work I would not need to answer this question for you. :) You would have a very hard time finding it just using code inspection since the bug was found several levels under the code since there were a number of Macros in place and the source of the problem was in the creation but the fix had to be elsewhere. I only found this leak because I had accidentally left on a debugging tool when test running on a Windows box and the debugging tool was causing it to break at the creation point. It took me quite a while to peel back the layers and find out why it was really complaining. The fix was easy, but understanding what needed to be fixed and why was not. Danny The short version is that right now the port is doing what it should be. hth, Doug ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Mon, 22 Dec 2008 16:13:10 -0800, Doug Barton do...@freebsd.org wrote: And can someone please state affirmatively that the patches to acl.c and iptable.c do the right thing, with or without the patch to the port? This patch completely fixed the hole in my test environment. With this patch you don't need the ARCH=x86_64 trick. I'm also pretty sure there's no bad side effect with this patch. So, basically, there's no reason not to apply this patch anyway. Alternatively, you may just want to wait until 9.5.1 ships, which will be released within a few days. --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
I have confirmed that the ARCH=x86_64 trick resolved the issues with my configuration. I have tested this with an authoritative and recursive dns/bind95 port with modified Makefile. I have not fully tested the acl.c and iptable.c since the patch suit my need. Thanks! --- On Tue, 12/23/08, Doug Barton do...@dougbarton.us wrote: From: Doug Barton do...@dougbarton.us Subject: Re: dnsperf and BIND memory consumption To: bind-us...@isc.org bind-us...@isc.org Date: Tuesday, December 23, 2008, 1:15 PM JINMEI Tatuya / wrote: At Tue, 2 Dec 2008 00:35:32 -0500, Vinny Abello vi...@tellurian.com wrote: For what it's worth, I just want to contribute that I can confirm this behavior on my systems as well. On BIND 9.5.0-P2, From an off-list discussion, I found there was indeed memory leak in the code of 9.5.0-P2 (so I was wrong in suspecting it might be the FreeBSD thread library). Can you try the patched copied below to see whether it solves the problem? Folks, I have been following this issue, but I haven't seen any conclusive statements that any of the proposed solutions are effective and I don't want to make a change to the port without knowing for sure that it's the right one (and unfortunately I don't have an amd64 system to do testing on). So, can someone please state affirmatively if the proposed patch to the port (ARCH= x86_64) results in the right stuff in config.log, AND results in a named that doesn't experience the leaks? And can someone please state affirmatively that the patches to acl.c and iptable.c do the right thing, with or without the patch to the port? Thanks, Doug -- This .signature sanitized for your protection ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
JINMEI Tatuya / 神明達哉 wrote: At Mon, 15 Dec 2008 09:53:23 +0300, Dmitry Rybin rybi...@post.ru wrote: Thank's to JINMEI Tatuya for support. I have over 40 views, defined in named.conf, max-memory for cache - 32Mb. Named daemon allocate over 2 Gb per 24 hours of work. Each view has a separate cache DB. So if each of these 40 views really needs to cache a certain amount of data, a footprint of 2GB is not a surprising situation, even with a 32MB of max-cache-size for each view. OK. I Just limit max-cache-size 16MB. 16MB * 50 views = over 800 MB of memory. How much total memory bind can accrue? -- ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Mon, 15 Dec 2008 09:53:23 +0300, Dmitry Rybin rybi...@post.ru wrote: Thank's to JINMEI Tatuya for support. I have over 40 views, defined in named.conf, max-memory for cache - 32Mb. Named daemon allocate over 2 Gb per 24 hours of work. Each view has a separate cache DB. So if each of these 40 views really needs to cache a certain amount of data, a footprint of 2GB is not a surprising situation, even with a 32MB of max-cache-size for each view. Have you any ideas how to limit memory usage? Unfortunately not, unless you can consolidate the caching views to a small number of views. --- JINMEI, Tatuya Internet Systems Consortium, Inc. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Wed, 17 Dec 2008 17:07:12 -0800,
JINMEI Tatuya jinmei_tat...@isc.org wrote:
At Mon, 15 Dec 2008 09:53:23 +0300,
Dmitry Rybin rybi...@post.ru wrote:
Thank's to JINMEI Tatuya for support.
I have over 40 views, defined in named.conf, max-memory for cache -
32Mb. Named daemon allocate over 2 Gb per 24 hours of work.
Each view has a separate cache DB. So if each of these 40 views
really needs to cache a certain amount of data, a footprint of 2GB is
not a surprising situation, even with a 32MB of max-cache-size for
each view.
Have you any ideas how to limit memory usage?
Unfortunately not, unless you can consolidate the caching views to a
small number of views.
If you can allow the multiple views to share a single cache, one
possibility is to create a separate localhost view as the single
caching view and forward all recursive queries to that view:
view localhost {
match-clients { ::1; };
};
view view1 {
match-clients { view1-clients; };
max-cache-ttl 0; //disable caching to save memory
zone . {
type forward;
forwarders { ::1; };
forward only;
};
// add other authoritative zones if necessary
};
view view2 {
match-clients { view2-clients; };
max-cache-ttl 0;
zone . {
type forward;
forwarders { ::1; };
forward only;
};
// add other authoritative zones if necessary
};
---
JINMEI, Tatuya
Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Thank's to JINMEI Tatuya for support.
I have over 40 views, defined in named.conf, max-memory for cache -
32Mb. Named daemon allocate over 2 Gb per 24 hours of work.
Have you any ideas how to limit memory usage?
Dmitry Rybin wrote:
max-cache-size 64M;
# /usr/bin/limits -v 1200M /usr/local/sbin/named-test -c
/etc/namedb/named.conf
Over 10 minutes of work and core dumped:
(gdb) bt
#0 0x0058c3fc in thr_kill ()
#1 0x005c5a68 in abort ()
#2 0x00597af7 in malloc ()
#3 0x0056645a in isc_mem_createx2 (init_max_size=0, target_size=0,
memalloc=0x564400 default_memalloc, memfree=0x563320
default_memfree, arg=0x0,
ctxp=0xcb29b978, flags=Variable flags is not available.
) at mem.c:790
#4 0x00566730 in isc_mem_create (init_max_size=Variable
init_max_size is not available.
) at mem.c:859
#5 0x004d83ae in dns_resolver_create (view=0xca46e800,
taskmgr=0x80828000,
ntasks=31, socketmgr=Variable socketmgr is not available.
) at resolver.c:6514
#6 0x004ee860 in dns_view_createresolver (view=0xca46e800,
taskmgr=0x80828000,
ntasks=31, socketmgr=0x8082b000, timermgr=0x80829000, options=0,
dispatchmgr=0x8083c000,
dispatchv4=0x864c9000, dispatchv6=0x864c9800) at view.c:580
#7 0x0041bba2 in configure_view (view=0xca46e800,
config=0x80abb4c0,
vconfig=0x8085ada8, mctx=0x8080d140, actx=0x7eff8860,
need_hints=isc_boolean_true)
at server.c:1290
#8 0x00420f42 in load_configuration (filename=Variable
filename is not available.
) at server.c:3285
#9 0x00422095 in loadconfig (server=0x8082f000) at server.c:4121
#10 0x00422426 in reload (server=0x8082f000) at server.c:4141
#11 0x004225c2 in ns_server_reloadcommand (server=0x8082f000,
args=Variable args is not available.
) at server.c:4334
#12 0x00407682 in ns_control_docommand (message=Variable
message is not available.
) at control.c:102
#13 0x0040a8b7 in control_recvmessage (task=0x80839000,
event=Variable event is not available.
) at controlconf.c:456
#14 0x0057052c in run (uap=Variable uap is not available.
) at task.c:862
#15 0x005868a7 in thread_start ()
#16 0x in ?? ()
Cannot access memory at address 0x7eff9000
JINMEI Tatuya / 神明達哉 wrote:
At Wed, 10 Dec 2008 15:50:22 +0300,
Dmitry Rybin kirg...@corbina.net wrote:
JINMEI Tatuya / 神明達哉 wrote:
At Tue, 09 Dec 2008 18:05:27 +0300,
Dmitry Rybin kirg...@corbina.net wrote:
I test patch, add to bind95/Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
Future versions of BIND9 will support amd64 in its configure script to
workaround the FreeBSD port for amd64.
Regarding the memory leak, I believe it's already solved in 9.5.1rc1
(even with threads and without atomic).
I just make port bind 9.5.1rc1. It has same problem with memory leak.
It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
port) so that we can separate FreeBSD-port specific issue and BIND9
specific leak?
Second, what if you stop named by 'rndc stop'? If there's memory leak
in BIND9, it normally detects it during a cleanup process and
indicates the bug by aborting (core dumping) itself.
If it doesn't cause an abort, please then try the diagnosing I
suggested before:
http://marc.info/?l=bind-usersm=121811979629090w=2
To summarize it:
1. create a symbolic link from /etc/malloc.conf to X:
# ln -s X /etc/malloc.conf
2. - start named with a moderate limitation of virtual memory size, e.g.
# /usr/bin/limits -v 384m $path_to_named/named command line options
(note that 384m should be reasonably large compared with
max-cache-size. I'd suggest setting max-cache-size to 128M and
setting 'limits -v' to 512m).
3. Then the named process will eventually abort itself with a core dump
due to malloc failure. Please show us the stack trace at that point.
Hopefully it will reveal the malloc call that keeps consuming memory.
In fact, I myself successfully identified one leak in 9.5.0-P2 with
FreeBSD port this way.
---
JINMEI, Tatuya
Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Thank's to JINMEI Tatuya for support.
I have over 40 views, defined in named.conf, max-memory for cache -
32Mb. Named daemon allocate over 2 Gb per 24 hours of work.
Have you any ideas how to limit memory usage?
Dmitry Rybin wrote:
max-cache-size 64M;
# /usr/bin/limits -v 1200M /usr/local/sbin/named-test -c
/etc/namedb/named.conf
Over 10 minutes of work and core dumped:
(gdb) bt
#0 0x0058c3fc in thr_kill ()
#1 0x005c5a68 in abort ()
#2 0x00597af7 in malloc ()
#3 0x0056645a in isc_mem_createx2 (init_max_size=0, target_size=0,
memalloc=0x564400 default_memalloc, memfree=0x563320
default_memfree, arg=0x0,
ctxp=0xcb29b978, flags=Variable flags is not available.
) at mem.c:790
#4 0x00566730 in isc_mem_create (init_max_size=Variable
init_max_size is not available.
) at mem.c:859
#5 0x004d83ae in dns_resolver_create (view=0xca46e800,
taskmgr=0x80828000,
ntasks=31, socketmgr=Variable socketmgr is not available.
) at resolver.c:6514
#6 0x004ee860 in dns_view_createresolver (view=0xca46e800,
taskmgr=0x80828000,
ntasks=31, socketmgr=0x8082b000, timermgr=0x80829000, options=0,
dispatchmgr=0x8083c000,
dispatchv4=0x864c9000, dispatchv6=0x864c9800) at view.c:580
#7 0x0041bba2 in configure_view (view=0xca46e800,
config=0x80abb4c0,
vconfig=0x8085ada8, mctx=0x8080d140, actx=0x7eff8860,
need_hints=isc_boolean_true)
at server.c:1290
#8 0x00420f42 in load_configuration (filename=Variable
filename is not available.
) at server.c:3285
#9 0x00422095 in loadconfig (server=0x8082f000) at server.c:4121
#10 0x00422426 in reload (server=0x8082f000) at server.c:4141
#11 0x004225c2 in ns_server_reloadcommand (server=0x8082f000,
args=Variable args is not available.
) at server.c:4334
#12 0x00407682 in ns_control_docommand (message=Variable
message is not available.
) at control.c:102
#13 0x0040a8b7 in control_recvmessage (task=0x80839000,
event=Variable event is not available.
) at controlconf.c:456
#14 0x0057052c in run (uap=Variable uap is not available.
) at task.c:862
#15 0x005868a7 in thread_start ()
#16 0x in ?? ()
Cannot access memory at address 0x7eff9000
JINMEI Tatuya / 神明達哉 wrote:
At Wed, 10 Dec 2008 15:50:22 +0300,
Dmitry Rybin kirg...@corbina.net wrote:
JINMEI Tatuya / 神明達哉 wrote:
At Tue, 09 Dec 2008 18:05:27 +0300,
Dmitry Rybin kirg...@corbina.net wrote:
I test patch, add to bind95/Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
Future versions of BIND9 will support amd64 in its configure script to
workaround the FreeBSD port for amd64.
Regarding the memory leak, I believe it's already solved in 9.5.1rc1
(even with threads and without atomic).
I just make port bind 9.5.1rc1. It has same problem with memory leak.
It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
port) so that we can separate FreeBSD-port specific issue and BIND9
specific leak?
Second, what if you stop named by 'rndc stop'? If there's memory leak
in BIND9, it normally detects it during a cleanup process and
indicates the bug by aborting (core dumping) itself.
If it doesn't cause an abort, please then try the diagnosing I
suggested before:
http://marc.info/?l=bind-usersm=121811979629090w=2
To summarize it:
1. create a symbolic link from /etc/malloc.conf to X:
# ln -s X /etc/malloc.conf
2. - start named with a moderate limitation of virtual memory size, e.g.
# /usr/bin/limits -v 384m $path_to_named/named command line options
(note that 384m should be reasonably large compared with
max-cache-size. I'd suggest setting max-cache-size to 128M and
setting 'limits -v' to 512m).
3. Then the named process will eventually abort itself with a core dump
due to malloc failure. Please show us the stack trace at that point.
Hopefully it will reveal the malloc call that keeps consuming memory.
In fact, I myself successfully identified one leak in 9.5.0-P2 with
FreeBSD port this way.
---
JINMEI, Tatuya
Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
JINMEI Tatuya / 神明達哉 wrote: At Thu, 11 Dec 2008 11:25:42 +0300, Dmitry Rybin kirg...@corbina.net wrote: OK. I just make bind from src with ./configure --enable-threads gcc option -static. file /usr/local/sbin/named-test /usr/local/sbin/named-test: ELF 64-bit LSB executable, x86-64, version 1 (FreeBSD), for FreeBSD 7.1 (701100), statically linked, FreeBSD-style, not stripped please let me check some points: 1. you used 9.5.1rc1 without any patch, right? 2. did you try 'rndc stop' at some point? If so, did named stop cleanly or did it abort itself? 3. were you periodically reloading the server during the test? I'm not sure if this is coincidence but the self-abort happened while reload operation in your both cases. 4. if the answer to question#2 is yes, is it possible to not reload the server and see if it changes anything? 5. is it possible to install (if not yet) libxml2 port to your system and enable statistics-channels? then you can see more detailed information about how named uses memory. 1. Yes. With STD_CDEFINES='-DFD_SETSIZE=16384' --enable-threads and static linking. 2. Cleanly 3. No. Then reloading result is the same. 4. 5. Yes. Building named with libxml2 done. Say me private your's IP to make access to it. ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
OK. I just make bind from src with ./configure --enable-threads gcc
option -static.
file /usr/local/sbin/named-test
/usr/local/sbin/named-test: ELF 64-bit LSB executable, x86-64, version 1
(FreeBSD), for FreeBSD 7.1 (701100), statically linked, FreeBSD-style,
not stripped
fresh system (yesterday cvsup to RELENG_7)
$ uname -a
FreeBSD XXX.XXX.XX 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #1: Wed Dec 10
17:07:03 MSK 2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/XXX amd64
2.
max-cache-size 128M;
start:
/usr/bin/limits -c 1000M -v 500M /usr/local/sbin/named-test -c
/etc/namedb/named.conf
$ gdb -c named-test.core -se /usr/local/sbin/named-test
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type show copying to see the conditions.
There is absolutely no warranty for GDB. Type show warranty for details.
This GDB was configured as amd64-marcel-freebsd...
Core was generated by `named-test'.
Program terminated with signal 6, Aborted.
#0 0x0058c3fc in thr_kill ()
[New Thread 0x80902f00 (LWP 100404)]
[New Thread 0x80902d80 (LWP 100400)]
[New Thread 0x80902c00 (LWP 100356)]
[New Thread 0x80902a80 (LWP 100318)]
[New Thread 0x80902900 (LWP 100239)]
[New Thread 0x80902780 (LWP 100237)]
[New Thread 0x80902600 (LWP 100222)]
[New Thread 0x80902480 (LWP 100209)]
[New Thread 0x80902300 (LWP 100175)]
[New Thread 0x80902180 (LWP 100092)]
[New Thread 0x80803180 (LWP 100177)]
(gdb) bt
#0 0x0058c3fc in thr_kill ()
#1 0x005c5a68 in abort ()
#2 0x00597af7 in malloc ()
#3 0x00564247 in mem_getunlocked (ctx=0x8080d140, size=94) at
mem.c:385
#4 0x00564b68 in isc__mem_get (ctx=0x8080d140, size=94,
file=0x61bd78 rbt.c, line=1425) at mem.c:1096
#5 0x00480121 in create_node (mctx=0x8080d140,
name=0x7fbfcff0, nodep=0x7fbfd2e0) at rbt.c:1424
#6 0x0048080f in dns_rbt_addnode (rbt=0x80a925e8,
name=0x88cf2020, nodep=0x7fbfd3a8) at rbt.c:624
#7 0x0048be53 in loading_addrdataset (arg=0x94b07ff0,
name=0x88cf2020, rdataset=0x7fbfd810) at rbtdb.c:5657
#8 0x00463761 in commit (callbacks=0x7fbfe5c0,
lctx=0x80834000, head=0x7fbfe480, owner=0x88cf2020,
source=0x94c2afd8 co/brand.bak, line=611215) at master.c:2729
#9 0x004668df in load_text (lctx=0x80834000) at master.c:1427
#10 0x0046b61b in dns_master_loadfile2 (master_file=0x878a7098
co/broad.bak, top=Variable top is not available.
)
at master.c:2350
#11 0x00506126 in zone_load (zone=0x878ec000, flags=Variable
flags is not available.
) at zone.c:1504
#12 0x005082b9 in load (zone=Variable zone is not available.
) at zt.c:246
#13 0x00507ec2 in dns_zt_apply2 (zt=Variable zt is not available.
) at zt.c:379
#14 0x00508144 in dns_zt_load (zt=0x86adb750,
stop=isc_boolean_false) at zt.c:237
#15 0x004223c7 in load_zones (server=0x8082f000,
stop=isc_boolean_false) at server.c:3659
#16 0x004232fc in run_server (task=Variable task is not available.
) at server.c:3751
#17 0x0057052c in run (uap=Variable uap is not available.
) at task.c:862
#18 0x005868a7 in thread_start ()
#19 0x in ?? ()
Cannot access memory at address 0x7fbff000
At normal situation after startup memory usage over 700 MB.
-
JINMEI Tatuya / 神明達哉 wrote:
At Wed, 10 Dec 2008 15:50:22 +0300,
Dmitry Rybin [EMAIL PROTECTED] wrote:
JINMEI Tatuya / 神明達哉 wrote:
At Tue, 09 Dec 2008 18:05:27 +0300,
Dmitry Rybin [EMAIL PROTECTED] wrote:
I test patch, add to bind95/Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
Future versions of BIND9 will support amd64 in its configure script to
workaround the FreeBSD port for amd64.
Regarding the memory leak, I believe it's already solved in 9.5.1rc1
(even with threads and without atomic).
I just make port bind 9.5.1rc1. It has same problem with memory leak.
It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
port) so that we can separate FreeBSD-port specific issue and BIND9
specific leak?
Second, what if you stop named by 'rndc stop'? If there's memory leak
in BIND9, it normally detects it during a cleanup process and
indicates the bug by aborting (core dumping) itself.
If it doesn't cause an abort, please then try the diagnosing I
suggested before:
http://marc.info/?l=bind-usersm=121811979629090w=2
To summarize it:
1. create a symbolic link from /etc/malloc.conf to X:
# ln -s X /etc/malloc.conf
2. - start named with a moderate limitation of virtual memory size, e.g.
# /usr/bin/limits -v 384m $path_to_named/named command line options
(note that 384m should be reasonably large compared with
max-cache-size. I'd suggest setting max-cache-size to 128M and
Re: dnsperf and BIND memory consumption
max-cache-size 64M;
# /usr/bin/limits -v 1200M /usr/local/sbin/named-test -c
/etc/namedb/named.conf
Over 10 minutes of work and core dumped:
(gdb) bt
#0 0x0058c3fc in thr_kill ()
#1 0x005c5a68 in abort ()
#2 0x00597af7 in malloc ()
#3 0x0056645a in isc_mem_createx2 (init_max_size=0, target_size=0,
memalloc=0x564400 default_memalloc, memfree=0x563320
default_memfree, arg=0x0,
ctxp=0xcb29b978, flags=Variable flags is not available.
) at mem.c:790
#4 0x00566730 in isc_mem_create (init_max_size=Variable
init_max_size is not available.
) at mem.c:859
#5 0x004d83ae in dns_resolver_create (view=0xca46e800,
taskmgr=0x80828000,
ntasks=31, socketmgr=Variable socketmgr is not available.
) at resolver.c:6514
#6 0x004ee860 in dns_view_createresolver (view=0xca46e800,
taskmgr=0x80828000,
ntasks=31, socketmgr=0x8082b000, timermgr=0x80829000, options=0,
dispatchmgr=0x8083c000,
dispatchv4=0x864c9000, dispatchv6=0x864c9800) at view.c:580
#7 0x0041bba2 in configure_view (view=0xca46e800,
config=0x80abb4c0,
vconfig=0x8085ada8, mctx=0x8080d140, actx=0x7eff8860,
need_hints=isc_boolean_true)
at server.c:1290
#8 0x00420f42 in load_configuration (filename=Variable
filename is not available.
) at server.c:3285
#9 0x00422095 in loadconfig (server=0x8082f000) at server.c:4121
#10 0x00422426 in reload (server=0x8082f000) at server.c:4141
#11 0x004225c2 in ns_server_reloadcommand (server=0x8082f000,
args=Variable args is not available.
) at server.c:4334
#12 0x00407682 in ns_control_docommand (message=Variable
message is not available.
) at control.c:102
#13 0x0040a8b7 in control_recvmessage (task=0x80839000,
event=Variable event is not available.
) at controlconf.c:456
#14 0x0057052c in run (uap=Variable uap is not available.
) at task.c:862
#15 0x005868a7 in thread_start ()
#16 0x in ?? ()
Cannot access memory at address 0x7eff9000
JINMEI Tatuya / 神明達哉 wrote:
At Wed, 10 Dec 2008 15:50:22 +0300,
Dmitry Rybin [EMAIL PROTECTED] wrote:
JINMEI Tatuya / 神明達哉 wrote:
At Tue, 09 Dec 2008 18:05:27 +0300,
Dmitry Rybin [EMAIL PROTECTED] wrote:
I test patch, add to bind95/Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
Future versions of BIND9 will support amd64 in its configure script to
workaround the FreeBSD port for amd64.
Regarding the memory leak, I believe it's already solved in 9.5.1rc1
(even with threads and without atomic).
I just make port bind 9.5.1rc1. It has same problem with memory leak.
It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
port) so that we can separate FreeBSD-port specific issue and BIND9
specific leak?
Second, what if you stop named by 'rndc stop'? If there's memory leak
in BIND9, it normally detects it during a cleanup process and
indicates the bug by aborting (core dumping) itself.
If it doesn't cause an abort, please then try the diagnosing I
suggested before:
http://marc.info/?l=bind-usersm=121811979629090w=2
To summarize it:
1. create a symbolic link from /etc/malloc.conf to X:
# ln -s X /etc/malloc.conf
2. - start named with a moderate limitation of virtual memory size, e.g.
# /usr/bin/limits -v 384m $path_to_named/named command line options
(note that 384m should be reasonably large compared with
max-cache-size. I'd suggest setting max-cache-size to 128M and
setting 'limits -v' to 512m).
3. Then the named process will eventually abort itself with a core dump
due to malloc failure. Please show us the stack trace at that point.
Hopefully it will reveal the malloc call that keeps consuming memory.
In fact, I myself successfully identified one leak in 9.5.0-P2 with
FreeBSD port this way.
---
JINMEI, Tatuya
Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Memory statistic
start - 570M
1 min - 913M
2 min - 958M
3 min - 1092M
4 min - 1074M
5 min - 1082M
10 min - 1217M
15 min - 1234M
60 min - 1513M
max-cache-size 800M;
Port installed only with Threads parameter, and patch in Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
===
# ps axw|grep named
/usr/local/sbin/named -t /var/named -u bind -c /etc/namedb/named.conf -t
/var/named -u bind
===
$ rndc status
version: 9.5.0-P2 (Unknown DNS1)
number of zones: 899
debug level: 0
xfers running: 0
xfers deferred: 0
soa queries in progress: 2
query logging is OFF
recursive clients: 286/9900/1
tcp clients: 0/100
server is up and running
===
(port installed)
$ldd /usr/local/sbin/named
/usr/local/sbin/named:
libcrypto.so.5 = /lib/libcrypto.so.5 (0x807bb000)
libthr.so.3 = /lib/libthr.so.3 (0x80a4d000)
libc.so.7 = /lib/libc.so.7 (0x80b63000)
(system standart)
$ldd /usr/sbin/named
/usr/sbin/named:
libcrypto.so.5 = /lib/libcrypto.so.5 (0x807a9000)
libthr.so.3 = /lib/libthr.so.3 (0x80a3b000)
libc.so.7 = /lib/libc.so.7 (0x80b51000)
===
ivan jr sy wrote:
Hi
can you verify if you're using the newly installed named.
did you configure your options to replace the base?
can you give us:
ldd /usr/sbin/named
ldd /usr/local/sbin/named
to my understanding, there should be no memory leak issue at all if you
disable threads..
this post has always been directed to the concern of FreeBSD + amd64 platform
+ FreeBSD port dns/bind95 (BIND 9.5.0-P2) + threading enabled
thanks!
--- On Wed, 12/10/08, Dmitry Rybin [EMAIL PROTECTED] wrote:
From: Dmitry Rybin [EMAIL PROTECTED]
Subject: Re: dnsperf and BIND memory consumption
To: Vinny Abello [EMAIL PROTECTED]
Cc: JINMEI Tatuya / 神明達哉 [EMAIL PROTECTED], [EMAIL PROTECTED] [EMAIL
PROTECTED], [EMAIL PROTECTED] [EMAIL PROTECTED]
Date: Wednesday, December 10, 2008, 4:05 AM
Hello!
I test patch, add to bind95/Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
work/bind-9.5.0-P2/config.log
uname -m = amd64
/usr/bin/uname -p = amd64
Target: amd64-undermydesk-freebsd
Configured with: FreeBSD/amd64 system compiler
ISC_ARCH_DIR='x86_32'
build='x86_64-portbld-freebsd7.0'
build_alias='x86_64-portbld-freebsd7.0'
build_cpu='x86_64'
host='x86_64-portbld-freebsd7.0'
host_cpu='x86_64'
I didn't find any affect, memory leak very quickly with
threads support,
and slowly without threads.
FreeBSD xxx 7.0-STABLE FreeBSD 7.0-STABLE #0: Wed Jul 2
14:18:35 MSD
2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/H1 amd64
Vinny Abello wrote:
so does this memory leak only occur if
@ISC_ARCH_DIR@ is noatomic under
FreeBSD amd64?
and not when its x86_32 ?
First off, note that I have no explicit evidence
of memory leak. But
*if there is indeed leak in the FreeBSD pthread
library*, the key is
noatomic. With this configuration
named will call pthread
locks/unlocks much, much heavier, so the problem
may be observable
more clearly. named still uses pthread locks Even
with x86_32, so it
may just be leaking memory more slowly.
Again, everything is just a guess and could be
wrong. We should seek
advice from someone who knows FreeBSD library
well.
Just out of curiosity, why in theory is this not seen
in prior versions of BIND such as 9.4.2-P2 or 9.4.3 on the
same FreeBSD 7.0 AMD64 platforms with threading enabled in
BIND?
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
JINMEI Tatuya / 神明達哉 wrote:
At Tue, 09 Dec 2008 18:05:27 +0300,
Dmitry Rybin [EMAIL PROTECTED] wrote:
I test patch, add to bind95/Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
Future versions of BIND9 will support amd64 in its configure script to
workaround the FreeBSD port for amd64.
Regarding the memory leak, I believe it's already solved in 9.5.1rc1
(even with threads and without atomic).
I just make port bind 9.5.1rc1. It has same problem with memory leak.
It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
grep x86 work/bind-9.5.1rc1/config.log
ISC_ARCH_DIR='x86_32'
build='x86_64-portbld-freebsd7.0'
build_alias='x86_64-portbld-freebsd7.0'
build_cpu='x86_64'
host='x86_64-portbld-freebsd7.0'
host_cpu='x86_64'
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Hi,
is it possible to see your named.conf
what is the methodology of the test? is it for authoritative queries?
recursive? or both? at the same time?
my patch for the port is the same as yours...
thanks!
===
.if ${ARCH} == amd64
ARCH=x86_64
.endif
--- On Thu, 12/11/08, Dmitry Rybin [EMAIL PROTECTED] wrote:
From: Dmitry Rybin [EMAIL PROTECTED]
Subject: Re: dnsperf and BIND memory consumption
To: JINMEI Tatuya / 神明達哉 [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED] [EMAIL PROTECTED], [EMAIL PROTECTED] [EMAIL
PROTECTED]
Date: Thursday, December 11, 2008, 1:50 AM
JINMEI Tatuya / 神明達哉 wrote:
At Tue, 09 Dec 2008 18:05:27 +0300,
Dmitry Rybin [EMAIL PROTECTED] wrote:
I test patch, add to bind95/Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
Future versions of BIND9 will support amd64 in its
configure script to
workaround the FreeBSD port for amd64.
Regarding the memory leak, I believe it's already
solved in 9.5.1rc1
(even with threads and without atomic).
I just make port bind 9.5.1rc1. It has same problem with
memory leak.
It grows from 670M on startup, to 1,4Gb after 20 minutes of
work.
grep x86 work/bind-9.5.1rc1/config.log
ISC_ARCH_DIR='x86_32'
build='x86_64-portbld-freebsd7.0'
build_alias='x86_64-portbld-freebsd7.0'
build_cpu='x86_64'
host='x86_64-portbld-freebsd7.0'
host_cpu='x86_64'
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
At Wed, 10 Dec 2008 15:50:22 +0300,
Dmitry Rybin [EMAIL PROTECTED] wrote:
JINMEI Tatuya / 神明達哉 wrote:
At Tue, 09 Dec 2008 18:05:27 +0300,
Dmitry Rybin [EMAIL PROTECTED] wrote:
I test patch, add to bind95/Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
Future versions of BIND9 will support amd64 in its configure script to
workaround the FreeBSD port for amd64.
Regarding the memory leak, I believe it's already solved in 9.5.1rc1
(even with threads and without atomic).
I just make port bind 9.5.1rc1. It has same problem with memory leak.
It grows from 670M on startup, to 1,4Gb after 20 minutes of work.
Can you first fall back to the vanilla 9.5.1rc1 (i.e., not FreeBSD
port) so that we can separate FreeBSD-port specific issue and BIND9
specific leak?
Second, what if you stop named by 'rndc stop'? If there's memory leak
in BIND9, it normally detects it during a cleanup process and
indicates the bug by aborting (core dumping) itself.
If it doesn't cause an abort, please then try the diagnosing I
suggested before:
http://marc.info/?l=bind-usersm=121811979629090w=2
To summarize it:
1. create a symbolic link from /etc/malloc.conf to X:
# ln -s X /etc/malloc.conf
2. - start named with a moderate limitation of virtual memory size, e.g.
# /usr/bin/limits -v 384m $path_to_named/named command line options
(note that 384m should be reasonably large compared with
max-cache-size. I'd suggest setting max-cache-size to 128M and
setting 'limits -v' to 512m).
3. Then the named process will eventually abort itself with a core dump
due to malloc failure. Please show us the stack trace at that point.
Hopefully it will reveal the malloc call that keeps consuming memory.
In fact, I myself successfully identified one leak in 9.5.0-P2 with
FreeBSD port this way.
---
JINMEI, Tatuya
Internet Systems Consortium, Inc.
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Hello!
I test patch, add to bind95/Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
work/bind-9.5.0-P2/config.log
uname -m = amd64
/usr/bin/uname -p = amd64
Target: amd64-undermydesk-freebsd
Configured with: FreeBSD/amd64 system compiler
ISC_ARCH_DIR='x86_32'
build='x86_64-portbld-freebsd7.0'
build_alias='x86_64-portbld-freebsd7.0'
build_cpu='x86_64'
host='x86_64-portbld-freebsd7.0'
host_cpu='x86_64'
I didn't find any affect, memory leak very quickly with threads support,
and slowly without threads.
FreeBSD xxx 7.0-STABLE FreeBSD 7.0-STABLE #0: Wed Jul 2 14:18:35 MSD
2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/H1 amd64
Vinny Abello wrote:
so does this memory leak only occur if
@ISC_ARCH_DIR@ is noatomic under FreeBSD amd64?
and not when its x86_32 ?
First off, note that I have no explicit evidence of memory leak. But
*if there is indeed leak in the FreeBSD pthread library*, the key is
noatomic. With this configuration named will call pthread
locks/unlocks much, much heavier, so the problem may be observable
more clearly. named still uses pthread locks Even with x86_32, so it
may just be leaking memory more slowly.
Again, everything is just a guess and could be wrong. We should seek
advice from someone who knows FreeBSD library well.
Just out of curiosity, why in theory is this not seen in prior versions of
BIND such as 9.4.2-P2 or 9.4.3 on the same FreeBSD 7.0 AMD64 platforms with
threading enabled in BIND?
--
Рыбин Дмитрий
Управление магистральной сети
Отдел Информационных Систем
Руководитель группы АВР
Corbina Telecom
Tel: +7(495) 728-4000
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
Re: dnsperf and BIND memory consumption
Hi
can you verify if you're using the newly installed named.
did you configure your options to replace the base?
can you give us:
ldd /usr/sbin/named
ldd /usr/local/sbin/named
to my understanding, there should be no memory leak issue at all if you disable
threads..
this post has always been directed to the concern of FreeBSD + amd64 platform +
FreeBSD port dns/bind95 (BIND 9.5.0-P2) + threading enabled
thanks!
--- On Wed, 12/10/08, Dmitry Rybin [EMAIL PROTECTED] wrote:
From: Dmitry Rybin [EMAIL PROTECTED]
Subject: Re: dnsperf and BIND memory consumption
To: Vinny Abello [EMAIL PROTECTED]
Cc: JINMEI Tatuya / 神明達哉 [EMAIL PROTECTED], [EMAIL PROTECTED] [EMAIL
PROTECTED], [EMAIL PROTECTED] [EMAIL PROTECTED]
Date: Wednesday, December 10, 2008, 4:05 AM
Hello!
I test patch, add to bind95/Makefile
.if (${ARCH} == amd64)
ARCH= x86_64
.endif
work/bind-9.5.0-P2/config.log
uname -m = amd64
/usr/bin/uname -p = amd64
Target: amd64-undermydesk-freebsd
Configured with: FreeBSD/amd64 system compiler
ISC_ARCH_DIR='x86_32'
build='x86_64-portbld-freebsd7.0'
build_alias='x86_64-portbld-freebsd7.0'
build_cpu='x86_64'
host='x86_64-portbld-freebsd7.0'
host_cpu='x86_64'
I didn't find any affect, memory leak very quickly with
threads support,
and slowly without threads.
FreeBSD xxx 7.0-STABLE FreeBSD 7.0-STABLE #0: Wed Jul 2
14:18:35 MSD
2008 [EMAIL PROTECTED]:/usr/obj/usr/src/sys/H1 amd64
Vinny Abello wrote:
so does this memory leak only occur if
@ISC_ARCH_DIR@ is noatomic under
FreeBSD amd64?
and not when its x86_32 ?
First off, note that I have no explicit evidence
of memory leak. But
*if there is indeed leak in the FreeBSD pthread
library*, the key is
noatomic. With this configuration
named will call pthread
locks/unlocks much, much heavier, so the problem
may be observable
more clearly. named still uses pthread locks Even
with x86_32, so it
may just be leaking memory more slowly.
Again, everything is just a guess and could be
wrong. We should seek
advice from someone who knows FreeBSD library
well.
Just out of curiosity, why in theory is this not seen
in prior versions of BIND such as 9.4.2-P2 or 9.4.3 on the
same FreeBSD 7.0 AMD64 platforms with threading enabled in
BIND?
--
Рыбин Дмитрий
Управление магистральной сети
Отдел Информационных Систем
Руководитель группы АВР
Corbina Telecom
Tel: +7(495) 728-4000
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
___
bind-users mailing list
bind-users@lists.isc.org
https://lists.isc.org/mailman/listinfo/bind-users
RE: dnsperf and BIND memory consumption
-Original Message- From: JINMEI Tatuya / 神明達哉 [mailto:[EMAIL PROTECTED] Sent: Tuesday, December 09, 2008 3:38 PM To: Vinny Abello Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: dnsperf and BIND memory consumption At Tue, 9 Dec 2008 15:26:25 -0500, Vinny Abello [EMAIL PROTECTED] wrote: Has anybody else tried this patch for you? I haven't had time to look into this at all. If nobody has tried this yet, I'll get around to it when I can and let you know the result. No one else other than by myself. It worked perfectly for me, i.e., I could reproduce the problem and I could completely eliminate the leak with the patch. One thing I was not certain about in an off-list discussion that led to this patch was that the patch reportedly solved the leak only partially. I've been hoping to confirm that, but unfortunately I've not got any followup since then. So, basically, I believe the problem was solved, it would also help if you could confirm it. Thanks, --- JINMEI, Tatuya Internet Systems Consortium, Inc. Jinmei, I'll try to confirm when I have some spare time and let you know. -Vinny ___ bind-users mailing list bind-users@lists.isc.org https://lists.isc.org/mailman/listinfo/bind-users
