Re: Porting patch(1) from NetBSD to FreeBSD (was Re: FreeBSD in Google Code-In 2012? You can help too!)
Thank you all for the inputs. I understand this is a long grueling process so I will attempt to do things in approximately following order: 1) prepare a new port for bsd patch 2) make sure new bsd patch has all options of existing gnu patch 3) merge outstanding patches: http://svnweb.freebsd.org/base/head/gnu/usr.bin/patch/?view=log 4) bug compatibility 5) performance cheers, Hiren ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
Hi All, Can anyone think of any quick pointers as to why some code originally written under 6.4 amd64 - when re-compiled under 9.0-stable amd64 takes up a *lot* more memory when running? The code involved is a sendmail Milter, and a TCP server type program (that runs up a large number of threads [~700] at startup). Both were previously compiled with: -O2 -pthread -lc_r They're now compiled under 9.0-S with just: -O2 -pthread As an example, under 6.4 the size/res for one reported by top is 81Mb/48Mb - and for the other is 44Mb/9Mb [this is after it's been running for weeks]. Under 9.0-stable the initial memory used by the processes just after starting rises to 625Mb/128Mb and 519Mb/65Mb respectively. Is that something I need to worry about? They've not been running longing enough yet to see if anything is 'leaking' (i.e. if size/res continues to go up). Just thought I'd ask if there's a simple/possible explanation for this - and if it's something I need to worry about... Thanks, -Karl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
- Original Message - From: Karl Pielorz kpielorz_...@tdx.co.uk To: freebsd-hackers@freebsd.org Sent: Tuesday, October 30, 2012 11:12 AM Subject: Threaded 6.4 code compiled under 9.0 uses a lot more memory?.. Hi All, Can anyone think of any quick pointers as to why some code originally written under 6.4 amd64 - when re-compiled under 9.0-stable amd64 takes up a *lot* more memory when running? The code involved is a sendmail Milter, and a TCP server type program (that runs up a large number of threads [~700] at startup). Both were previously compiled with: -O2 -pthread -lc_r They're now compiled under 9.0-S with just: -O2 -pthread As an example, under 6.4 the size/res for one reported by top is 81Mb/48Mb - and for the other is 44Mb/9Mb [this is after it's been running for weeks]. Under 9.0-stable the initial memory used by the processes just after starting rises to 625Mb/128Mb and 519Mb/65Mb respectively. Is that something I need to worry about? They've not been running longing enough yet to see if anything is 'leaking' (i.e. if size/res continues to go up). Just thought I'd ask if there's a simple/possible explanation for this - and if it's something I need to worry about... amd64 vs i386? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
Hi, On Tue, 30 Oct 2012 11:12:22 + Karl Pielorz kpielorz_...@tdx.co.uk wrote: Can anyone think of any quick pointers as to why some code originally written under 6.4 amd64 - when re-compiled under 9.0-stable amd64 takes up a *lot* more memory when running? is it still the same compiler? As an example, under 6.4 the size/res for one reported by top is 81Mb/48Mb - and for the other is 44Mb/9Mb [this is after it's been running for weeks]. I assume that it is a plain FreeBSD program without X. Erich ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
--On 30 October 2012 11:21 + Steven Hartland kill...@multiplay.co.uk wrote: They've not been running longing enough yet to see if anything is 'leaking' (i.e. if size/res continues to go up). Just thought I'd ask if there's a simple/possible explanation for this - and if it's something I need to worry about... amd64 vs i386? Nice try :) - But as I said in my original email (he says, checking again - yup it's in there ;) both the 6.4 and 9.0 systems are amd64... I'd expect the memory usage to 'go up' between the versions for the programs, I'm just concerned by the size increase... -Karl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
Karl Pielorz kpielorz_...@tdx.co.uk wrote: Can anyone think of any quick pointers as to why some code originally written under 6.4 amd64 - when re-compiled under 9.0-stable amd64 takes up a *lot* more memory when running? 6.4 comes with phkmalloc while 9.0 uses jemalloc. Maybe you are allocating memory in a way that is less space-efficiently handled by jemalloc's default configuration. Fabian signature.asc Description: PGP signature
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
If this is only difference between gcc34 v gcc42 it's quite spectacular... -- View this message in context: http://freebsd.1045724.n5.nabble.com/Threaded-6-4-code-compiled-under-9-0-uses-a-lot-more-memory-tp5756466p5756476.html Sent from the freebsd-hackers mailing list archive at Nabble.com. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
--On 30 October 2012 18:27 +0700 Erich Dollansky erichfreebsdl...@ovitrap.com wrote: is it still the same compiler? Depends how you mean 'the same' - on the 6.4 system it shows: cc (GCC) 3.4.6 [FreeBSD] 20060305 And, on the 9.0-S it shows: cc (GCC) 4.2.1 20070831 patched [FreeBSD] So 'same' - but different versions. I assume that it is a plain FreeBSD program without X. Yes. They are 'plain' programs - no X or anything. Now they've been running for an hour or so - they've gotten a little larger 552M/154M and 703M/75M. If it's not harmful I can live with it - it was just a bit of a surprise. -Karl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
--On 30 October 2012 22:59 +1100 Jan Mikkelsen j...@transactionware.com wrote: -O2 -pthread -lc_r They're now compiled under 9.0-S with just: -O2 -pthread libc_r is a user mode implementation of pthreads, so there is one actual kernel thread with a stack. You now have ~700 kernel threads on startup. Per-thread stack allocation will be different, and you could quite easily explain differences that way. That seems the most fitting explanation so far - aside from seeing if I can cut back on the number of threads, I presume there's no issue with having that many kicking around - the RES size is still quite 'small' (still waiting to see if anything is 'leaking') - and if ~700 threads happily ran under user mode pthreads - it should still perform at least 'similarly' with kernel threading? -Karl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
Hi, On 30/10/2012, at 10:12 PM, Karl Pielorz kpielorz_...@tdx.co.uk wrote: Hi All, Can anyone think of any quick pointers as to why some code originally written under 6.4 amd64 - when re-compiled under 9.0-stable amd64 takes up a *lot* more memory when running? The code involved is a sendmail Milter, and a TCP server type program (that runs up a large number of threads [~700] at startup). Both were previously compiled with: -O2 -pthread -lc_r They're now compiled under 9.0-S with just: -O2 -pthread libc_r is a user mode implementation of pthreads, so there is one actual kernel thread with a stack. You now have ~700 kernel threads on startup. Per-thread stack allocation will be different, and you could quite easily explain differences that way. Regards, Jan. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
Hi, On Tue, 30 Oct 2012 11:59:46 + Karl Pielorz kpielorz_...@tdx.co.uk wrote: --On 30 October 2012 18:27 +0700 Erich Dollansky erichfreebsdl...@ovitrap.com wrote: is it still the same compiler? Depends how you mean 'the same' - on the 6.4 system it shows: cc (GCC) 3.4.6 [FreeBSD] 20060305 And, on the 9.0-S it shows: cc (GCC) 4.2.1 20070831 patched [FreeBSD] So 'same' - but different versions. did you check the default data sizes? I assume that it is a plain FreeBSD program without X. Yes. They are 'plain' programs - no X or anything. Now they've been running for an hour or so - they've gotten a little larger 552M/154M and 703M/75M. If it's not harmful I can live with it - it was just a bit of a surprise. And a reason to spend more money on memory. Knowing the real reason would be better. I can understand your surprise. Erich ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
On Tue, 2012-10-30 at 13:46 +0100, Fabian Keil wrote: Karl Pielorz kpielorz_...@tdx.co.uk wrote: Can anyone think of any quick pointers as to why some code originally written under 6.4 amd64 - when re-compiled under 9.0-stable amd64 takes up a *lot* more memory when running? 6.4 comes with phkmalloc while 9.0 uses jemalloc. Maybe you are allocating memory in a way that is less space-efficiently handled by jemalloc's default configuration. Fabian jemalloc is certainly the first thing that came to my mind. Does MALLOC_PRODUCTION need to be defined on a 9.0 system, or is that something that gets turned on automatically in an official release build? (I'm always working with non-release stuff so I'm not sure how that gets handled). -- Ian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
On 30/10/2012 15:47, Ian Lepore wrote: On Tue, 2012-10-30 at 13:46 +0100, Fabian Keil wrote: Karl Pielorz kpielorz_...@tdx.co.uk wrote: Can anyone think of any quick pointers as to why some code originally written under 6.4 amd64 - when re-compiled under 9.0-stable amd64 takes up a *lot* more memory when running? 6.4 comes with phkmalloc while 9.0 uses jemalloc. Maybe you are allocating memory in a way that is less space-efficiently handled by jemalloc's default configuration. Fabian jemalloc is certainly the first thing that came to my mind. Does MALLOC_PRODUCTION need to be defined on a 9.0 system, or is that something that gets turned on automatically in an official release build? (I'm always working with non-release stuff so I'm not sure how that gets handled). It is turned on by default on -stable, by a commit from release engineers. signature.asc Description: OpenPGP digital signature
pxeboot slowness when run in vmware
hi, as soon as I 'initialize' a virtual disk via gpart, even if nothing is mounted, the pxeboot adds around 60s delay to show the boot menu, - I don't know if the delay is in boot or pxeboot. if I destroy the geom, the the boot menu appears inmediately. any insight? danny ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
--On 30 October 2012 19:43 +0700 Erich Dollansky erichfreebsdl...@ovitrap.com wrote: Depends how you mean 'the same' - on the 6.4 system it shows: cc (GCC) 3.4.6 [FreeBSD] 20060305 And, on the 9.0-S it shows: cc (GCC) 4.2.1 20070831 patched [FreeBSD] So 'same' - but different versions. did you check the default data sizes? How do you mean? Now they've been running for an hour or so - they've gotten a little larger 552M/154M and 703M/75M. If it's not harmful I can live with it - it was just a bit of a surprise. And a reason to spend more money on memory. Knowing the real reason would be better. I can understand your surprise. Hehe, more 'concern' than surprise I guess now... The sendmail milter has grown to a SIZE/RES of 1045M / 454M under 9.0. The original 6.4 machine under heaver load (more connections) shows a SIZE/RES of 85M/52M. The TCP listener code is now showing a SIZE/REZ of 815M/80M under 9.0 with the original 6.4 box showing 44M/9.5M The 9.0 box says it has 185M active, 472M inactive, 693M wired, 543M buf, and 4554M free. At this stage I'm just a bit concerned that at least the milter code is going to grow, and grow - and die. I would think it would last over night so I'll see what the figures are in the morning. Thanks for the replies... -Karl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
Some suggestions here, jemalloc, kernel threads are good ones. Another issue may just be some change for default thread stack size. This would explain why the RESIDENT set is the same, but the VIRTUAL grew. -Alfred On 10/30/12 9:56 AM, Karl Pielorz wrote: --On 30 October 2012 19:43 +0700 Erich Dollansky erichfreebsdl...@ovitrap.com wrote: Depends how you mean 'the same' - on the 6.4 system it shows: cc (GCC) 3.4.6 [FreeBSD] 20060305 And, on the 9.0-S it shows: cc (GCC) 4.2.1 20070831 patched [FreeBSD] So 'same' - but different versions. did you check the default data sizes? How do you mean? Now they've been running for an hour or so - they've gotten a little larger 552M/154M and 703M/75M. If it's not harmful I can live with it - it was just a bit of a surprise. And a reason to spend more money on memory. Knowing the real reason would be better. I can understand your surprise. Hehe, more 'concern' than surprise I guess now... The sendmail milter has grown to a SIZE/RES of 1045M / 454M under 9.0. The original 6.4 machine under heaver load (more connections) shows a SIZE/RES of 85M/52M. The TCP listener code is now showing a SIZE/REZ of 815M/80M under 9.0 with the original 6.4 box showing 44M/9.5M The 9.0 box says it has 185M active, 472M inactive, 693M wired, 543M buf, and 4554M free. At this stage I'm just a bit concerned that at least the milter code is going to grow, and grow - and die. I would think it would last over night so I'll see what the figures are in the morning. Thanks for the replies... -Karl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Threaded 6.4 code compiled under 9.0 uses a lot more memory?..
On Tue, Oct 30, 2012 at 10:48:03AM -0700, Alfred Perlstein wrote: Some suggestions here, jemalloc, kernel threads are good ones. Another issue may just be some change for default thread stack size. This would explain why the RESIDENT set is the same, but the VIRTUAL grew. I suggest to take a look at where the actual memory goes. Start with procstat -v. -Alfred On 10/30/12 9:56 AM, Karl Pielorz wrote: --On 30 October 2012 19:43 +0700 Erich Dollansky erichfreebsdl...@ovitrap.com wrote: Depends how you mean 'the same' - on the 6.4 system it shows: cc (GCC) 3.4.6 [FreeBSD] 20060305 And, on the 9.0-S it shows: cc (GCC) 4.2.1 20070831 patched [FreeBSD] So 'same' - but different versions. did you check the default data sizes? How do you mean? Now they've been running for an hour or so - they've gotten a little larger 552M/154M and 703M/75M. If it's not harmful I can live with it - it was just a bit of a surprise. And a reason to spend more money on memory. Knowing the real reason would be better. I can understand your surprise. Hehe, more 'concern' than surprise I guess now... The sendmail milter has grown to a SIZE/RES of 1045M / 454M under 9.0. The original 6.4 machine under heaver load (more connections) shows a SIZE/RES of 85M/52M. The TCP listener code is now showing a SIZE/REZ of 815M/80M under 9.0 with the original 6.4 box showing 44M/9.5M The 9.0 box says it has 185M active, 472M inactive, 693M wired, 543M buf, and 4554M free. At this stage I'm just a bit concerned that at least the milter code is going to grow, and grow - and die. I would think it would last over night so I'll see what the figures are in the morning. Thanks for the replies... -Karl ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org pgpGsStsnYyLq.pgp Description: PGP signature
