Re: gcc 4.2 miscompilation with -O2 -fno-omit-frame-pointer on amd64
On Sat, 19 Nov 2011 12:01:50 +0200 Gleb Kurtsou gleb.kurt...@gmail.com mentioned: Hi, I was lucky to write a bit of code which gcc 4.2 fails to compile correctly with -O2. Too keep long story short the code fails for gcc from base system and last gcc 4.2 snapshot from ports. It works with gcc 4.3, gcc 4.4 on FreeBSD and Linux. Clang from base is also good. -O and -Os optimization levels are fine (I've tried with all -f* flags mentioned in documentation) -O2 -fno-omit-frame-pointer combination is troublesome on amd64. I presume i386 should be fine. These options are also used for compilation of kernel (with debugging enabled) and modules. I'm not able to share the code, but have a test case reproducing the bug. I've encountered the issue over a week ago and tried narrowing it down to a simple test I could share but without much success. The code itself is very common: initialize two structs on stack, call a function with pointers to those stucts as arguments. A number of inlined assertion functions. gcc fails to correctly optimize struct assignments with -fno-omit-frame-pointer, I have a number of small structs assigned, gcc decides not to use data coping but to assign fields directly. I've tried disabling sra, tweaking sra parameters -- no luck in forcing it to copy data. Replacing one particular assignment with memcpy produces correct code, but that's not a solution. -O2 -fno-omit-frame-pointer -fno-inline is buggy -O2 -fno-omit-frame-pointer -frename-registers is buggy I found similar issue with gcc 4.6, but I'm not able to reproduce it with gcc test case: https://bugzilla.redhat.com/show_bug.cgi?id=679924 http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47893 I'll be glad to help debugging it and will be hanging on #bsddev during weekend as glk. Hi! I'm not sure this is relevant to your case, but our base gcc used to have a bug with strict aliasing, which was fixed only in a GPLv3 version of it. That's why we have -fno-strict-aliasing in default CFALGS. So you might try to build using -fno-strict-aliasing. -- Stanislav Sedov ST4096-RIPE () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments pgpuO922JsvHw.pgp Description: PGP signature
Re: gcc 4.2 miscompilation with -O2 -fno-omit-frame-pointer on amd64
On (02/12/2011 01:56), Stanislav Sedov wrote: On Sat, 19 Nov 2011 12:01:50 +0200 Gleb Kurtsou gleb.kurt...@gmail.com mentioned: Hi, I was lucky to write a bit of code which gcc 4.2 fails to compile correctly with -O2. Too keep long story short the code fails for gcc from base system and last gcc 4.2 snapshot from ports. It works with gcc 4.3, gcc 4.4 on FreeBSD and Linux. Clang from base is also good. -O and -Os optimization levels are fine (I've tried with all -f* flags mentioned in documentation) -O2 -fno-omit-frame-pointer combination is troublesome on amd64. I presume i386 should be fine. These options are also used for compilation of kernel (with debugging enabled) and modules. I'm not able to share the code, but have a test case reproducing the bug. I've encountered the issue over a week ago and tried narrowing it down to a simple test I could share but without much success. The code itself is very common: initialize two structs on stack, call a function with pointers to those stucts as arguments. A number of inlined assertion functions. gcc fails to correctly optimize struct assignments with -fno-omit-frame-pointer, I have a number of small structs assigned, gcc decides not to use data coping but to assign fields directly. I've tried disabling sra, tweaking sra parameters -- no luck in forcing it to copy data. Replacing one particular assignment with memcpy produces correct code, but that's not a solution. -O2 -fno-omit-frame-pointer -fno-inline is buggy -O2 -fno-omit-frame-pointer -frename-registers is buggy I found similar issue with gcc 4.6, but I'm not able to reproduce it with gcc test case: https://bugzilla.redhat.com/show_bug.cgi?id=679924 http://gcc.gnu.org/bugzilla/show_bug.cgi?id=47893 I'll be glad to help debugging it and will be hanging on #bsddev during weekend as glk. Hi! I'm not sure this is relevant to your case, but our base gcc used to have a bug with strict aliasing, which was fixed only in a GPLv3 version of it. That's why we have -fno-strict-aliasing in default CFALGS. So you might try to build using -fno-strict-aliasing. I always have -fno-strict-aliasing, the whole idea of misusing undefined behaviour to perform optimization is crazy. I guess it seemed evident to me so I've skipped the flag above. Besides gcc was barking with aliasing warnings on 3rd party party code in my case, I had to change warnings flags to run tests without -fno-strict-aliasing. I've dropped -fno-omit-frame-pointer, async unwind tables are ok for userland. Another work around was adding -finline-functions. Kernel and modules won't build with -finline-functions. So we are just lucky not to catch it. Thanks, Gleb. -- Stanislav Sedov ST4096-RIPE () ascii ribbon campaign - against html e-mail /\ www.asciiribbon.org - against proprietary attachments ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
Yeah On Fri, Dec 02, 2011 at 03:19:53PM +0800, Adrian Chadd wrote: .. where are these statistics coming from? top? Adrian ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
on 30/11/2011 14:39 Steven Hartland said the following: We're seeing some impossible memory usage stats reported on machines here from vmstat and sysctl vm.vmtotal. We have machines reporting to be using 31GB total when they only have 8GB physical and are not using any swap. Here's an output from one of our machines:- vmstat -c 2 -w 1 -n 0 procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 31768M 2112M 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 31768M 2112M 2 0 0 0 0 0 370 8139 3996 0 1 99 The raw output is:- vmstat -c 2 -w 1 -n 0 -H procs memory page faults cpu r b w avmfre flt re pi pofr sr in sy cs us sy id 0 0 0 32530228 2162524 586 0 0 0 421 0 106 270 569 0 6 94 0 0 0 32530228 2162524 2 0 0 0 0 0 286 8234 4347 0 1 99 Top shows:- last pid: 6665; load averages: 0.00, 0.00, 0.01 up 80+01:24:12 09:35:28 1893 processes:1 running, 1892 sleeping CPU: 0.0% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.7% idle Mem: 3754M Active, 84M Inact, 1976M Wired, 4K Cache, 2109M Free Swap: 4096M Total, 4096M Free sysctl vm.vmtotal vm.vmtotal: System wide totals computed every five seconds: (values in kilobytes) === Processes: (RUNQ: 1 Disk Wait: 0 Page Wait: 0 Sleep: 1893) Virtual Memory: (Total: 1106403532K Active: 32540260K) Real Memory:(Total: 4563648K Active: 3921644K) Shared Virtual Memory: (Total: 19976K Active: 16396K) Shared Real Memory: (Total: 9040K Active: 8436K) Free Memory Pages: 2161740K As mentioned this machine has 8GB of ram and according to both top and swapinfo is using no swap at all From dmesg:- real memory = 8589934592 (8192 MB) avail memory = 823536 (7873 MB) swapinfo Device 1K-blocks UsedAvail Capacity /dev/gptid/09f211f7-39ce-11e0-8 41943040 4194304 0% uname -a FreeBSD test 8.2-RELEASE FreeBSD 8.2-RELEASE #2: Thu Mar 24 17:28:55 UTC 2011 root@test:/usr/obj/usr/src/sys/MULTIPLAY amd64 sysctl hw.pagesize hw.pagesize: 4096 It looks like it may be out by a factor of 4, possibly due to the fact the its a 4k page size not 1k as indicated by the vmstat man page:- memory Information about the usage of virtual and real memory. Virtual pages (reported in units of 1024 bytes) are considered active if they belong to processes which are running or have run in the last 20 seconds. avm active virtual pages fre size of the free list I think that the description of avm is not sufficiently complete as it doesn't explain meaning of the virtual memory it refers too. Reference to units of 1024 bytes when talking about pages also sounds a bit vague. The values reported are actually byte sizes, non-humanized numbers are in KB (units of 1024 bytes in the poetic speech), humanized numbers carry an appropriate suffix. Totalling up RSS from ps axo rss gives a total in the region of that if the vm stats are out by a factor of 4, in this case it should be: 8132557 which is 7.75GB a much more realistic value. Am I totally missing something or is there problem here? Likely more of the former than of latter. Those virtual sizes are not sufficiently explained, but you have been warned that those are not physical sizes, so I am not sure why you try to compare the virtual figures with the physical figures. Here's an example. Let' say you mmap-ed a 1GB file into a process memory space, here you immediately increased your virtual size counts by 1GB, even if you hadn't accessed any bytes in the file yet and so none of them were in physical memory. The same applies to anonymous memory. P.S. the above is reveled by a cursory look through the code (which is publicly available btw) :-) -- Andriy Gapon ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
On 12/1/11 1:05 PM, Jason Hellenthal wrote: On Thu, Dec 01, 2011 at 10:44:58AM -, Steven Hartland wrote: - Original Message - From: Jason Hellenthaljh...@dataix.net This goes along with the thoughts I had about 4 months ago tending to some zfs statistics as well top showing greater than 100% actual CPU usage. This is a big pet peave of mine. Its like saying you ate 134% of a bannanna when in all reallity it is impossible. You can never have more than 100% usage of anything and when seen is a clear notice that some math is considerably incorrect leading to other such miscalculations to be performed. Things like the above already have checks in place that ensure no boundries are being crossed/overflowed or underrun but it surely makes processing results building future products a bitch. One instance is the calculation of threads for example firefox can be seen using upto or more 338% of the CPU. Thats impossible its like saying anyones CPU grew by 400%. I could understand a bit of overflow as stats are snapshots which may not be instuntanious, but 31GB instead of under 8GB is hardly a rounding issue / overflow. I agree With respect to top showing greater than 100% by how much are you talking? Do your realise that each core = 100%? So if you have a quad core your system total will be 400% not 100%? Yeah I realize that but it still would lead you to believe that if a proccessor has 4 cores on the same die then total for each core could only be 25% usage. And the usage for a proccess only consuming full usage of 1 core is 100%. But you can start firefox on a single uniproccessor and like stated above see large usage percents near 338% or greater which is impossible and leads me to believe were forcing calculation for the entire proccess of threads onto tthread 0. This makes accounting pretty difficult. A single-package machine with 4 cores on the die is not a uniprocessor machine. It is an SMP machine. Try booting a kernel without SMP, _that_ will give you a UP machine. The %CPU usage is actually very simple if you stop trying to make it so complicated. Each potential concurrent thread of execution is mapped to 100%. Thus, if you have a system with 16 potential threads (either due to 4 quad-core packages, or 2 quad-core package where each core has 2 threads via HTT, etc.), then you have a total CPU usage of 1600%. Why does this make sense? Because a machine with 16 concurrent threads can (theoretically) do 16x the work of an otherwise identical machine with a single thread of execution within a given unit of wall time. Another way to look at it is that in FreeBSD, each thread of execution is treated as a CPU in top, etc. Thus, 100% CPU means that a given thread is using all of the available cycles on a CPU. If you have a multithreaded app (like Firefox) that is using all of the available cycles on 4 CPUs, then that would be 400% CPU (it's using 4 CPUs). It may only be using 25% of the available system-wide CPU cycles, but that is not what %CPU measures. Anyways, pretty much everyone I've ever talked to about this gets it right away, so the current arrangement is fine for the majority of folks. It has also been that way since FreeBSD first added SMP over a decade ago. It is also true on other OS's such as OS X, so the current arrangement is here to stay. -- John Baldwin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
- Original Message - From: Jason Hellenthal jh...@dataix.net Just to put some visuals to this... . `-- DIE |-- Core1 [Idle] |-- Core2 [35% ] | `-- thread127 |-- Core3 [40% ] | `-- thread127 `-- Core4 [100%] `-- thread127 In this case you would say the DIE should be at a total of 175% ? I think your getting confused there; it sounds like your referring to a single CPU capable of multiple tasks via either multiple cores or HTT as an UP machine? If so that's your problem this isn't UP it SMP. Have a look on your machine in /var/run/dmesg.boot if your see it reporting more than one core then your SMP not UP hence the confusion as each of these cores be they real, virtual or physical represents a possible thread of 100% so even if you have a single physical CPU with 4 cores that still represents a possible total of 400% e.g the following shows a machine capable of 1600% if all cores are busy. FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 SMT threads cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 cpu4 (AP): APIC ID: 4 cpu5 (AP): APIC ID: 5 cpu6 (AP): APIC ID: 6 cpu7 (AP): APIC ID: 7 cpu8 (AP): APIC ID: 16 cpu9 (AP): APIC ID: 17 cpu10 (AP): APIC ID: 18 cpu11 (AP): APIC ID: 19 cpu12 (AP): APIC ID: 20 cpu13 (AP): APIC ID: 21 cpu14 (AP): APIC ID: 22 cpu15 (AP): APIC ID: 23 If you want proper UP which will total 100% you could remove SMP from your kernel but I wouldnt advise that ;-) Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
- Original Message - From: Andriy Gapon a...@freebsd.org Totalling up RSS from ps axo rss gives a total in the region of that if the vm stats are out by a factor of 4, in this case it should be: 8132557 which is 7.75GB a much more realistic value. Am I totally missing something or is there problem here? Likely more of the former than of latter. Those virtual sizes are not sufficiently explained, but you have been warned that those are not physical sizes, so I am not sure why you try to compare the virtual figures with the physical figures. My miss-understanding was due to what virtual actually meant. Here's an example. Let' say you mmap-ed a 1GB file into a process memory space, here you immediately increased your virtual size counts by 1GB, even if you hadn't accessed any bytes in the file yet and so none of them were in physical memory. The same applies to anonymous memory. P.S. the above is reveled by a cursory look through the code (which is publicly available btw) :-) Yer I did have a dig around before posting and ended up the code for vm.vmtotal, which is where vmstat gets its info from but that's just a summation of each object's size from vm_object_list. Thats where I got lost without an insight into what a vm_object.size actually represents. Your info about mmap'ed files helped point me in the right direction as it identified space that shows as virtual but doesn't show in swap or real ram, which is what I was missing. Given this starting point the following links provided me with addtional information:- http://www.freebsd.org/doc/en/books/arch-handbook/vm.html http://www.freebsd.org/doc/en/books/design-44bsd/overview-memory-management.html http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/ http://www.cse.chalmers.se/edu/course/EDA203/unix4.pdf I was under the incorrect impression that Virtual Memory (VM) was so named as it was a unified physical memory and swap (virtual memory), but its not that simple, as other items such as file-backed objects also count to this total which would never show in physical or swap allocation of other tools such as top and swapinfo. So what I believe is now the big cause of virtual memory uplift vs the memory totals shown by ps / top is that the vm totals include things like file backed memory mapped process binaries, shared libs etc many multiple times. This would explain why this specific machine shows the applification more than others here as it runs thousands of very small lightweight processes. Thanks for pointer Andy, I now understand a lot more about the BSD VMS :) What do people think about expanding that entry in the man page of vmstat to clarify just what active virtual pages really means? Regards Steve This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it. In the event of misdirection, illegible or incomplete transmission please telephone +44 845 868 1337 or return the E.mail to postmas...@multiplay.co.uk. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
rtld and noexec
Hi,
Here is a patch I use to prevent loading a shared object from a noexec
mountpoint. It's an easy way, I found, after the last root exploit
((http://seclists.org/fulldisclosure/2011/Nov/452), to enhance the
security of my web servers (with /home, /tmp and /var/tmp mounted with
noexec).
- the last ftpd/porftpd (libc ?) exploit does not work (indirect use
of rtld via nsswitch)
- the previous rtld security issue should have been more difficult to
use in a noexec context.
- It may help to prevent some miscellaneous usage of common softwares
using dlopen like apache or php.
I think it also makes sens because loading a shared object sounds like
a kind of execution.
What do you think about this patch and the opportunity to open a PR on
this subject?
Cheers
Joris
--- libexec/rtld-elf/rtld.c.orig2011-12-02 12:09:40.0 +0100
+++ libexec/rtld-elf/rtld.c 2011-12-02 13:45:18.0 +0100
@@ -1123,32 +1123,50 @@
{
char *pathname;
char *name;
+struct statfs mnt;
if (strchr(xname, '/') != NULL) { /* Hard coded pathname */
+ name = NULL;
if (xname[0] != '/' !trust) {
_rtld_error(Absolute pathname required for shared object \%s\,
xname);
return NULL;
}
if (refobj != NULL refobj-z_origin)
- return origin_subst(xname, refobj-origin_path);
+ pathname = origin_subst(xname, refobj-origin_path);
else
- return xstrdup(xname);
+ pathname = xstrdup(xname);
+}
+else { /* xname is not a path */
+ if (libmap_disable || (refobj == NULL) ||
+ (name = lm_find(refobj-path, xname)) == NULL)
+ name = (char *)xname;
+
+ dbg( Searching for \%s\, name);
+
+ pathname = search_library_path(name, ld_library_path);
+ if (pathname == NULL refobj != NULL)
+pathname = search_library_path(name, refobj-rpath);
+ if (pathname == NULL)
+pathname = search_library_path(name, gethints());
+ if (pathname == NULL)
+pathname = search_library_path(name, STANDARD_LIBRARY_PATH);
+}
+
+if (pathname != NULL) { /* noexec mountpoint in pathname */
+ if (statfs(pathname, mnt) != 0)
+free(pathname);
+ else {
+if (mnt.f_flags MNT_NOEXEC) {
+ _rtld_error(noexec violation for shared object
\%s\, pathname);
+ free(pathname);
+ return NULL;
+}
+else
+ return pathname;
+ }
}
-if (libmap_disable || (refobj == NULL) ||
- (name = lm_find(refobj-path, xname)) == NULL)
- name = (char *)xname;
-
-dbg( Searching for \%s\, name);
-
-if ((pathname = search_library_path(name, ld_library_path)) != NULL ||
- (refobj != NULL
- (pathname = search_library_path(name, refobj-rpath)) != NULL) ||
- (pathname = search_library_path(name, gethints())) != NULL ||
- (pathname = search_library_path(name, STANDARD_LIBRARY_PATH)) != NULL)
- return pathname;
-
if(refobj != NULL refobj-path != NULL) {
_rtld_error(Shared object \%s\ not found, required by \%s\,
name, basename(refobj-path));
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: rtld and noexec
On Fri, 2 Dec 2011 18:22:57 +0100
joris dedieu joris.ded...@gmail.com wrote:
Hi,
Here is a patch I use to prevent loading a shared object from a noexec
mountpoint. It's an easy way, I found, after the last root exploit
((http://seclists.org/fulldisclosure/2011/Nov/452), to enhance the
security of my web servers (with /home, /tmp and /var/tmp mounted with
noexec).
- the last ftpd/porftpd (libc ?) exploit does not work (indirect use
of rtld via nsswitch)
- the previous rtld security issue should have been more difficult to
use in a noexec context.
- It may help to prevent some miscellaneous usage of common softwares
using dlopen like apache or php.
I think it also makes sens because loading a shared object sounds like
a kind of execution.
What do you think about this patch and the opportunity to open a PR on
this subject?
Cheers
Joris
--- libexec/rtld-elf/rtld.c.orig2011-12-02 12:09:40.0
+0100 +++ libexec/rtld-elf/rtld.c 2011-12-02 13:45:18.0
+0100 @@ -1123,32 +1123,50 @@
{
char *pathname;
char *name;
+struct statfs mnt;
if (strchr(xname, '/') != NULL) { /* Hard coded pathname */
+ name = NULL;
if (xname[0] != '/' !trust) {
_rtld_error(Absolute pathname required for shared object
\%s\, xname);
return NULL;
}
if (refobj != NULL refobj-z_origin)
- return origin_subst(xname, refobj-origin_path);
+ pathname = origin_subst(xname, refobj-origin_path);
else
- return xstrdup(xname);
+ pathname = xstrdup(xname);
+}
+else { /* xname is not a path */
+ if (libmap_disable || (refobj == NULL) ||
+ (name = lm_find(refobj-path, xname)) == NULL)
+ name = (char *)xname;
+
+ dbg( Searching for \%s\, name);
+
+ pathname = search_library_path(name, ld_library_path);
+ if (pathname == NULL refobj != NULL)
+pathname = search_library_path(name, refobj-rpath);
+ if (pathname == NULL)
+pathname = search_library_path(name, gethints());
+ if (pathname == NULL)
+pathname = search_library_path(name,
STANDARD_LIBRARY_PATH);
+}
+
+if (pathname != NULL) { /* noexec mountpoint in pathname */
+ if (statfs(pathname, mnt) != 0)
+free(pathname);
+ else {
+if (mnt.f_flags MNT_NOEXEC) {
+ _rtld_error(noexec violation for shared object
\%s\, pathname);
+ free(pathname);
+ return NULL;
+}
+else
+ return pathname;
+ }
}
-if (libmap_disable || (refobj == NULL) ||
- (name = lm_find(refobj-path, xname)) == NULL)
- name = (char *)xname;
-
-dbg( Searching for \%s\, name);
-
-if ((pathname = search_library_path(name, ld_library_path)) !=
NULL ||
- (refobj != NULL
- (pathname = search_library_path(name, refobj-rpath)) != NULL)
||
- (pathname = search_library_path(name, gethints())) != NULL ||
- (pathname = search_library_path(name,
STANDARD_LIBRARY_PATH)) != NULL)
- return pathname;
-
if(refobj != NULL refobj-path != NULL) {
_rtld_error(Shared object \%s\ not found, required by
\%s\, name, basename(refobj-path));
___
1. There is a race using statfs and then loading the file.
2. We already have the check in do_load_object
--
Alexander Kabaev
signature.asc
Description: PGP signature
Re: To implement RFC 5848 (Signed Syslog Messages)?
On 12/01/11 10:01, Zhihao Yuan wrote: trivial. How about to implement RFC 5848 in our syslogd? In 2008 I implemented the syslog RFCs for NetBSD's syslogd, so if you are interested please take a look at the syslog code in NetBSD-current and at my report, linked under http://mschuette.name/wp/gsoc-syslogd/ I've always wanted to create a FreeBSD port for it, but never found enough time to continue the development :( Albert Mietus made a nice presentation in 2002 http://www.slideshare.net/SoftwareBeterMaken.nl/securing-syslog-on-freebsd Not sure whether his code is accessible or not. Albert Mietus' code is available at http://sourceforge.net/projects/syslog-sec/ -- Martin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: To implement RFC 5848 (Signed Syslog Messages)?
2011/12/2 Martin Schütte li...@mschuette.name: On 12/01/11 10:01, Zhihao Yuan wrote: trivial. How about to implement RFC 5848 in our syslogd? In 2008 I implemented the syslog RFCs for NetBSD's syslogd, so if you are interested please take a look at the syslog code in NetBSD-current and at my report, linked under http://mschuette.name/wp/gsoc-syslogd/ That's an amazing work. Did you compared those documents (they were drafts in 08') with the final versions? Any differences? I've always wanted to create a FreeBSD port for it, but never found enough time to continue the development :( Albert Mietus made a nice presentation in 2002 http://www.slideshare.net/SoftwareBeterMaken.nl/securing-syslog-on-freebsd Not sure whether his code is accessible or not. Albert Mietus' code is available at http://sourceforge.net/projects/syslog-sec/ -- Martin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org -- Zhihao Yuan, nickname lichray The best way to predict the future is to invent it. ___ 4BSD -- http://4bsd.biz/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: To implement RFC 5848 (Signed Syslog Messages)?
On 12/02/11 23:45, Zhihao Yuan wrote: In 2008 I implemented the syslog RFCs for NetBSD's syslogd, so if you That's an amazing work. Did you compared those documents (they were drafts in 08') with the final versions? Any differences? I followed the IETF process and as far as I know there are two major differences: a) For syslog-sign I encoded signatures with a PEM format in the way of X.509/OpenSSL, but the final RFC specifies an OpenPGP-like encoding. b) For TLS transport the rules for peer certificate verification (always a very confusing problem) were discussed and modified in the later drafts. Most notably the RFC requires support for wildcards in DNS names, which is not implemented. -- Martin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
Re: Invalid memory stats from vmstat and sysctl vm.vmtotal?
On Fri, Dec 02, 2011 at 05:13:05PM -, Steven Hartland wrote: - Original Message - From: Andriy Gapon a...@freebsd.org Totalling up RSS from ps axo rss gives a total in the region of that if the vm stats are out by a factor of 4, in this case it should be: 8132557 which is 7.75GB a much more realistic value. Am I totally missing something or is there problem here? Likely more of the former than of latter. Those virtual sizes are not sufficiently explained, but you have been warned that those are not physical sizes, so I am not sure why you try to compare the virtual figures with the physical figures. My miss-understanding was due to what virtual actually meant. Here's an example. Let' say you mmap-ed a 1GB file into a process memory space, here you immediately increased your virtual size counts by 1GB, even if you hadn't accessed any bytes in the file yet and so none of them were in physical memory. The same applies to anonymous memory. P.S. the above is reveled by a cursory look through the code (which is publicly available btw) :-) Yer I did have a dig around before posting and ended up the code for vm.vmtotal, which is where vmstat gets its info from but that's just a summation of each object's size from vm_object_list. Thats where I got lost without an insight into what a vm_object.size actually represents. Your info about mmap'ed files helped point me in the right direction as it identified space that shows as virtual but doesn't show in swap or real ram, which is what I was missing. Given this starting point the following links provided me with addtional information:- http://www.freebsd.org/doc/en/books/arch-handbook/vm.html http://www.freebsd.org/doc/en/books/design-44bsd/overview-memory-management.html http://www.freebsd.org/doc/en_US.ISO8859-1/articles/vm-design/ http://www.cse.chalmers.se/edu/course/EDA203/unix4.pdf I was under the incorrect impression that Virtual Memory (VM) was so named as it was a unified physical memory and swap (virtual memory), but its not that simple, as other items such as file-backed objects also count to this total which would never show in physical or swap allocation of other tools such as top and swapinfo. So what I believe is now the big cause of virtual memory uplift vs the memory totals shown by ps / top is that the vm totals include things like file backed memory mapped process binaries, shared libs etc many multiple times. This would explain why this specific machine shows the applification more than others here as it runs thousands of very small lightweight processes. Thanks for pointer Andy, I now understand a lot more about the BSD VMS :) What do people think about expanding that entry in the man page of vmstat to clarify just what active virtual pages really means? Regards Steve Thanks for your research Steve. That makes perfect sense and additions to the documentation are surely needed. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to freebsd-hackers-unsubscr...@freebsd.org
