Re: Question about adding flags to mmap system call / NVIDIA amd64 driver implementation
Robert Noland wrote: On Tue, 2009-04-28 at 16:48 -0500, Kevin Day wrote: On Apr 28, 2009, at 3:19 PM, Julian Bangert wrote: Hello, I am currently trying to work a bit on the remaining "missing feature" that NVIDIA requires ( http://wiki.freebsd.org/NvidiaFeatureRequests or a back post in this ML) - the improved mmap system call. you might check with jhb (john Baldwin) as I think (from his p4 work) that he may be doing something in this area in p4. For now, I am trying to extend the current system call and implementation to add cache control ( the type of memory caching used) . This feature inherently is very architecture specific- but it can lead to enormous performance improvements for memmapped devices ( useful for drivers, etc). I would do this at the user site by adding 3 flags to the mmap system call (MEM_CACHE__ATTR1 to MEM_CACHE__ATTR3 ) which are a single octal digit corresponding to the various caching options ( like Uncacheable,Write Combining, etc... ) with the same numbers as the PAT_* macros from i386/include/ specialreg.h except that the value 0 ( PAT_UNCACHEABLE ) is replaced with value 2 ( undefined), whereas value 0 ( all 3 flags cleared) is assigned the meaning "feature not used, use default cache control". For each cache behaviour there would of course also be a macro expanding to the rigth combination of these flags for enhanced useability. The mmap system call would, if any of these flags are set, decode them and get a corresponding PAT_* value, perform the mapping and then call into the pmap module to modify the cache attributes for every page. Have you looked at mem(4) yet? Several architectures allow attributes to be associated with ranges of physical memory. These attributes can be manipulated via ioctl() calls performed on /dev/mem. Declarations and data types are to be found in . The specific attributes, and number of programmable ranges may vary between architectures. The full set of supported attributes is: MDF_UNCACHEABLE The region is not cached. MDF_WRITECOMBINE Writes to the region may be combined or performed out of order. MDF_WRITETHROUGH Writes to the region are committed synchronously. MDF_WRITEBACK Writes to the region are committed asynchronously. MDF_WRITEPROTECT The region cannot be written to. This requires knowledge of the physical addresses, but I believe that's probably already necessary for what it sounds like you're trying to accomplish. Back in the FreeBSD-3.0 days, I was writing a custom driver for an AGP graphics controller, and setting the MTRR flags for the exposed buffer was a definite improvement (200-1200% faster in most cases). This is MTRR, which is what we currently do, when we can. The issue is that often times the BIOS maps ranges in a way that prevents us from using MTRR. This is generally ideal for things like agp and framebuffers when it works, since they have a specific physical range that you want to work with. With PCI(E) cards it isn't as cut and dry... In the ATI and Nouveau cases, we map scatter gather pages into the GART, which generally are allocated using contigmalloc behind the scenes, so it is also possible for it to work in that case. Moving forward, we may actually be mapping random pages into and out of the GART (GEM / TTM). In those cases we really don't have a large contiguous range that we could set MTRR on. Intel CPUs are limited to 8 MTRR registers for the entire system also, so that can become an issue quickly if you are trying to manipulate several areas of memory. With PAT we can manipulate the caching properties on a page level. PAT also allows for some overlap conditions that MTRR won't, such as mapping a page write-combining on top on an UNCACHEABLE MTRR. jhb@ has started some work on this, since I've been badgering him about this recently as well. robert. -- Kevin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Question about adding flags to mmap system call / NVIDIA amd64 driver implementation
On Tue, 2009-04-28 at 16:48 -0500, Kevin Day wrote: > On Apr 28, 2009, at 3:19 PM, Julian Bangert wrote: > > > Hello, > > > > I am currently trying to work a bit on the remaining "missing > > feature" that NVIDIA requires ( > > http://wiki.freebsd.org/NvidiaFeatureRequests > > or a back post in this ML) - the improved mmap system call. > > For now, I am trying to extend the current system call and > > implementation to add cache control ( the type of memory caching > > used) . This feature inherently is very architecture specific- but > > it can lead to enormous performance improvements for memmapped > > devices ( useful for drivers, etc). I would do this at the user site > > by adding 3 flags to the mmap system call (MEM_CACHE__ATTR1 to > > MEM_CACHE__ATTR3 ) which are a single octal digit corresponding to > > the various caching options ( like Uncacheable,Write Combining, > > etc... ) with the same numbers as the PAT_* macros from i386/include/ > > specialreg.h except that the value 0 ( PAT_UNCACHEABLE ) is replaced > > with value 2 ( undefined), whereas value 0 ( all 3 flags cleared) is > > assigned the meaning "feature not used, use default cache control". > > For each cache behaviour there would of course also be a macro > > expanding to the rigth combination of these flags for enhanced > > useability. > > > > The mmap system call would, if any of these flags are set, decode > > them and get a corresponding PAT_* value, perform the mapping and > > then call into the pmap module to modify the cache attributes for > > every page. > > Have you looked at mem(4) yet? > > Several architectures allow attributes to be associated with > ranges of > physical memory. These attributes can be manipulated via > ioctl() calls > performed on /dev/mem. Declarations and data types are to be > found in > . > > The specific attributes, and number of programmable ranges may > vary > between architectures. The full set of supported attributes is: > > MDF_UNCACHEABLE > The region is not cached. > > MDF_WRITECOMBINE > Writes to the region may be combined or performed out of > order. > > MDF_WRITETHROUGH > Writes to the region are committed synchronously. > > MDF_WRITEBACK > Writes to the region are committed asynchronously. > > MDF_WRITEPROTECT > The region cannot be written to. > > This requires knowledge of the physical addresses, but I believe > that's probably already necessary for what it sounds like you're > trying to accomplish. > > Back in the FreeBSD-3.0 days, I was writing a custom driver for an AGP > graphics controller, and setting the MTRR flags for the exposed buffer > was a definite improvement (200-1200% faster in most cases). This is MTRR, which is what we currently do, when we can. The issue is that often times the BIOS maps ranges in a way that prevents us from using MTRR. This is generally ideal for things like agp and framebuffers when it works, since they have a specific physical range that you want to work with. With PCI(E) cards it isn't as cut and dry... In the ATI and Nouveau cases, we map scatter gather pages into the GART, which generally are allocated using contigmalloc behind the scenes, so it is also possible for it to work in that case. Moving forward, we may actually be mapping random pages into and out of the GART (GEM / TTM). In those cases we really don't have a large contiguous range that we could set MTRR on. Intel CPUs are limited to 8 MTRR registers for the entire system also, so that can become an issue quickly if you are trying to manipulate several areas of memory. With PAT we can manipulate the caching properties on a page level. PAT also allows for some overlap conditions that MTRR won't, such as mapping a page write-combining on top on an UNCACHEABLE MTRR. jhb@ has started some work on this, since I've been badgering him about this recently as well. robert. > -- Kevin > > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" -- Robert Noland FreeBSD signature.asc Description: This is a digitally signed message part
Re: Question about adding flags to mmap system call / NVIDIA amd64 driver implementation
On Apr 28, 2009, at 3:19 PM, Julian Bangert wrote: Hello, I am currently trying to work a bit on the remaining "missing feature" that NVIDIA requires ( http://wiki.freebsd.org/NvidiaFeatureRequests or a back post in this ML) - the improved mmap system call. For now, I am trying to extend the current system call and implementation to add cache control ( the type of memory caching used) . This feature inherently is very architecture specific- but it can lead to enormous performance improvements for memmapped devices ( useful for drivers, etc). I would do this at the user site by adding 3 flags to the mmap system call (MEM_CACHE__ATTR1 to MEM_CACHE__ATTR3 ) which are a single octal digit corresponding to the various caching options ( like Uncacheable,Write Combining, etc... ) with the same numbers as the PAT_* macros from i386/include/ specialreg.h except that the value 0 ( PAT_UNCACHEABLE ) is replaced with value 2 ( undefined), whereas value 0 ( all 3 flags cleared) is assigned the meaning "feature not used, use default cache control". For each cache behaviour there would of course also be a macro expanding to the rigth combination of these flags for enhanced useability. The mmap system call would, if any of these flags are set, decode them and get a corresponding PAT_* value, perform the mapping and then call into the pmap module to modify the cache attributes for every page. Have you looked at mem(4) yet? Several architectures allow attributes to be associated with ranges of physical memory. These attributes can be manipulated via ioctl() calls performed on /dev/mem. Declarations and data types are to be found in . The specific attributes, and number of programmable ranges may vary between architectures. The full set of supported attributes is: MDF_UNCACHEABLE The region is not cached. MDF_WRITECOMBINE Writes to the region may be combined or performed out of order. MDF_WRITETHROUGH Writes to the region are committed synchronously. MDF_WRITEBACK Writes to the region are committed asynchronously. MDF_WRITEPROTECT The region cannot be written to. This requires knowledge of the physical addresses, but I believe that's probably already necessary for what it sounds like you're trying to accomplish. Back in the FreeBSD-3.0 days, I was writing a custom driver for an AGP graphics controller, and setting the MTRR flags for the exposed buffer was a definite improvement (200-1200% faster in most cases). -- Kevin ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SoC 2009: BSD-licensed libiconv in base system
On Tue, Apr 28, 2009, Gabor Kovesdan wrote: > Another idea to consider. Are all of our utilities wchar-clean? What > about library functions? (regex is surely not) Do we lack any important > utility or library? (we still do lack iconv and gettext and what > else...?) What about standards, like C99 wchar functions? Is there > something missing? What about POSIX if it has something related? > Personally, I think that these are more important questions than support > of some extremely rare languages. It's worth to consider how to deal > with them later but the basic problems need a higher priority. There's a fair number of utilities that are not locale-aware, and we don't have any of the *_l() functions in POSIX.1-2008. There are also some second-order issues in libraries, e.g., they assume multibyte encodings don't contain embedded low ASCII characters, or they are far less efficient than they could be. > I hope my work will be useful for the community. Btw, I suspected that > you might be interested in this and I wrote a mail personally to you > when I was looking for a mentor. Then I didn't resend it because > delphij@ volunteered to be my mentor. Have you ever got that mail? If > you find this an interesting project your comments, review, etc. are > still highly welcome. I did reply on 3/19, basically to say that I'd be happy to offer advice, but it's not really my area. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Question about adding flags to mmap system call / NVIDIA amd64 driver implementation
On Tue, Apr 28, 2009 at 22:19, Julian Bangert wrote: > Hello, > > I am currently trying to work a bit on the remaining "missing feature" that > NVIDIA requires ( http://wiki.freebsd.org/NvidiaFeatureRequests or a back > post in this ML) - the improved mmap system call. > For now, I am trying to extend the current system call and implementation > to add cache control ( the type of memory caching used) . This feature > inherently is very architecture specific- but it can lead to enormous > performance improvements for memmapped devices ( useful for drivers, etc). I > would do this at the user site by adding 3 flags to the mmap system call > (MEM_CACHE__ATTR1 to MEM_CACHE__ATTR3 ) which are a single octal digit > corresponding to the various caching options ( like Uncacheable,Write > Combining, etc... ) with the same numbers as the PAT_* macros from > i386/include/specialreg.h except that the value 0 ( PAT_UNCACHEABLE ) is > replaced with value 2 ( undefined), whereas value 0 ( all 3 flags cleared) > is assigned the meaning "feature not used, use default cache control". > For each cache behaviour there would of course also be a macro expanding to > the rigth combination of these flags for enhanced useability. Hmm, I don't like that. What about using something like PAT_WC directly for the userland? Afaik a userland app that uses stuff like this is md anyway. > The mmap system call would, if any of these flags are set, decode them and > get a corresponding PAT_* value, perform the mapping and then call into the > pmap module to modify the cache attributes for every page. > > My first question is if there is a more elegant way of solving that - the 3 > flags would be architecture specific ( they could be used for other things > on other architectures though if need be ) and I do not know the policy on > architecture specific syscall flags, therefore I appreciate any input. > > The second question goes to all those great VM/pmap gurus out there: As far > as I understand, at the moment the pmap_change_attr can only cange the cache > flags for kernel pages. Is there a particular reason why this function might > not be adapted/extended to userspace mappings? If not, I would either add a > new function to iterate over all pages and set cache flags for a particular > region or add a new member (possibly just add the 3 flags again ? ) to the > md part of vm_page_t. Or one could just keep track and return errors as soon > as someone tries to map a memory region ( cache-customized mapping is > usually done to device memory ) already mapped with different cache > behaviour. Do you know how other OS handle this stuff? Maybe there is some inspiration there for a clean interface. I'm not sure if I remember correctly but there is something in my mind that we must take care that no virtual pages have different PAT settings for the same physical page. Maybe I read something like this in the AMD's documentation of PAT. Sorry I don't remember exactly but perhaps someone else can explain it better. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Question about adding flags to mmap system call / NVIDIA amd64 driver implementation
Hello, I am currently trying to work a bit on the remaining "missing feature" that NVIDIA requires ( http://wiki.freebsd.org/NvidiaFeatureRequests or a back post in this ML) - the improved mmap system call. For now, I am trying to extend the current system call and implementation to add cache control ( the type of memory caching used) . This feature inherently is very architecture specific- but it can lead to enormous performance improvements for memmapped devices ( useful for drivers, etc). I would do this at the user site by adding 3 flags to the mmap system call (MEM_CACHE__ATTR1 to MEM_CACHE__ATTR3 ) which are a single octal digit corresponding to the various caching options ( like Uncacheable,Write Combining, etc... ) with the same numbers as the PAT_* macros from i386/include/specialreg.h except that the value 0 ( PAT_UNCACHEABLE ) is replaced with value 2 ( undefined), whereas value 0 ( all 3 flags cleared) is assigned the meaning "feature not used, use default cache control". For each cache behaviour there would of course also be a macro expanding to the rigth combination of these flags for enhanced useability. The mmap system call would, if any of these flags are set, decode them and get a corresponding PAT_* value, perform the mapping and then call into the pmap module to modify the cache attributes for every page. My first question is if there is a more elegant way of solving that - the 3 flags would be architecture specific ( they could be used for other things on other architectures though if need be ) and I do not know the policy on architecture specific syscall flags, therefore I appreciate any input. The second question goes to all those great VM/pmap gurus out there: As far as I understand, at the moment the pmap_change_attr can only cange the cache flags for kernel pages. Is there a particular reason why this function might not be adapted/extended to userspace mappings? If not, I would either add a new function to iterate over all pages and set cache flags for a particular region or add a new member (possibly just add the 3 flags again ? ) to the md part of vm_page_t. Or one could just keep track and return errors as soon as someone tries to map a memory region ( cache-customized mapping is usually done to device memory ) already mapped with different cache behaviour. I thank you for your assistance & happy coding, --Julian Bangert ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SoC 2009: BSD-licensed libiconv in base system
On Tue, Apr 28, 2009 at 03:53:10PM +0200, G?bor K?vesd?n wrote: > >> It's possible that there are little poor countries with an own writing > >> system but probably their writing system is unsupported because the > >> starvation, poorness and lack of water and electricity are more serious > >> problems there. > > > > I wouldn't call all both parts of Korea little poor countries and it is > > a wonderful example for why UCS 4 Level 1 can be problematic. > > > > > I didn't know about Korea. Then I insist even more on that Unicode should > be extended to cover them instead of making implementation-specific > solutions. Unicode covers Korean. It just violates the "one logic character equals one UCS-4 character" or however you want to put. More trivial example can be obtained when looking at both your and my name. Diacrets have historically been part of the character, but it is possible to use combining characters in unicode for the cleaner description. Joerg ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: C99: Suggestions for style(9)
On Tue, 28 Apr 2009, Peter Jeremy wrote: On 2009-Apr-26 09:02:36 +0200, Christoph Mallon wrote: as some of you may have noticed, several years ago a new millenium started and a decade ago there was a new C standard. Your implication that FreeBSD is therefore a decade behind the times is unfair. Whilst the C99 standard was published a decade ago, compilers implementing that standard are still not ubiquitous. HEAD recently switched to C99 as default (actually gnu99, but that's rather close). Note that gcc 4.2 (the FreeBSD base compiler) states that it is not C99 compliant. However, if you take a look at http://gcc.gnu.org/gcc-4.2/c99status.html , you will see that it is very close. The vast majority of C99 features are implemented and working correctly. Even those which are marked as "broken" generally work in most cases, and fail only in rather obscure corner cases that real programs are unlikely to encounter. In particular, the features Christoph proposes to use work fine. -- Nate Eldredge neldre...@math.ucsd.edu ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SoC 2009: BSD-licensed libiconv in base system
>> It's possible that there are little poor countries with an own writing >> system but probably their writing system is unsupported because the >> starvation, poorness and lack of water and electricity are more serious >> problems there. > > I wouldn't call all both parts of Korea little poor countries and it is > a wonderful example for why UCS 4 Level 1 can be problematic. > I didn't know about Korea. Then I insist even more on that Unicode should be extended to cover them instead of making implementation-specific solutions. -- Gábor Kövesdán ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SoC 2009: BSD-licensed libiconv in base system
>> >> My ex-girlfriend is working in Nepal [...] Even this country's encoding >> is >> supported. > > > Probably because Nepali language doesn't have a separate script, they use > Devanagari! :-) I know, what I wanted to say was that even Nepali is covered by the supported scripts. -- Gábor Kövesdán ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: C99: Suggestions for style(9)
I like the part about using as many variables as possible because
of documentation and performance enhancements. I tend to like
the other changes as well..
On Sun, Apr 26, 2009 at 09:02:36AM +0200, Christoph Mallon wrote:
> Hi hackers@,
>
> as some of you may have noticed, several years ago a new millenium
> started and a decade ago there was a new C standard. HEAD recently
> switched to C99 as default (actually gnu99, but that's rather close). So
> it's finally time to re-examine wether style(9) needs to be accomodated.
> The diff with the proposed changes is attached. Below are the changes
> with some further explanations. They are in the order as they appear in
> style(9).
> Maybe using all of these changes is a bit too big change at once, but
> I'd like some of them applied to style(9). So, please consider the
> proposed changes individually and not a as a all-or-nothing package.
>
> >-Do not put declarations
> >-inside blocks unless the routine is unusually complicated.
> >+Prefer declaring loop iterators in the for-statement to limit their scope.
> > .Bd -literal
> >-for (; cnt < 15; cnt++) {
> >+for (int cnt = 0; cnt < 15; cnt++) {
> [...]
> >-When declaring variables in functions declare them sorted by size,
> >-then in alphabetical order; multiple ones per line are okay.
> >+When declaring variables in functions declare them sorted in alphabetical
> >order;
> >+prefer one declaration per line, especially when pointer declarators or
> >+initializations are involved.
> > If a line overflows reuse the type keyword.
> > .Pp
> >-Be careful to not obfuscate the code by initializing variables in
> >-the declarations.
> >-Use this feature only thoughtfully.
> >-DO NOT use function calls in initializers.
> >+Prefer initializing variables right at their declaration.
> >+When the value is not supposed to change in the function, make the
> >variable
> >+const.
> >+This makes it easier for a reader to identify the where the value of a
> >variable
> >+originates.
> >+Do not reuse the same variable in a different context, declare a new
> >variable.
> > .Bd -literal
> >-struct foo one, *two;
> >-double three;
> >-int *four, five;
> >-char *six, seven, eight, nine, ten, eleven, twelve;
> >+struct foo *bar;
> >+struct foo baz = { 42, "qux" };
> >
> >-four = myfunction();
> >+FILE *const f = fopen("name", "r");
> >+if (f == NULL)
> >+err("Failed to open file");
> >+/* We can safely assume that f is not changed anymore, even if the
> >+ * function is a hundred lines long */
>
> This change is to reduce the scope of local variables. For reasons why
> this does not cost anything in terms of stack space, please see the last
> change, which adds explanations about local variables.
> Smaller scopes and distinct variables for distinct contexts make it
> easier for readers of the code to identify the def-use-chains of
> variables, because there are less places with assignments to a variable
> and the scope is smaller. Also, when a variable is initialised right at
> its declaration and is not supposed to change, you can emphasize this by
> making it const.
> I looked at older discussions regarding this topic and identified
> several concerns, which were raised. I'll address them below:
>
> - It does not add anything to the language.
> Well, yes, just like if, do, for, goto, the comma operator, compound
> assignment (+=, ...), ++/-- and many other things. All you need to be
> Turing complete is lambda calculus - there hardly can be less syntax.
> But, like all mentioned constructs, it makes the code more concise.
>
> - It leads to sloppy code. You could reuse another variable or should
> think again whether you really need this variable.
> Reusing variables in different contexts is error prone and bad for
> maintainability. You could reuse a variable, which is not as dead as you
> thought. More local variables cost nothing, especially no stack space,
> see the last proposed changed below. It is good to use more local
> variables, because it gives the reader a name, which carries
> information. Also it makes it easier for a reader (and the compiler!) to
> identify, which expressions are the same. You could repeat
> foo->long_name->even_longer_name[2 * i + 1] five times or just declare a
> local variable and cache the value to make it easier to read.
> It also enables the compiler to produce better warnings: If you declare
> a new variable, it is not initialised and if you do not initialise it on
> all paths before a use, the compiler will warn you. But if you reuse an
> older variable, it is already initialised by its former uses and so you
> get no warning, but operate on garbage values (the old value from the
> previous use).
> So it does not lead to slopyy code, but better code, which is easier to
> comprehend, the compiler can better help you to prevent bugs, and as a
> side effect the compiler can produce better
Re: SoC 2009: BSD-licensed libiconv in base system
On Tue, Apr 28, 2009 at 11:08:49AM +0200, Gabor Kovesdan wrote: > Citrus is based on UCS-4 as an internal encoding, just like the another > BSD-licensed iconv library. This is a barrier to support encodings that > aren't supported by UCS-4. More precisely is that Citrus can and will use UCS-4 as appropiate. It does not enforce it. Which is an important difference. One reason that it is often used is that it helps to avoid exponential growth of the translation tables. It just isn't worth the time to write ISO-8859-1 to ISO-8859-15 (trivial), if the translation to and from UCS4 gives the same result with marginally more work. > It's possible that there are little poor countries with an own writing > system but probably their writing system is unsupported because the > starvation, poorness and lack of water and electricity are more serious > problems there. I wouldn't call all both parts of Korea little poor countries and it is a wonderful example for why UCS 4 Level 1 can be problematic. Joerg ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: IPsec in GENERIC kernel config
On Tue, Apr 28, 2009 at 09:36:26AM +0300, Jan Melen wrote: > Hi, [...] > Just to understand the problem correctly I guess you are talking about > performance hit on outgoing packets as the IPsec tries to find a > security policy even for packets that should not be encrypted? For > incoming traffic I don't see any reason for performance hit. The (more or less) same check is done for incoming packets, because we NEED to ensure that IPsec traffic comes from the appropriate IPsec tunnel, and non IPsec traffic comes without IPsec > Has anyone done any measurements on magnitude of performance loss we get > from trying to match the outgoing packets for non-existent IPsec > policies? I would guess that if you have zero SPD entries in your system > it can't be a lot as it a matter of calling: > ip_ipsec_output -> ipsec4_checkpolicy -> ipsec_getpolicybyaddr/sock -> > key_allocsp which in turn searches through an empty list. We (my company) already tried such a hack, which completely skips IPsec process if we know that SPD (both in and out) is empty. It works, and has the expected impact on performance loss. Yvan. ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SoC 2009: BSD-licensed libiconv in base system
> > My ex-girlfriend is working in Nepal [...] Even this country's encoding is > supported. Probably because Nepali language doesn't have a separate script, they use Devanagari! :-) On Tue, Apr 28, 2009 at 2:38 PM, Gabor Kovesdan wrote: > David Schultz escribió: > >> On Mon, Apr 27, 2009, Joerg Sonnenberger wrote: >> >> >>> On Mon, Apr 27, 2009 at 11:49:41AM -0700, Tim Kientzle wrote: >>> >>> David Schultz wrote: > ... whether it would make more sense to standardize on something like > UCS-4 for the internal representation. > > YES. Without this, wchar_t is useless. >>> I strongly disagree. Everything can be represented as UCS-4 is a bad >>> assumption, but something Americans and Europeans naturally don't have >>> to care about. >>> >>> >> >> ...but isn't this moot at present because there are no >> widely-accepted encodings that include characters that >> aren't supported by UCS-4? Citrus doesn't seem to support >> any such encodings in any case. >> >> > Citrus is based on UCS-4 as an internal encoding, just like the another > BSD-licensed iconv library. This is a barrier to support encodings that > aren't supported by UCS-4. > >> If this ever really becomes an issue, we could always stuff >> locale-dependent encodings into unused UCS-4 code pages. >> However, it doesn't seem worthwhile to deliberately burden >> programmers over concerns that are presently, and for the >> foreseeable future, hypothetical. >> >> > I'm not a Unicode expert, but isn't the reason of periodical standard > reviews and changes to cover more and more human languages? We could just > support the latest Unicode standard and let the Unicode workgroups map those > new characters into unused code points. The Latin-based, Cyrillic, > Devanagari and CJK encodings are well-supported, I think. I don't know too > much about CJK encondings, though, if the thousands of ideographs are all > supported or not. But I'd say the most significant languages that are used > on the Internet are supported, the rest might have another problems... > > [OFF] > It's possible that there are little poor countries with an own writing > system but probably their writing system is unsupported because the > starvation, poorness and lack of water and electricity are more serious > problems there. My ex-girlfriend is working in Nepal in a cooperation > program (it's kinda scholarship) and she told me that they only have > electricity in 8 hours a day, 4 during the night and 4 during the day. There > are no sidewalks for pedestrians, they go along with the cars on the street > and the pollution is extremely high. Even this country's encoding is > supported. What I am trying to say is that countries with unsupported > languages probably won't really care about character encodings if they > rarely have computers... I can just hope that their living conditions will > get better and their language will be supported. I can also hope that the > Unicode people will focus more on these countries instead of fucking up the > time with fictionary languages from fairy tales... [1] > Probably I'll go to visit her in Nepal in January, it will be an > interesting experience. I'll check if I can help the IT world there with > anything. > [ON] > > Another idea to consider. Are all of our utilities wchar-clean? What about > library functions? (regex is surely not) Do we lack any important utility or > library? (we still do lack iconv and gettext and what else...?) What about > standards, like C99 wchar functions? Is there something missing? What about > POSIX if it has something related? Personally, I think that these are more > important questions than support of some extremely rare languages. It's > worth to consider how to deal with them later but the basic problems need a > higher priority. > > > [1] http://en.wikipedia.org/wiki/Tengwar#Unicode > > > Cheers, > > -- > Gabor Kovesdan > FreeBSD Volunteer > > EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org > WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org > > ___ > freebsd-hackers@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: C99: Suggestions for style(9)
On 2009-Apr-26 09:02:36 +0200, Christoph Mallon wrote: >as some of you may have noticed, several years ago a new millenium >started and a decade ago there was a new C standard. Your implication that FreeBSD is therefore a decade behind the times is unfair. Whilst the C99 standard was published a decade ago, compilers implementing that standard are still not ubiquitous. > HEAD recently >switched to C99 as default (actually gnu99, but that's rather close). Note that gcc 4.2 (the FreeBSD base compiler) states that it is not C99 compliant. >Maybe using all of these changes is a bit too big change at once, but >I'd like some of them applied to style(9). So, please consider the >proposed changes individually and not a as a all-or-nothing package. One area you do not address is code consistency. Currently, the FreeBSD kernel (and, to a lesser extent, userland) are mostly style(9) compliant - ie the entire codebase is mostly written in the same style. Whilst you may not like it (and I don't know that anyone completely agrees with everything in style(9)), it does mean that the code is consistent. Changing style(9) in a way that is not consistent with current code means that either existing code must be brought into line with the new standard - which incurs a one-off cost - or the code base becomes split into "old" and "new" style - incurring an ongoing maintenance cost as maintainers switch between styles. Both approaches incur a risk of introducing new bugs. Note that I'm not saying that style(9) can never be changed, I'm just saying that any change _will_ incur a cost and the cost as well as the potential benefits need to be considered. [Reduce declaration scope as far as possible, initialise variables where they are declared, sort by name] Whilst my personal preference is for the style suggested by Christoph (and I generally agree with the points he makes), this is also the change that incurs the biggest stylistic change. This is not a change that can be practically retrofitted to existing code and therefore its implementation would result in a mixture of code styles - increasing the risk of bugs due to confusion as to which style was being used. I am not yet sure whether the benefits outweigh the costs, [Don't parenthesize return values] >Removed, because it does not improve maintainability in any way. This change could be made and tested mechanically. But there is no justification for making it - stating that the existing rule is no better than the proposed rule is no reason to change. [No K&R declarations] >Removed, because there is no need to use them anymore. Whilst this is a change that could be performed mechanically, there are some gotchas relating to type promotion (as you point out). The kernel already contains a mixture of ANSI & K&R declarations and style(9) recommends using ANSI. IMHO, there is no need to make this change until all K&R code is removed from FreeBSD. [ Don't insert an empty line if the function has no local variables.] This change could be made and tested mechanically. IMHO, this change has neglible risk and could be safely implemented. >> +.Sh LOCAL VARIABLES >Last, but definitely not least, I added this paragraph about the use of >local variables. This is to clarify, how today's compilers handle >unaliased local variables. Every version of gcc that FreeBSD has ever used would do this for primitive types when optimisation was enabled. This approach can become expensive in stack (and this is an issue for kernel code) when using non-primitive types or when optimisation is not enabled (though I'm not sure if this is supported). Assuming that gcc (and icc and clang) behaves as stated in all supported optimisation modes, this change would appear to be quite safe to make. -- Peter Jeremy pgp4M8f6oRjRW.pgp Description: PGP signature
Re: IPsec in GENERIC kernel config
Hi, Bjoern A. Zeeb wrote: On Mon, 27 Apr 2009, Sam Leffler wrote: Hi, Jan Melen wrote: Hi, Again when I compiled a custom kernel just to enable IPsec in the FreeBSD kernel it came to my mind why is it so that the IPsec is not enabled by default in the GENERIC kernel configuration file? At least for me the GENERIC kernel configuration would do just fine if the IPsec would be enabled in it by default. Now I have to build a custom kernel just for IPsec btw IPsec is even mandatory for a host supporting IPv6. IPsec incurs a performance hit. Fix that and it can be enabled in GENERIC. There is even a PR for this: http://www.freebsd.org/cgi/query-pr.cgi?pr=conf/128030 Just to understand the problem correctly I guess you are talking about performance hit on outgoing packets as the IPsec tries to find a security policy even for packets that should not be encrypted? For incoming traffic I don't see any reason for performance hit. Has anyone done any measurements on magnitude of performance loss we get from trying to match the outgoing packets for non-existent IPsec policies? I would guess that if you have zero SPD entries in your system it can't be a lot as it a matter of calling: ip_ipsec_output -> ipsec4_checkpolicy -> ipsec_getpolicybyaddr/sock -> key_allocsp which in turn searches through an empty list. Jan ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Improving geom_mirror(4)'s read balancing
Ivan Voras wrote: Maxim Sobolev wrote: The patch is available here: http://sobomax.sippysoft.com/~sobomax/geom_mirror.diff. I would like to get input on the functionality/code itself, as well on what is the best way to add this functionality. Right now, it's part of the round-robin balancing code. Technically, it could be added as a separate new balancing method, but for the reasons outlined above I really doubt having "pure" round-robin has any practical value now. The only case where previous behavior might be beneficial is with solid-state/RAM disks where there is virtually no seek time, so that by reading close sectors from two separate disks one could actually get a better speed. At the very least, the new method should become default, while "old round-robin" be another option with clearly documented shortcomings. I would really like to hear what people think about that. Have you perhaps seen this: http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/113885 I'm using the patch in the PR and it helps a bit, similar to what you have seen. Pawel is silent about the issue so I guess it can also be taken as silent approval :) Oh, great! I am curious as to if there is any background behind "distance to use delay" metric? To me it seems the current number of outstanding requests is much more important when selecting between disk X and disk Y. I am not a storage expert, so that I could be wrong though. One way or another the load-balancing has be improved and the new more intelligent scheduling IMHO should be the default one. -Maxim ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SoC 2009: BSD-licensed libiconv in base system
On Mon, Apr 27, 2009 at 10:42:46PM -0400, David Schultz wrote: > On Mon, Apr 27, 2009, Joerg Sonnenberger wrote: > > On Mon, Apr 27, 2009 at 03:49:04PM -0400, David Schultz wrote: > > > ...but isn't this moot at present because there are no > > > widely-accepted encodings that include characters that > > > aren't supported by UCS-4? Citrus doesn't seem to support > > > any such encodings in any case. > > > > "Just" using UCS-4 not necessarily buys you the desired affect. > > Keep in mind that UCS-4 is still a variable width encoding, as soon as > > you factor combining characters and some other interesting parts in. > > This is true, but unfortunately C99 wasn't really designed to > support combining characters. I don't understand how this relates > to the present discussion. The main reason people want to use wchar_t is because they want to use a fixed width presentation of characters. Just using UCS-4 doesn't give you that if you ever want to support Level 2 and higher. It also highlights UCS-4 is not that state independent as it is often thought to be. That is again something undesirable. Joerg ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SoC 2009: BSD-licensed libiconv in base system
David Schultz escribió: On Mon, Apr 27, 2009, Joerg Sonnenberger wrote: On Mon, Apr 27, 2009 at 11:49:41AM -0700, Tim Kientzle wrote: David Schultz wrote: ... whether it would make more sense to standardize on something like UCS-4 for the internal representation. YES. Without this, wchar_t is useless. I strongly disagree. Everything can be represented as UCS-4 is a bad assumption, but something Americans and Europeans naturally don't have to care about. ...but isn't this moot at present because there are no widely-accepted encodings that include characters that aren't supported by UCS-4? Citrus doesn't seem to support any such encodings in any case. Citrus is based on UCS-4 as an internal encoding, just like the another BSD-licensed iconv library. This is a barrier to support encodings that aren't supported by UCS-4. If this ever really becomes an issue, we could always stuff locale-dependent encodings into unused UCS-4 code pages. However, it doesn't seem worthwhile to deliberately burden programmers over concerns that are presently, and for the foreseeable future, hypothetical. I'm not a Unicode expert, but isn't the reason of periodical standard reviews and changes to cover more and more human languages? We could just support the latest Unicode standard and let the Unicode workgroups map those new characters into unused code points. The Latin-based, Cyrillic, Devanagari and CJK encodings are well-supported, I think. I don't know too much about CJK encondings, though, if the thousands of ideographs are all supported or not. But I'd say the most significant languages that are used on the Internet are supported, the rest might have another problems... [OFF] It's possible that there are little poor countries with an own writing system but probably their writing system is unsupported because the starvation, poorness and lack of water and electricity are more serious problems there. My ex-girlfriend is working in Nepal in a cooperation program (it's kinda scholarship) and she told me that they only have electricity in 8 hours a day, 4 during the night and 4 during the day. There are no sidewalks for pedestrians, they go along with the cars on the street and the pollution is extremely high. Even this country's encoding is supported. What I am trying to say is that countries with unsupported languages probably won't really care about character encodings if they rarely have computers... I can just hope that their living conditions will get better and their language will be supported. I can also hope that the Unicode people will focus more on these countries instead of fucking up the time with fictionary languages from fairy tales... [1] Probably I'll go to visit her in Nepal in January, it will be an interesting experience. I'll check if I can help the IT world there with anything. [ON] Another idea to consider. Are all of our utilities wchar-clean? What about library functions? (regex is surely not) Do we lack any important utility or library? (we still do lack iconv and gettext and what else...?) What about standards, like C99 wchar functions? Is there something missing? What about POSIX if it has something related? Personally, I think that these are more important questions than support of some extremely rare languages. It's worth to consider how to deal with them later but the basic problems need a higher priority. [1] http://en.wikipedia.org/wiki/Tengwar#Unicode Cheers, -- Gabor Kovesdan FreeBSD Volunteer EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: SoC 2009: BSD-licensed libiconv in base system
David Schultz escribió: On Thu, Apr 23, 2009, Gábor Kövesdán wrote: Hello all, my name is Gábor Kövesdán. I'm a Hungarian student and I'll be working on a BSD-licensed libiconv implementation for FreeBSD during this year's Summer of Code program. It'll be based on NetBSD's Citrus iconv but there is a lot to do besides porting. My mentor is Xin Li. Nice. I'm sure many people will thank you for this. I hope my work will be useful for the community. Btw, I suspected that you might be interested in this and I wrote a mail personally to you when I was looking for a mentor. Then I didn't resend it because delphij@ volunteered to be my mentor. Have you ever got that mail? If you find this an interesting project your comments, review, etc. are still highly welcome. One complaint I've heard about both our wide character implementation and Citrus iconv is that the internal (wchar_t) encoding depends on the current locale. (Basically it uses a packed binary version of whatever the external representation is.) The relevant standards allow this, but it can be a pain for application and library writers. One thing to think about is whether it would make more sense to standardize on something like UCS-4 for the internal representation. I haven't got to such details yet, so I didn't know this. But UCS-4 seems to be reasonable for me. Cheers, -- Gabor Kovesdan FreeBSD Volunteer EMAIL: ga...@freebsd.org .:|:. ga...@kovesdan.org WEB: http://people.FreeBSD.org/~gabor .:|:. http://kovesdan.org ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
