Re: tr A-Z a-z in locales other than C
Jilles Tjoelker wrote: > On Tue, Jun 07, 2011 at 04:24:43AM +0400, Andrey Chernov wrote: ... > > Back to the ports: it is not hard to run _any_ port's make > > or configure with LANG=C directly by the ports Mk system to > > eliminate that problem. > > True, but some ports install scripts with problematic tr calls. So part of the porting effort may be to provide a patch that prepends something along the lines of "env LANG=C" to tr calls in those scripts. It would surely not be the only kind of situation in which a port needed to patch the ported code to get it to run correctly on FreeBSD :) ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: tr A-Z a-z in locales other than C
On Tue, Jun 07, 2011 at 11:17:12PM +0200, Jilles Tjoelker wrote: > In FreeBSD, upper case sorts before lower case, so cases can be > distinguished this way but all letters may require two ranges. In most > other operating systems the cases go together so a single range is > sufficient, but cases cannot be distinguished. Making such things work > on multiple operating systems requires careful testing. Such thing can't work consistenly on multiple operating systems by definition, because POSIX states "undefined" here. So the best we can is to concentrace on our system. No program should relay on that until POSIX define that somehow. > > Moreover, having differently treated regex ranges in tr vs other places > > you mention will produce additional chaos. > > I think this is already inconsistent because some programs do not enable > locale or use different locale code. I say the same, producing additional chaos is not bringing chaos from nowhere. AFAIK nobody use different locale code but often different regex implemetation. > > Back to the ports: it is not hard to run _any_ port's make or configure > > with LANG=C directly by the ports Mk system to eliminate that problem. > > True, but some ports install scripts with problematic tr calls. What count says, how many ports do that? Summarizing I suggest to consider two models: 1) Developer/programer etc. tr coderange does good for it. 2) Working with national language docs/end user/ tr coderange does bad for it. Sacrificing model 2) for 1) is not the thing we need, if such ports number is low. If such ports number is significant, we can consider additional options like automatically search and replace such tr's through pkg-plist (similar scanning we already do for security reasons). -- http://ache.vniz.net/ ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: tr A-Z a-z in locales other than C
On Wed, 8 Jun 2011, Jilles Tjoelker wrote: maybe another option would be modifying tr to recognize other [new] environment variables... TR_LANG, TR_LC_ALL, TR_LC_CTYPE and TR_LC_COLLATE. done that way, things could be set in /etc/make.conf (or sys.mk), not need any patching, and not interfere with other uses of locale. That would be rather ugly. If tr a-z A-Z is supposed to be deceiving in some locales, then let it remain so unconditionally. = it can still be as ugly as one wants it to be, and in some ports that might be fine. but this option would provide a very simple option to reign in how ugly it is. -- ...atom http://atom.smasher.org/ 762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808 - "The livestock sector is a major player [in climate change], responsible for 18% of greenhouse gas emissions measured in CO2 equivalent. This is a higher share than transport." -- Livestock's long shadow, 2006 UN report sponsored by WTO, EU, AS-AID, FAO, et al ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Testing a change to printf(9)
On 6/7/11 6:33 PM, Dieter BSD wrote: I've been working on fixing problems with printf(9), log(9) and related functions. Today I tried converting printf(9) to write to the log rather than directly to the console, unless the log is not open, in which case the message is also sent to the console. Printf(...) is now the same as log(LOG_INFO, ...). oh please no! from my perspective, I want my printf output to go to the console. immediately, regardless of where it goes after that. I don't care if there is or is not a log. I do NOT want to EVER have the problem I've had on linux where the last vital bit of output never made it out before the system stopped. once it's been shown on the console I don't care what happens to it.. Printfs to the console cause data loss. Easily repeatable. Absolutely unacceptable. You are welcome to fix the actual underlying problem. I would love to see the underlying problem fixed! I've asked a few times before, but I'll ask again. Why does a driver for one piece of hardware have to block interrupts for all hardware? I thought changing from spl to mutex was supposed to fix this? My changes do not prevent a sysadmin from having printf output go to the console, it gives them a choice. Currently there is no choice. NO! a kernel printf goes to the console however it may be redirected. It MAY also decide to go somewhere else. But not if it means unreliable delivery to the serial port. The console MUST get the output in a completely reliable manner unless it's been disabled. do not do anything that gets between the console and the problem. if you want to send it on to some other secondary sevice such s a log, then disable the console and take the priority service but do NOT do any of the silly tricks that some people have tried in the past like making the console just one of several things on a mux. all with equal ability to screw up the other. I want the console to be the last refuge in a dying system. if a send a char there I KNOW it's gone out. it is synchronous and it doesn't lie. if you don't want a reliable console then turn it off but don't make something else that is unreliable and call it the console. call it anything else but don't screw up the console. I commented out the line in /etc/syslog.conf that sends some log messages to the console. In multiuser mode, normal printfs go to log, but not the console, as expected. Bootup messages, shutdown messages, and panic messages all show up on the console as expected. Are there any other special cases to test? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Testing a change to printf(9)
>> I've been working on fixing problems with printf(9), log(9) and >> related functions. Today I tried converting printf(9) to write >> to the log rather than directly to the console, unless the log is >> not open, in which case the message is also sent to the console. >> Printf(...) is now the same as log(LOG_INFO, ...). > oh please no! > > from my perspective, I want my printf output to go to the console. > immediately, regardless of where it goes after that. > I don't care if there is or is not a log. > I do NOT want to EVER have the problem I've had on linux where > the last vital bit of output never made it out before the system stopped. > > once it's been shown on the console I don't care what happens to it.. Printfs to the console cause data loss. Easily repeatable. Absolutely unacceptable. You are welcome to fix the actual underlying problem. I would love to see the underlying problem fixed! I've asked a few times before, but I'll ask again. Why does a driver for one piece of hardware have to block interrupts for all hardware? I thought changing from spl to mutex was supposed to fix this? My changes do not prevent a sysadmin from having printf output go to the console, it gives them a choice. Currently there is no choice. >> I commented out the line in /etc/syslog.conf that sends >> some log messages to the console. In multiuser mode, >> normal printfs go to log, but not the console, as expected. >> >> Bootup messages, shutdown messages, and panic messages all >> show up on the console as expected. >> >> Are there any other special cases to test? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: sizeof(function pointer)
On 01/06/2011 00:07, m...@freebsd.org wrote: > I am looking into potentially MFC'ing r212367 and related, that adds > drains to sbufs. The reason for MFC is that several pieces of new > code in CURRENT are using the drain functionality and it would make > MFCing those changes much easier. > > The problem is that r212367 added a pointer to a drain function in the > sbuf (it replaced a pointer to void). The C standard doesn't > guarantee that a void * and a function pointer have the same size, > though its true on amd64, i386 and I believe PPC. What I'm wondering > is, though not guaranteed by the standard, is it *practically* true > that sizeof(void *) == sizeof(int(*)(void)), such that an MFC won't > break binary compatibility for any supported architecture? (The > standard does guarantee, though not in words, that all function > pointers have the same size, since it guarantees that pointers to > functions can be cast to other pointers to functions and back without > changing the value). > > Another possibility is to malloc a blob that is sizeof(int(*)(void)) > and store that in a renamed s_unused; this is a bit messier but > guaranteed to work. I'd just rather the code be an MCF instead of a > partial re-write. You could add a static-assert to check for this at compile-time: http://stackoverflow.com/questions/3385515/static-assert-in-c e.g. something like: STATIC_ASSERT(sizeof(void *) == sizeof(int(*)(void)),ptr_to_func_expected_to_be_same_size_as_data_ptr); I've only used static-assert-like constructs in C++, so don't know how well they behave in C... -- Sent from my ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Check for 0 ino_t in readdir(3)
mdf wrote:
> There is a check in the function implementing readdir(3) for a zero
> inode number:
>
> struct dirent *
> _readdir_unlocked(dirp, skip)
> DIR *dirp;
> int skip;
> {
> /* ... */
> if (dp->d_ino == 0 && skip)
> continue;
> /* ... */
> }
>
> "skip" is 1 except for when coming from _seekdir(3).
>
> I don't recall any requirement that a filesystem not use an inode
> numbered 0, though for obvious reasons it's a poor choice for a file's
> inode. So... is this code in libc incorrect? Or is there
> documentation that 0 cannot be a valid inode number for a filesystem?
>
Well, my recollection (if I'm incorrect, please correct me:-) is that, for
real BSD directories (the ones generated by UFS/FFS, which everything else
is expected to emulate), the d_ino field is set to 0 when the first entry
in a directory block is unlink'd. This is because directory entries are not
permitted to straddle blocks, so the first entry can not be subsumed by the
last dirent in the previous block.
In other words, when d_ino == 0, the dirent is free.
rick
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: tr A-Z a-z in locales other than C
On Wed, Jun 08, 2011 at 09:56:39AM +1200, Atom Smasher wrote: > the man page makes it clear... > Translate the contents of file1 to upper-case. > tr "[:lower:]" "[:upper:]" < file1 > (This should be preferred over the traditional UNIX idiom of ``tr a-z > A-Z'', since it works correctly in all locales.) > for any other uses, either build the port with locale specified as "C" as > mentioned, or patch the port so: > tr '[a-z]' '[A-Z]' > becomes: > env LC_ALL=C tr '[a-z]' '[A-Z]' > the only change that would be appropriate to the tr utility would be a > command-line option to select a locale... something like: > tr -l C '[a-z]' '[A-Z]' > i don't think anyone would object to that, but it would still require > patching some ports under some locales... That new option would provide zero benefit. If things are going to be patched anyway then patch them to be standards compliant. > maybe another option would be modifying tr to recognize other [new] > environment variables... TR_LANG, TR_LC_ALL, TR_LC_CTYPE and > TR_LC_COLLATE. done that way, things could be set in /etc/make.conf (or > sys.mk), not need any patching, and not interfere with other uses of > locale. That would be rather ugly. If tr a-z A-Z is supposed to be deceiving in some locales, then let it remain so unconditionally. -- Jilles Tjoelker ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: tr A-Z a-z in locales other than C
the man page makes it clear... Translate the contents of file1 to upper-case. tr "[:lower:]" "[:upper:]" < file1 (This should be preferred over the traditional UNIX idiom of ``tr a-z A-Z'', since it works correctly in all locales.) for any other uses, either build the port with locale specified as "C" as mentioned, or patch the port so: tr '[a-z]' '[A-Z]' becomes: env LC_ALL=C tr '[a-z]' '[A-Z]' the only change that would be appropriate to the tr utility would be a command-line option to select a locale... something like: tr -l C '[a-z]' '[A-Z]' i don't think anyone would object to that, but it would still require patching some ports under some locales... maybe another option would be modifying tr to recognize other [new] environment variables... TR_LANG, TR_LC_ALL, TR_LC_CTYPE and TR_LC_COLLATE. done that way, things could be set in /etc/make.conf (or sys.mk), not need any patching, and not interfere with other uses of locale. -- ...atom http://atom.smasher.org/ 762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808 - "We in the West must bear in mind that the poor countries are poor primarily because we have exploited them through political or economic colonialism." -- Martin Luther King, Jr ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: Testing a change to printf(9)
On 6/7/11 12:57 PM, Dieter BSD wrote: I've been working on fixing problems with printf(9), log(9) and related functions. Today I tried converting printf(9) to write to the log rather than directly to the console, unless the log is not open, in which case the message is also sent to the console. Printf(...) is now the same as log(LOG_INFO, ...). oh please no! from my perspective, I want my printf output to go to the console. immediately, regardless of where it goes after that. I don't care if there is or is not a log. I do NOT want to EVER have the problem I've had on linux where the last vital bit of output never made it out before the system stopped. once it's been shown on the console I don't care what happens to it.. I commented out the line in /etc/syslog.conf that sends some log messages to the console. In multiuser mode, normal printfs go to log, but not the console, as expected. Bootup messages, shutdown messages, and panic messages all show up on the console as expected. Are there any other special cases to test? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org" ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: tr A-Z a-z in locales other than C
On Tue, Jun 07, 2011 at 04:24:43AM +0400, Andrey Chernov wrote: > On Tue, Jun 07, 2011 at 12:41:05AM +0200, Jilles Tjoelker wrote: > > There is a related issue with ranges in regular expressions, glob and > > fnmatch (likewise unspecified by POSIX outside the POSIX locale), but > > this is less likely to cause problems. > You care about ports, but suggested change is americano-centrism which > kills tr usage for national language documents due to impossibility to > specify whole national alphabet easily, just by two letters. Hmm, so that's with translation to a constant, or with the -d and/or -s options. In such cases, there may be a range for all letters with collation order, but not with codeset order (mainly if "all letters" includes letters with diacritical marks). In FreeBSD, upper case sorts before lower case, so cases can be distinguished this way but all letters may require two ranges. In most other operating systems the cases go together so a single range is sufficient, but cases cannot be distinguished. Making such things work on multiple operating systems requires careful testing. > Moreover, having differently treated regex ranges in tr vs other places > you mention will produce additional chaos. I think this is already inconsistent because some programs do not enable locale or use different locale code. With UTF-8 or other multibyte character sets, this is even more so because functions like isalpha work very poorly by definition and there is no collation support for such character sets in FreeBSD. > Back to the ports: it is not hard to run _any_ port's make or configure > with LANG=C directly by the ports Mk system to eliminate that problem. True, but some ports install scripts with problematic tr calls. -- Jilles Tjoelker ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Testing a change to printf(9)
I've been working on fixing problems with printf(9), log(9) and related functions. Today I tried converting printf(9) to write to the log rather than directly to the console, unless the log is not open, in which case the message is also sent to the console. Printf(...) is now the same as log(LOG_INFO, ...). I commented out the line in /etc/syslog.conf that sends some log messages to the console. In multiuser mode, normal printfs go to log, but not the console, as expected. Bootup messages, shutdown messages, and panic messages all show up on the console as expected. Are there any other special cases to test? ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Check for 0 ino_t in readdir(3)
There is a check in the function implementing readdir(3) for a zero
inode number:
struct dirent *
_readdir_unlocked(dirp, skip)
DIR *dirp;
int skip;
{
/* ... */
if (dp->d_ino == 0 && skip)
continue;
/* ... */
}
"skip" is 1 except for when coming from _seekdir(3).
I don't recall any requirement that a filesystem not use an inode
numbered 0, though for obvious reasons it's a poor choice for a file's
inode. So... is this code in libc incorrect? Or is there
documentation that 0 cannot be a valid inode number for a filesystem?
If 0 cannot be a valid inode number, my GSoC mentee Gleb suggested
d_fileno == 0 be skipped in dirent_exists() as used by
vop_stdvptocnp(9), and also skipped by unionfs.
Thanks,
matthew
___
freebsd-hackers@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-hackers
To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
Re: FreeBSD I/OAT (QuickData now?) driver
All 10GigE NICs and some newer 10 GigE NICs have multiple hardware queues with a separate MSI-x vector per queue, where each vector is directed to a different CPU. The current operating model is to have a separate interrupt thread per vector. This obviously gets bogged down if one has multiple cards as the interrupt threads end up requiring the scheduler to distribute work fairly between cards as multiple threads will end up running on the same CPUs. Nokia had a reasonable interface for coping with this that was reminiscent of NAPI whereby cooperative sharing between interfaces was provided by having a single taskqueue thread per-core and the cards would queue tasks (which would be re-queued if more than a certain amount of work were required) as interrupts were delivered. There has been talk off and on of porting this "net_task" interface to freebsd. None of this addresses PF_RING's facility for pushing packets in to userland - but presumably Rizzo's netmap work addresses those in need of that sufficiently. Cheers, Kip On Tue, Jun 7, 2011 at 4:13 AM, grarpamp wrote: > Is this work part of what's needed to enable the FreeBSD > equivalent of TNAPI? > > I know we've got polling. And probably MSI-X in a couple drivers. > Pretty sure there is still one CPU doing the interrupt work? > And none of the multiple queue thread spreading tech exists? > > http://www.ntop.org/blog > http://www.ntop.org/TNAPI.html > TNAPI attempts to solve the following problems: > * Distribute the traffic across cores (i.e. the more core the more > scalable is your networking application) for improving scalability. > * Poll packets simultaneously from each RX queue (contraty to > sequential NAPI polling) for fetching packets as fast as possible > hence improve performance. > * Through PF_RING, expose the RX queues to the userland so that > the application can spawn one thread per queue hence avoid using > semaphores at all. > TNAPI achieves all this by starting one thread per RX queue. Received > packets are then pushed to PF_RING (if available) or through the > standard Linux stack. However in order to fully exploit this > technology it is necessary to use PF_RING as it provides a straight > packet path from kernel to userland. Furthermore it allows to create a > virtual ethernet card per RX queue. > ___ > freebsd-...@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org" > ___ freebsd-hackers@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-hackers To unsubscribe, send any mail to "freebsd-hackers-unsubscr...@freebsd.org"
