Travis Vitek wrote:
[...]
I think it might be simpler to keep things abstract but given my
specification above a simple query string would look like this:
"en_US.* 1\n"
"zh_*.UTF-8 2\n"
"zh_*.UTF-8 3\n"
"zh_*.UTF-8 4\n"
for the equivalent of:
locale == "en_US.*" && MB_CUR_MAX == 1
|| locale == "zh_*.UTF-8" && MB_CUR_MAX == 2
|| locale == "zh_*.UTF-8" && MB_CUR_MAX == 3
|| locale == "zh_*.UTF-8" && MB_CUR_MAX == 4
I'm totally confused. If we're going to write out each of the expansions,
then why did I take the time to implement brace expansion?
I thought the idea was to allow us to select locales using a brace expanded
query string. If we are explicitly writing out each of the names, then we
wasted a bunch of time writing brace expansion code.
Yes, the idea was/is to use brace expansion to provide alternation.
I'm not dismissing the idea, just observing that it doesn't neatly
fit into the grammar I outlined above in the absence of anything
else. Can you show your grammar and explain how it works so that
I can better understand what we're discussing?
I'm not sure how you could use brace expressions here. Maybe it
should be the other way around (<SP> should be OR and <NL> AND).
But then the grep -e idea is out the window.
Well, if we're going down the road of rewriting this _again_
Usually design precedes implementation. If you opt to implement first
you need to be prepared to make changes based on flaws in your (or
anyone else's) ideas that you use to ground your implementation in.
then how
about using something like '&&' and '||', or even 'and' and 'or' for the
logical operations and then '(' and ')' for grouping? Almost like the
'equivalent of' that you wrote above. Something that is readable by a
C/C++ programmer or the average guy off of the street?
We discussed and rejected this syntax.
The truth is that not every guy knows grep,
Anyone who works with this aspect of stdcxx will need to know it.
and I'm sure that those who
do wouldn't expect to see a grammar that used '\n' and ' ' to represent
logical operations.
I agree. I will welcome a better alternative that's in line with
the "grep" spirit of the solution we agreed on (so that it can
be made use of in the Expected Failures project).
Or maybe we need
a way to denote/group terms. Then we might be able to say:
"en_US.* 1\n"
"zh_*.UTF-8 ({2..4})"
expand it to
"en_US.* 1\n"
"zh_*.UTF-8 (2 3 4)"
and "know" that the spaces in "2 3 4" denote OR even though the
space in "zh_*.UTF-8 (" denotes an AND. Ugh. I'm not sure I lik
this all that much better.
const char* locales =
rw_locale_query ("en_US.* 1\nzh_*.UTF-8 {2..4}", 10);
I don't see why that would be necessary. You can do it with the
following query using normal brace expansion, and it's human readable.
const char* locales =
rw_locale_query ("{en_US.* 1,zh_*.UTF-8 {2..4}}", 10);
What's "{en_US.* 1,zh_*.UTF-8 {2..4}}" supposed to expand to?
Bash 3.2 doesn't expand it. I suppose it could be
"en_US.* 1 zh_*.UTF-8 2 3 4" or
"en_US.* 1 zh_*.UTF-8 2 zh_*.UTF-8 3 zh_*.UTF-8 4"
I believe that this is _exactly_ what you suggested in our meetings when
I was in Boulder the last time. Maybe I'm just confused, but I am pretty
sure that was what was presented.
The whiteboard has been wiped clean so unless you wrote it down
we'll never know because I myself don't remember anymore. But if
you re-read the "long hanging fruit" thread I also suggested that
for example this string:
"*_{JP,CN}.* {3,4}"
expand into
"*_JP.* 4\n*_CN.* 4\n*_JP.* 3\n*_CN.* 3\n"
and probably a whole number of other things before then. The point
is that it was just a suggestion and not a fully baked design, and
like many other initial ideas it might have been flawed in more
than one way. Which is why we need to come up with a design spec
first, review it, and implement only if it makes sense to everyone
and if it deals with the use cases we're interested in dealing with.
The shell does whitespace collapse and tokenization before it does the
expansion. To use whitespace in a brace expansion in the shell you have
to escape it.
So the following expansion should work just fine in csh...
{en_US.*\ 1,zh_*.UTF-8\ {2..4}}
It should expand to
en_US.* 1
zh_*.UTF-8 2
zh_*.UTF-8 3
zh_*.UTF-8 4
With newlines at the ends or without? I assume without, and that's
precisely why I'm not happy with it. Because the string above really
is
"en_US.* 1 zh_*.UTF-8 2 zh_*.UTF-8 3 zh_*.UTF-8 4"
where each odd space means AND and every even space means OR. That
seems very unintuitive to me.
Remember that I originally provided rw_brace_expand() that doesn't do all
of that. It treats whitespace like any other character. Of course if you
insist on 100% compatibility with shell brace expansion, then feel free to
escape the spaces. Personally I prefer strings without escapes.
So the explicit spaces aren't special but the ones that result from
brace expansion are? (This is an honest question.)
Either way I think I'm getting confused by the lack of distinction
between what's OR and what's AND.
I give an example above of how a brace expansion already solves the
problem.
If the brace expansion routine I've written returns a null terminated
buffer of null terminated strings that are the brace expansions and we
have a function for doing primitive string matching [rw_fnmatch], then
this is a pretty simple problem to solve.
This is exactly what you are doing with the xfail.txt thing. The platform
string is just a brace expansion and grep-like expression...
aix-5.3-*-vacpp-9.0-{12,15}?
Why can't ours be seperated by spaces, or some other character? Is it so
different?
You mean spaces instead of the dashes above? Because brace expansion
also produces spaces and because brace expansion happens before any
other processing. How would the "other processing" distinguish our
spaces from those produced from brace expansion?
Maybe we should ask the question the other way: why can't we use
dashes (or some other non-delimiting characters) in the locale spec
instead of spaces? Would that make me remove my objection? (I think
it might but I'd want to think about it some more).
I suppose the big difference is that the format above is rigid and well
defined, whereas the locale match format is still in flux.
I know that the '\n' is how you'd use `grep -e', but does it really make
sense? We aren't using `grep -e' here.
I'm trying to model the interface on something we all know how
to use. grep -e seemed the closest example of an known interface
that would let us do what we want that I could think of.
Maybe it would help to go back to the basics and try to approach
this by analyzing the problem first. Would putting this together
be helpful?
That depends on how you define helpful. It will not be helpful in getting
this task done in reasonable time.
That depends on how you define reasonable ;-) I had hoped we'd be
done in two to four weeks when we started.
It may be helpful in convincing me to
reimplement this functionality for a third time.
1. the set of locale attributes we want to keep track of
in our locale "database"
What details are necessary to reduce the number of locales tested? The
honest answer to this is _none_. We could just pick N random locales and
run the test with them. That would satisfy the original issue of testing
to many locales.
But it would, in some cases, most likely compromise the effectiveness
of the tests, and if truly random, make failures hard to reproduce.
That idea has been discarded, so the next best thing to do is to have it
include a list of installed locales, and the language, territory and
codeset canonical names as well as the MB_CUR_LEN value for each. Those
are the only criteria that we currently use for selecting locales in the
tests.
Sounds reasonable. I see you added to the wiki. Great! What about
the OS each native name goes with? (e.g., utf8 on HP-UX vs UTF-8
on Linux). And locale aliases (e.g., en -> english -> en_US ->
en_US.ISO8859-1 on Linux -- I made up the names but the concept
is real)?
I don't see anything else useful. If there is some detail that is useful,
most likely we could check it by loading the locale and getting the data
directly instead of caching that data ourselves.
2. one or more possible formats of the database
Because of all of the differences between similarly named locales on
different systems, I don't think it makes sense to keep the locale
data in revision control. It should probably be generated at runtime
and flushed to a file for reuse by later tests.
Maybe. That to me seems like an implementation detail, but...
Given that, I don't feel that the format of the data is significant. It
might be nice for it to be human readable, but that is about it.
...this seems relevant because it will dictate the structure of
the query. Unless you want to do a whole lot of massaging of the
query before you apply it to the file. And if you take the
prototype suggestion I made -- to implement it first using grep
or some such standard tool -- it will matter a whole lot. Also
if you want to get optimum performance out of your code and
minimize its complexity the format will matter. Structure it
too much and you'll end up writing a database API to parse it.
3. the kinds of queries done in our locale tests, and the
ones we expect to do in future tests
This is the important question. As mentioned above, the only thing that
I see being used is selecting locales by name by MB_CUR_LEN.
The codecvt or ctype tests do that. The numeric, monetary, and
time tests will probably be interested in the language (e.g.,
find any German locale so that I use German days of months to
test time_put). I'm saying we should try to come up with the
*actual* queries for these tests to see if what we're doing
here will be helpful there.
With that, we can create a prototype solution using an existing
query language of our choice (such as grep). Once that works,
the grammar should naturally fall out and we can reimplement
the prototype in the test driver.
Isn't that what you did while I was in Boulder? That is how we arrived
at this system of brace expansion and name matching that we are talking
about now.
By prototype I mean a solution that actually works. It could be
written in shell or Python and work on just a single platform,
and with limitations, but it proves the concept. What we did
when you visited in Boulder was sketch out a rough idea on the
whiteboard.
Your prototype boils down to something like this, where the fictional
'my_locale' utility lists the names of all installed locales followed
by a seperator and then the MB_CUR_LEN value.
for i in `echo $brace_expr`;
do
my_locale -a | grep -e $i
done
This is a sketch, not a prototype. I can't take it and run it to
see how it works. There is no locale file so I have no idea what
$i should look like.
Honestly, I don't care what the grammar is. I don't care what the format
of the file is, and I don't care what shell utility we are trying to fake
today.
All I care about is finishing up this task. Two months is more than enough
time for something like this to be designed and implemented.
Yet we still don't have the final grammar. It seems to me that we
need to start with it. I don't see how anyone could write any amount
of code unless they know what the grammar looks like.
Martin
