Re: [Re: Re: [naviserver-devel] ns_parseargs example]
On 5/30/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote:
>
> Am 30.05.2005 um 12:23 schrieb Stephen Deasey:
>
> >
> > The other option is -eightbit:flag syntax. It's not perfect, but it's
> > better than nested lists and has ~5 years of usage in the ACS behind
> > it.
> >
>
> Another try (I can't just let go) ...
>
> I do not see much sense in trying to keep the Tcl argument list
> format under all costs. Give me a good reason why we should do that?
> Anyways, we are about to define a new "proc"-type of command
> (ns_proc) aren't we? What is the sense then to force ourselves
> to the old format?
>
> Why don't we *extend* Tcl-proc arg-treatment like this:
>
> {name typecheck ?default_value?}
>
> This is *not* Tcl-proc compatible but *is* in the spirit of
> the language. The "typecheck" above can/will be definitely
> a list of one or more elements, depending on the check itself.
> The typecheck is *forced*. You must give a type. If you need
> no checking at all use "string" which will cover all possiblities.
> By *forcing* the type-check, many of the programming
> errors and double-checks in the procedure itself can be saved.
>
> Also, all people needing no extra checking can always use plain Tcl-proc
> calls. Others who'd need checking would use exteneded ns_proc syntax.
> This is pretty clear and I see no problems with that at all.
>
> For example:
> ns_parseargs {{-retries int 0} {-blockcopy flag} {device string}
> args}
>
> or
> ns_proc dosomething {{-retries int 0} {-blockcopy flag} {device
> string}} {
># ...
> }
>
> The most natural Tcl-way is to use lists. We've seen that the
> desperate try to keep the Tcl-proc syntax leads to too complicated
> nested lists which humans are not easily assembling.
> So, this is a plain list which is easy to write and read by humans.
>
> We can evolve typecheck over time to include more elaborate
> tests, but a handful of those (string, int, flag, bool, list) are most
> useful and complete for about all the needs I can think of
> at the moment.
>
> What is wrong with that (except that it perhaps does not follow
> openacs model)?
ns_proc test {{-flag f}} {
if {$flag} {
ns_log notice ok
}
}
test -flag t ;# ok
test -flag true ;# ok
test -flag yes ;# ok
test -flag 1 ;# ok
The result of the expression passed to Tcl's if is converted to
boolean so there's no need to use [string is true $x]. The
difference between '-flag' and '-flag t' is two characters -- I don't
think it's worth it.
As I mentioned before, in general I think it's a bad idea to create
boolean flags as it makes it cumbersome to call such a proc where the
flag is computed. e.g.:
if {$x} {
foo -flag ...
} else {
foo ...
}
It gets really ugly when there are multiple flags. I added boolean
flags to the C implementation as there is already existing code which
uses them which could benefit from the arg parsing.
Regarding the 'oneof' type, maybe what's actually needed is some
syntactic sugar? e.g. something like:
proc assert_oneof {varname list} {
upvar $varname x
if {![lsearch -exact $list $x]} {
error "assertion failed: $varname must be one of $list"
}
}
proc test {{x foo}} {
assert_oneof x {foo bar baz}
...
}
The disadvantage of extending the proc syntax as you've suggested
above is that users are then forced to always provide a type, even if
it's just the 'null' type, string. This kind of type checking is
likely to be the exception rather than the rule, but I imagine option
parsing will be quite common. The ordering of the default argument
has changed which is incompatible with what the programmer expects
from standard proc usage, even if the mechanical incompatibility does
not matter.
The oneof check shows that things will get more complicated than
simple 'is int', 'is bool' checks. Min and max range checks on
integers would be nice, regexp checks on strings would be cool etc...
What about applying more than one check to an arg? There would need
to be syntax for it as you can't specify the arg more than once as you
can specify multiple assertions.
Other languages solve this with assertions or pre-conditions and
post-conditions, and that feels more like what we're doing here. Tcl
*does* have type checking: an integer argument passed to a proc which
is then passed to the expr command will cause a conversion failure if
the value is 'foo'. The type checking on the proc argument will just
cause the failure earlier (and will result in double-checking).
What we're catching here are programmer errors. Checking is more
urgently required for malicious attacks. You really want to be able
to say something like:
ns_queryget -assert int -- life 42
It was always in the back of my mind that we needed some kind of
unified checking mechanism to handle both procs and ns_queryget (and
forms, as in Vlad's nstk). Rather than shoe-horn the type checking of
proc into the proc args, perhaps it would be better to u
Re: [naviserver-devel] Re: RFE #1202462
On 5/30/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: > > Am 30.05.2005 um 12:01 schrieb Stephen Deasey: > > > > > What's the return value of Ns_FSOpen? > > > > Something that can be fed to Ns_FSRead and/or Ns_FSClose > for example. I haven't started the implementation yet > but this is first what comes to mind. > > Presumably, it will be Tcl_Channel in VFS or a regular > OS filehandle w/o VFS. The return type would be unknown unless the caller also used #ifdef? The issue is not preferring open over Tcl_VFSOpen, and it's not mmap per se. mmap is just one example of a function which requires a file descriptor or real files in the file system. sendfile is another, as are readv/writev, epoll, kqueue, inotify, etc. The Tcl VFS is a lowest common denominator abstraction over file systems. It provides the common things, but can't provide implementations of everything, nor can it return a useful file descriptor from a Tcl_Channel, at least in the case where everything is contained within one big file. I think Ns_TclGetOpenFd is going to fail. I don't see anything wrong with using Tcl_VFS* directly to read the config file, write to a log file etc., and the tcl commands in nsd/tclfile.c should be replaced by simple Tcl wrappers anyway. It's the stuff in nsd/fastpath.c that's important. File descriptors need to be available for use with modern, high performance system calls with a minimum of conditional compilation/execution. This also has the potential to be a never-ending bug hunt. Absolutely every call to the file system has to be converted to Tcl_VFS* or Ns_VFS*, including all modules and the libraries they wrap, or people will be surprised... I'm not against this feature, but it's pretty esoteric and the implementation is invasive so the quality bar needs to be set high. I'm sure you can do it :-)
Re: [Re: Re: [naviserver-devel] ns_parseargs example]
To be as much close to current Tcl convention in proc syntax we can add
optional type AFTER default value
ns_proc test { {-flag 0 flag} {-list a oneof {a b c}} {-name ""} { value
"" } } {
}
This way i do not have to remember 2 different ways to provide default
value and if i want type checking, i will give optional typoe, if not
then no type cheking.
Stephen Deasey wrote:
On 5/30/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote:
Am 30.05.2005 um 12:23 schrieb Stephen Deasey:
The other option is -eightbit:flag syntax. It's not perfect, but it's
better than nested lists and has ~5 years of usage in the ACS behind
it.
Another try (I can't just let go) ...
I do not see much sense in trying to keep the Tcl argument list
format under all costs. Give me a good reason why we should do that?
Anyways, we are about to define a new "proc"-type of command
(ns_proc) aren't we? What is the sense then to force ourselves
to the old format?
Why don't we *extend* Tcl-proc arg-treatment like this:
{name typecheck ?default_value?}
This is *not* Tcl-proc compatible but *is* in the spirit of
the language. The "typecheck" above can/will be definitely
a list of one or more elements, depending on the check itself.
The typecheck is *forced*. You must give a type. If you need
no checking at all use "string" which will cover all possiblities.
By *forcing* the type-check, many of the programming
errors and double-checks in the procedure itself can be saved.
Also, all people needing no extra checking can always use plain Tcl-proc
calls. Others who'd need checking would use exteneded ns_proc syntax.
This is pretty clear and I see no problems with that at all.
For example:
ns_parseargs {{-retries int 0} {-blockcopy flag} {device string}
args}
or
ns_proc dosomething {{-retries int 0} {-blockcopy flag} {device
string}} {
# ...
}
The most natural Tcl-way is to use lists. We've seen that the
desperate try to keep the Tcl-proc syntax leads to too complicated
nested lists which humans are not easily assembling.
So, this is a plain list which is easy to write and read by humans.
We can evolve typecheck over time to include more elaborate
tests, but a handful of those (string, int, flag, bool, list) are most
useful and complete for about all the needs I can think of
at the moment.
What is wrong with that (except that it perhaps does not follow
openacs model)?
ns_proc test {{-flag f}} {
if {$flag} {
ns_log notice ok
}
}
test -flag t ;# ok
test -flag true ;# ok
test -flag yes ;# ok
test -flag 1 ;# ok
The result of the expression passed to Tcl's if is converted to
boolean so there's no need to use [string is true $x]. The
difference between '-flag' and '-flag t' is two characters -- I don't
think it's worth it.
As I mentioned before, in general I think it's a bad idea to create
boolean flags as it makes it cumbersome to call such a proc where the
flag is computed. e.g.:
if {$x} {
foo -flag ...
} else {
foo ...
}
It gets really ugly when there are multiple flags. I added boolean
flags to the C implementation as there is already existing code which
uses them which could benefit from the arg parsing.
Regarding the 'oneof' type, maybe what's actually needed is some
syntactic sugar? e.g. something like:
proc assert_oneof {varname list} {
upvar $varname x
if {![lsearch -exact $list $x]} {
error "assertion failed: $varname must be one of $list"
}
}
proc test {{x foo}} {
assert_oneof x {foo bar baz}
...
}
The disadvantage of extending the proc syntax as you've suggested
above is that users are then forced to always provide a type, even if
it's just the 'null' type, string. This kind of type checking is
likely to be the exception rather than the rule, but I imagine option
parsing will be quite common. The ordering of the default argument
has changed which is incompatible with what the programmer expects
from standard proc usage, even if the mechanical incompatibility does
not matter.
The oneof check shows that things will get more complicated than
simple 'is int', 'is bool' checks. Min and max range checks on
integers would be nice, regexp checks on strings would be cool etc...
What about applying more than one check to an arg? There would need
to be syntax for it as you can't specify the arg more than once as you
can specify multiple assertions.
Other languages solve this with assertions or pre-conditions and
post-conditions, and that feels more like what we're doing here. Tcl
*does* have type checking: an integer argument passed to a proc which
is then passed to the expr command will cause a conversion failure if
the value is 'foo'. The type checking on the proc argument will just
cause the failure earlier (and will result in double-checking).
What we're catching here are programmer errors. Checking is more
urgently required for malicious attacks. You really want to be able
to say something like:
ns_queryget -assert int -- li
Re: [Re: Re: [naviserver-devel] ns_parseargs example]
Am 01.06.2005 um 15:45 schrieb Vlad Seryakov:
To be as much close to current Tcl convention in proc syntax we can
add optional type AFTER default value
ns_proc test { {-flag 0 flag} {-list a oneof {a b c}} {-name ""}
{ value "" } } {
}
This way i do not have to remember 2 different ways to provide
default value and if i want type checking, i will give optional
typoe, if not then no type cheking.
The only side-effect is of course that you'd need to give the default
argument if you need type-checking. I believe I can live with that.
Zoran
Re: [Re: Re: [naviserver-devel] ns_parseargs example]
Am 01.06.2005 um 13:24 schrieb Stephen Deasey: So, how about assertions? Good. So, that means we scratch the idea of putting it in the proc/ns_proc syntax and keep it as is, i.e. we parse-out only the options from the argument list and do no extra type checking there. Type-checking would be implemented as a separate command then. I understand the blues with the boolean arguments and I am willing to scratch them from the specs. So, if you need "-dothat" you'd be actually saying "-dothat true" or "-dothat false". All that leaves us with the ns_parseargs pretty much as it is now, or? Apropos assertions: http://www.tcl.tk/cgi-bin/tct/tip/53.html This was withdrawn because of the fact that it already exist in tcllib. So, there is very little to do except "package req tcllib" and use what's there. Zoran
Re: [naviserver-devel] Re: RFE #1202462
Am 01.06.2005 um 14:10 schrieb Stephen Deasey: On 5/30/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: Am 30.05.2005 um 12:01 schrieb Stephen Deasey: What's the return value of Ns_FSOpen? Something that can be fed to Ns_FSRead and/or Ns_FSClose for example. I haven't started the implementation yet but this is first what comes to mind. Presumably, it will be Tcl_Channel in VFS or a regular OS filehandle w/o VFS. The return type would be unknown unless the caller also used #ifdef? The return value is of course opaque and should be returned to the FS abstraction layer. But I read on... I don't see anything wrong with using Tcl_VFS* directly to read the config file, write to a log file etc., and the tcl commands in nsd/tclfile.c should be replaced by simple Tcl wrappers anyway. It's the stuff in nsd/fastpath.c that's important. File descriptors need to be available for use with modern, high performance system calls with a minimum of conditional compilation/execution. This also has the potential to be a never-ending bug hunt. Absolutely every call to the file system has to be converted to Tcl_VFS* or Ns_VFS*, including all modules and the libraries they wrap, or people will be surprised... I'm not against this feature, but it's pretty esoteric and the implementation is invasive so the quality bar needs to be set high. I'm sure you can do it :-) Of course, all you say holds. If you need to go to the max when accesing files, then Tcl VFS will not be the fastest kid on the block. But what about this: we make a Ns_FS wrappers and put them in place, even in the fastpath and adp code. We also add Ns_FSGetOSHandle which will then return invalid handle to the caller if the file is not located on the native filesystem. This is trivial and costs no performance. Now, the caller can accomodate for that and decide what to do. A very good example is the actual fastpath code which does the same with the mmap. It works-around the fact that mmap is not defined in the server config and uses simple open/read/close sequence. This way you'd be free to add whatever fancy roaring implementation (kaio for example) to get the max-speed, if you like. We need only cover access to files, no sockets, pipes, etc. What we'd need is just a handful of calls: open, close, stat, seek, read, write, opendir, readdir, closedir, getcwd (did I forgot something?) Most of them would use the opaque file-handle from which you can obtain the OS handle to make any private speed optimizations. If the handle cannot be obtained, then you provide the best-effort fallback. What would/could be wrong with that? Zoran
Re: [naviserver-devel] Re: RFE #1202462
On 6/1/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: > > Am 01.06.2005 um 14:10 schrieb Stephen Deasey: > > > On 5/30/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: > > > >> > >> Am 30.05.2005 um 12:01 schrieb Stephen Deasey: > >> > >> > >>> > >>> What's the return value of Ns_FSOpen? > >>> > >>> > >> > >> Something that can be fed to Ns_FSRead and/or Ns_FSClose > >> for example. I haven't started the implementation yet > >> but this is first what comes to mind. > >> > >> Presumably, it will be Tcl_Channel in VFS or a regular > >> OS filehandle w/o VFS. > >> > > > > > > The return type would be unknown unless the caller also used #ifdef? > > The return value is of course opaque and should be returned > to the FS abstraction layer. But I read on... > > > > > I don't see anything wrong with using Tcl_VFS* directly to read the > > config file, write to a log file etc., and the tcl commands in > > nsd/tclfile.c should be replaced by simple Tcl wrappers anyway. It's > > the stuff in nsd/fastpath.c that's important. File descriptors need > > to be available for use with modern, high performance system calls > > with a minimum of conditional compilation/execution. > > > > This also has the potential to be a never-ending bug hunt. Absolutely > > every call to the file system has to be converted to Tcl_VFS* or > > Ns_VFS*, including all modules and the libraries they wrap, or people > > will be surprised... > > > I'm not against this feature, but it's pretty esoteric and the > > implementation is invasive so the quality bar needs to be set high. > > I'm sure you can do it :-) > > > > Of course, all you say holds. If you need to go to the max when > accesing files, then Tcl VFS will not be the fastest kid on the block. > > But what about this: we make a Ns_FS wrappers and put them in place, > even in the fastpath and adp code. We also add Ns_FSGetOSHandle which > will then return invalid handle to the caller if the file is not located > on the native filesystem. This is trivial and costs no performance. > Now, the caller can accomodate for that and decide what to do. > A very good example is the actual fastpath code which does the same > with the mmap. It works-around the fact that mmap is not defined in > the server config and uses simple open/read/close sequence. > > This way you'd be free to add whatever fancy roaring implementation > (kaio for example) to get the max-speed, if you like. > We need only cover access to files, no sockets, pipes, etc. > What we'd need is just a handful of calls: > >open, close, stat, seek, read, write, opendir, readdir, closedir, > getcwd > > (did I forgot something?) > > Most of them would use the opaque file-handle from which you can > obtain the OS handle to make any private speed optimizations. If the > handle cannot be obtained, then you provide the best-effort fallback. > > What would/could be wrong with that? Sounds good.
Re: [Re: Re: [naviserver-devel] ns_parseargs example]
On 6/1/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: > > Am 01.06.2005 um 13:24 schrieb Stephen Deasey: > > > So, how > > about assertions? > > > > Good. > > So, that means we scratch the idea of putting it in the > proc/ns_proc syntax and keep it as is, i.e. we parse-out > only the options from the argument list and do no extra > type checking there. Type-checking would be implemented > as a separate command then. > > I understand the blues with the boolean arguments and > I am willing to scratch them from the specs. So, if you > need "-dothat" you'd be actually saying "-dothat true" > or "-dothat false". > > All that leaves us with the ns_parseargs pretty much as > it is now, or? > > Apropos assertions: > http://www.tcl.tk/cgi-bin/tct/tip/53.html > > This was withdrawn because of the fact that it already > exist in tcllib. So, there is very little to do except > "package req tcllib" and use what's there. Looks good. For program bug detection, use tcllib assertions in cooperation with the option parsing we have now. For input validation, some other mechanism should be used.
Re: [Re: Re: [naviserver-devel] ns_parseargs example]
Am 01.06.2005 um 21:30 schrieb Stephen Deasey: For input validation, some other mechanism should be used. Input of what kind? From Tcl or from C? Do you mean form-input? Why you can't apply the tcllib assertion on form-input, if this is what you mean? Zoran
Re: [Re: Re: [naviserver-devel] ns_parseargs example]
On 6/1/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: > > Am 01.06.2005 um 21:30 schrieb Stephen Deasey: > > > For input > > validation, some other mechanism should be used. > > > > Input of what kind? From Tcl or from C? > Do you mean form-input? Why you can't apply the > tcllib assertion on form-input, if this is what you mean? One of the features of tcllib assertions is that you can turn them off, or into no-ops when in production mode, just like standard C assertions. That's a useful feature, but not when you're checking form input!
Re: [Re: Re: [naviserver-devel] ns_parseargs example]
Am 01.06.2005 um 22:23 schrieb Stephen Deasey: That's a useful feature, but not when you're checking form input! Why not?
Re: [naviserver-devel] Re: RFE #1202462
Am 01.06.2005 um 21:27 schrieb Stephen Deasey: Sounds good. OK. I will start the work and give diffs for review when I come to some point. Zoran
Re: [Re: Re: [naviserver-devel] ns_parseargs example]
On 6/1/05, Zoran Vasiljevic <[EMAIL PROTECTED]> wrote: > > Am 01.06.2005 um 22:23 schrieb Stephen Deasey: > > > That's a useful feature, but not when you're checking > > form input! > > Why not? Because form input always needs to be checked. You can't declare it debugged and turn off exception checking. People will continue to feed your program broken/malicious input.
Re: [Re: Re: [naviserver-devel] ns_parseargs example]
Am 01.06.2005 um 22:50 schrieb Stephen Deasey: Because form input always needs to be checked. You can't declare it debugged and turn off exception checking. People will continue to feed your program broken/malicious input. Ah, this is what you mean! What about fixing tcllib to allow two types of asserts? The one with off/on switch and the one without? I haven't looked into there, but this can't be rocket science. Zoran
