Re: [9fans] typed sh (was: what features would you like in a shell?)
2009/4/6 Bakul Shah bakul+pl...@bitblocks.com: On Thu, 02 Apr 2009 20:28:57 BST roger peppe rogpe...@gmail.com wrote: a pipeline is an amazingly powerful thing considering that it's not a turing-complete abstraction. f | g is basically function composition, where f and g are stream functions. Of course, this simple analogy breaks down the moment we add more input/output channels -- may be that is why anything beyond a simple pipeline seems to get people in trouble (see the rc output redirection thread). actually, the analogy works fine if we add more input channels - it's multiple output channels that make things hard, as they mean that you have an arbitrary directed graph rather than a tree, which doesn't have such a convenient textual representation and is harder to comprehend to boot. in alphabet, i had a diagnostic channel, which was strictly textual and not easily accessible from the language, which was arguably not the best solution, but i didn't want things to get too complex. To go beyond simple char streams, one can for example build a s-expr pipeline: a stream of self identifying objects of a few types (chars, numbers, symbols, lists, vectors). the difficulty with s-exprs (and most nested structures, e.g. XML) from a pipeline point of view is that their nested nature means that any branch might contain unlimited quantities of stuff, so you can't always process in O(1) space, which is one of the things i really like about pipeline processing. i found a nice counter-example in the fs stuff - the fundamental type was based around a conditional-push protocol for sending trees of files - the sender sends some information on a file/directory and the receiver replies whether to descend into that file or not. the tree had a canonical order (alphabetical on name), so tree merging could be done straightforwardly in O(1) space. this kind of streaming feels like a regular pipeline, but you can't do this with a regular pipeline. for instance, a later element in the pipeline can prevent an earlier from descending into a part of the file system that might block indefinitely. every language has a trade-off between typed and untyped representations; with alphabet i was trying to create something where it was *possible* to create new kinds of types where necessary (as in the fs example), but where it wouldn't be customary or necessary to do so in the vast majority of cases. perhaps it was folly, but i still think it was an interesting experiment, and i don't know of anything similar.
Re: [9fans] typed sh (was: what features would you like in a shell?)
On Thu, 16 Apr 2009 18:24:36 BST roger peppe rogpe...@gmail.com wrote: 2009/4/6 Bakul Shah bakul+pl...@bitblocks.com: On Thu, 02 Apr 2009 20:28:57 BST roger peppe rogpe...@gmail.com =C2=A0w= rote: a pipeline is an amazingly powerful thing considering that it's not a turing-complete abstraction. f | g is basically function composition, where f and g are stream functions. Of course, this simple analogy breaks down the moment we add more input/output channels -- may be that is why anything beyond a simple pipeline seems to get people in trouble (see the rc output redirection thread). actually, the analogy works fine if we add more input channels - it's multiple output channels that make things hard, as they mean that you have an arbitrary directed graph rather than a tree, which doesn't have such a convenient textual representation and is harder to comprehend to boot. True in general but certain graphs are relatively easy to comprehend depending on what you are doing (trees, hub spokes, rings). Shells don't provide you a convenient mechanism for constructing these graphs (I'd use macros in Scheme/Lisp, or a graphics editor). For DAGs you can use something like the example below but it doesn't have the nice aesthetics of a pipeline! let s0,s1 = function-with-two-output-streams function-with-two-input-streams(f0(s0), f1(s1), ...) To go beyond simple char streams, one can for example build a s-expr pipeline: a stream of self identifying objects of a few types (chars, numbers, symbols, lists, vectors). the difficulty with s-exprs (and most nested structures, e.g. XML) from a pipeline point of view is that their nested nature means that any branch might contain unlimited quantities of stuff, so you can't always process in O(1) space, which is one of the things i really like about pipeline processing. You can have arbitrarily long lines in a text file so if you operate on lines, you need arbitrary buffer space. It is the same problem. Also note that I was talking about a stream of s-exprs, not one s-expr as a stream (which makes no sense). For example, (attach ...) (walk ...) (open ...) (read ...) (clunk ...) i found a nice counter-example in the fs stuff - the fundamental type was based around a conditional-push protocol for sending trees of files - the sender sends some information on a file/directory and the receiver replies whether to descend into that file or not. the tree had a canonical order (alphabetical on name), so tree merging could be done straightforwardly in O(1) space. this kind of streaming feels like a regular pipeline, but you can't do this with a regular pipeline. for instance, a later element in the pipeline can prevent an earlier from descending into a part of the file system that might block indefinitely. every language has a trade-off between typed and untyped representations; with alphabet i was trying to create something where it was *possible* to create new kinds of types where necessary (as in the fs example), but where it wouldn't be customary or necessary to do so in the vast majority of cases. perhaps it was folly, but i still think it was an interesting experiment, and i don't know of anything similar.
Re: [9fans] typed sh (was: what features would you like in a shell?)
Nitpick: the output type of one command and the input type of the next command in the pipeline has to match, not every command. i think this is wrong. there's no requirement that the programs participating in a pipeline are compatable at all; that's the beauty of pipes. you can do things that were not envisioned at the time the programs were written. To go beyond simple char streams, one can for example build a s-expr pipeline: a stream of self identifying objects of a few types (chars, numbers, symbols, lists, vectors). In Q (from kx.com) over an IPC connection you can send strings, vectors, dictionaries, tables, or arbitray Q expressions. But there the model is more of a client/server. or ntfs where files are databases. not sure if streams can look the same way. - erik
Re: [9fans] typed sh (was: what features would you like in a shell?)
If program A outputs numbers in big-endian order and B expects input in little-endian order, A|B won't do the right thing. non-marshaled data considered harmful. film at 11. ☺ what i said was not that A|B makes sense for all A and B and for any data but rather that using text streams makes A|B possible for any A and any B and any input. the output might not be useful, but that is a problem on a completely different semantic level, one that computers are usually no good at. alsi, i don't think that type compatability is sufficient to insure that the output makes sense. what if A produces big-endian times in ms while B expects big-endian times in µs. Even for programs like wc have a concept of a 'character' and if the prev prog. produces something else you will be counting something meaningless. that's why plan 9 uses a single character set. but forcing compability seems worse. where are these decisions centralized? how do you change decisions? can you override these decisions (cast)? how does the output of, say, awk get typed? - erik
Re: [9fans] typed sh (was: what features would you like in a shell?)
On Mon, 06 Apr 2009 12:02:21 EDT erik quanstrom quans...@quanstro.net wrote:
If program A outputs numbers in big-endian order and B
expects input in little-endian order, A|B won't do the right
thing.
non-marshaled data considered harmful. film at 11. âº
In effect you are imposing a constraint (a type discipline).
Even if the programs themselves check such constraints, the
compatibility idea exists.
what i said was not that A|B makes sense for all A and B
and for any data but rather that using text streams makes
A|B possible for any A and any B and any input. the output
might not be useful, but that is a problem on a completely
different semantic level, one that computers are usually no good at.
alsi, i don't think that type compatability is sufficient
to insure that the output makes sense. what if A produces
big-endian times in ms while B expects big-endian times in µs.
In effect you are saying that text streams allow nonsensical
pipelines as well as sensible ones and anything other than
text streams would imply giving up freedom to create sensible
pipelines as yet unthought of. No disagreement there but see
below.
Even for programs like wc have a concept of a
'character' and if the prev prog. produces something else you
will be counting something meaningless.
that's why plan 9 uses a single character set.
but forcing compability seems worse. where are these decisions
centralized? how do you change decisions? can you override
these decisions (cast)? how does the output of, say, awk get
typed?
I am not suggesting forcing anything; I am suggesting
experimenting with s-expr streams (in the context of typed
sh idea). I don't know if that buys you anything more or if
you give up any essential freedom. My guess is you'd build
something more scalable, more composable but I wouldn't
really know until it is tried. I imagine s-expr-{grep,awk}
would look quite different from {grep,awk}. May be you'd end
up with something like a Lisp machine.
Re: [9fans] typed sh (was: what features would you like in a shell?)
but forcing compability seems worse. where are these decisions centralized? how do you change decisions? can you override these decisions (cast)? how does the output of, say, awk get typed? The output of awk is a byte stream, same as its input. The same holds for any program. If you want to give it some other type then you need at least a casting mechanism. You probably also want a splitting mechanism, with a regular expression for the field separator. -- John Stalker School of Mathematics Trinity College Dublin tel +353 1 896 1983 fax +353 1 896 2282
Re: [9fans] typed sh (was: what features would you like in a shell?)
On Thu, 02 Apr 2009 20:28:57 BST roger peppe rogpe...@gmail.com wrote: 2009/4/2 fge...@gmail.com: i wanted to go a little beyond sh while stopping short of the type profligacy of most other languages, hoping to create a situation where many commands used exactly the same types, and hence were viable to pipeline together. Nitpick: the output type of one command and the input type of the next command in the pipeline has to match, not every command. a pipeline is an amazingly powerful thing considering that it's not a turing-complete abstraction. f | g is basically function composition, where f and g are stream functions. Of course, this simple analogy breaks down the moment we add more input/output channels -- may be that is why anything beyond a simple pipeline seems to get people in trouble (see the rc output redirection thread). To go beyond simple char streams, one can for example build a s-expr pipeline: a stream of self identifying objects of a few types (chars, numbers, symbols, lists, vectors). In Q (from kx.com) over an IPC connection you can send strings, vectors, dictionaries, tables, or arbitray Q expressions. But there the model is more of a client/server.
Re: [9fans] typed sh (was: what features would you like in a shell?)
2009/4/2 fge...@gmail.com: On Thu, Apr 2, 2009 at 8:41 PM, John Stalker stal...@maths.tcd.ie wrote: What I most often miss in shell programming is a proper type system. You should have a look at alphabet. It is cool. http://www.vitanuova.com/inferno/man/1/sh-alphabet.html i certainly enjoyed creating it. unfortunately it's unfinished - i ran into unwarranted-complexity problems trying to simulate polymorphism with processes and channels... limbo's not a great language for writing interpreters in when you don't know all your types in advance. for those thinking of ideas for new shells, i still think there's mileage in some of alphebet's ideas, principally the way it lives in a half-way house between almost no types (sh) and unrestricted types (most other languages). that idea came from the thought that part of what makes sh so powerful is the fact that so many commands use the same types (string, stream of bytes + exit status). i wanted to go a little beyond sh while stopping short of the type profligacy of most other languages, hoping to create a situation where many commands used exactly the same types, and hence were viable to pipeline together. a pipeline is an amazingly powerful thing considering that it's not a turing-complete abstraction. alphabet was actually a generalisation of the fs command (see http://www.vitanuova.com/inferno/man/1/fs.html), that i'd found really useful (and more powerful than other filesystem traversal tools i've seen, for minimal code). some time i'll finish it!
Re: [9fans] typed sh (was: what features would you like in a shell?)
I don't know if others have already hit this kind of problematic, but I was dealing with a fair amount of C code, usable both as a library and accessible by a shell. Plus debugging needs. So I was, again and again, writing a wrapper to access a C function from the shell. So I ended concluding that I needed a kind of C interpreter as a shell. (I have an implementation, but it is not pure C---sentential calculus is distinct; it's a 4 values logic (NONSENSE, TRUE, FALSE, UNDECIDABLE) that has already real application in geometrical calculus; and integer and real calculus is added too for mathematical tasks---but it is not ready for prime time and I have still unanswered questions for special things I want to be present.) typedef, i.e. the ability to define other types above primary ones is perhaps what you are looking for? ISTR that on the early PCees, there was a basic interpreter in BIOS to let you play with the almost bare machine. I'd like to have a shell that lets me play with the bare OS. -- Thierry Laronde (Alceste) tlaronde +AT+ polynum +dot+ com http://www.kergis.com/ Key fingerprint = 0FF7 E906 FBAF FE95 FD89 250D 52B1 AE95 6006 F40C
