Hi Roland,
Thank you for this information!
Some questions & comments below...
April
> Date: Tue, 20 Jun 2006 06:05:52 +0200
> From: Roland Mainz <roland.mainz at nrubsig.org>
> X-Accept-Language: en
> MIME-Version: 1.0
> To: ksh93-integration-discuss <ksh93-integration-discuss at opensolaris.org>,
April Chin <April.Chin at eng.sun.com>, Mike Kupfer <mike.kupfer at sun.com>
> Subject: An incomplete tourist guide (version 001) to the OS/Net
ksh93-integration prototype002...
> Content-Transfer-Encoding: 7bit
> X-ID: Ee0UWiZAge9Ibz9A+igWVv+fkxQcmAwAcdXTqAtpaxoIvpCnu-3hUs at t-dialin.net
> X-TOI-MSGID: 0bdddbf8-77b6-4eab-88ba-6bd1ffd83d48
>
>
> Hi!
>
> ----
>
> This is mainly for April as an overview for writing the ARC fasttrack
> document:
>
> * Notes about the Makefiles and the ksh93 built system used in OS/Net
> The makefiles have a couple of unusual features (partially because they
> are modelled after the original AST build system), including:
>
> - The ksh93/AST sources use some headers which are generated based on
> templates. The OS/Net tree does not use this mechanism for now, instead
> these headers are pulled-over manually from a standalone build and
> integrated into OS/Net. This choice was made to simplify regression
> testing, otherwise we may end-up in problems such that the automated
> feature testing enables/disabled features based on the built machine
> even between minor OS/Net build versions without being noticed. The
> manual update of these files allows the engineers to run the regression
> tests after this update in a controlled environment.
>
> - libshell, libcmd, libdll and libast have their sources split into
> platform-independent parts located in
> usr/src/lib/name_of_library/common/ and platform-specific files in
> usr/src/lib/name_of_library/$(MACH)/, noteable automatically generated
> headers
>
> - The libshell and libast Makefiles build the object files (*.o) in
> subdirs under usr/src/lib/name_of_library/$(MACH)/pics/ and
> usr/src/lib/name_of_library/$(MACH)/objs/, making it easier to navigate
> (e.g. it's easier to have only ten or twenty object files in one dir
> (which is named after the subdir where the sources are located in) than
> trying to deal with 300 of them in one directory).
>
> - The ksh93/AST sources are currently not lint clean. We follow the
> "perl" approach for such a case:
> -- snip --
> #
> # ksh is not lint-clean yet. Fake up a target.
> #
> lint:
> @ $(ECHO) "usr/src/cmd/ksh is not lint-clean: skipping"
> @ $(TRUE)
> -- snip --
> The new libraries still provide the "lintcheck" Makefile target to allow
> manual lint checking.
This sounds fine.
>
> - usr/src/cmd/ksh/ is just the ksh93 frontend binary which directly
> calls into libshell.so (which is ksh93 in a shared library). Since the
> AST sources keep the frontend source file (sh/pmain.c) which contains
> the |main()| function together with the other libshell sources we're
> doing simply the same (e.g. usr/src/cmd/ksh/ fetches the source file
> from usr/src/lib/libshell/common/)
>
> - "src/cmd/ksh/Makefile.ksh93switch" is a small hack for
> OpenSolaris-based distributions and testing. It contains a small switch
> which can be set to install ksh93 as /usr/bin/ksh and enable a version
> of |libc::wordexp()| which uses ksh93. The switch is OFF by default but
> can be enabled on demand, e.g. for testing the migration of /usr/bin/ksh
> to ksh93 status and for OpenSolaris-based distributions who wish to
> avoid shipping the propiretary Solaris ksh monstrosity.
> AFAIK it doesn't really fit into OS/Net but it would still be nice to
> take it as it is highly usefull for the ksh93-migration and
> OpenSolaris-based distributions.
I think this is very useful as well. Since this makes no change for
Solaris builds I'd like to include this and the switch-dependent
wordexp.c change in the putback.
>
> - libcmd is a merge of Solaris libcmd (which only contains the API to
> read files in /etc/default/, e.g. |defopen()|, |defread()|, |defcntl()|)
> and ksh93's libcmd. The API added from the ksh93 version only contains
> function names starting with "b_" (for "b_" for (shell) builtin), e.g.
> |b_basename()|. A quick scan through an installed Solaris 10U1 machine
> (full install+OEM support) shows no other applications+libraries
> employing a similar naming scheme except one binary: dtksh (because it
> is ksh93-base, too :-) ). Since dtksh does not use or reference
> libcmd.so there should be no problem (the Solaris libcmd API is marked
> as SUNWprivate, e.g Sun owns this API, it's private and all sources
> which use it are under control by Sun).
I'm wondering what method you used here...I'm assuming something
like elfdump of binaries on the installed system?
Thanks for doing this--this will help support the case for merging libcmd.
>
> - The ksh93 files are currently stored in the same packages as Solaris
> ksh (mainly to avoid making the patch more compliciated since at some
> point libc may depend on ksh93)
Okay, sounds good.
>
> - "ksh93" uses "isaexec" to call a 64bit version of ksh93 on 64bit
> platforms. This has become neccesary as ksh93 provides binary interfaces
> (libshell.so and the ability to load more builtins at runtime) which may
> require a 64bit address space, for example a loadable builtin command
> which uses |mmap()| to map a file larger than 2GB into the address space
> or very large arrays (for example ksh93 has no problems to handle an
> associate array with multiple gigabytes of data).
> Currently the 64bit version is not ready for integration into OS/Net
> (see below) but we already install the neccesary infrastructure (e.g.
> "isaexec" links etc.) that we can add 64bit versions of the shell in a
> later putback (and hopefully without requiring an extra ARC fasttrack
> case for that).
> A seperate work will be done to force isaexec(1) to use a 32bit version
> of an executable via environment variables on demand (this is unrelated
> to the ksh93-integration project, but it's on my ToDo list...).
>
>
>
> * Currently missing features:
> - Localisation is currently not implemented. The binaries are compiled
> with localisation support enabled but we do not intend to support this
> with the initial putback (mainly to simplify things, because
> localisation sources do not seem to be part of the OS/Net repository and
> we simply only have two languages right now: germany (me) and english
> (the builtin language))
>
> - No localised version of the manual page is provided yet
>
> - 64bit versions of ksh93 and it's libraries are currently not provided
> because there is a small glitch which causes the automatically generated
> headers to differ between 32bit and 64bit builds. Upstream is working on
> getting this fixed but it is unlikely that this will be done before the
> initial putback,
>
>
>
> * Prototype002 adds the following files/dirs/links to Solaris:
> -- snip --
> -- snip --
>
>
>
> * The following OS/Net source files are added or modified by the
> ksh93-integration project (based on the output of % svn diff
> svn://svn.genunix.org/on/tags/b37/usr
>
svn://svn.genunix.org/on/branches/ksh93/gisburn/prototype002/m1_ast_ast_imported
/usr
> # ):
> -- snip --
> -- snip --
>
>
I'll send you separate e-mail
to see if I have the correct list of the new library interfaces.
>
> * Interfaces:
>
> The following interfaces are added with the ksh93-integration
> prototype002:
>
> Interface Notes
> --------- -----
> /usr/bin/ksh93 Executable location, isaexec link
> /usr/bin/pfksh93 Executable location, isaexec link
> /usr/bin/rksh93 Executable location, isaexec link
> /usr/bin/krsh93 Executable location, isaexec link
> /usr/bin/$(ISA)/ksh93 Executable location of ksh93
> /usr/bin/$(ISA)/pfksh93 Hard link to /usr/bin/$(ISA)/ksh93
> /usr/bin/$(ISA)/rksh93 Hard link to /usr/bin/$(ISA)/ksh93
> /usr/bin/$(ISA)/krsh93 Hard link to /usr/bin/$(ISA)/ksh93
> /lib/libshell.so.1 Shared library containing ksh93's core
> /lib/libast.so.1 Shared library containg utility functions
> /lib/libdll.so.1 Shared library containg utility functions
> /lib/libcmd.so.1 Shared library containg shell builtins
> /usr/demo/ksh/ Place to put demo files and test suite
>
> The components in /usr/bin/ have the stabilty status "Commited" (I hope
> this is the right term... :-) ), the library interfaces are "Private"
> for now (actually "semi-private" as ksh93-based shells shipped with
> Solaris such as dtksh, tksh etc. should be able to use it, however Sun
> does not seem to have a concept of "Semi-private" for such a case)
For the private library interfaces we can make them:
Project Private, which is for use only by the ksh93 project;
Consolidation Private, for use only by our OS-Net consolidation
(the source we have access to); or
Sun Private, for use also by other consolidations in Sun with
separate source gates, such as those for dbx or dtksh.
Since we want to allow other Sun applications to use libshell, we'd want to
make it Sun Private.
The existing libcmd interfaces (defread, defopen, defcntl)
are Sun Private, but I assume this does not require the
new interfaces in libcmd to have this same scope...someone can correct me
if this is wrong.
So my question is, what is the proper scope for the
new libcmd interfaces, as well as those for libast and libdll?
Sun Private, or one of the more limited categories?
I know you mentioned libast's usefulness by external applications,
but I don't think we want to commit these interfaces for external use yet.
Would an internal application such as dbx, require any of the other
interfaces other than libshell's?
>
> All other components of ksh93 which appear to be interfaces (paths,
> environment variables, builtin commands, builtin command paths, user
> interface (e.g. shell editing modes) etc.) are Volatile. In particular,
> even though the AST/ksh93 community is committed to maintaining
> backwards compatibility of these interfaces are also considered
> Volatile. Future project maturity may result in the upgrading of the
> stability of these interfaces.
>
>
>
> * References:
> 1. http://www.opensolaris.org/os/project/ksh93-integration
> 2. http://www.kornshell.com/
> 3.
>
http://www.opensolaris.org/os/project/ksh93-integration/docs/ksh93r/man/man1/sh/
> 4.
>
http://www.opensolaris.org/os/project/ksh93-integration/docs/ksh93r/man/man3/she
ll/
>
>
>
>
> AFAIK that's all for now... :-)
>
> ----
>
> Bye,
> Roland
>
> --
> __ . . __
> (o.\ \/ /.o) roland.mainz at nrubsig.org
> \__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
> /O /==\ O\ TEL +49 641 7950090
> (;O/ \/ \O;)
