Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-09-01 Thread Irek Szczesniak
On Wed, Aug 29, 2012 at 7:37 PM, David Korn d...@research.att.com wrote:
 cc:  ast-developers@research.att.com
 Subject: Re: Re: Re: [ast-developers] Fwd: Per thread open(), stat(), 
 rename() and  so on, and *at() API
 

 Excluding tksh and dtksh - has anyone ever managed to use libshell for
 anything else? Anyone? I think the answer is a bold NO because the
 libshell API is NOT suited for tasks outside making more shell
 variants. We've been there and figured that libshell can not be used
 in ANY larger project because it fiddles with so many global resources
 around that it sooner, more sooner blows your application up. The
 libshell code is NOT designed to share control with anything else than
 itself.



 Many years ago I had a summer student embed ksh93 into a windows scripting
 interface.

That doesn't mean libshell has a good API for embedding. Actually it's
quite unsuited to use for anything than tksh, dtksh or biosh. The
whole point of having an API for embedding is to make it an
universally easy task to embed something into an existing application
and mix it with other APIs. libshell makes this VERY hard since it
acts like the sole owner of a lot of resources without asking or
having callbacks to notify the consumer. Taking option {a} will make
this even worse.

Irek
___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Lionel Cons
On 29 August 2012 10:01, Roland Mainz roland.ma...@nrubsig.org wrote:
 [Forwarding email since somehow the ast-developers@ list has issues
 with postings from Olga's address... ;-( ]
 -- Forwarded message --
 From: ольга крыжановская olga.kryzhanov...@gmail.com
 Date: Wed, Aug 29, 2012 at 7:36 AM
 Subject: Fwd: Per thread open(), stat(), rename() and so on, and *at() API
 To: Roland Mainz roland.ma...@nrubsig.org


 Wenn du wach bist schick es bitte nochmal an
 ast-developers@research.att.com. Meine mail ist im spam-filter
 hangengeblieben.
 -- Forwarded message --
 From: ольга крыжановская olga.kryzhanov...@gmail.com
 Date: Wed, Aug 29, 2012 at 6:54 AM
 Subject: Per thread open(), stat(), rename() and so on, and *at() API
 To: Glenn Fowler g...@research.att.com, David Korn
 d...@research.att.com, Phong Vo k...@research.att.com
 Cc: ast-developers@research.att.com


 Glenn, I have an alternative for your per thread open(), stat(),
 rename() and so on API plans:

 Instead of creating AST wrapper code for open(), stat(), rename() and
 so we could use the following macros, defined in ast/fcntl.h:

 --cutme
 #define unlink(path)
 unlinkat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), 0)
 #define rmdir(path)
 unlinkat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), AT_REMOVEDIR)
 #define chown(path, uid,
 gid)fchownat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (uid), 
 (gid),
 0)
 #define lchown(path, uid, gid)
 fchownat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (uid), (gid),
 AT_SYMLINK_NOFOLLOW)
 #define stat(path, sb)  
 fstatat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (sb), 0)
 #define lstat(path, sb) 
 fstatat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (sb), AT_SYMLINK_NOFOLLOW)
 #define rename(oldname,
 newname)renameat(AST_GET_CURRENT_THREAD_CWD_FD(), (oldname),
 AST_GET_CURRENT_THREAD_CWD_FD(), (newname))
 #define access(path,
 amode)  faccessat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (amode), 0)
 #define eaccess(path,
 amode)  faccessat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (amode),
 AT_EACCESS)
 #define mkdir(path, amode)  
 mkdirat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (amode))
 #define mkfifo(path, amode)
 mkfifoat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (amode))
 #define mknod(path, amode,
 adev)   mknodat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (amode),
 (adev))
 #define readlink(path, buf,
 bufsize)readlinkat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (buf),
 (bufsize))
 #define symlink(oldpath, newpath)   symlinkat((oldpath),
 AST_GET_CURRENT_THREAD_CWD_FD(), (newpath))
 --cutme

 AST_GET_CURRENT_THREAD_CWD_FD() would be defined to be a preprocessor
 macro, which will be a function returning the current thread's cwd fd;
 if there is none allocated for this thread yet it will be done by
 within that function, using open(., O_search).

 However, consumers of the ast/fcntl.h header are _free_ to redefine
 this macro, for example src/cmd/ksh93/include/defs.h could define
 AST_GET_CURRENT_THREAD_CWD_FD() as
 #define AST_GET_CURRENT_THREAD_CWD_FD() (shp-pwdfd)
 This would, at least, save the function call overhead, and finally
 make use of (shp-pwdfd) as it was intended.

 What do you think? It would make things easier for you to implement,
 i.e. you only have to implement the per thread
 AST_GET_CURRENT_THREAD_CWD_FD(), and applications are free to use the
 *at() API if they wish to.

What is the purpose of the change? Wouldn't it be easier to just to
make a sweep of the codebase using an awk script and replace the
functions with the new functions, instead of adding yet another
#define for standard filesystem calls? I think the current number of
cpp redirects for such calls has reached a number of nesting levels
which has become unsustainable.

Lionel

___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Cedric Blancher
On 29 August 2012 10:01, Roland Mainz roland.ma...@nrubsig.org wrote:
 [Forwarding email since somehow the ast-developers@ list has issues
 with postings from Olga's address... ;-( ]
 -- Forwarded message --
 From: ольга крыжановская olga.kryzhanov...@gmail.com
 Date: Wed, Aug 29, 2012 at 7:36 AM
 Subject: Fwd: Per thread open(), stat(), rename() and so on, and *at() API
 To: Roland Mainz roland.ma...@nrubsig.org


 Wenn du wach bist schick es bitte nochmal an
 ast-developers@research.att.com. Meine mail ist im spam-filter
 hangengeblieben.
 -- Forwarded message --
 From: ольга крыжановская olga.kryzhanov...@gmail.com
 Date: Wed, Aug 29, 2012 at 6:54 AM
 Subject: Per thread open(), stat(), rename() and so on, and *at() API
 To: Glenn Fowler g...@research.att.com, David Korn
 d...@research.att.com, Phong Vo k...@research.att.com
 Cc: ast-developers@research.att.com


 Glenn, I have an alternative for your per thread open(), stat(),
 rename() and so on API plans:

 Instead of creating AST wrapper code for open(), stat(), rename() and
 so we could use the following macros, defined in ast/fcntl.h:

 --cutme
 #define unlink(path)
 unlinkat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), 0)
 #define rmdir(path)
 unlinkat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), AT_REMOVEDIR)
 #define chown(path, uid,
 gid)fchownat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (uid), 
 (gid),
 0)
 #define lchown(path, uid, gid)
 fchownat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (uid), (gid),
 AT_SYMLINK_NOFOLLOW)
 #define stat(path, sb)  
 fstatat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (sb), 0)
 #define lstat(path, sb) 
 fstatat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (sb), AT_SYMLINK_NOFOLLOW)
 #define rename(oldname,
 newname)renameat(AST_GET_CURRENT_THREAD_CWD_FD(), (oldname),
 AST_GET_CURRENT_THREAD_CWD_FD(), (newname))
 #define access(path,
 amode)  faccessat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (amode), 0)
 #define eaccess(path,
 amode)  faccessat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (amode),
 AT_EACCESS)
 #define mkdir(path, amode)  
 mkdirat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (amode))
 #define mkfifo(path, amode)
 mkfifoat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (amode))
 #define mknod(path, amode,
 adev)   mknodat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (amode),
 (adev))
 #define readlink(path, buf,
 bufsize)readlinkat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (buf),
 (bufsize))
 #define symlink(oldpath, newpath)   symlinkat((oldpath),
 AST_GET_CURRENT_THREAD_CWD_FD(), (newpath))
 --cutme

 AST_GET_CURRENT_THREAD_CWD_FD() would be defined to be a preprocessor
 macro, which will be a function returning the current thread's cwd fd;
 if there is none allocated for this thread yet it will be done by
 within that function, using open(., O_search).

 However, consumers of the ast/fcntl.h header are _free_ to redefine
 this macro, for example src/cmd/ksh93/include/defs.h could define
 AST_GET_CURRENT_THREAD_CWD_FD() as
 #define AST_GET_CURRENT_THREAD_CWD_FD() (shp-pwdfd)
 This would, at least, save the function call overhead, and finally
 make use of (shp-pwdfd) as it was intended.

 What do you think? It would make things easier for you to implement,
 i.e. you only have to implement the per thread
 AST_GET_CURRENT_THREAD_CWD_FD(), and applications are free to use the
 *at() API if they wish to.

What do you intend to do for platforms which do not have the POSIX
openat() and friends API? Old Solaris, old Linux comes in mind (not
that I care about any OS older than four years)...

Ced
-- 
Cedric Blancher cedric.blanc...@googlemail.com
Institute Pasteur

___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Cedric Blancher
On 29 August 2012 13:54, Roland Mainz roland.ma...@nrubsig.org wrote:
 [Forwarding since ATT's spam filter is on drugs... ;-(( ]
 -- Forwarded message --
 From: ольга крыжановская olga.kryzhanov...@gmail.com
 Date: Wed, Aug 29, 2012 at 1:20 PM
 Subject: Re: [ast-developers] Fwd: Per thread open(), stat(), rename()
 and so on, and *at() API
 To: Lionel Cons lionelcons1...@googlemail.com
 Cc: Roland Mainz roland.ma...@nrubsig.org, ast-developers@research.att.com


 Lionel, there are 2 purposes:

 1. Allow that ksh93/libshell can be embedded in other applications
 (Roland calls this 'back end usage', i.e. the user is not aware that
 he is interacting with a shell, and neither do any other apis know
 that, or at least should not have to know about that), with out
 interfering the calling application by changing the current working
 directory.
 2. Allow that ksh93/libshell and the libast api can have a per thread
 current working directory.

 There are 2 proposals how to accomplish that:
 a) Add wrapper functions for open(), stat(), lstat(), chdir(),
 fchdir(), access(), eaccess(), rename() and so on, and redirect them
 to their *at() counterparts, with dirfd set to the current threads cwd
 fd.

 or

 b) Add wrapper cpp macros which redirect open(), stat(), lstat(),
 chdir(), fchdir(), access(), eaccess(), rename() and so on to their
 *at() counterparts, but define an extra macro which is AT_FDCWD for
 non threaded applications, and calls a function to obtain the current
 threads working directory fd for the current thread, but allows to
 override this macro on a per source file basis, so that applications
 like libshell, who maintain their own cwd directory fd, can use that.

 David and Glenn seems to prefer {a}, while I and Roland prefer {b},
 because {b} will allow an application to work as back end api without
 interfering with the global cwd or any other, may be libast based apis
 cwd. This includes nested usage of libshell, like for system() or
 wordexp().

I'd prefer option {b}, too. Strongly (sorry David and Glenn :)).
It sounds like the more sane and flexible way to do this. The big road
block is that you have to find a solution for old platforms which do
not have openat() and friends, or EOS (end-of-support) them (I'd
support that if no platform younger than four years lacks openat() and
friends).

Ced
-- 
Cedric Blancher cedric.blanc...@googlemail.com
Institute Pasteur

___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Glenn Fowler

we're going to take some research leeway and investigate the implications of {a}
we're not strangers to doing stuff like that (3d(1))
and we're not strangers to having research prove us wrong

On Wed, 29 Aug 2012 14:49:33 +0200 Cedric Blancher wrote:
 On 29 August 2012 13:54, Roland Mainz roland.ma...@nrubsig.org wrote:
  [Forwarding since ATT's spam filter is on drugs... ;-(( ]
  -- Forwarded message --
  From: ÏÌØÇÁ ËÒÙÖÁÎÏ×ÓËÁÑ olga.kryzhanov...@gmail.com
  Date: Wed, Aug 29, 2012 at 1:20 PM
  Subject: Re: [ast-developers] Fwd: Per thread open(), stat(), rename()
  and so on, and *at() API
  To: Lionel Cons lionelcons1...@googlemail.com
  Cc: Roland Mainz roland.ma...@nrubsig.org, ast-developers@research.att.com
 
 
  Lionel, there are 2 purposes:
 
  1. Allow that ksh93/libshell can be embedded in other applications
  (Roland calls this 'back end usage', i.e. the user is not aware that
  he is interacting with a shell, and neither do any other apis know
  that, or at least should not have to know about that), with out
  interfering the calling application by changing the current working
  directory.
  2. Allow that ksh93/libshell and the libast api can have a per thread
  current working directory.
 
  There are 2 proposals how to accomplish that:
  a) Add wrapper functions for open(), stat(), lstat(), chdir(),
  fchdir(), access(), eaccess(), rename() and so on, and redirect them
  to their *at() counterparts, with dirfd set to the current threads cwd
  fd.
 
  or
 
  b) Add wrapper cpp macros which redirect open(), stat(), lstat(),
  chdir(), fchdir(), access(), eaccess(), rename() and so on to their
  *at() counterparts, but define an extra macro which is AT_FDCWD for
  non threaded applications, and calls a function to obtain the current
  threads working directory fd for the current thread, but allows to
  override this macro on a per source file basis, so that applications
  like libshell, who maintain their own cwd directory fd, can use that.
 
  David and Glenn seems to prefer {a}, while I and Roland prefer {b},
  because {b} will allow an application to work as back end api without
  interfering with the global cwd or any other, may be libast based apis
  cwd. This includes nested usage of libshell, like for system() or
  wordexp().

 I'd prefer option {b}, too. Strongly (sorry David and Glenn :)).
 It sounds like the more sane and flexible way to do this. The big road
 block is that you have to find a solution for old platforms which do
 not have openat() and friends, or EOS (end-of-support) them (I'd
 support that if no platform younger than four years lacks openat() and
 friends).

 Ced
 -- 
 Cedric Blancher cedric.blanc...@googlemail.com
 Institute Pasteur

 ___
 ast-developers mailing list
 ast-developers@research.att.com
 https://mailman.research.att.com/mailman/listinfo/ast-developers

___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Glenn Fowler

one example of intercepting/renaming syscalls is ast.h already, and for a 
while,
translates all foo() syscalls to foo64() where foo64() is avaliable

On Wed, 29 Aug 2012 09:10:34 -0400 Glenn Fowler wrote:
 we're going to take some research leeway and investigate the implications of 
 {a}
 we're not strangers to doing stuff like that (3d(1))
 and we're not strangers to having research prove us wrong

 On Wed, 29 Aug 2012 14:49:33 +0200 Cedric Blancher wrote:
  On 29 August 2012 13:54, Roland Mainz roland.ma...@nrubsig.org wrote:
   [Forwarding since ATT's spam filter is on drugs... ;-(( ]
   -- Forwarded message --
   From: ÏÌØÇÁ ËÒÙÖÁÎÏ×ÓËÁÑ olga.kryzhanov...@gmail.com
   Date: Wed, Aug 29, 2012 at 1:20 PM
   Subject: Re: [ast-developers] Fwd: Per thread open(), stat(), rename()
   and so on, and *at() API
   To: Lionel Cons lionelcons1...@googlemail.com
   Cc: Roland Mainz roland.ma...@nrubsig.org, 
   ast-developers@research.att.com
  
  
   Lionel, there are 2 purposes:
  
   1. Allow that ksh93/libshell can be embedded in other applications
   (Roland calls this 'back end usage', i.e. the user is not aware that
   he is interacting with a shell, and neither do any other apis know
   that, or at least should not have to know about that), with out
   interfering the calling application by changing the current working
   directory.
   2. Allow that ksh93/libshell and the libast api can have a per thread
   current working directory.
  
   There are 2 proposals how to accomplish that:
   a) Add wrapper functions for open(), stat(), lstat(), chdir(),
   fchdir(), access(), eaccess(), rename() and so on, and redirect them
   to their *at() counterparts, with dirfd set to the current threads cwd
   fd.
  
   or
  
   b) Add wrapper cpp macros which redirect open(), stat(), lstat(),
   chdir(), fchdir(), access(), eaccess(), rename() and so on to their
   *at() counterparts, but define an extra macro which is AT_FDCWD for
   non threaded applications, and calls a function to obtain the current
   threads working directory fd for the current thread, but allows to
   override this macro on a per source file basis, so that applications
   like libshell, who maintain their own cwd directory fd, can use that.
  
   David and Glenn seems to prefer {a}, while I and Roland prefer {b},
   because {b} will allow an application to work as back end api without
   interfering with the global cwd or any other, may be libast based apis
   cwd. This includes nested usage of libshell, like for system() or
   wordexp().

  I'd prefer option {b}, too. Strongly (sorry David and Glenn :)).
  It sounds like the more sane and flexible way to do this. The big road
  block is that you have to find a solution for old platforms which do
  not have openat() and friends, or EOS (end-of-support) them (I'd
  support that if no platform younger than four years lacks openat() and
  friends).

  Ced
  -- 
  Cedric Blancher cedric.blanc...@googlemail.com
  Institute Pasteur

  ___
  ast-developers mailing list
  ast-developers@research.att.com
  https://mailman.research.att.com/mailman/listinfo/ast-developers

 ___
 ast-developers mailing list
 ast-developers@research.att.com
 https://mailman.research.att.com/mailman/listinfo/ast-developers

___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Cedric Blancher
On 29 August 2012 15:10, Glenn Fowler g...@research.att.com wrote:

 we're going to take some research leeway and investigate the implications of 
 {a}
 we're not strangers to doing stuff like that (3d(1))
 and we're not strangers to having research prove us wrong

What's wrong with {b}? It sounds like the easier and more flexible
solution - {a} adds a lot of functions and macros and {b} only adds
macros and a single function.

Ced
-- 
Cedric Blancher cedric.blanc...@googlemail.com
Institute Pasteur
___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Cedric Blancher
On 29 August 2012 15:13, Glenn Fowler g...@research.att.com wrote:

 one example of intercepting/renaming syscalls is ast.h already, and for a 
 while,
 translates all foo() syscalls to foo64() where foo64() is avaliable

I know, but you do that with #define like Olga's proposal. Adding more
functions sounds bad since a lot of platforms have a hefty penalty for
function calls with more than two arguments.

Ced
-- 
Cedric Blancher cedric.blanc...@googlemail.com
Institute Pasteur
___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Lionel Cons
On 29 August 2012 14:35, Cedric Blancher cedric.blanc...@googlemail.com wrote:
 On 29 August 2012 10:01, Roland Mainz roland.ma...@nrubsig.org wrote:
 [Forwarding email since somehow the ast-developers@ list has issues
 with postings from Olga's address... ;-( ]
 -- Forwarded message --
 From: ольга крыжановская olga.kryzhanov...@gmail.com
 Date: Wed, Aug 29, 2012 at 7:36 AM
 Subject: Fwd: Per thread open(), stat(), rename() and so on, and *at() API
 To: Roland Mainz roland.ma...@nrubsig.org


 Wenn du wach bist schick es bitte nochmal an
 ast-developers@research.att.com. Meine mail ist im spam-filter
 hangengeblieben.
 -- Forwarded message --
 From: ольга крыжановская olga.kryzhanov...@gmail.com
 Date: Wed, Aug 29, 2012 at 6:54 AM
 Subject: Per thread open(), stat(), rename() and so on, and *at() API
 To: Glenn Fowler g...@research.att.com, David Korn
 d...@research.att.com, Phong Vo k...@research.att.com
 Cc: ast-developers@research.att.com


 Glenn, I have an alternative for your per thread open(), stat(),
 rename() and so on API plans:

 Instead of creating AST wrapper code for open(), stat(), rename() and
 so we could use the following macros, defined in ast/fcntl.h:

 --cutme
 #define unlink(path)
 unlinkat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), 0)
 #define rmdir(path)
 unlinkat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), AT_REMOVEDIR)
 #define chown(path, uid,
 gid)fchownat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (uid), 
 (gid),
 0)
 #define lchown(path, uid, gid)
 fchownat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (uid), (gid),
 AT_SYMLINK_NOFOLLOW)
 #define stat(path, sb)  
 fstatat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (sb), 0)
 #define lstat(path, sb) 
 fstatat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (sb), AT_SYMLINK_NOFOLLOW)
 #define rename(oldname,
 newname)renameat(AST_GET_CURRENT_THREAD_CWD_FD(), (oldname),
 AST_GET_CURRENT_THREAD_CWD_FD(), (newname))
 #define access(path,
 amode)  faccessat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (amode), 
 0)
 #define eaccess(path,
 amode)  faccessat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (amode),
 AT_EACCESS)
 #define mkdir(path, amode)  
 mkdirat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (amode))
 #define mkfifo(path, amode)
 mkfifoat(AST_GET_CURRENT_THREAD_CWD_FD(),
 (path), (amode))
 #define mknod(path, amode,
 adev)   mknodat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (amode),
 (adev))
 #define readlink(path, buf,
 bufsize)readlinkat(AST_GET_CURRENT_THREAD_CWD_FD(), (path), (buf),
 (bufsize))
 #define symlink(oldpath, newpath)   symlinkat((oldpath),
 AST_GET_CURRENT_THREAD_CWD_FD(), (newpath))
 --cutme

 AST_GET_CURRENT_THREAD_CWD_FD() would be defined to be a preprocessor
 macro, which will be a function returning the current thread's cwd fd;
 if there is none allocated for this thread yet it will be done by
 within that function, using open(., O_search).

 However, consumers of the ast/fcntl.h header are _free_ to redefine
 this macro, for example src/cmd/ksh93/include/defs.h could define
 AST_GET_CURRENT_THREAD_CWD_FD() as
 #define AST_GET_CURRENT_THREAD_CWD_FD() (shp-pwdfd)
 This would, at least, save the function call overhead, and finally
 make use of (shp-pwdfd) as it was intended.

 What do you think? It would make things easier for you to implement,
 i.e. you only have to implement the per thread
 AST_GET_CURRENT_THREAD_CWD_FD(), and applications are free to use the
 *at() API if they wish to.

 What do you intend to do for platforms which do not have the POSIX
 openat() and friends API? Old Solaris, old Linux comes in mind (not
 that I care about any OS older than four years)...

Search your inbox for emails with the title [ast-developers]
|*at()|-emulation, 3rd prototype It contains emulation functions
for all *at() calls, using the old POSIX functions to emulate them.

Lionel

___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Lionel Cons
On 29 August 2012 15:10, Glenn Fowler g...@research.att.com wrote:

 we're going to take some research leeway and investigate the implications of 
 {a}
 we're not strangers to doing stuff like that (3d(1))
 and we're not strangers to having research prove us wrong

Well, if you use {a} then please remove the mamfile code which
generates libshell.so - libshell.a is sufficient to build ksh. So far
a shared library of libshell doesn't make sense until it gets actually
usable for embedding. Right now it doesn't work for that purpose and
if you pick {a} over {b} then I don't see a chance that it'll get any
better anytime soon.

Lionel
___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Glenn Fowler

On Wed, 29 Aug 2012 15:14:47 +0200 Cedric Blancher wrote:
 On 29 August 2012 15:10, Glenn Fowler g...@research.att.com wrote:
 
  we're going to take some research leeway and investigate the implications 
  of {a}
  we're not strangers to doing stuff like that (3d(1))
  and we're not strangers to having research prove us wrong

 What's wrong with {b}? It sounds like the easier and more flexible
 solution - {a} adds a lot of functions and macros and {b} only adds
 macros and a single function.

lets add threads to the standard
hmm, resolving conflicts between process global resources and threads is too 
hard,
let that be handled in user space (e.g., 1Mi different ways)
oh, what about pwd
lets add ~100 new syscalls to handle that
oh, and *at()ify the library apis too -- stdio, opendir, did I forget anything?
 ...
every new resource conflict seems to result in a flurry of new syscalls
and the old pardigms get thrown out
what if there were solutions that preserved some of the old paradigms?

or
*at() is not a solution to the thread problem
unless every stinking api ever written that deals with pathnames has an *at() 
variant
otherwise how do you get them to agree what the thread pwd is?
seems like a perfect place for standardization
but the standards gave us *at() instead

now, ranting a bit less
suppose you have n threads each with its own thread-pwd
how do those threads spawn processes, each with the thread-pwd of the parent?

___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Glenn Fowler

On Wed, 29 Aug 2012 15:23:25 +0200 Lionel Cons wrote:
 On 29 August 2012 15:10, Glenn Fowler g...@research.att.com wrote:
 
  we're going to take some research leeway and investigate the implications 
  of {a}
  we're not strangers to doing stuff like that (3d(1))
  and we're not strangers to having research prove us wrong

 Well, if you use {a} then please remove the mamfile code which
 generates libshell.so - libshell.a is sufficient to build ksh. So far
 a shared library of libshell doesn't make sense until it gets actually
 usable for embedding. Right now it doesn't work for that purpose and
 if you pick {a} over {b} then I don't see a chance that it'll get any
 better anytime soon.

I'm not sure if you are pointing out problems with bootstrap build from source
or libshell.so built the right way with ast nmake -- the ksh93 Mamfile has
no actions for generating shared libs -- it does have the capability to set
cc -c options to generate .o's suitable for use in generating a shared lib

if you plan on using ast shared libs and plugins then you must build with at 
least ast-base
this will build nmake first and then build everything else with nmake
its nmake that figures out the native incantations for building shared libs and 
dlls

linux is not the only target
there are uses for shared/dll libshell
PACKAGE_OPTIONS=optimize-space uses it for a smaller ksh a.out footprint
and uwin builds shell.dll by default

___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Lionel Cons
On 29 August 2012 15:41, Glenn Fowler g...@research.att.com wrote:

 On Wed, 29 Aug 2012 15:23:25 +0200 Lionel Cons wrote:
 On 29 August 2012 15:10, Glenn Fowler g...@research.att.com wrote:
 
  we're going to take some research leeway and investigate the implications 
  of {a}
  we're not strangers to doing stuff like that (3d(1))
  and we're not strangers to having research prove us wrong

 Well, if you use {a} then please remove the mamfile code which
 generates libshell.so - libshell.a is sufficient to build ksh. So far
 a shared library of libshell doesn't make sense until it gets actually
 usable for embedding. Right now it doesn't work for that purpose and
 if you pick {a} over {b} then I don't see a chance that it'll get any
 better anytime soon.

 I'm not sure if you are pointing out problems with bootstrap build from source
 or libshell.so built the right way with ast nmake -- the ksh93 Mamfile has
 no actions for generating shared libs -- it does have the capability to set
 cc -c options to generate .o's suitable for use in generating a shared lib

The problem is not packaging. I was trying to make a pun on the
libshell API. It's just useless for anything except using it for
writing a shell.

The pun was: If it useless, why ship a shared library version of libshell? Why?

I say that because the usage of chdir() in libshell is a pain in the
arse since other APIs may want to do the same (in other threads,
libshell just running and used in the main thread) at the same time.
We have that problem for example with the Gnome and KDE file selection
dialogue which treat the cwd as their own property. Add libshell with
another chdir() at the same time and you have a nice, confused
application which bites itself into it's own arse. Add rm to the mix
and the system gets bitten into its arse.

That's why I like Olga's option {b}. It is a step forward into the
right direction and even covers the case that someone might want to
use libshell in a nested way (system() and wordexp() were given as
examples).

Lionel
___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread David Korn
Subject: Re: Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and 
so  on, and *at() API


 What is the purpose of the change? Wouldn't it be easier to just to
 make a sweep of the codebase using an awk script and replace the
 functions with the new functions, instead of adding yet another
 #define for standard filesystem calls? I think the current number of
 cpp redirects for such calls has reached a number of nesting levels
 which has become unsustainable.
 
 Lionel
 
 

This would not allow the code to work on systems that don't support the
@ functions.

Also, people who have added builtins would have to change their code
as well.

Our solution will make them functions not macros.

David Korn
d...@research.att.com
___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread David Korn
Subject: Re: Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and 
so  on, and *at() API


 I know, but you do that with #define like Olga's proposal. Adding more
 functions sounds bad since a lot of platforms have a hefty penalty for
 function calls with more than two arguments.
 
 Ced
 

In this case the penalty is dwarfed by the time for the system call itself
since each function wraps a system call.

David Korn
d...@research.att.com
___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread David Korn
Subject: Re: Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and 
so  on, and *at() API



 Well, if you use {a} then please remove the mamfile code which
 generates libshell.so - libshell.a is sufficient to build ksh. So far
 a shared library of libshell doesn't make sense until it gets actually
 usable for embedding. Right now it doesn't work for that purpose and
 if you pick {a} over {b} then I don't see a chance that it'll get any
 better anytime soon.
 
 Lionel
 

It does work for embedding.  It is embedded in tksh.

David Korn
d...@research.att.com
___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Irek Szczesniak
On Wed, Aug 29, 2012 at 3:10 PM, Glenn Fowler g...@research.att.com wrote:

 we're going to take some research leeway and investigate the implications of 
 {a}

Glenn, this is supposed to be my free month. Holiday. I'm only
intervening because I see DOOM coming straight ahead.
Your solution of having an inaccessible per thread cwd is the solution
Windows tried. They FAILED. Please do not repeat that history.

 we're not strangers to doing stuff like that (3d(1))
 and we're not strangers to having research prove us wrong

Microsoft already tried option {a} in Windows as convenient, backwards
compatible and easy path for developers. I think everyone agreed it
was the worst possible solution - a lesson hard learned in a decade of
problems spawned from that. By overall measure, you're making libast
impossible to use for outside projects with that because you can no
longer mix libast calls safely with calls from other utility
libraries.

Irek
___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Glenn Fowler

On Wed, 29 Aug 2012 15:55:15 +0200 Lionel Cons wrote:
 On 29 August 2012 15:41, Glenn Fowler g...@research.att.com wrote:
 
  On Wed, 29 Aug 2012 15:23:25 +0200 Lionel Cons wrote:
  On 29 August 2012 15:10, Glenn Fowler g...@research.att.com wrote:
  
   we're going to take some research leeway and investigate the 
   implications of {a}
   we're not strangers to doing stuff like that (3d(1))
   and we're not strangers to having research prove us wrong
 
  Well, if you use {a} then please remove the mamfile code which
  generates libshell.so - libshell.a is sufficient to build ksh. So far
  a shared library of libshell doesn't make sense until it gets actually
  usable for embedding. Right now it doesn't work for that purpose and
  if you pick {a} over {b} then I don't see a chance that it'll get any
  better anytime soon.
 
  I'm not sure if you are pointing out problems with bootstrap build from 
  source
  or libshell.so built the right way with ast nmake -- the ksh93 Mamfile has
  no actions for generating shared libs -- it does have the capability to set
  cc -c options to generate .o's suitable for use in generating a shared lib

 The problem is not packaging. I was trying to make a pun on the
 libshell API. It's just useless for anything except using it for
 writing a shell.

 The pun was: If it useless, why ship a shared library version of libshell? 
 Why?

 I say that because the usage of chdir() in libshell is a pain in the
 arse since other APIs may want to do the same (in other threads,
 libshell just running and used in the main thread) at the same time.
 We have that problem for example with the Gnome and KDE file selection
 dialogue which treat the cwd as their own property. Add libshell with
 another chdir() at the same time and you have a nice, confused
 application which bites itself into it's own arse. Add rm to the mix
 and the system gets bitten into its arse.

 That's why I like Olga's option {b}. It is a step forward into the
 right direction and even covers the case that someone might want to
 use libshell in a nested way (system() and wordexp() were given as
 examples).

the theory is that although any threaded app compiled with ast.h
may look like its using chdir()/fchdir() at the process global level will
in effect be using them at the thread local level in a thread safe manner

other libs could the real chdir()/fchdir() it in a global non-thread-safe
way and not affect the state of the ast.h. compiled code

___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers


Re: [ast-developers] Fwd: Per thread open(), stat(), rename() and so on, and *at() API

2012-08-29 Thread Roland Mainz
On Wed, Aug 29, 2012 at 6:44 PM, Glenn Fowler g...@research.att.com wrote:

 On Wed, 29 Aug 2012 15:55:15 +0200 Lionel Cons wrote:
 On 29 August 2012 15:41, Glenn Fowler g...@research.att.com wrote:
 
  On Wed, 29 Aug 2012 15:23:25 +0200 Lionel Cons wrote:
  On 29 August 2012 15:10, Glenn Fowler g...@research.att.com wrote:
  
   we're going to take some research leeway and investigate the 
   implications of {a}
   we're not strangers to doing stuff like that (3d(1))
   and we're not strangers to having research prove us wrong
 
  Well, if you use {a} then please remove the mamfile code which
  generates libshell.so - libshell.a is sufficient to build ksh. So far
  a shared library of libshell doesn't make sense until it gets actually
  usable for embedding. Right now it doesn't work for that purpose and
  if you pick {a} over {b} then I don't see a chance that it'll get any
  better anytime soon.
 
  I'm not sure if you are pointing out problems with bootstrap build from 
  source
  or libshell.so built the right way with ast nmake -- the ksh93 Mamfile has
  no actions for generating shared libs -- it does have the capability to set
  cc -c options to generate .o's suitable for use in generating a shared lib

 The problem is not packaging. I was trying to make a pun on the
 libshell API. It's just useless for anything except using it for
 writing a shell.

 The pun was: If it useless, why ship a shared library version of libshell? 
 Why?

 I say that because the usage of chdir() in libshell is a pain in the
 arse since other APIs may want to do the same (in other threads,
 libshell just running and used in the main thread) at the same time.
 We have that problem for example with the Gnome and KDE file selection
 dialogue which treat the cwd as their own property. Add libshell with
 another chdir() at the same time and you have a nice, confused
 application which bites itself into it's own arse. Add rm to the mix
 and the system gets bitten into its arse.

 That's why I like Olga's option {b}. It is a step forward into the
 right direction and even covers the case that someone might want to
 use libshell in a nested way (system() and wordexp() were given as
 examples).

 the theory is that although any threaded app compiled with ast.h
 may look like its using chdir()/fchdir() at the process global level will
 in effect be using them at the thread local level in a thread safe manner

 other libs could the real chdir()/fchdir() it in a global non-thread-safe
 way and not affect the state of the ast.h. compiled code

Grumpf... I've discussed this with Olga (who can't really partake in
the discussion because the research.att.com spam filter is on illegal
drugs) ...
... one of the *many* problems with hidden per-thread globals is
that resources (like a |Shell_t*| object) created by one thread are
locked to that thread... which means they can no longer be borrowed
or moved to another thread without adding extra code or extra API...
something which should never be neccesary for clean, thread-enabled
API.

The 2nd issues are related to proper embedding: IMO it's very
desireable to wean libshell off from using the global cwd and
instead use an internal directory fd. The advantage is that a
|Shell_t*| object no longer interferes with other APIs using that
global cwd and we get closer to the goal of allowing _nested_ usage of
multiple |Shell_t*| objects in the _same_ thread, for example to
implement something like |system()| without actually creating a new
process. |system()| would create a new |Shell*t| (while the previous
instance waits), execute the code and returns to the caller. We can do
that easily with option {b} ... but not with option {a}.

Also... both options are very very similar to each other... that's why
I don't understand the fierce resistance from your side to try first
{b} and switch to {a} if that doesn't work (as said... I've written
(thread-safe (only mutex needs to be added)) |*at()| emulation code
which works on all platforms).



Bye,
Roland

-- 
  __ .  . __
 (o.\ \/ /.o) roland.ma...@nrubsig.org
  \__\/\/__/  MPEG specialist, CJAVASunUnix programmer
  /O /==\ O\  TEL +49 641 3992797
 (;O/ \/ \O;)
___
ast-developers mailing list
ast-developers@research.att.com
https://mailman.research.att.com/mailman/listinfo/ast-developers