Re: system long commands and stderr redirection.
There is an alternative to execvp and lib$spawn; this involves going to a lower level of call - to the underlying VMS system services. This also overcomes the line length problem. I've used sys$creprc. You loose argv,argc,envp end everything else taken for granted with crtl. It would be alot of work to use it with crtl based applications.
Re: system long commands and stderr redirection.
Douglas B Rupp wrote: Very interesting. So gcc on VMS is under active development? Absolutely. My company is working on the IA64 VMS port of GNAT under contract to HP. GNAT as you may know, is the Gnu Ada compiler and Gnu C is required to build the backend. I don't really recommend Gnu C for serious development on VMS, since its not compatible with the standard DEC C header files. Only the ones needed for building GNAT and bootstrapping Gnu C have been ported. If such information can be made available publicly, One of the projects on my list is to build a GCC 6809 cross complier that runs on OpenVMS. The one that I currently have only works under cygwin. The latter version uses Perl. So far I've run into two problems with the Perl system function. One is stderr can't be redirected properly, Define properly. What do you want it to do and what does it do instead? For example this command: system $m4 --help /dev/null 21 | grep reload-state /dev/null; writes the help output to the screen, when it should be going to /dev/null. That may need to be handled by GNV, or HP C RTL folks as the redirection is being handled by m4, presumably a C program in GNV, not by the Perl system function. another is long command lines don't work. I suspect system is implemented with lib$spawn? Yes, system is implemented via lib$spawn, so as far as I know, there's no way around DCL command line limits. You can see the implementation in the Perl_do_spawn() function in [.vms]vms.c in the Perl sources. The line limits are increased in 8.2, do you have access to the pre-release code before the field test? You may want to contact HP to see if they can give you a backported DCL, if it is possible for it to run on an older version. I believe that a better solution would be for system to look at the program(s) being executed, and if they are CRTL based executables then vfork/exec should be used, otherwise falling back to lib$spawn. This is what my ports of bash and Gnu make do, and it works quite well. Are these separate from GNV? I used to have an anonymous ftp site and some of my Gnu tools for VMS made their way eventually to DEC-Compaq-HP and I think now reside on sourceforge.net. They cleaned up and rewrote the VMS specific piping in bash, and ended up breaking some things. I've sent them a few bug reports but I don't know if they ever got fixed. As far as I know, the current GNV is: 1.5-6. There is ongoing work on the make application and with pipes. I think that there is an e-mail contact for feedband on the GNV download page. I do not have time to look for it at the moment. You can join Encompasserve.org (free) and put files in the [.www] directory protected w:re, with the files protected also w:re and they will be available for download through a web server via: http://encompasserve.org/~username/ -John [EMAIL PROTECTED] Personal Opinion Only
Re: system long commands and stderr redirection.
For example this command: system $m4 --help /dev/null 21 | grep reload-state /dev/null; writes the help output to the screen, when it should be going to /dev/null. That may need to be handled by GNV, or HP C RTL folks as the redirection is being handled by m4, presumably a C program in GNV, not by the Perl system function. I think that is wrong. The Perl system function should handle redirection that same on Unix as it does on VMS. I was able to work around the problem by rewriting the above as the following: system bash -c \$m4 --help /dev/null 21 | grep reload-state /dev/null\; which demonstrates the problem doesn't need to be handled by m4. another is long command lines don't work. I suspect system is implemented with lib$spawn? Yes, system is implemented via lib$spawn, so as far as I know, there's no way around DCL command line limits. You can see the implementation in the Perl_do_spawn() function in [.vms]vms.c in the Perl sources. The line limits are increased in 8.2, do you have access to the pre-release code before the field test? I think that is also wrong. The method I described handles virtually unlimited command line lengths. Unless the DCL command line length liimit has been completely removed (which I find very unlikely but maybe worth asking about), it won't be enough for other than trivial scripts.
Re: system long commands and stderr redirection.
The fact that we always use lib$spawn and never use execvp() like the docs say we should do is a separate but related problem. Switching to use execvp() when we can would not, as far as I can tell, help with redirection. There is nothing in the CRTL docs about redirection operators being supported as arguments to the exec() family. The actual redirection needs to be handled separately, that's true but using exec makes it at least possible. The code implementing redirection can be copied from bash. POSIX specifies a 2K limit for the shell command buffer, though I believe many unices support more or even have no concrete limit. I have no idea what bash on VMS supports. The cases you are talking about that can be done with the exec() family probably can bypass that limit, but strictly speaking they aren't really commands; they are images invoked with C-style argv arrays passed to them. Distinguishing between when you can use exec() and when not strikes me as a bit tricky; it's hard to parse a shell command (or determine whether it even is a shell command) without using the command shell to do it. A perfectly reasonable solution would be to provide an alternate version of system that uses exec, and let the user decide which one to use.
Re: system long commands and stderr redirection.
Douglas B Rupp wrote: For example this command: system $m4 --help /dev/null 21 | grep reload-state /dev/null; writes the help output to the screen, when it should be going to /dev/null. That may need to be handled by GNV, or HP C RTL folks as the redirection is being handled by m4, presumably a C program in GNV, not by the Perl system function. I think that is wrong. The Perl system function should handle redirection that same on Unix as it does on VMS. I was able to work around the problem by rewriting the above as the following: While I do not know what the Perl system function is doing, it would look to me that it is simply passing the command through as a DCL command, and as such is not attempting to interpret the redirection codes at all. system bash -c \$m4 --help /dev/null 21 | grep reload-state /dev/null\; which demonstrates the problem doesn't need to be handled by m4. I will have to confess that I know little of bash, m4, or perl. However, it now appears that the redirection is being done by bash instead of md4 or dcl. The system function appears to assume that the underlying operation system will understand the quoted command, and makes no attempt to process it. The question is is if the command: m4 --help /dev/null 21 | grep reload-state /dev/null will work under DCL. The behavior seems to be explained in these comments in VMS.C. * Note on command arguments to perl 'exec' and 'system': When handled * in 'VMSish fashion' (i.e. not after a call to vfork) The args * are concatenated to form a DCL command string. If the first arg * begins with '$' (i.e. the perl script had \$ Type or some such), * the command string is handed off to DCL directly. Otherwise, * the first token of the command is taken as the filespec of an image * to run. The filespec is expanded using a default type of '.EXE' and * the process defaults for device, directory, etc., and if found, the resultant * filespec is invoked using the DCL verb 'MCR', and passed the rest of * the command string as parameters. This is perhaps a bit complicated, * but I hope it will form a happy medium between what VMS folks expect * from lib$spawn and what Unix folks expect from exec. So it appears one way to implement what you may need is a modification for Perl for VMS to use the BASH shell instead of DCL, or to run the program with out a shell at all. Maybe one of the Perl maintainers can figure out a patch. Perl can detect if it is running under bash and should use it instead DCL by looking at the environment variable SHELL to see if perl is being run under bash another is long command lines don't work. I suspect system is implemented with lib$spawn? Yes, system is implemented via lib$spawn, so as far as I know, there's no way around DCL command line limits. You can see the implementation in the Perl_do_spawn() function in [.vms]vms.c in the Perl sources. The line limits are increased in 8.2, do you have access to the pre-release code before the field test? I think that is also wrong. The method I described handles virtually unlimited command line lengths. Unless the DCL command line length liimit has been completely removed (which I find very unlikely but maybe worth asking about), it won't be enough for other than trivial scripts. The DCL line length has not been completely removed. And i was in error. The new line lengths are in 7.3-2. A line can now be 4095 bytes in length. Perl has an internal limit of (4*MAX_DCL_SYMBOL-4) for the line lengths and since MAX_DCL_SYMBOL is 255, this limits it to 1016 characters. With VMS 7.3-2 and later on Alpha/IA64 DCL symbols are limited to 8192 bytes, and previously they are limited to 1024 bytes. As I do not know the history of the perl limitations, I do not know what would have to be changed to have them take advantage of the new size. -John [EMAIL PROTECTED] Personal Opinion Only
Re: system long commands and stderr redirection.
Craig A. Berry wrote: The way I see the problem is that we are running the command under the wrong shell (DCL instead of bash). One possibility here would be to have Perl detect whether it is running under bash and if it is, then prepend bash to the command for system(), exec(), and backtick operations. That would be looking at the environment variable SHELL and using it's value if it is present. If it is not present, fall back to the current behavior. Better yet, if there were some way to make bash a true-blue CLI from a VMS standpoint, then lib$spawn would automatically use whatever CLI is in use by the parent. That's out of my league and I have no idea how possible or difficult that would be. I do not know either. I do not think that making it a true CLI is high on the current priority lists. Bash currently uses DCL in a fallback mode, and so it would have to be able to detect if DCL was not there. It may already be able to do that. -John [EMAIL PROTECTED] Personal Opinion Only
Re: system long commands and stderr redirection.
However, it now appears that the redirection is being done by bash instead of md4 or dcl. In my latter example, that is correct. However I think it wrong to require a bash -c wrapper to achieve proper redirection. Perl is a shell, or at least a wannabe shell and should handle redirection internally to be maximally useful. Consider the case of a long command line requiring this redirection. Even with a bash -c wrapper, it can't work because the lib$spawn will overflow trying to call bash.
Re: system long commands and stderr redirection.
Douglas B Rupp wrote: However, it now appears that the redirection is being done by bash instead of md4 or dcl. In my latter example, that is correct. However I think it wrong to require a bash -c wrapper to achieve proper redirection. Perl is a shell, or at least a wannabe shell and should handle redirection internally to be maximally useful. Consider the case of a long command line requiring this redirection. Even with a bash -c wrapper, it can't work because the lib$spawn will overflow trying to call bash. I disagree. Perl is not a shell, it is a scripting language. It is no more a shell than a C compiler or a BASIC interpreter is. One of the security weaknesses in Unix is the presense of shell metacharacters that can completely alter the function of the command and the predilection of programmers to simply pass user input onto things like the system function call. Because of this, programs have been made to do things completely different from what the programmer intended. DCL shares very little of this weakness. If a program calls system with a command using user supplied arguments, the program can be assured that the command issued is the one that will be executed. The user supplied arguments cannot alter what verb got executed. If you wish a script to alter the way a command is normally interpreted then it is my opinion that it is the job of your script to make the necessary adjustments (e.g. prepend PIPE to the command; prepend bash -c to the command; etc.). The system function is documented as simply passing a string to the host environment to be executed however the host interprets the command. If you wish to create a new Perl function that means evaluate this string as if it were issued on a Unix system and then set up an environment and execute it accordingly please feel free to do so. But I'd be opposed to modifying the expectations or operation of system. Mark Berryman Mark Berryman
Re: system long commands and stderr redirection.
If you wish to create a new Perl function that means evaluate this string as if it were issued on a Unix system and then set up an environment and execute it accordingly please feel free to do so. But I'd be opposed to modifying the expectations or operation of system. As I said previously, that would be a perfectly acceptable solution. But as it is now Perl isn't up the to task of running scripts on VMS for my application.
Re: system long commands and stderr redirection.
At 9:52 PM -0700 4/21/04, Douglas B Rupp wrote: I recently joined the mailing list. I did a quick search of the archive but couldn't find any relevant information on the problem. Welcome. Recently Gcc changed from requiring autoconf version 2.13 to version 2.57. Very interesting. So gcc on VMS is under active development? The latter version uses Perl. So far I've run into two problems with the Perl system function. One is stderr can't be redirected properly, Define properly. What do you want it to do and what does it do instead? another is long command lines don't work. I suspect system is implemented with lib$spawn? Yes, system is implemented via lib$spawn, so as far as I know, there's no way around DCL command line limits. You can see the implementation in the Perl_do_spawn() function in [.vms]vms.c in the Perl sources. I believe that a better solution would be for system to look at the program(s) being executed, and if they are CRTL based executables then vfork/exec should be used, otherwise falling back to lib$spawn. This is what my ports of bash and Gnu make do, and it works quite well. That's intriguing. I take it by a CRTL based executable you mean one that calls DECC$CRTL_INIT? How do you determine that at run time? Also, you seem to be suggesting that vfork/exec do not use lib$spawn. Any idea how they create a subprocess if they don't do that? Please let me know if perhaps this has already been implemented, or if someone has a patch that does this, or failing that suggestions or opinions on how easy this would be to do. Patches are welcome, but an overhaul of system() does not sound particularly easy, so I would like to understand a bit more about how your approach works. Are the bash or make implementations you mention above open source? -- Craig A. Berry mailto:[EMAIL PROTECTED] ... getting out of a sonnet is much more difficult than getting in. Brad Leithauser
Re: system long commands and stderr redirection.
Very interesting. So gcc on VMS is under active development? Absolutely. My company is working on the IA64 VMS port of GNAT under contract to HP. GNAT as you may know, is the Gnu Ada compiler and Gnu C is required to build the backend. I don't really recommend Gnu C for serious development on VMS, since its not compatible with the standard DEC C header files. Only the ones needed for building GNAT and bootstrapping Gnu C have been ported. The latter version uses Perl. So far I've run into two problems with the Perl system function. One is stderr can't be redirected properly, Define properly. What do you want it to do and what does it do instead? For example this command: system $m4 --help /dev/null 21 | grep reload-state /dev/null; writes the help output to the screen, when it should be going to /dev/null. another is long command lines don't work. I suspect system is implemented with lib$spawn? Yes, system is implemented via lib$spawn, so as far as I know, there's no way around DCL command line limits. You can see the implementation in the Perl_do_spawn() function in [.vms]vms.c in the Perl sources. I believe that a better solution would be for system to look at the program(s) being executed, and if they are CRTL based executables then vfork/exec should be used, otherwise falling back to lib$spawn. This is what my ports of bash and Gnu make do, and it works quite well. That's intriguing. I take it by a CRTL based executable you mean one that calls DECC$CRTL_INIT? How do you determine that at run time? There are a couple of ways to check, by looking at the location of the executable (e.g. /gnu/bin) or by looking at some bits in the header. That's the easy part. Also, you seem to be suggesting that vfork/exec do not use lib$spawn. Any idea how they create a subprocess if they don't do that? Exec uses lib$spawn but CRTL also sets up some piping infrastructure. The basic idea is to do redirect stdin, stderr, stdout as needed to temporary files to read and write the command line as needed for communicating it to the next process in the pipeline. The temporary files can be created on a ramdisk to give reasonable performance. Unfortunately its not possible to uses pipes directly due to the semantics of vfork on VMS. It gets kind of messy but I can give you the code from bash that does it. Patches are welcome, but an overhaul of system() does not sound particularly easy, so I would like to understand a bit more about how your approach works. Are the bash or make implementations you mention above open source? I used to have an anonymous ftp site and some of my Gnu tools for VMS made their way eventually to DEC-Compaq-HP and I think now reside on sourceforge.net. They cleaned up and rewrote the VMS specific piping in bash, and ended up breaking some things. I've sent them a few bug reports but I don't know if they ever got fixed.
