Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Jeremy Begg]

Hi Craig,


Is it possible using the CRTL?  I'm not sure it's reasonable to expect
Perl to do better than C can do.  Here's what seems to me like the
relevant section from the CRTL manual:


No, it's not reasonable to expect Perl to do better than CRTL.  I hadn't
consulted the CRTL manual before making my comments.


When I'm writing programs I assume that each 'print' statement will
result
in one output record.  I susppose you could arrange for each 'print'
to be
followed by a flush to disk at that point, but of course performance
would
probably suffer.


What about printing a list?  Should each list item produce a new
record, or should all elements be coalesced into a single (potentially
humongous) write because they all came from one print statement?  I'm
not sure a print statement that can function as a list operator maps
very well onto your assumption; Perl ain't Fortran.


Fair enough!

Jeremy

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Craig A. Berry]

On Nov 16, 2010, at 5:02 PM, Jeremy Begg wrote:

Thanks for the feedback, Jeremy.


I'm assuming that if the record format is FAB$C_VAR or FAB$C_VFC, the
records will never contain binary data with embedded newlines.  Is
that true?   What other assumptions am I making that I shouldn't?


The great advantage of the VAR and VFC formats (compared to the  
various
Stream formats) is that you *can* have embedded newlines and other  
non-

printable characters in the record.  So it's quite conceivable that a
program might right "binary" data to a VAR or VFC file.  On the other
hand it would not be common Perl practice to do so.


Is it possible using the CRTL?  I'm not sure it's reasonable to expect  
Perl to do better than C can do.  Here's what seems to me like the  
relevant section from the CRTL manual:


=
1.8.2.2.1 Accessing Variable-Length or VFC Record Files in Record Mode

When you access a variable-length or VFC record file in record mode,  
many I/O
functions behave differently than they would if they were being used  
with stream

mode. This section describes these differences.

In general, the new-line character (\n) is the record separator for  
all record

modes. On output, when a new-line character is encountered, a record is
generated unless you specify an optional argument (such as "ctx=bin" or
"ctx=xplct") that affects the interpretation of new lines.
=

I'm pretty sure what I'm proposing will do in Perl exactly what the  
CRTL says it will do, but, as always, I'm grateful when people catch  
my mistakes before I'm finished making them.


FAB$L_CTX is not available in the stat buffer and thus would be  
difficult to get at, and in fact we know in this case what the open  
call looked like and that it did not include "ctx=bin".  I think the  
following demonstrates that you cannot embed newlines in a record  
using unix I/O from C, at least without ctx=bin:


$ type record_test.c
#include 
#include 
#include 
#include 

main()
{
int file;
int flags;

flags = O_RDWR | O_CREAT;

file = open("record_test.dat", flags, 0, "rfm=var");
if (file == -1)
perror("open failed"), exit(1);

if (write(file, "This record contains tab (\t) and newline(\n) in  
the middle\n", 58) < 0)

perror("write failed"), exit(1);

close(file);
}
$ cc record_test
$ link record_test
$ r record_test
$ type record_test.dat
This record contains tab (  ) and newline(
  ) in the middle

$ dump/record record_test.dat

Dump of file D0:[CRAIG.TEST]RECORD_TEST.DAT;1 on 16-NOV-2010 22:08:19.68
File ID (116860,63,0)   End of file block 1 / Allocated 33

Record number 1 (0001), 42 (002A) bytes, RFA(0001,,)

 646E6120 29092820 62617420 736E6961 746E6F63 2064726F 63657220  
73696854 This record contains tab (.) and 00
  0A28 656E696C  
77656E20  newline(... 20


Record number 2 (0002), 16 (0010) bytes, RFA(0001,,002C)

 0A656C64 64696D20 65687420  
6E692029 ) in the middle. 00



When I'm writing programs I assume that each 'print' statement will  
result
in one output record.  I susppose you could arrange for each 'print'  
to be
followed by a flush to disk at that point, but of course performance  
would

probably suffer.



What about printing a list?  Should each list item produce a new  
record, or should all elements be coalesced into a single (potentially  
humongous) write because they all came from one print statement?  I'm  
not sure a print statement that can function as a list operator maps  
very well onto your assumption; Perl ain't Fortran.


Cheers,
  Craig

Craig A. Berry
mailto:craigbe...@mac.com

"... getting out of a sonnet is much more
 difficult than getting in."
 Brad Leithauser

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Jeremy Begg]

Hi Craig,

I saw this a week or so ago but didn't have time to respond earlier ... I
hope this message is still useful.


This is right after the perlio layer has called down to the unixio
layer to get the file open.  We have an fd, so we can do an fstat() on
that and retrieve the record format from the VMS-specific bits of the
stat structure.  Then I check to see if it's a regular file (not a
device like a mailbox that may need to carry binary data) and that the
record format is either variable or variable with fixed control.  If
these conditions are met, I enable the line buffering option on that
filehandle.


So far so good ...


I have tested this and it works for situations similar to Martin's
original report, and it does not introduce any new test failures in
the test suite.  But what situations, if any, does this break?  I'm
assuming that if the record format is FAB$C_VAR or FAB$C_VFC, the
records will never contain binary data with embedded newlines.  Is
that true?   What other assumptions am I making that I shouldn't?


The great advantage of the VAR and VFC formats (compared to the various
Stream formats) is that you *can* have embedded newlines and other non-
printable characters in the record.  So it's quite conceivable that a
program might right "binary" data to a VAR or VFC file.  On the other
hand it would not be common Perl practice to do so.

When I'm writing programs I assume that each 'print' statement will result
in one output record.  I susppose you could arrange for each 'print' to be
followed by a flush to disk at that point, but of course performance would
probably suffer.

Just my 2c worth.

Regards,

Jeremy Begg

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Craig A. Berry]

On Apr 21, 2010, at 5:29 PM, martin.zin...@deutsche-boerse.com wrote:

If you open a text file with Carriage return carriage control for  
output

(based off an existing file) and populate the new file with longer
records, at some point gratuitous
line breaks are added to the file.


Finally getting back to this after six months.  And I think I have a  
solution.  To review, what happens when you use the Perl "open"  
operator is that it calls into its own buffered I/O layer named  
"perlio" which sits on top of another layer called "unixio" which is  
implemented in terms of the CRTL read/write functions.  This  
arrangement was new in about 5.6 but became the default in 5.10, and  
that's where we started seeing the problem Martin describes on VMS.


The problem is that while the perlio layer is buffered, the unixio  
layer is not.  When the buffer in the perlio layer gets filled up, it  
triggers a flush to the lower layer.  The flush in the perlio layer  
causes a write() in the unixio layer, and when you do that you go all  
the way to disk, and if writing to a record-oriented file, you'll  
likely introduce an extra record boundary in the file unless you had  
the extreme good fortune to hit the end of a line at the same time you  
hit the end of the buffer.  Part of the problem is that the buffer in  
the perlio layer is hard-wired to 4K.  With a larger buffer, you would  
typically not see as many extra records, but you would still see them.


It turns out the perlio layer has some knobs and switches on it, and  
one of them is a "line buffering" option.  If this option is enabled,  
then the flush to the lower layer happens whenever a newline character  
appears in the data.  As long as your lines are shorter than the  
length of the buffer, you write them out whole, which empties the  
buffer in the upper layer making room for more data, and everything is  
peachy.


So, where and how to enable this line buffering?  Here's my proposed  
patch:


--- perlio.c;-0 2010-10-21 07:58:15 -0500
+++ perlio.c2010-11-02 21:32:41 -0500
@@ -3758,6 +3758,22 @@ PerlIOBuf_open(pTHX_ PerlIO_funcs *self,
 */
PerlLIO_setmode(fd, O_BINARY);
 #endif
+#ifdef VMS
+#include 
+   /* Enable line buffering with record-oriented regular  
files
+* so we don't introduce an extraneous record boundary  
when

+* the buffer fills up.
+*/
+   if (PerlIOBase(f)->flags & PERLIO_F_CANWRITE) {
+   Stat_t st;
+   if (PerlLIO_fstat(fd, &st) == 0
+   && S_ISREG(st.st_mode)
+   && (st.st_fab_rfm == FAB$C_VAR
+   || st.st_fab_rfm == FAB$C_VFC)) {
+   PerlIOBase(f)->flags |= PERLIO_F_LINEBUF;
+   }
+   }
+#endif
}
}
 }

[end]


This is right after the perlio layer has called down to the unixio  
layer to get the file open.  We have an fd, so we can do an fstat() on  
that and retrieve the record format from the VMS-specific bits of the  
stat structure.  Then I check to see if it's a regular file (not a  
device like a mailbox that may need to carry binary data) and that the  
record format is either variable or variable with fixed control.  If  
these conditions are met, I enable the line buffering option on that  
filehandle.


I have tested this and it works for situations similar to Martin's  
original report, and it does not introduce any new test failures in  
the test suite.  But what situations, if any, does this break?  I'm  
assuming that if the record format is FAB$C_VAR or FAB$C_VFC, the  
records will never contain binary data with embedded newlines.  Is  
that true?   What other assumptions am I making that I shouldn't?



Craig A. Berry
mailto:craigbe...@mac.com

"... getting out of a sonnet is much more
 difficult than getting in."
 Brad Leithauser

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Craig A. Berry]

On Apr 22, 2010, at 2:39 PM, martin.zin...@deutsche-boerse.com wrote:


Unfortunatly the suggested solution I really would like to implement
(since it requires not code/setup changes) does give me a headache.
Setting stdio as the standard during using configure.com does cause  
the

compilation of perl to fail.

Here is what I did:

* Unpack perl into a clean directory
* @configure
 ...
  --> Default answers for everything but
 Use the PerlIO abstraction layer? [y] n
 Ok, doing things the stdio way.


It should be possible to simply say:

$ @configure -"Uuseperlio" -"des"

However, that was broken until the following change:




But in any case, choosing to turn off perlio by answering the  
interactive question explicitly as you did was successful in producing  
the desired configuration.



CC/DECC/NOANSI_ALIAS
/Include=[]/Standard=Relaxed_ANSI/Prefix=All/Obj=.obj/NoList
/float=ieee/ieee=denorm/Define=(PERL_CORE,_USE_STD_STAT=1) VMS.C

&& PL_perlio_fd_refcnt)
^
%CC-E-UNDECLARED, In this statement, "PL_perlio_fd_refcnt" is not
declared.
at line number 3080 in file DSA0:[PUBLIC.GNU.PERLSTD]vms.c;1



As I noted earlier, that is (was) a bug.  It's now fixed in:




I did a build with perlio disabled and these patches in place.  Your  
original test now succeeds.  Mine doesn't because we get a new record  
for each print statement, which worked well for your case but not for  
mine.  That's because disabling perlio entirely disables any buffering  
so each print statement goes all the way down to fwrite().


It may be what we really need is instead of the default being perlio  
and its buffering on top of unix I/O (write() and friends), it should  
be perlio on top of stdio (fwrite() and friends) for record-oriented  
files and devices and perlio on top of unix I/O for everything else.   
This would require some understanding of how the PerlIO scheme chooses  
its base layers and also detection of whether we've got a record- 
oriented gizmo or not, probably by hooking a callback onto fopen() or  
fdopen().  Which sounds like real work.




Craig A. Berry
mailto:craigbe...@mac.com

"... getting out of a sonnet is much more
 difficult than getting in."
 Brad Leithauser

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Craig A. Berry]

On Apr 22, 2010, at 2:39 PM, martin.zin...@deutsche-boerse.com wrote:

First of all thanks a lot for your insightfull comments. A couple of  
the

suggestions do restore the previous behaviour, in particular

* define PERLIO "stdio"
* use open OUT => ':stdio';
* open(TXT, '>:stdio', 'test.lis');

all produce the same output as 5.8.x .

Setting stdio as the standard during using configure.com does cause  
the

compilation of perl to fail.



CC/DECC/NOANSI_ALIAS
/Include=[]/Standard=Relaxed_ANSI/Prefix=All/Obj=.obj/NoList
/float=ieee/ieee=denorm/Define=(PERL_CORE,_USE_STD_STAT=1) VMS.C

&& PL_perlio_fd_refcnt)
^
%CC-E-UNDECLARED, In this statement, "PL_perlio_fd_refcnt" is not
declared.
at line number 3080 in file DSA0:[PUBLIC.GNU.PERLSTD]vms.c;1


This is a bug.  Obviously the non-perlio case hasn't been tested  
recently.



Any further ideas?



I think we're just getting started on a proper analysis.  Looking at:



I believe on VMS we are getting the perlio layers "unix perlio" when  
what we want (and what happens on other platforms) is "unix stdio".   
Without the stdio layer we are not getting whatever RMS-aware magic is  
in the CRTL to try to Do The Right Thing with record-oriented files.   
At least that's my current working theory, which I haven't really  
proven yet.


I do have a simpler reproducer:

$ type rectest.pl
use strict;
use warnings;
use VMS::Stdio qw( vmsopen );

1 while unlink 'rectest.lis';

my $fh = vmsopen('>rectest.lis', 'rfm=var', 'rat=cr')
or die "Couldn't open rectest.lis: $!";

for (1..4097) { print $fh 'Z'; }

close $fh;

[end]

What you get with this is a file with 4096 Zs on the first line, then  
a record boundary, then one Z on a second line.



Craig A. Berry
mailto:craigbe...@mac.com

"... getting out of a sonnet is much more
 difficult than getting in."
 Brad Leithauser

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Martin . Zinser]

Hello Craig,

almost there ;-)

First of all thanks a lot for your insightfull comments. A couple of the 
suggestions do restore the previous behaviour, in particular

* define PERLIO "stdio"
* use open OUT => ':stdio';
* open(TXT, '>:stdio', 'test.lis');

all produce the same output as 5.8.x .

Unfortunatly the suggested solution I really would like to implement 
(since it requires not code/setup changes) does give me a headache.
Setting stdio as the standard during using configure.com does cause the 
compilation of perl to fail.

Here is what I did:

* Unpack perl into a clean directory 
* @configure 
  ...
   --> Default answers for everything but
  Use the PerlIO abstraction layer? [y] n
  Ok, doing things the stdio way.
 ...
* mmk

which fails in vms.c with the following error:

Copy/Log/Noconfirm [.vms]vms.c []
%COPY-S-COPIED, DSA0:[PUBLIC.GNU.PERLSTD.vms]vms.c;1 copied to 
DSA0:[PUBLIC.GNU.
PERLSTD]vms.c;1 (864 blocks)
CC/DECC/NOANSI_ALIAS 
/Include=[]/Standard=Relaxed_ANSI/Prefix=All/Obj=.obj/NoList
/float=ieee/ieee=denorm/Define=(PERL_CORE,_USE_STD_STAT=1) VMS.C

 && PL_perlio_fd_refcnt)
^
%CC-E-UNDECLARED, In this statement, "PL_perlio_fd_refcnt" is not 
declared.
at line number 3080 in file DSA0:[PUBLIC.GNU.PERLSTD]vms.c;1

  && PL_perlio_fd_refcnt)
.^
%CC-E-UNDECLARED, In this statement, "PL_perlio_fd_refcnt" is not 
declared.
at line number 4688 in file DSA0:[PUBLIC.GNU.PERLSTD]vms.c;1

 && PL_perlio_fd_refcnt)
^
%CC-E-UNDECLARED, In this statement, "PL_perlio_fd_refcnt" is not 
declared.
at line number 4715 in file DSA0:[PUBLIC.GNU.PERLSTD]vms.c;1
%MMK-F-ERRUPD, error status %X10B91262 occurred when updating target 
VMS.OBJ

Now, as far as I can see in the code/listing PL_perlio_fd_refcnt is either 
defined in embedvar.h or perlapi.h, but in perlapi.h only if Multiplicity 
is defined (which I guess is related to the non-default multiplicity 
option in configure.com)  and in embedvar.h the definition is skipped 
because PERL_GLOBAL_STRUCT is not defined, so this is where the error 
seems to
originate.

Any further ideas?

Greetings, Martin



"Craig A. Berry"  
22.04.2010 05:44

To
"John E. Malmberg" 
cc
martin.zin...@deutsche-boerse.com, vmsperl@perl.org
Subject
Re: Problem with VMS Carriage return carriage control files in 5.10 and 
5.12








On Apr 21, 2010, at 8:04 PM, John E. Malmberg wrote:

> martin.zin...@deutsche-boerse.com wrote:
>> Hello Colleagues,
>> I am currently trying to drag our development team kicking and 
>> screaming into the century of the fruitbat, but what looks like a 
>> bug in 5.10.1 and 5.12.0 is a stumbling block.
>> Problem description:
>> If you open a text file with Carriage return carriage control for 
>> output (based off an existing file) and populate the new file with 
>> longer records, at some point gratuitous line breaks are added to 
>> the file.
>> This does not happen with either Perl 5.8.0 on Alpha, nor Perl 
>> 5.8.5 on Itanium (in that case the hp build). It does happen with 
>> both 5.10.1 and 5.12.0 on Itanium.

Thanks for the report, and especially for the concise reproducer.  The 
first thing that springs to mind in this area regarding changes 
between 5.8.x and 5.10.x is the default PerlIO behavior.  I don't 
remember exactly when it changed, and the change might not have been 
documented as well as it should have been since it wouldn't have made 
any difference with stream-oriented files, but basically it's now 
doing the unixio write() rather than the stdio fwrite() by default. 
It's not especially surprising that fwrite() is a bit more simpatico 
with record-oriented files than is write().

There are umpteen different ways to get the old behavior or 
accommodate the new behavior.  There is a configure-time option to 
disable perlio, which is equivalent to doing:

$ define PERLIO "stdio"

at run-time.  Or you can add:

use open OUT => ':stdio';

before the open statement (<http://perldoc.perl.org/open.html>).

Or push the stdio layer as part of the open statement itself (<
http://perldoc.perl.org/functions/open.html 
 >):

--- mztest.com;-0   2010-04-21 19:33:50 -0500
+++ mztest.com  2010-04-21 22:30:11 -0500
@@ -23,7 +23,7 @@ $  ENDIF
  $!
  $  perl -w SYS$INPUT: 'p1'
  $  DECK
-open( TXT, ">test.lis");
+open TXT, '>:stdio', 'test.lis';

  for ($i=0;$i<=120;$i++){
  printf( TXT
[end]

Or you can force the file to be a stream file using vmssysopen.

Or you can experiment with syswrite or vmssyswrite instead of print so 
that you control the size of the record you are writing.

Or you could (possibly, depending on the real-world problem behind the 
reproducer) cre

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Mudiaga Obada]

Hi,

we are currently testing perl 5.10.1 in our development environment. We
also ran into some issues which i did some initial analysis on.
Unfortunately, i am not directly involved in testing 5.10.1 and didn't
find time to get as far as collecting exact details in an isolated test
case yet.

There are two issues i have off the back of my mind:

1)  Random problems with carriage return and vmsopen()

I suspected it had something with buffering in connection with vmsopen().
For our purposes, i changed the record format from "rfm=var" to
"rfm=stmlf" and files looked fine
(Although this does smell like a regression from 5.8.6, the modules in
our case probably should not have been using "rfm=var" in the first
place...)



2) File::Spec::VMS->catpath() was making some "vmsify path" routine create
illegal VMS filenames like "LOG:[]filename.txt" instead of  LOG:filename.txt
(which made File::Copy->copy() break)

To continue our tests, we added something similar to "s/:[]/:/" as a
kludge in File/Spec/VMS.pm
 

Regards,

Mudiaga Obada
 

On 22.04.2010 16:25, Craig A. Berry wrote:
>
> On Apr 21, 2010, at 11:55 PM, Carl Friedberg wrote:
>
>> About 7 weeks ago, I put 5.10.1 into production. It looked good until
>> the overnight production ran, and then all hell broke loose. I did
>> not have time to poke around, but the symptom was that completely
>> trivial perl scripts failed, with extra line breaks being thrown into
>> the output. I put the old version back, and everyone was happy again.
>> The offending (trivial) perl script was replaced with an awk script.
>
>
> First of all, it's good to see people using Perl in production on
> VMS.  Obviously it's a bit distressing to find out about an impediment
> to using the current or previous version just after a new major
> version has been released.  I think the next steps are to narrow down
> further where the behavior changed by testing against 5.8.9 and
> 5.10.0.  If anyone has the opportunity to do that, please speak up.
>
> The unwanted line break in Martin's example occurs at exactly 4048
> bytes into the file.  There is likely a buffer involved, but not a
> mailbox buffer since as far as I can tell there's no mailbox
> involved.  Of course it's possible this example dodged a bullet by
> hitting a record boundary earlier on and the real magic number is
> something less than 4048.  I'm going to have a crack at narrowing this
> down further.
> 
> Craig A. Berry
>

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Craig A. Berry]

On Apr 21, 2010, at 11:55 PM, Carl Friedberg wrote:

About 7 weeks ago, I put 5.10.1 into production. It looked good  
until the overnight production ran, and then all hell broke loose. I  
did not have time to poke around, but the symptom was that  
completely trivial perl scripts failed, with extra line breaks being  
thrown into the output. I put the old version back, and everyone was  
happy again. The offending (trivial) perl script was replaced with  
an awk script.



First of all, it's good to see people using Perl in production on  
VMS.  Obviously it's a bit distressing to find out about an impediment  
to using the current or previous version just after a new major  
version has been released.  I think the next steps are to narrow down  
further where the behavior changed by testing against 5.8.9 and  
5.10.0.  If anyone has the opportunity to do that, please speak up.


The unwanted line break in Martin's example occurs at exactly 4048  
bytes into the file.  There is likely a buffer involved, but not a  
mailbox buffer since as far as I can tell there's no mailbox  
involved.  Of course it's possible this example dodged a bullet by  
hitting a record boundary earlier on and the real magic number is  
something less than 4048.  I'm going to have a crack at narrowing this  
down further.


Craig A. Berry
mailto:craigbe...@mac.com

"... getting out of a sonnet is much more
 difficult than getting in."
 Brad Leithauser

RE: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Carl Friedberg]

Martin, Craig & John,

About 7 weeks ago, I put 5.10.1 into production. It looked good until the 
overnight production ran, and then all hell broke loose. I did not have time to 
poke around, but the symptom was that completely trivial perl scripts failed, 
with extra line breaks being thrown into the output. I put the old version 
back, and everyone was happy again. The offending (trivial) perl script was 
replaced with an awk script.

Thanks for describing this. I've been way too busy with tasks unrelated to 
Perl, and I'm sorry I didn't report this when it hit (on Mar 3rd, when I put 
5.10.1 into production after extensive ( but not sufficient) testing.

My client reported:

" I use a perl script that gave me back unexpected output tonight.  Several 
records were not handled as expected in regards to the end of line.  I will 
discuss with Carl in the AM/PM.

"If Carl could put back the old version in the AM, I can rerun the processing 
script using last night's games and see if the problem goes away and then we 
can go from there."

And indeed, the client is still back at 5.8.6...

> 
> On Apr 21, 2010, at 8:04 PM, John E. Malmberg wrote:
> 
> > martin.zin...@deutsche-boerse.com wrote:

... all deleted...

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Craig A. Berry]

On Apr 21, 2010, at 8:04 PM, John E. Malmberg wrote:


martin.zin...@deutsche-boerse.com wrote:

Hello Colleagues,
I am currently trying to drag our development team kicking and  
screaming into the century of the fruitbat, but what looks like a  
bug in 5.10.1 and 5.12.0 is a stumbling block.

Problem description:
If you open a text file with Carriage return carriage control for  
output (based off an existing file) and populate the new file with  
longer records, at some point gratuitous line breaks are added to  
the file.
This does not happen with either Perl 5.8.0 on Alpha, nor Perl  
5.8.5 on Itanium (in that case the hp build). It does happen with  
both 5.10.1 and 5.12.0 on Itanium.


Thanks for the report, and especially for the concise reproducer.  The  
first thing that springs to mind in this area regarding changes  
between 5.8.x and 5.10.x is the default PerlIO behavior.  I don't  
remember exactly when it changed, and the change might not have been  
documented as well as it should have been since it wouldn't have made  
any difference with stream-oriented files, but basically it's now  
doing the unixio write() rather than the stdio fwrite() by default.   
It's not especially surprising that fwrite() is a bit more simpatico  
with record-oriented files than is write().


There are umpteen different ways to get the old behavior or  
accommodate the new behavior.  There is a configure-time option to  
disable perlio, which is equivalent to doing:


$ define PERLIO "stdio"

at run-time.  Or you can add:

use open OUT => ':stdio';

before the open statement ().

Or push the stdio layer as part of the open statement itself ():


--- mztest.com;-0   2010-04-21 19:33:50 -0500
+++ mztest.com  2010-04-21 22:30:11 -0500
@@ -23,7 +23,7 @@ $  ENDIF
 $!
 $  perl -w SYS$INPUT: 'p1'
 $  DECK
-open( TXT, ">test.lis");
+open TXT, '>:stdio', 'test.lis';

 for ($i=0;$i<=120;$i++){
 printf( TXT
[end]

Or you can force the file to be a stream file using vmssysopen.

Or you can experiment with syswrite or vmssyswrite instead of print so  
that you control the size of the record you are writing.


Or you could (possibly, depending on the real-world problem behind the  
reproducer) create the existing file as STREAM_LF (or convert it to  
such) before Perl ever sees it and inherits its default record  
attributes.


The issue is that the pipe emulation code in perl adds extra new- 
lines to output from subprocesses.


Since there is no piping in Martin's reproducer, I'd say there's about  
a 0% chance pipes have anything to do with it.


The size of the mailbox buffer may have something to do with when  
the new-line is added.  It may be that something changed there.  In  
PERLVMS.POD, it documents that PERL_MBX_SIZE controls this with a  
default of 512 bytes.



That's the most common cause of newlines where you don't want them,  
but not in this case.  BTW, the default is now 8192 bytes in 5.12.0.



Craig A. Berry
mailto:craigbe...@mac.com

"... getting out of a sonnet is much more
 difficult than getting in."
 Brad Leithauser

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[John E. Malmberg]

martin.zin...@deutsche-boerse.com wrote:

Hello Colleagues,

I am currently trying to drag our development team kicking and screaming 
into the century of the fruitbat, but what looks like a bug in 5.10.1 and 
5.12.0 is a stumbling block.


Problem description:

If you open a text file with Carriage return carriage control for output 
(based off an existing file) and populate the new file with longer 
records, at some point gratuitous 
line breaks are added to the file.


This does not happen with either Perl 5.8.0 on Alpha, nor Perl 5.8.5 on 
Itanium (in that case the hp build). It does happen with both 5.10.1 and 
5.12.0 on Itanium.


The bug should be present in the older 5.8.0, it just may be something 
is causing you to miss it.


The issue is that the pipe emulation code in perl adds extra new-lines 
to output from subprocesses.


The size of the mailbox buffer may have something to do with when the 
new-line is added.  It may be that something changed there.  In 
PERLVMS.POD, it documents that PERL_MBX_SIZE controls this with a 
default of 512 bytes.


I do not recall any changes to that code since I started working with 
Perl internals, and that was with perl 5.8.6.


The only changes that I can recall is that the size of the command 
buffer and parameters was adjusted to match what VMS supports, and there 
were some improvements in launching subcommands.


But obviously something is different between the two versions.

Regards,
-John
wb8...@qsl.net
Personal Opinion Only

Re: Problem with VMS Carriage return carriage control files in 5.10 and 5.12

[Martin . Zinser]

Hello,

sorry, looks like the list processor strips the attachment. The canned 
example is at 
http://zinser.no-ip.info/www/trans/rms51x.zip

Greetings, Martin




martin.zin...@deutsche-boerse.com 
22.04.2010 00:29

To
vmsperl@perl.org
cc

Subject
Problem with VMS Carriage return carriage control files in 5.10 and 5.12







Hello Colleagues,

I am currently trying to drag our development team kicking and screaming 
into the century of the fruitbat, but what looks like a bug in 5.10.1 and 
5.12.0 is a stumbling block.

Problem description:

If you open a text file with Carriage return carriage control for output 
(based off an existing file) and populate the new file with longer 
records, at some point gratuitous 
line breaks are added to the file.

This does not happen with either Perl 5.8.0 on Alpha, nor Perl 5.8.5 on 
Itanium (in that case the hp build). It does happen with both 5.10.1 and 
5.12.0 on Itanium.

The attached Zip file contains a simple DCL to reproduce the problem 
(essentially it creates a test file and then in an embedded Perl script 
does a printf to file).  The 
reproducer is an extremely simplified version of the actual code. It also 
contains the output from 5.8.x and 5.12 (5.10.1 looks the same).

Any ideas why this happens and how to fix it? 

Greetings, Martin

-
Diese E-Mail enthaelt vertrauliche oder rechtlich geschuetzte 
Informationen.
Wenn Sie nicht der beabsichtigte Empfaenger sind, informieren Sie bitte
sofort den Absender und loeschen Sie diese E-Mail. Das unbefugte Kopieren
dieser E-Mail oder die unbefugte Weitergabe der enthaltenen Informationen
ist nicht gestattet.

The information contained in this message is confidential or protected by
law. If you are not the intended recipient, please contact the sender and
delete this message. Any unauthorised copying of this message or
unauthorised distribution of the information contained herein is 
prohibited.

Legally required information for business correspondence/
Gesetzliche Pflichtangaben fuer Geschaeftskorrespondenz:
http://deutsche-boerse.com/letterhead


-
Diese E-Mail enthaelt vertrauliche oder rechtlich geschuetzte Informationen.
Wenn Sie nicht der beabsichtigte Empfaenger sind, informieren Sie bitte
sofort den Absender und loeschen Sie diese E-Mail. Das unbefugte Kopieren
dieser E-Mail oder die unbefugte Weitergabe der enthaltenen Informationen
ist nicht gestattet.

The information contained in this message is confidential or protected by
law. If you are not the intended recipient, please contact the sender and
delete this message. Any unauthorised copying of this message or
unauthorised distribution of the information contained herein is prohibited.

Legally required information for business correspondence/
Gesetzliche Pflichtangaben fuer Geschaeftskorrespondenz:
http://deutsche-boerse.com/letterhead