Re: VBS file read in windows - end of record issue

[Paul Gilmartin]

On Thu, 18 May 2023 11:37:18 -0500, Kirk Wolf wrote:
>
>To answer your other question (preserving SDW):  if you copy a RECFM=VBS 
>dataset using DCB=RECFM=U, you will get BDWs and SDWs.
> 
Yes.  I did that with REXX in an era when REXX supported nether
VBS nor U.  I first used a utility (REPRO, IIRC) to copy (overriden)
RECFM=U to RECFM=V, adding generated BDW/RDW in a
temp file, then using EXECIO.  REXX is better nowadays.  But
LRECL=X?

Yes.  But PDSE prohibits such overrides.  Why!?  (I suspect some
limitation of NOTE/POINT.)

Beware FTP!  I've fount it undoes allocation overrides rather than
using QSAM with allocated attributes.

What's recommended for continuous stream data, as for lab
instruments?

o RECFM=U?

o RECFM=VBS,LRECL=X?

o zFS?

o Other (specify)?

-- 
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Kirk Wolf]

I apologize if this has already been covered on this thread, but using LF/CRLF 
etc on a VBS file will result in problems with binary records since they could 
easily contain a line terminator.

To answer your other question (preserving SDW):  if you copy a RECFM=VBS 
dataset using DCB=RECFM=U, you will get BDWs and SDWs.

Kirk Wolf
Dovetailed Technologies

PS>  Co:Z SFTP and Co:Z Dataset Pipes have options for converting z/OS records 
like RECFM=VBS into streams of bytes with various terminators, like  cr, crlf, 
rdw, l4 (4 byte length prefix), mfrdw (Microfocus), an aribtrary hex separator, 
or no separators.   For binary records you would want to use rdw, l4, or mfrdw 
options.  You can download and use it under the terms of our free Community 
License.
https://coztoolkit.com/docs/dspipes/dsp-ref_fromdsn.html


You might also consider writing a z/OS program yourself that converts RECFM=VBS 
into a simple length prefixed stream.  If you are a little comfortable with 
Java, the com.ibm.jzos.RecordReader class makes that super simple.

On Mon, May 15, 2023, at 11:20 PM, Prashant Joshi wrote:
> Thank for your response, Paul.
> 
> I agree, CRLF irrelevant to binary files. Surprisingly, only after using CRLF 
> option, I get those binary records in proper record length.
> I tried to force it to U format but did not work for m. I will try again with 
> different methods.
> 
> I can preserve RDW and BDW. How to preserve SDW? I did not see mention of SDW 
> in these options. May be SDW is the key to maintain record length.
> 
> Thank you,
> Prashant Joshi
> Mainframe Architect
> +91 9743 440 503 (Mobile) 
> 
> 
> -Original Message-
> From: IBM Mainframe Discussion List  On Behalf Of 
> Paul Gilmartin
> Sent: Tuesday, May 16, 2023 1:38 AM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: [EXTERNAL] Re: VBS file read in windows - end of record issue
> 
> On Mon, 15 May 2023 18:54:40 +, Prashant Joshi  wrote:
> >...
> >For some files, when I transfer using IND$FILE (option 6) using Binary and 
> >CRLF option checked, I gets record properly. But it does not work for every 
> >file. And due to different file sizes, IND$FILE may not be option for me.
> >
> CRLF should be irrelevant for binary files.
> 
> 
> >Need your help to find, what am I missing?
> >
> Allocate your data set overriding to RECFM=U.  Tne BDWs and SDWs should 
> appear ad data.
> 
> Transfer the file with a method that preserves those BDWs and SDWs.  (I don't 
> recommend
> IND$FILE.)
> 
> Decode the SDWs and translate to UTF-8  with your Python.
> 
> I hate EBCDIC!
> 
> --
> gil
> 
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
> lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> 
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Michael Oujesky]

Long time ago IFASMFDP had a default BLKSIZE of 
4096, but today uses SDB to match to device 
characteristics.  However, LRECL is set to 32767 
and SMF did write records that long.  At last 
check, still did use DCBE, so  features like LBI 
could not be used for it's output files.


Note that if you SMS compress the output file, 
BLKIZE becomes logical as the data is physically 
written full track (roughly 56K bytes).


Michael


At 10:09 AM 5/18/2023, Lennie Dymoke-Bradshaw wrote:


Radoslaw,

If you specify the LRECL and/or BLKSIZE in your 
program, then you can set a  value that appears 
to flout the JCL rules. For example it used to 
be that IFASMFDP wrote data sets with a BLKSIZE 
of 32767. I am unsure if it still does.


Lennie Dymoke-Bradshaw
https://rsclweb.com
‘Dance like no one is watching. Encrypt like everyone is.’

-Original Message-
From: IBM Mainframe Discussion List 
 On Behalf Of Radoslaw Skorupka

Sent: 18 May 2023 10:36
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: VBS file read in windows - end of record issue

Well, can you show me a JCL job to allocate such dataset?
My experience say that any attempt to create PS 
dataset with LRECL > ~32kB ends with JCL error 
and IEF638I JCL Reference clearly say that for 
non-VSAM dataset the maximum is 32760. For VSAM it is 32761.
"Additional Syntax" says it is possible to use 
LRECL=nK when n is up to 16,383. However 
it is possible only for ISO/ANSI V3 tapes.
And there is LRECL=X, which is applicable only 
to QSAM VS/VBS. It is not cheating in the 
meaning I provided earlier, but it is not quite simple dataset usage.



Note: despite of the above it is possible to 
allocate PS VBS file with LRECL=32767, but the 
LRECL cannot be specified in JCL. LIKE is the trick.



Regarding a little bit off-topic compressed extended format datasets:
system reports "legal" BLKSIZE 32760 (SDB was used). What's inside - it
is covered by media manager IMHO.


(irrelevant)
My local "cheating" definition used before: user creates
records/segments, *including* SDW. IMHO tools like File Manager allow
such cheating. However it is just track editing play, not dataset usage.


Regards
--
Radoslaw Skorupka
Lodz, Poland




W dniu 17.05.2023 o 21:14, Michael Oujesky pisze:
> Having read and written records longer than three MB, it is not
> "cheating". Especially with RMF 74.5 records with 59 "broken" (split)
> records to reassemble into one very long record. See the SMF manual on
> RMF record reassembly area.
>
> JCL allows LRECL=16384K.  And SMS compressed files write full tracks
> (roughly 56KB).
>
> Michael
>
> At 01:20 PM 5/17/2023, Radoslaw Skorupka wrote:
>> Content-Transfer-Encoding: 8bit
>>
>> Well, not really.
>> There is LRECL=X, but besides we have not very strict limitation of
>> LRECL. It is 32760 or 32767.
>> First value is limited by JCL syntax, but the second is available
>> when allocation PS using LIKE= keyword.
>> Of course one may automagically write segments with custom-created
>> SDWs, but I would call it cheating.
>>
>> BTW: The purpose of VBS was not veeery long record, but records
>> up to 32k, even on DASD with shorter track. Hint: the track is
>> natural limit of BLKSIZE. It is no longer important since 3380
>> (80's), because track size exceeded 32k.
>>
>>
>> --
>> Radoslaw Skorupka
>> Lodz, Poland
>>
>>
>>
>>
>> W dniu 16.05.2023 o 18:52, Michael Oujesky pisze:
>>> Just another tidbit, but when combining the record segments, while
>>> the VBS architecture does not specify a maximum record length, you
>>> can expect the full records to be up to 16,777,215 (16384K - 1)
>>> bytes in length.
>>>
>>> Realizing that the RDW is actually a SDW.
>>>
>>> Michael
>>>
>>> At 01:26 AM 5/16/2023, Michael Stein wrote:
>>>> On Tue, May 16, 2023 at 04:14:07AM +, Prashant Joshi wrote:
>>>> >> Did you specify binary mode on the python open call? --
>>>>
>>>> > Yes. And I can read the data.
>>>>
>>>> How are you reading the data.  Assuming an open like:
>>>>
>>>> myfile = open("filename", "rb")
>>>>
>>>> You need to either read it all into memory:
>>>>
>>>> alldata = myfile.read()
>>>>
>>>> or read specific lengths which is messier as you need to read specific
>>>> lengths, first 4 bytes for the RDW and then the length of the record
>>>> in the RDW-4 (as you already read the RDW).
>>>>
>>>> The 4 byte RDW includes the length of the record in the first 2 bytes
>>&g

Re: VBS file read in windows - end of record issue

[Michael Oujesky]

See the references on XLRI.

This should be a good start - 
Processing 
Records Longer than 32 760 Bytes - IBM Documentation


This also might help - 
Additional 
syntax for LRECL=(bytes) - IBM Documentation


You will perhaps need to review transmittal modes in - 
z/OS: 
z/OS DFSMS Using Data Sets (ibm.com)


LRECL=16384K in JCL ends up providing a lrecl of 16384k-1, but guess 
the folks coding JCL interpretation thought 16384K was enough of a keyword.


LRECL=X gives you a lrecl of 32768 which appears to signal that SDW 
are present and that BFTEK=A (which doesn't support logical records 
longer than 32767) will not work to reassemble the logical 
record.  For such file, reassembly (de-segmentation) of the logical 
record is left to the application program.


Note that when working with records longer than 32767 the length of 
the record is now three bytes long (i.e. 16384K - 1). Also see DCB 
field DCBXLREC.


Michael


At 04:35 AM 5/18/2023, Radoslaw Skorupka wrote:

Content-Transfer-Encoding: 8bit

Well, can you show me a JCL job to allocate such dataset?
My experience say that any attempt to create PS dataset with LRECL > 
~32kB ends with JCL error and IEF638I
JCL Reference clearly say that for non-VSAM dataset the maximum is 
32760. For VSAM it is 32761.
"Additional Syntax" says it is possible to use LRECL=nK when 
n is up to 16,383. However it is possible only for ISO/ANSI V3 tapes.
And there is LRECL=X, which is applicable only to QSAM VS/VBS. It is 
not cheating in the meaning I provided earlier, but it is not quite 
simple dataset usage.



Note: despite of the above it is possible to allocate PS VBS file 
with LRECL=32767, but the LRECL cannot be specified in JCL. LIKE is the trick.



Regarding a little bit off-topic compressed extended format 
datasets: system reports "legal" BLKSIZE 32760 (SDB was used). 
What's inside - it is covered by media manager IMHO.



(irrelevant)
My local "cheating" definition used before: user creates 
records/segments, *including* SDW. IMHO tools like File Manager 
allow such cheating. However it is just track editing play, not dataset usage.



Regards
--
Radoslaw Skorupka
Lodz, Poland




W dniu 17.05.2023 o 21:14, Michael Oujesky pisze:
Having read and written records longer than three MB, it is not 
"cheating". Especially with RMF 74.5 records with 59 "broken" 
(split) records to reassemble into one very long record. See the 
SMF manual on RMF record reassembly area.


JCL allows LRECL=16384K.  And SMS compressed files write full 
tracks (roughly 56KB).


Michael

At 01:20 PM 5/17/2023, Radoslaw Skorupka wrote:

Content-Transfer-Encoding: 8bit

Well, not really.
There is LRECL=X, but besides we have not very strict limitation 
of LRECL. It is 32760 or 32767.
First value is limited by JCL syntax, but the second is available 
when allocation PS using LIKE= keyword.
Of course one may automagically write segments with custom-created 
SDWs, but I would call it cheating.


BTW: The purpose of VBS was not veeery long record, but 
records up to 32k, even on DASD with shorter track. Hint: the 
track is natural limit of BLKSIZE. It is no longer important since 
3380 (80's), because track size exceeded 32k.



--
Radoslaw Skorupka
Lodz, Poland




W dniu 16.05.2023 o 18:52, Michael Oujesky pisze:
Just another tidbit, but when combining the record segments, 
while the VBS architecture does not specify a maximum record 
length, you can expect the full records to be up to 16,777,215 
(16384K - 1) bytes in length.


Realizing that the RDW is actually a SDW.

Michael

At 01:26 AM 5/16/2023, Michael Stein wrote:

On Tue, May 16, 2023 at 04:14:07AM +, Prashant Joshi wrote:
>> Did you specify binary mode on the python open call? --

> Yes. And I can read the data.

How are you reading the data.  Assuming an open like:

myfile = open("filename", "rb")

You need to either read it all into memory:

alldata = myfile.read()

or read specific lengths which is messier as you need to read specific
lengths, first 4 bytes for the RDW and then the length of the record
in the RDW-4 (as you already read the RDW).

The 4 byte RDW includes the length of the record in the first 2 bytes
(bigendian order) and the spanning bits in the last 2 bytes.

Either way you need to walk your way through the binary data, any code
looking for CR or NL or space isn't correct.

A description of VBS records formats:

z/OS 2.4 DFSMS Using Data Sets SC23-6855-40
https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R4sc236855/$file/idad400_v2r4.pdf 



pdf page 273 Variable-Length Record Formats
(near bottom of page and continues on to more pages)

> its just that I don't get proper end of 

Re: VBS file read in windows - end of record issue

[Lennie Dymoke-Bradshaw]

Radoslaw,

If you specify the LRECL and/or BLKSIZE in your program, then you can set a  
value that appears to flout the JCL rules. For example it used to be that 
IFASMFDP wrote data sets with a BLKSIZE of 32767. I am unsure if it still does.

Lennie Dymoke-Bradshaw
https://rsclweb.com 
‘Dance like no one is watching. Encrypt like everyone is.’

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Radoslaw Skorupka
Sent: 18 May 2023 10:36
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: VBS file read in windows - end of record issue

Well, can you show me a JCL job to allocate such dataset?
My experience say that any attempt to create PS dataset with LRECL > ~32kB ends 
with JCL error and IEF638I JCL Reference clearly say that for non-VSAM dataset 
the maximum is 32760. For VSAM it is 32761.
"Additional Syntax" says it is possible to use LRECL=nK when n is up to 
16,383. However it is possible only for ISO/ANSI V3 tapes.
And there is LRECL=X, which is applicable only to QSAM VS/VBS. It is not 
cheating in the meaning I provided earlier, but it is not quite simple dataset 
usage.


Note: despite of the above it is possible to allocate PS VBS file with 
LRECL=32767, but the LRECL cannot be specified in JCL. LIKE is the trick.


Regarding a little bit off-topic compressed extended format datasets: 
system reports "legal" BLKSIZE 32760 (SDB was used). What's inside - it 
is covered by media manager IMHO.


(irrelevant)
My local "cheating" definition used before: user creates 
records/segments, *including* SDW. IMHO tools like File Manager allow 
such cheating. However it is just track editing play, not dataset usage.


Regards
-- 
Radoslaw Skorupka
Lodz, Poland




W dniu 17.05.2023 o 21:14, Michael Oujesky pisze:
> Having read and written records longer than three MB, it is not 
> "cheating". Especially with RMF 74.5 records with 59 "broken" (split) 
> records to reassemble into one very long record. See the SMF manual on 
> RMF record reassembly area.
>
> JCL allows LRECL=16384K.  And SMS compressed files write full tracks 
> (roughly 56KB).
>
> Michael
>
> At 01:20 PM 5/17/2023, Radoslaw Skorupka wrote:
>> Content-Transfer-Encoding: 8bit
>>
>> Well, not really.
>> There is LRECL=X, but besides we have not very strict limitation of 
>> LRECL. It is 32760 or 32767.
>> First value is limited by JCL syntax, but the second is available 
>> when allocation PS using LIKE= keyword.
>> Of course one may automagically write segments with custom-created 
>> SDWs, but I would call it cheating.
>>
>> BTW: The purpose of VBS was not veeery long record, but records 
>> up to 32k, even on DASD with shorter track. Hint: the track is 
>> natural limit of BLKSIZE. It is no longer important since 3380 
>> (80's), because track size exceeded 32k.
>>
>>
>> -- 
>> Radoslaw Skorupka
>> Lodz, Poland
>>
>>
>>
>>
>> W dniu 16.05.2023 o 18:52, Michael Oujesky pisze:
>>> Just another tidbit, but when combining the record segments, while 
>>> the VBS architecture does not specify a maximum record length, you 
>>> can expect the full records to be up to 16,777,215 (16384K - 1) 
>>> bytes in length.
>>>
>>> Realizing that the RDW is actually a SDW.
>>>
>>> Michael
>>>
>>> At 01:26 AM 5/16/2023, Michael Stein wrote:
>>>> On Tue, May 16, 2023 at 04:14:07AM +, Prashant Joshi wrote:
>>>> >> Did you specify binary mode on the python open call? --
>>>>
>>>> > Yes. And I can read the data.
>>>>
>>>> How are you reading the data.  Assuming an open like:
>>>>
>>>> myfile = open("filename", "rb")
>>>>
>>>> You need to either read it all into memory:
>>>>
>>>> alldata = myfile.read()
>>>>
>>>> or read specific lengths which is messier as you need to read specific
>>>> lengths, first 4 bytes for the RDW and then the length of the record
>>>> in the RDW-4 (as you already read the RDW).
>>>>
>>>> The 4 byte RDW includes the length of the record in the first 2 bytes
>>>> (bigendian order) and the spanning bits in the last 2 bytes.
>>>>
>>>> Either way you need to walk your way through the binary data, any code
>>>> looking for CR or NL or space isn't correct.
>>>>
>>>> A description of VBS records formats:
>>>>
>>>> z/OS 2.4 DFSMS Using Data Sets SC23-6855-40
>>>> https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R4sc236855/

Re: VBS file read in windows - end of record issue

[Radoslaw Skorupka]

Well, can you show me a JCL job to allocate such dataset?
My experience say that any attempt to create PS dataset with LRECL > 
~32kB ends with JCL error and IEF638I
JCL Reference clearly say that for non-VSAM dataset the maximum is 
32760. For VSAM it is 32761.
"Additional Syntax" says it is possible to use LRECL=nK when n 
is up to 16,383. However it is possible only for ISO/ANSI V3 tapes.
And there is LRECL=X, which is applicable only to QSAM VS/VBS. It is not 
cheating in the meaning I provided earlier, but it is not quite simple 
dataset usage.



Note: despite of the above it is possible to allocate PS VBS file with 
LRECL=32767, but the LRECL cannot be specified in JCL. LIKE is the trick.



Regarding a little bit off-topic compressed extended format datasets: 
system reports "legal" BLKSIZE 32760 (SDB was used). What's inside - it 
is covered by media manager IMHO.



(irrelevant)
My local "cheating" definition used before: user creates 
records/segments, *including* SDW. IMHO tools like File Manager allow 
such cheating. However it is just track editing play, not dataset usage.



Regards
--
Radoslaw Skorupka
Lodz, Poland




W dniu 17.05.2023 o 21:14, Michael Oujesky pisze:
Having read and written records longer than three MB, it is not 
"cheating". Especially with RMF 74.5 records with 59 "broken" (split) 
records to reassemble into one very long record. See the SMF manual on 
RMF record reassembly area.


JCL allows LRECL=16384K.  And SMS compressed files write full tracks 
(roughly 56KB).


Michael

At 01:20 PM 5/17/2023, Radoslaw Skorupka wrote:

Content-Transfer-Encoding: 8bit

Well, not really.
There is LRECL=X, but besides we have not very strict limitation of 
LRECL. It is 32760 or 32767.
First value is limited by JCL syntax, but the second is available 
when allocation PS using LIKE= keyword.
Of course one may automagically write segments with custom-created 
SDWs, but I would call it cheating.


BTW: The purpose of VBS was not veeery long record, but records 
up to 32k, even on DASD with shorter track. Hint: the track is 
natural limit of BLKSIZE. It is no longer important since 3380 
(80's), because track size exceeded 32k.



--
Radoslaw Skorupka
Lodz, Poland




W dniu 16.05.2023 o 18:52, Michael Oujesky pisze:
Just another tidbit, but when combining the record segments, while 
the VBS architecture does not specify a maximum record length, you 
can expect the full records to be up to 16,777,215 (16384K - 1) 
bytes in length.


Realizing that the RDW is actually a SDW.

Michael

At 01:26 AM 5/16/2023, Michael Stein wrote:

On Tue, May 16, 2023 at 04:14:07AM +, Prashant Joshi wrote:
>> Did you specify binary mode on the python open call? --

> Yes. And I can read the data.

How are you reading the data.  Assuming an open like:

    myfile = open("filename", "rb")

You need to either read it all into memory:

    alldata = myfile.read()

or read specific lengths which is messier as you need to read specific
lengths, first 4 bytes for the RDW and then the length of the record
in the RDW-4 (as you already read the RDW).

The 4 byte RDW includes the length of the record in the first 2 bytes
(bigendian order) and the spanning bits in the last 2 bytes.

Either way you need to walk your way through the binary data, any code
looking for CR or NL or space isn't correct.

A description of VBS records formats:

z/OS 2.4 DFSMS Using Data Sets SC23-6855-40
https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R4sc236855/$file/idad400_v2r4.pdf 



pdf page 273 Variable-Length Record Formats
(near bottom of page and continues on to more pages)

> its just that I don't get proper end of reord. Hence every time I
> read record, I get random record length (multiple records/half 
records

> combined)

There aren't any "end of records" in a VBS file.  At the begining of
the file you know you are at the start of a RDW (Well, BDW/RDW but I'm
assuming the FTP removed the BDWs).

You can find the next record by going the length specified in the RDW
into the file -- that is the start of the next RDW. Continue until
you've processed all the records.

More help will likely require you to show some code and/or data so
we can see what is going on...




--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Michael Oujesky]

Having read and written records longer than three MB, it is not 
"cheating". Especially with RMF 74.5 records with 59 "broken" (split) 
records to reassemble into one very long record. See the SMF manual 
on RMF record reassembly area.


JCL allows LRECL=16384K.  And SMS compressed files write full tracks 
(roughly 56KB).


Michael

At 01:20 PM 5/17/2023, Radoslaw Skorupka wrote:

Content-Transfer-Encoding: 8bit

Well, not really.
There is LRECL=X, but besides we have not very strict limitation of 
LRECL. It is 32760 or 32767.
First value is limited by JCL syntax, but the second is available 
when allocation PS using LIKE= keyword.
Of course one may automagically write segments with custom-created 
SDWs, but I would call it cheating.


BTW: The purpose of VBS was not veeery long record, but records 
up to 32k, even on DASD with shorter track. Hint: the track is 
natural limit of BLKSIZE. It is no longer important since 3380 
(80's), because track size exceeded 32k.



--
Radoslaw Skorupka
Lodz, Poland




W dniu 16.05.2023 o 18:52, Michael Oujesky pisze:
Just another tidbit, but when combining the record segments, while 
the VBS architecture does not specify a maximum record length, you 
can expect the full records to be up to 16,777,215 (16384K - 1) 
bytes in length.


Realizing that the RDW is actually a SDW.

Michael

At 01:26 AM 5/16/2023, Michael Stein wrote:

On Tue, May 16, 2023 at 04:14:07AM +, Prashant Joshi wrote:
>> Did you specify binary mode on the python open call? --

> Yes. And I can read the data.

How are you reading the data.  Assuming an open like:

myfile = open("filename", "rb")

You need to either read it all into memory:

alldata = myfile.read()

or read specific lengths which is messier as you need to read specific
lengths, first 4 bytes for the RDW and then the length of the record
in the RDW-4 (as you already read the RDW).

The 4 byte RDW includes the length of the record in the first 2 bytes
(bigendian order) and the spanning bits in the last 2 bytes.

Either way you need to walk your way through the binary data, any code
looking for CR or NL or space isn't correct.

A description of VBS records formats:

z/OS 2.4 DFSMS Using Data Sets SC23-6855-40
https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R4sc236855/$file/idad400_v2r4.pdf 



pdf page 273 Variable-Length Record Formats
(near bottom of page and continues on to more pages)

> its just that I don't get proper end of reord. Hence every time I
> read record, I get random record length (multiple records/half records
> combined)

There aren't any "end of records" in a VBS file.  At the begining of
the file you know you are at the start of a RDW (Well, BDW/RDW but I'm
assuming the FTP removed the BDWs).

You can find the next record by going the length specified in the RDW
into the file -- that is the start of the next RDW.  Continue until
you've processed all the records.

More help will likely require you to show some code and/or data so
we can see what is going on...


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Steve Thompson]

Not sure this is correct (VBS sizing). If I remember correctly 
VBS was handling telemetry data which could be quite long and was 
also variable in size per record.


Steve Thompson

On 5/17/2023 2:20 PM, Radoslaw Skorupka wrote:

Well, not really.
There is LRECL=X, but besides we have not very strict 
limitation of LRECL. It is 32760 or 32767.
First value is limited by JCL syntax, but the second is 
available when allocation PS using LIKE= keyword.
Of course one may automagically write segments with 
custom-created SDWs, but I would call it cheating.


BTW: The purpose of VBS was not veeery long record, but 
records up to 32k, even on DASD with shorter track. Hint: the 
track is natural limit of BLKSIZE. It is no longer important 
since 3380 (80's), because track size exceeded 32k.





--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Paul Gilmartin]

On Wed, 17 May 2023 20:20:24 +0200, Radoslaw Skorupka wrote:
>...
>BTW: The purpose of VBS was not veeery long record, but records up
>to 32k, even on DASD with shorter track. Hint: the track is natural
>limit of BLKSIZE. It is no longer important since 3380 (80's), because
>track size exceeded 32k.
>
I thought it was to support a FORTRAN tradition.

And I notice
:
- RECFM=VS or VBS cannot be specified for a UNIX file.

That's bizarre!  UNIX files are naturally spanned.

--n
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Radoslaw Skorupka]

Well, not really.
There is LRECL=X, but besides we have not very strict limitation of 
LRECL. It is 32760 or 32767.
First value is limited by JCL syntax, but the second is available when 
allocation PS using LIKE= keyword.
Of course one may automagically write segments with custom-created SDWs, 
but I would call it cheating.


BTW: The purpose of VBS was not veeery long record, but records up 
to 32k, even on DASD with shorter track. Hint: the track is natural 
limit of BLKSIZE. It is no longer important since 3380 (80's), because 
track size exceeded 32k.



--
Radoslaw Skorupka
Lodz, Poland




W dniu 16.05.2023 o 18:52, Michael Oujesky pisze:
Just another tidbit, but when combining the record segments, while the 
VBS architecture does not specify a maximum record length, you can 
expect the full records to be up to 16,777,215 (16384K - 1) bytes in 
length.


Realizing that the RDW is actually a SDW.

Michael

At 01:26 AM 5/16/2023, Michael Stein wrote:

On Tue, May 16, 2023 at 04:14:07AM +, Prashant Joshi wrote:
>> Did you specify binary mode on the python open call? --

> Yes. And I can read the data.

How are you reading the data.  Assuming an open like:

    myfile = open("filename", "rb")

You need to either read it all into memory:

    alldata = myfile.read()

or read specific lengths which is messier as you need to read specific
lengths, first 4 bytes for the RDW and then the length of the record
in the RDW-4 (as you already read the RDW).

The 4 byte RDW includes the length of the record in the first 2 bytes
(bigendian order) and the spanning bits in the last 2 bytes.

Either way you need to walk your way through the binary data, any code
looking for CR or NL or space isn't correct.

A description of VBS records formats:

z/OS 2.4 DFSMS Using Data Sets SC23-6855-40
https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R4sc236855/$file/idad400_v2r4.pdf 



pdf page 273 Variable-Length Record Formats
(near bottom of page and continues on to more pages)

> its just that I don't get proper end of reord. Hence every time I
> read record, I get random record length (multiple records/half records
> combined)

There aren't any "end of records" in a VBS file.  At the begining of
the file you know you are at the start of a RDW (Well, BDW/RDW but I'm
assuming the FTP removed the BDWs).

You can find the next record by going the length specified in the RDW
into the file -- that is the start of the next RDW.  Continue until
you've processed all the records.

More help will likely require you to show some code and/or data so
we can see what is going on...


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[David Crayford]

> On 17 May 2023, at 2:53 am, Paul Gilmartin 
> <042bfe9c879d-dmarc-requ...@listserv.ua.edu> wrote:
> 
> On Tue, 16 May 2023 11:26:06 -0700, Michael Stein wrote:
>> 
>> read can be supplied with an operand saying the maximum amount to
>> read:  read(4) to read 4 bytes.  It might read less either because
>> 
> So: "read_data = f.read( ntohs( SDW ) - 4 )"

There is no ntohs in Python. Network protocols are abstract in the libraries. 
The way to read/write binary data in different endians is to use the struct 
package https://docs.python.org/3/library/struct.html.

> 
>> it reached the end of the input file or because the input is
>> "interactive" (console? network?) and it just doesn't have any
>> more right now.
>> 
> I suppose there's some distinction between EAGAIN and EOF.
> In REXX I have successfully (not necessarily usefully) switched
> dynamically between blocking and non blocking and handled
> EAGAIN.
> 
> I don't know that REXX has SYSCALL SELECT.  Does Python?
> 
> -- 
> gil
> 
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Farley, Peter]

No SELECT per se, but python does have an "asyncio" module that I believe 
accomplishes that task.  Haven't had occasion to use it myself so far, but I 
know it exists.

Peter

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Paul Gilmartin
Sent: Tuesday, May 16, 2023 2:54 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: VBS file read in windows - end of record issue

On Tue, 16 May 2023 11:26:06 -0700, Michael Stein wrote:
>
>read can be supplied with an operand saying the maximum amount to
>read:  read(4) to read 4 bytes.  It might read less either because
>
So: "read_data = f.read( ntohs( SDW ) - 4 )"

>it reached the end of the input file or because the input is 
>"interactive" (console? network?) and it just doesn't have any more 
>right now.
> 
I suppose there's some distinction between EAGAIN and EOF.
In REXX I have successfully (not necessarily usefully) switched dynamically 
between blocking and non blocking and handled EAGAIN.

I don't know that REXX has SYSCALL SELECT.  Does Python?

--

This message and any attachments are intended only for the use of the addressee 
and may contain information that is privileged and confidential. If the reader 
of the message is not the intended recipient or an authorized representative of 
the intended recipient, you are hereby notified that any dissemination of this 
communication is strictly prohibited. If you have received this communication 
in error, please notify us immediately by e-mail and delete the message and any 
attachments from your system.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Paul Gilmartin]

On Tue, 16 May 2023 11:26:06 -0700, Michael Stein wrote:
>
>read can be supplied with an operand saying the maximum amount to
>read:  read(4) to read 4 bytes.  It might read less either because
>
So: "read_data = f.read( ntohs( SDW ) - 4 )"

>it reached the end of the input file or because the input is
>"interactive" (console? network?) and it just doesn't have any
>more right now.
> 
I suppose there's some distinction between EAGAIN and EOF.
In REXX I have successfully (not necessarily usefully) switched
dynamically between blocking and non blocking and handled
EAGAIN.

I don't know that REXX has SYSCALL SELECT.  Does Python?

-- 
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Michael Stein]

On Tue, May 16, 2023 at 11:23:46AM -0500, Paul Gilmartin wrote:
> On Tue, 16 May 2023 11:06:59 -0500, John McKown wrote:
> 
> >In one of my C programs, I first read the RDW, did a ntohs() to convert
> >from mainframe to Intel integer, subtracted 4, then read that number of
> >bytes into a char[32768].
> >
> I'm Python-naive.  But trying to educate myself with the example,
> :
> with open('workfile', encoding="utf-8") as f:
> read_data = f.read()

The default mode for python open is "rt" which is read text, not
binary.  So python will look for CR LF and alter your binary data.
The encoding="utf-8" is also invalid as the binary parts aren't utf-8
either and converting them will also break things (or raise an exception).

> ... I see no explicit byte count.  Is that implied, perhaps by a declaration
> such as "char read_data[ LRECL ]"?

No, read() without an operand says to read to the end of the file,
everything...

read can be supplied with an operand saying the maximum amount to
read:  read(4) to read 4 bytes.  It might read less either because
it reached the end of the input file or because the input is
"interactive" (console? network?) and it just doesn't have any
more right now.

Here's what the python description says:

 |  read(self, size=-1, /)
 |  Read and return up to n bytes.
 |  
 |  If the argument is omitted, None, or negative, reads and
 |  returns all data until EOF.
 |  
 |  If the argument is positive, and the underlying raw stream is
 |  not 'interactive', multiple raw reads may be issued to satisfy
 |  the byte count (unless EOF is reached first).  But for
 |  interactive raw streams (as well as sockets and pipes), at most
 |  one raw read will be issued, and a short result does not imply
 |  that EOF is imminent.
 |  
 |  Returns an empty bytes object on EOF.
 |  
 |  Returns None if the underlying raw stream was open in non-blocking
 |  mode and no data is available at the moment.

In the mainframe world, many files are "binary" of some sort and dealing
with record boundries is normal and assumed.  This means for fixed length
records counting of LRECL bytes and for variable length records using
the length in the RDW to determine the record length.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Michael Oujesky]

Just another tidbit, but when combining the record segments, while 
the VBS architecture does not specify a maximum record length, you 
can expect the full records to be up to 16,777,215 (16384K - 1) bytes 
in length.


Realizing that the RDW is actually a SDW.

Michael

At 01:26 AM 5/16/2023, Michael Stein wrote:

On Tue, May 16, 2023 at 04:14:07AM +, Prashant Joshi wrote:
>> Did you specify binary mode on the python open call? --

> Yes. And I can read the data.

How are you reading the data.  Assuming an open like:

myfile = open("filename", "rb")

You need to either read it all into memory:

alldata = myfile.read()

or read specific lengths which is messier as you need to read specific
lengths, first 4 bytes for the RDW and then the length of the record
in the RDW-4 (as you already read the RDW).

The 4 byte RDW includes the length of the record in the first 2 bytes
(bigendian order) and the spanning bits in the last 2 bytes.

Either way you need to walk your way through the binary data, any code
looking for CR or NL or space isn't correct.

A description of VBS records formats:

z/OS 2.4 DFSMS Using Data Sets SC23-6855-40
https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R4sc236855/$file/idad400_v2r4.pdf

pdf page 273 Variable-Length Record Formats
(near bottom of page and continues on to more pages)

> its just that I don't get proper end of reord. Hence every time I
> read record, I get random record length (multiple records/half records
> combined)

There aren't any "end of records" in a VBS file.  At the begining of
the file you know you are at the start of a RDW (Well, BDW/RDW but I'm
assuming the FTP removed the BDWs).

You can find the next record by going the length specified in the RDW
into the file -- that is the start of the next RDW.  Continue until
you've processed all the records.

More help will likely require you to show some code and/or data so
we can see what is going on...

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Seymour J Metz]

The code

with open('workfile', encoding="utf-8") as f:
read_data = f.read()

is for reading text and you would normally be concerned with the character 
count. Try
with open('workfile', "rb") as f:
read_data = f.read()


From: IBM Mainframe Discussion List  on behalf of 
Paul Gilmartin <042bfe9c879d-dmarc-requ...@listserv.ua.edu>
Sent: Tuesday, May 16, 2023 12:23 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: VBS file read in windows - end of record issue

On Tue, 16 May 2023 11:06:59 -0500, John McKown wrote:

>In one of my C programs, I first read the RDW, did a ntohs() to convert
>from mainframe to Intel integer, subtracted 4, then read that number of
>bytes into a char[32768].
>
I'm Python-naive.  But trying to educate myself with the example,
<https://docs.python.org/3/tutorial/inputoutput.html>:
with open('workfile', encoding="utf-8") as f:
read_data = f.read()

... I see no explicit byte count.  Is that implied, perhaps by a declaration
such as "char read_data[ LRECL ]"?

--
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Paul Gilmartin]

On Tue, 16 May 2023 11:06:59 -0500, John McKown wrote:

>In one of my C programs, I first read the RDW, did a ntohs() to convert
>from mainframe to Intel integer, subtracted 4, then read that number of
>bytes into a char[32768].
>
I'm Python-naive.  But trying to educate myself with the example,
:
with open('workfile', encoding="utf-8") as f:
read_data = f.read()

... I see no explicit byte count.  Is that implied, perhaps by a declaration
such as "char read_data[ LRECL ]"?

-- 
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[John McKown]

In one of my C programs, I first read the RDW, did a ntohs() to convert
from mainframe to Intel integer, subtracted 4, then read that number of
bytes into a char[32768].

On Tue, May 16, 2023 at 7:46 AM Paul Gilmartin <
042bfe9c879d-dmarc-requ...@listserv.ua.edu> wrote:

> On Tue, 16 May 2023 09:03:56 +, Prashant Joshi wrote:
> >
> >I was trying to read file using "with open as".  This way I was expecting
> to pull single record at every iteration of loop.
> >With your suggestion, I opened entire file in memory and read each record
> based on record length in RDW. Now I can read each record as expected.
> >
> Are the line breaks then not within the binary file but an artifact of the
> read()  method of the file object created by "with open() as"?
>
> If the file is too large to open in memory, is it possible to read it in
> small pieces of an arbitrary number of byres and reassemble the
> VBS logical records with a concatenation operator according to
> the SDWs (RDWs)?
>
> --
> gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Paul Gilmartin]

On Tue, 16 May 2023 09:03:56 +, Prashant Joshi wrote:
>
>I was trying to read file using "with open as".  This way I was expecting to 
>pull single record at every iteration of loop. 
>With your suggestion, I opened entire file in memory and read each record 
>based on record length in RDW. Now I can read each record as expected.
> 
Are the line breaks then not within the binary file but an artifact of the
read()  method of the file object created by "with open() as"?

If the file is too large to open in memory, is it possible to read it in
small pieces of an arbitrary number of byres and reassemble the
VBS logical records with a concatenation operator according to
the SDWs (RDWs)?

-- 
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Prashant Joshi]

Michael Stein, 

You are right. Thank very much for pointing to the mistake.

I was trying to read file using "with open as".  This way I was expecting to 
pull single record at every iteration of loop. 
With your suggestion, I opened entire file in memory and read each record based 
on record length in RDW. Now I can read each record as expected.

Once again thank you to all member for taking time to read my query and 
responding. Its good learning.

Thank you,
Prashant Joshi

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Michael Stein
Sent: Tuesday, May 16, 2023 11:56 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: [EXTERNAL] Re: VBS file read in windows - end of record issue

On Tue, May 16, 2023 at 04:14:07AM +, Prashant Joshi wrote:
>> Did you specify binary mode on the python open call? --

> Yes. And I can read the data.

How are you reading the data.  Assuming an open like:

myfile = open("filename", "rb")

You need to either read it all into memory:

alldata = myfile.read()

or read specific lengths which is messier as you need to read specific lengths, 
first 4 bytes for the RDW and then the length of the record in the RDW-4 (as 
you already read the RDW).

The 4 byte RDW includes the length of the record in the first 2 bytes 
(bigendian order) and the spanning bits in the last 2 bytes.

Either way you need to walk your way through the binary data, any code looking 
for CR or NL or space isn't correct.

A description of VBS records formats:

z/OS 2.4 DFSMS Using Data Sets SC23-6855-40 
https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R4sc236855/$file/idad400_v2r4.pdf
 

pdf page 273 Variable-Length Record Formats (near bottom of page and continues 
on to more pages)

> its just that I don't get proper end of reord. Hence every time I read 
> record, I get random record length (multiple records/half records
> combined)

There aren't any "end of records" in a VBS file.  At the begining of the file 
you know you are at the start of a RDW (Well, BDW/RDW but I'm assuming the FTP 
removed the BDWs).

You can find the next record by going the length specified in the RDW into the 
file -- that is the start of the next RDW.  Continue until you've processed all 
the records.

More help will likely require you to show some code and/or data so we can see 
what is going on...

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Prashant Joshi]

Hello Paul,

"ignores 0D0A" means it treat is as any other data.
And by "line break" I mean end of line/record and so next record will start 
from new line.

Thank you,
Prashant Joshi


-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Paul Gilmartin
Sent: Tuesday, May 16, 2023 11:52 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: [EXTERNAL] Re: VBS file read in windows - end of record issue

On Tue, 16 May 2023 04:45:32 +, Prashant Joshi  wrote:

>... If it completely ignores 0D0A and treat entire file as single 
> record,
>
By "ignores 0D0A" do you mean it deletes them, or treats them as ordinary 
characters?


>I... My problem is, it adds random line break and so when I read each 
>record in loop, 
>
What's a "line break"?

I don't know Python, but it might help others if you show your code.

--
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Michael Stein]

On Tue, May 16, 2023 at 04:14:07AM +, Prashant Joshi wrote:
>> Did you specify binary mode on the python open call? -- 

> Yes. And I can read the data.

How are you reading the data.  Assuming an open like:

myfile = open("filename", "rb")

You need to either read it all into memory:

alldata = myfile.read()

or read specific lengths which is messier as you need to read specific
lengths, first 4 bytes for the RDW and then the length of the record
in the RDW-4 (as you already read the RDW).

The 4 byte RDW includes the length of the record in the first 2 bytes
(bigendian order) and the spanning bits in the last 2 bytes.

Either way you need to walk your way through the binary data, any code
looking for CR or NL or space isn't correct.

A description of VBS records formats:

z/OS 2.4 DFSMS Using Data Sets SC23-6855-40
https://www-40.ibm.com/servers/resourcelink/svc00100.nsf/pages/zOSV2R4sc236855/$file/idad400_v2r4.pdf

pdf page 273 Variable-Length Record Formats 
(near bottom of page and continues on to more pages)

> its just that I don't get proper end of reord. Hence every time I
> read record, I get random record length (multiple records/half records
> combined) 

There aren't any "end of records" in a VBS file.  At the begining of
the file you know you are at the start of a RDW (Well, BDW/RDW but I'm
assuming the FTP removed the BDWs).

You can find the next record by going the length specified in the RDW
into the file -- that is the start of the next RDW.  Continue until
you've processed all the records.

More help will likely require you to show some code and/or data so
we can see what is going on...

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Paul Gilmartin]

On Tue, 16 May 2023 04:45:32 +, Prashant Joshi  wrote:

>... If it completely ignores 0D0A and treat entire file as single record, 
>
By "ignores 0D0A" do you mean it deletes them, or treats them as ordinary 
characters?


>I... My problem is, it adds random line break and so when I read each 
>record in loop, 
>
What's a "line break"?

I don't know Python, but it might help others if you show your code.

-- 
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Prashant Joshi]

Thank you for your response, Charles,

I preserve RDW and able to read record length field. I am also able to 
successfully covert EBCDIC to ASCII and Packed to decimal. So there is no issue 
with data conversion. As you correctly point out, intel/windows does not 
understand 0D0A. If it completely ignores 0D0A and treat entire file as single 
record, I am still ok with that as I can read records, based on record length 
field in RDW. My problem is, it adds random line break and so when I read each 
record in loop, it returns me records with record length between 1 to 
thousands. My records are in range of 150 to 250 length.

Thanks,
Prashant 


-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Charles Mills
Sent: Tuesday, May 16, 2023 5:33 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: [EXTERNAL] Re: VBS file read in windows - end of record issue

There are so many layers of issues here ...

If the data is "binary" (any or nearly any byte value is possible) then no 
approach that involves CRLF is going to work. Period. Software does not have a 
way to distinguish between a 0D0A that signifies the end of a record versus a 
0D0A that just happens to appear in the data -- say a halfword of 3338 decimal.

If you preserve the RDW keep in mind that the Z is a "big-endian" machine: 
x'0050' represents decimal 80, but Intel is "little-endian": decimal 80 is 
x'5000'. You will have to account for that in processing the data.

If the records contain decimal packed data there are not many programs in 
Windows that can process packed data.

If there are character fields in the data then they are going to have to be 
translated from EBCDIC to ASCII in order to be generally usable. That's a whole 
topic of its own.

I think without a real good understanding of the record conventions on both 
machines that success is unlikely.

Here is how I have done this

ftp> quote type i
200 Representation type is Image
ftp> quote mode s
200 Data transfer mode is Stream
ftp> quote site rdw
200 SITE command was accepted
ftp> get zos.vfromat.file

Then in my C++ code I read 4 bytes, decode the RDW accounting for the "endian" 
issue, and then read the rest of the record.

Charles

On Mon, 15 May 2023 18:24:12 -0500, Michael Oujesky  
wrote:

>Depend on what you need to do and there at least two levels to be addressed:
>* How variable length records appear on Windows
>* How to deal with SDW (spanned long record segments If you want  
>both BDW and RDW/SDW present on Windows at least two approaches:
>* TERSE the file, binary transfer to windows, WWUNTERSE from Watson 
>& Walker) on windows (-V option removes BDW)
>* FTP binary transfer from z/OS to Windows using //DD: and JCL to 
>force RDCFM=U Regular FTP binary transfer with "SITE RDW" will give you 
>the file with just RDW/SDW (BDW is stripped off).  Without the "SITE 
>RDW", you just get the data with no indication of where the record 
>ends).
>
>Then there is the issue of how to re-combine the segments into the full 
>record length.

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Prashant Joshi]

Thank you, Michael,

I used WWUNTERSE for my terse file. I am able to get RDW & BDW and also able to 
read data after proper conversion. Problem is when I read file for each record 
(using loop), I get different record lengths for each record, in range of 
single digit to thousands. Seems it adds "new line" randomly in file. I can use 
record length field in RDW to read each records but this random "new line" 
breaks my loop.
I also tried to STRIP (remove additional space from end of record) these 
records to remove "new line" character and converted entire file as one single 
record. Now I get only one record and can use record length field in RDW to 
split file. But then this STRIP option removes some record information as well 
along with "new line" character. Hence it affects my real record length.



Thanks,
Prashant


-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Michael Oujesky
Sent: Tuesday, May 16, 2023 4:54 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: [EXTERNAL] Re: VBS file read in windows - end of record issue

Depend on what you need to do and there at least two levels to be addressed:
* How variable length records appear on Windows
* How to deal with SDW (spanned long record segments If you want  both BDW 
and RDW/SDW present on Windows at least two approaches:
* TERSE the file, binary transfer to windows, WWUNTERSE from Watson & 
Walker) on windows (-V option removes BDW)
* FTP binary transfer from z/OS to Windows using //DD: and JCL to force 
RDCFM=U Regular FTP binary transfer with "SITE RDW" will give you the file with 
just RDW/SDW (BDW is stripped off).  Without the "SITE RDW", you just get the 
data with no indication of where the record ends).

Then there is the issue of how to re-combine the segments into the full record 
length.

Does that help any?

Michael


At 01:54 PM 5/15/2023, Prashant Joshi wrote:
>I am trying to read VB & VBS binary file in windows using python. I 
>tried FTPing file directly, then using Terse and then XMIT but every 
>time when I read the file in windows, I get random new line (CR/LF) 
>inserted in record. I am not able to read complete record.
>
>For some files, when I transfer using IND$FILE (option 6) using Binary 
>and CRLF option checked, I gets record properly. But it does not work 
>for every file. And due to different file sizes, IND$FILE may not be 
>option for me.
>
>
>
>Need your help to find, what am I missing?
>
>
>Thanks,
>Prashant
>
>--
>For IBM-MAIN subscribe / signoff / archive access instructions, send 
>email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Prashant Joshi]

Thank for your response, Paul.

I agree, CRLF irrelevant to binary files. Surprisingly, only after using CRLF 
option, I get those binary records in proper record length.
I tried to force it to U format but did not work for m. I will try again with 
different methods.

I can preserve RDW and BDW. How to preserve SDW? I did not see mention of SDW 
in these options. May be SDW is the key to maintain record length.

Thank you,
Prashant Joshi
Mainframe Architect
+91 9743 440 503 (Mobile) 


-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Paul Gilmartin
Sent: Tuesday, May 16, 2023 1:38 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: [EXTERNAL] Re: VBS file read in windows - end of record issue

On Mon, 15 May 2023 18:54:40 +, Prashant Joshi  wrote:
>...
>For some files, when I transfer using IND$FILE (option 6) using Binary and 
>CRLF option checked, I gets record properly. But it does not work for every 
>file. And due to different file sizes, IND$FILE may not be option for me.
>
CRLF should be irrelevant for binary files.


>Need your help to find, what am I missing?
>
Allocate your data set overriding to RECFM=U.  Tne BDWs and SDWs should appear 
ad data.

Transfer the file with a method that preserves those BDWs and SDWs.  (I don't 
recommend
IND$FILE.)

Decode the SDWs and translate to UTF-8  with your Python.

I hate EBCDIC!

--
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Prashant Joshi]

Thank you for your response, Michael,

1. FTP -> what options did you use? -- I use Binary and QUOTE SITE RDW

2. Terse & XMIT and then ? how did it get to windows? -- I used WWUNTERSE 
(>From Watson & Walker) for TERSE and XMIT manager (from CBT tape) for XMIT 

3. IND$FILE which you decided isn't really a candidate due to file sizes. -- 
That's correct

Which python: python 2 or python 3? -- Python 3

Did you specify binary mode on the python open call? -- Yes. And I can read the 
data. Its just that I don't get proper end of record. Hence every time I read 
record, I get random record length (multiple records/half records combined)

Thanks,
Prashant

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Michael Stein
Sent: Tuesday, May 16, 2023 1:37 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: [EXTERNAL] Re: VBS file read in windows - end of record issue

On Mon, May 15, 2023 at 06:54:40PM +, Prashant Joshi wrote:
> I am trying to read VB & VBS binary file in windows using python. I 
> tried FTPing file directly, then using Terse and then XMIT but every 
> time when I read the file in windows, I get random new line (CR/LF) 
> inserted in record. I am not able to read complete record.

Let's see if I understand the problem, here's a summary I see:

You tried 3 transfer mechanisms for the z/OS to your windows machine (I'm 
assuming it's from z/OS since this is the mainframe list).

1. FTP -> what options did you use?

2. Terse & XMIT and then ? how did it get to windows?

   What do you have on windows which can process this combination
   of format?

3. IND$FILE which you decided isn't really a candidate due to file sizes.

On the windows machine you are trying to read the resulting file with python.

Which python: python 2 or python 3?

Did you specify binary mode on the python open call?

The raw VB & VBS files sound relatively easy to read with python by reading in 
binary mode and using the BDW/RDW information to separate the records.  

I'm not sure how you would deal with undoing the XMIT and Terse on windows once 
you got an intact file there.

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Charles Mills]

There are so many layers of issues here ...

If the data is "binary" (any or nearly any byte value is possible) then no 
approach that involves CRLF is going to work. Period. Software does not have a 
way to distinguish between a 0D0A that signifies the end of a record versus a 
0D0A that just happens to appear in the data -- say a halfword of 3338 decimal.

If you preserve the RDW keep in mind that the Z is a "big-endian" machine: 
x'0050' represents decimal 80, but Intel is "little-endian": decimal 80 is 
x'5000'. You will have to account for that in processing the data.

If the records contain decimal packed data there are not many programs in 
Windows that can process packed data.

If there are character fields in the data then they are going to have to be 
translated from EBCDIC to ASCII in order to be generally usable. That's a whole 
topic of its own.

I think without a real good understanding of the record conventions on both 
machines that success is unlikely.

Here is how I have done this

ftp> quote type i
200 Representation type is Image
ftp> quote mode s
200 Data transfer mode is Stream
ftp> quote site rdw
200 SITE command was accepted
ftp> get zos.vfromat.file

Then in my C++ code I read 4 bytes, decode the RDW accounting for the "endian" 
issue, and then read the rest of the record.

Charles

On Mon, 15 May 2023 18:24:12 -0500, Michael Oujesky  
wrote:

>Depend on what you need to do and there at least two levels to be addressed:
>* How variable length records appear on Windows
>* How to deal with SDW (spanned long record segments
>If you want  both BDW and RDW/SDW present on Windows at least two approaches:
>* TERSE the file, binary transfer to windows, WWUNTERSE from
>Watson & Walker) on windows (-V option removes BDW)
>* FTP binary transfer from z/OS to Windows using //DD: and JCL to
>force RDCFM=U
>Regular FTP binary transfer with "SITE RDW" will give you the file
>with just RDW/SDW (BDW is stripped off).  Without the "SITE RDW", you
>just get the data with no indication of where the record ends).
>
>Then there is the issue of how to re-combine the segments into the
>full record length.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Michael Oujesky]

Depend on what you need to do and there at least two levels to be addressed:
   * How variable length records appear on Windows
   * How to deal with SDW (spanned long record segments
If you want  both BDW and RDW/SDW present on Windows at least two approaches:
   * TERSE the file, binary transfer to windows, WWUNTERSE from 
Watson & Walker) on windows (-V option removes BDW)
   * FTP binary transfer from z/OS to Windows using //DD: and JCL to 
force RDCFM=U
Regular FTP binary transfer with "SITE RDW" will give you the file 
with just RDW/SDW (BDW is stripped off).  Without the "SITE RDW", you 
just get the data with no indication of where the record ends).


Then there is the issue of how to re-combine the segments into the 
full record length.


Does that help any?

Michael


At 01:54 PM 5/15/2023, Prashant Joshi wrote:
I am trying to read VB & VBS binary file in windows using python. I 
tried FTPing file directly, then using Terse and then XMIT but every 
time when I read the file in windows, I get random new line (CR/LF) 
inserted in record. I am not able to read complete record.


For some files, when I transfer using IND$FILE (option 6) using 
Binary and CRLF option checked, I gets record properly. But it does 
not work for every file. And due to different file sizes, IND$FILE 
may not be option for me.




Need your help to find, what am I missing?


Thanks,
Prashant

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Paul Gilmartin]

On Mon, 15 May 2023 18:54:40 +, Prashant Joshi  wrote:
>...
>For some files, when I transfer using IND$FILE (option 6) using Binary and 
>CRLF option checked, I gets record properly. But it does not work for every 
>file. And due to different file sizes, IND$FILE may not be option for me.
>
CRLF should be irrelevant for binary files.


>Need your help to find, what am I missing?
>
Allocate your data set overriding to RECFM=U.  Tne BDWs and SDWs should appear 
ad data.

Transfer the file with a method that preserves those BDWs and SDWs.  (I don't 
recommend
IND$FILE.)

Decode the SDWs and translate to UTF-8  with your Python.

I hate EBCDIC!

-- 
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: VBS file read in windows - end of record issue

[Michael Stein]

On Mon, May 15, 2023 at 06:54:40PM +, Prashant Joshi wrote:
> I am trying to read VB & VBS binary file in windows using python. I
> tried FTPing file directly, then using Terse and then XMIT but every
> time when I read the file in windows, I get random new line (CR/LF)
> inserted in record. I am not able to read complete record.

Let's see if I understand the problem, here's a summary I see:

You tried 3 transfer mechanisms for the z/OS to your windows machine
(I'm assuming it's from z/OS since this is the mainframe list).

1. FTP -> what options did you use?

2. Terse & XMIT and then ? how did it get to windows?

   What do you have on windows which can process this combination
   of format?

3. IND$FILE which you decided isn't really a candidate due to file sizes.

On the windows machine you are trying to read the resulting file with
python.

Which python: python 2 or python 3?

Did you specify binary mode on the python open call?

The raw VB & VBS files sound relatively easy to read with python by
reading in binary mode and using the BDW/RDW information to separate
the records.  

I'm not sure how you would deal with undoing the XMIT and Terse
on windows once you got an intact file there.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

VBS file read in windows - end of record issue

[Prashant Joshi]

I am trying to read VB & VBS binary file in windows using python. I tried 
FTPing file directly, then using Terse and then XMIT but every time when I read 
the file in windows, I get random new line (CR/LF) inserted in record. I am not 
able to read complete record.

For some files, when I transfer using IND$FILE (option 6) using Binary and CRLF 
option checked, I gets record properly. But it does not work for every file. 
And due to different file sizes, IND$FILE may not be option for me.



Need your help to find, what am I missing?


Thanks,
Prashant

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN