Re: Improve OMVS cp performance?

[David Crayford]

On 18/11/2020 11:47 pm, Kirk Wolf wrote:

Hey David,

Thank you for rescuing this thread from the ash heap of old hardware model
numbers and tape formats :-)


Topic drift makes this forum almost unreadable these days!



We already have on our list to add ISPF-style enqueue serialization to
putpds.  It just didn't make the first release.
Currently putpds allocates the data set with DISP=OLD  (SYSDSN EXCL).


Awesome. We have coz installed on all LPARS on multiple plexes. I'll 
install the lastest version locally and wait until the next release

with ISPF-style ENQs before requesting a global roll-out.

Cheers




On Tue, Nov 17, 2020 at 11:02 PM David Crayford  wrote:


This is really cool. We could use this right now for our SCLM/Git
Integration tooling.

Q: How does it handle member ENQs. Does it ENQ using SPFEDIT or SYSDSN?
One of the problems we ran into with "cp" copying an entire data set is
it fails if one member is in use.
We worked around this by writing a cp_pds_to_dir command.

On 13/11/2020 9:54 pm, Kirk Wolf wrote:

I wanted to update this thread with a bit of news.   We have had the
enjoyable experience of working with Lionel and Henri (please check out:
http://zigi.rocks) on requirements for some new shell commands for the

Co:Z

Toolkit V6.2.0, which we released this week.

The new commands (getpds and putpds) use BPAM+BSAM along with our

existing

Co:Z record <-> stream processing framework to allow you to copy PDS
members to and from z/OS UNIX files.  Also included are extensive options
for ISPF stats processing.

The performance (when compared to "/bin/cp") is better than we could have
expected.  The following example copies all of the members of SYS1.MACLIB
to text files in the current directory:

$ getpds //sys1.maclib .
getpds(SYS1.MACLIB)[N]: 2015 members/2435218 records/194817440 bytes

read;

  194011101 bytes written in 1.791 seconds

(103.307

MBytes/sec).

For more information:
https://dovetail.com/docs/zos-utilities/dsp-ref_getpds.html
https://dovetail.com/docs/zos-utilities/dsp-ref_putpds.html
https://dovetail.com/docs/cozinstall/changes.html
Co:Z is available free under our Community License
.

Kirk Wolf
http://dovetail.com


On Mon, Jun 22, 2020 at 9:30 AM Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:


On Wed, 17 Jun 2020 07:14:39 -0500, Lionel B Dyck wrote:


Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see https://zigi.rocks) where I need to

copy PDS members to/from USS so that Git can manage them. With small
projects this isn't an issue but with larger projects it could take

enough

time for you to go to lunch ☹

Btw. I voted your RFE.


I notice that replies this thread have focused on Classic PDS as the
performance culprit.  Has someone tried the benchmarks:

  cp UNIX -> UNIX  vs.  cp PDS -> PDS

... and compared performance?  I suspect Classic OS was optimized
for a few large data sets; UNIX for many small files.

Might there be an argument here for maintaining the data where it
works best and omitting the copying?


-Original Message-
From: Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM

FWIW: It's a pity that the IBM C library doesn't have any support for

BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you

agree:


https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Kirk Wolf]

Hey David,

Thank you for rescuing this thread from the ash heap of old hardware model
numbers and tape formats :-)

We already have on our list to add ISPF-style enqueue serialization to
putpds.  It just didn't make the first release.
Currently putpds allocates the data set with DISP=OLD  (SYSDSN EXCL).


On Tue, Nov 17, 2020 at 11:02 PM David Crayford  wrote:

> This is really cool. We could use this right now for our SCLM/Git
> Integration tooling.
>
> Q: How does it handle member ENQs. Does it ENQ using SPFEDIT or SYSDSN?
> One of the problems we ran into with "cp" copying an entire data set is
> it fails if one member is in use.
> We worked around this by writing a cp_pds_to_dir command.
>
> On 13/11/2020 9:54 pm, Kirk Wolf wrote:
> > I wanted to update this thread with a bit of news.   We have had the
> > enjoyable experience of working with Lionel and Henri (please check out:
> > http://zigi.rocks) on requirements for some new shell commands for the
> Co:Z
> > Toolkit V6.2.0, which we released this week.
> >
> > The new commands (getpds and putpds) use BPAM+BSAM along with our
> existing
> > Co:Z record <-> stream processing framework to allow you to copy PDS
> > members to and from z/OS UNIX files.  Also included are extensive options
> > for ISPF stats processing.
> >
> > The performance (when compared to "/bin/cp") is better than we could have
> > expected.  The following example copies all of the members of SYS1.MACLIB
> > to text files in the current directory:
> >
> > $ getpds //sys1.maclib .
> > getpds(SYS1.MACLIB)[N]: 2015 members/2435218 records/194817440 bytes
> read;
> >  194011101 bytes written in 1.791 seconds
> (103.307
> > MBytes/sec).
> >
> > For more information:
> > https://dovetail.com/docs/zos-utilities/dsp-ref_getpds.html
> > https://dovetail.com/docs/zos-utilities/dsp-ref_putpds.html
> > https://dovetail.com/docs/cozinstall/changes.html
> > Co:Z is available free under our Community License
> > .
> >
> > Kirk Wolf
> > http://dovetail.com
> >
> >
> > On Mon, Jun 22, 2020 at 9:30 AM Paul Gilmartin <
> > 000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
> >
> >> On Wed, 17 Jun 2020 07:14:39 -0500, Lionel B Dyck wrote:
> >>
> >>> Kirk - thank you for the ideas.
> >>>
> >>> What I'm doing is in the ZIGI (see https://zigi.rocks) where I need to
> >> copy PDS members to/from USS so that Git can manage them. With small
> >> projects this isn't an issue but with larger projects it could take
> enough
> >> time for you to go to lunch ☹
> >>> Btw. I voted your RFE.
> >>>
> >> I notice that replies this thread have focused on Classic PDS as the
> >> performance culprit.  Has someone tried the benchmarks:
> >>
> >>  cp UNIX -> UNIX  vs.  cp PDS -> PDS
> >>
> >> ... and compared performance?  I suspect Classic OS was optimized
> >> for a few large data sets; UNIX for many small files.
> >>
> >> Might there be an argument here for maintaining the data where it
> >> works best and omitting the copying?
> >>
> >>> -Original Message-
> >>> From: Kirk Wolf
> >>> Sent: Wednesday, June 17, 2020 7:03 AM
> >>>
> >>> FWIW: It's a pity that the IBM C library doesn't have any support for
> >> BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
> >>> agree:
> >>>
> >>
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811
> >>
> >> -- gil
> >>
> >> --
> >> For IBM-MAIN subscribe / signoff / archive access instructions,
> >> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> >>
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Wed, 18 Nov 2020 13:01:53 +0800, David Crayford wrote:

>This is really cool. We could use this right now for our SCLM/Git
>Integration tooling.
>
>Q: How does it handle member ENQs. Does it ENQ using SPFEDIT or SYSDSN?
>One of the problems we ran into with "cp" copying an entire data set is
>it fails if one member is in use.
>We worked around this by writing a cp_pds_to_dir command.
> 
Further, does LMCOPY use BPAM, avoiding the multiple OPEN/CLOSE overhead
and SPFEDIT, avoiding the EXC SYSDSN lockout?

But the killers are that LMCOPY requires LMINIT, not compatible with UNIX
files, and LMPUT requires EXC SYSDSN, even for PDSE.

Grrr...

There are several possible RFEs, none of which I ever submitted:

o LMINIT should support UNIX files.

o fopen() should optionally support the SPFEDIT alternative.
  This has been discussed previously in these fora.

o SHR should suffice for LMPUT to PDSE.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[David Crayford]

This is really cool. We could use this right now for our SCLM/Git 
Integration tooling.


Q: How does it handle member ENQs. Does it ENQ using SPFEDIT or SYSDSN? 
One of the problems we ran into with "cp" copying an entire data set is 
it fails if one member is in use.

We worked around this by writing a cp_pds_to_dir command.

On 13/11/2020 9:54 pm, Kirk Wolf wrote:

I wanted to update this thread with a bit of news.   We have had the
enjoyable experience of working with Lionel and Henri (please check out:
http://zigi.rocks) on requirements for some new shell commands for the Co:Z
Toolkit V6.2.0, which we released this week.

The new commands (getpds and putpds) use BPAM+BSAM along with our existing
Co:Z record <-> stream processing framework to allow you to copy PDS
members to and from z/OS UNIX files.  Also included are extensive options
for ISPF stats processing.

The performance (when compared to "/bin/cp") is better than we could have
expected.  The following example copies all of the members of SYS1.MACLIB
to text files in the current directory:

$ getpds //sys1.maclib .
getpds(SYS1.MACLIB)[N]: 2015 members/2435218 records/194817440 bytes read;
 194011101 bytes written in 1.791 seconds (103.307
MBytes/sec).

For more information:
https://dovetail.com/docs/zos-utilities/dsp-ref_getpds.html
https://dovetail.com/docs/zos-utilities/dsp-ref_putpds.html
https://dovetail.com/docs/cozinstall/changes.html
Co:Z is available free under our Community License
.

Kirk Wolf
http://dovetail.com


On Mon, Jun 22, 2020 at 9:30 AM Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:


On Wed, 17 Jun 2020 07:14:39 -0500, Lionel B Dyck wrote:


Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see https://zigi.rocks) where I need to

copy PDS members to/from USS so that Git can manage them. With small
projects this isn't an issue but with larger projects it could take enough
time for you to go to lunch ☹

Btw. I voted your RFE.


I notice that replies this thread have focused on Classic PDS as the
performance culprit.  Has someone tried the benchmarks:

 cp UNIX -> UNIX  vs.  cp PDS -> PDS

... and compared performance?  I suspect Classic OS was optimized
for a few large data sets; UNIX for many small files.

Might there be an argument here for maintaining the data where it
works best and omitting the copying?


-Original Message-
From: Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM

FWIW: It's a pity that the IBM C library doesn't have any support for

BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you

agree:


https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[R.S.]

W dniu 17.11.2020 o 01:57, Wayne Bickerdike pisze:

Atlas rang a bell and then I remembered it was used at the Rutherford labs
in Chilton.

I worked at AERE (Atomic Energy Research Establishment) which was next
door. They were basically the same campus but AERE was secure and
Rutherford wasn't.


[...]

Chilton!
Huge set of old pictures!
And rare gem: picture of 2305 DASD.

Also some pictures of Atlas parts, S/360, first Cray, ICL machines, etc.

--
Radoslaw Skorupka
Lodz, Poland





==

Jeśli nie jesteś adresatem tej wiadomości:

- powiadom nas o tym w mailu zwrotnym (dziękujemy!),
- usuń trwale tę wiadomość (i wszystkie kopie, które wydrukowałeś lub zapisałeś 
na dysku).
Wiadomość ta może zawierać chronione prawem informacje, które może wykorzystać 
tylko adresat.Przypominamy, że każdy, kto rozpowszechnia (kopiuje, rozprowadza) 
tę wiadomość lub podejmuje podobne działania, narusza prawo i może podlegać 
karze.

mBank S.A. z siedzibą w Warszawie, ul. Senatorska 18, 00-950 
Warszawa,www.mBank.pl, e-mail: kont...@mbank.pl. Sąd Rejonowy dla m. st. 
Warszawy XII Wydział Gospodarczy Krajowego Rejestru Sądowego, KRS 025237, 
NIP: 526-021-50-88. Kapitał zakładowy (opłacony w całości) według stanu na 
01.01.2020 r. wynosi 169.401.468 złotych.

If you are not the addressee of this message:

- let us know by replying to this e-mail (thank you!),
- delete this message permanently (including all the copies which you have 
printed out or saved).
This message may contain legally protected information, which may be used 
exclusively by the addressee.Please be reminded that anyone who disseminates 
(copies, distributes) this message or takes any similar action, violates the 
law and may be penalised.

mBank S.A. with its registered office in Warsaw, ul. Senatorska 18, 00-950 
Warszawa,www.mBank.pl, e-mail: kont...@mbank.pl. District Court for the Capital 
City of Warsaw, 12th Commercial Division of the National Court Register, KRS 
025237, NIP: 526-021-50-88. Fully paid-up share capital amounting to PLN 
169.401.468 as at 1 January 2020.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Attila Fogarasi]

One reason that 48 bit words were so popular in this class of machine is
that there are some significant differential equation solutions which
converge at 48 bit but diverge at 32 bit floating point.  So on the S/360
you would have to use 64 bit floating point to solve them.  When memory
cost USD 1 per bit that difference was a big impact.

On Tue, Nov 17, 2020 at 11:58 AM Wayne Bickerdike  wrote:

> Atlas rang a bell and then I remembered it was used at the Rutherford labs
> in Chilton.
>
> I worked at AERE (Atomic Energy Research Establishment) which was next
> door. They were basically the same campus but AERE was secure and
> Rutherford wasn't.
>
> Ferranti sold two other Atlas installations, one to a joint consortium
> of London
> University <https://en.wikipedia.org/wiki/London_University> and British
> Petroleum <https://en.wikipedia.org/wiki/British_Petroleum> in 1963, and
> another to the Atomic Energy Research Establishment
> <https://en.wikipedia.org/wiki/Atomic_Energy_Research_Establishment>
> (Harwell)
> in December 1964. The AEA machine was later moved to the Atlas Computer
> Laboratory at Chilton, a few yards outside the boundary fence of Harwell,
> which placed it on civilian lands and thus much easier to access. This
> installation grew to be the largest Atlas, containing 48 kWords of 48-bit
> core
> memory <https://en.wikipedia.org/wiki/Core_memory> and 32 tape drives.
>
> Funny story:
>
> Somebody stole a lathe from the Rutherford labs (c. 1968). After that
> happened they installed a gate and security guards at the lab.
>
> At Harwell, we were checked for ID on entry, at Rutherford you were checked
> for lathes on exit!
>
>
>
>
> On Tue, Nov 17, 2020 at 8:42 AM Bernd Oppolzer  >
> wrote:
>
> > Well, I didn't know much about Atlas until now, so I had to do some
> > research;
> > very impressive, indeed. Atlas is about 3 to 5 times faster than the TR
> 4.
> > But in contrast to Atlas, the TR 4 was produced in a sort of small
> > industrial series (35 machines);
> > Atlas seems to be a super computer with very few installations.
> >
> > What strikes me most are the similarities:
> >
> > - 48 bit words
> > - two 24 bit signed integers or six 8 bit-bytes or one 48 bit integer or
> > floating point
> > - or one instruction (two on TR 4 / TR 440)
> > - 24 bit address space (only 16 on the TR 4, later 22 on the TR 440)
> > - 16 k words of core store (32 k max on TR 4, 192 k words on the TR 440,
> > maybe more)
> > - 8 k words of read only memory for supervisor etc. (4 k on TR 4,
> > nothing on TR 440,
> > was loaded at startup time)
> >
> > Atlas was the first implementation of virtual memory,
> > but the first paper on virtual memory is from Fritz-Rudolf Güntsch from
> > 1957
> > https://de.wikipedia.org/wiki/Fritz-Rudolf_G%C3%BCntsch -
> > German wikipedia says: Güntsch is the inventor of virtual memory
> > (German title: F.-R. Güntsch: /Logischer Entwurf eines digitalen
> > Rechengeräts
> > mit mehreren asynchron laufenden Trommeln und automatischem
> > Schnellspeicherbetrieb./
> > TU Berlin, 1957 (Dissertation)) - and: Fritz-Rudolf Güntsch later (in
> > the 1960s)
> > worked for Telefunken and was the designer of the TR 4 and the TR 440 :-)
> >
> > Kind regards
> >
> > Bernd
> >
> >
> >
> > Am 16.11.2020 um 20:46 schrieb Seymour J Metz:
> > > Faster than Atlas?
> > >
> > >
> > > --
> > > Shmuel (Seymour J.) Metz
> > > http://mason.gmu.edu/~smetz3
> > >
> > >
> > > 
> > > From: IBM Mainframe Discussion List  on
> > behalf of Bernd Oppolzer 
> > > Sent: Sunday, November 15, 2020 5:43 PM
> > > To: IBM-MAIN@LISTSERV.UA.EDU
> > > Subject: Re: Improve OMVS cp performance?
> > >
> > > Telefunken TR 4, designed 1958, first delivered in 1962. This predates
> > > IBM/360
> > > by at least 4 years. The fastest mainframe built in Europe at that
> time.
> > > The internal code ("Zentralcode") was an 8-bit code using 256
> characters.
> > > Word structure, a word had 48 bits plus 2 tag bits (tagged
> architecture)
> > > plus some parity bits, not seen by the programmer.
> > > Mostly used with ALGOL; the TR 4 was "a Hardware implementation of
> ALGOL"
> > > (quote from E.J. Dijkstra).
> > > A word could hold up to six characters, but some later languages
> > > (like Fortran) decided to store only 4 characters in one word,
> > &

Re: Improve OMVS cp performance?

[Wayne Bickerdike]

Atlas rang a bell and then I remembered it was used at the Rutherford labs
in Chilton.

I worked at AERE (Atomic Energy Research Establishment) which was next
door. They were basically the same campus but AERE was secure and
Rutherford wasn't.

Ferranti sold two other Atlas installations, one to a joint consortium
of London
University <https://en.wikipedia.org/wiki/London_University> and British
Petroleum <https://en.wikipedia.org/wiki/British_Petroleum> in 1963, and
another to the Atomic Energy Research Establishment
<https://en.wikipedia.org/wiki/Atomic_Energy_Research_Establishment> (Harwell)
in December 1964. The AEA machine was later moved to the Atlas Computer
Laboratory at Chilton, a few yards outside the boundary fence of Harwell,
which placed it on civilian lands and thus much easier to access. This
installation grew to be the largest Atlas, containing 48 kWords of 48-bit core
memory <https://en.wikipedia.org/wiki/Core_memory> and 32 tape drives.

Funny story:

Somebody stole a lathe from the Rutherford labs (c. 1968). After that
happened they installed a gate and security guards at the lab.

At Harwell, we were checked for ID on entry, at Rutherford you were checked
for lathes on exit!




On Tue, Nov 17, 2020 at 8:42 AM Bernd Oppolzer 
wrote:

> Well, I didn't know much about Atlas until now, so I had to do some
> research;
> very impressive, indeed. Atlas is about 3 to 5 times faster than the TR 4.
> But in contrast to Atlas, the TR 4 was produced in a sort of small
> industrial series (35 machines);
> Atlas seems to be a super computer with very few installations.
>
> What strikes me most are the similarities:
>
> - 48 bit words
> - two 24 bit signed integers or six 8 bit-bytes or one 48 bit integer or
> floating point
> - or one instruction (two on TR 4 / TR 440)
> - 24 bit address space (only 16 on the TR 4, later 22 on the TR 440)
> - 16 k words of core store (32 k max on TR 4, 192 k words on the TR 440,
> maybe more)
> - 8 k words of read only memory for supervisor etc. (4 k on TR 4,
> nothing on TR 440,
> was loaded at startup time)
>
> Atlas was the first implementation of virtual memory,
> but the first paper on virtual memory is from Fritz-Rudolf Güntsch from
> 1957
> https://de.wikipedia.org/wiki/Fritz-Rudolf_G%C3%BCntsch -
> German wikipedia says: Güntsch is the inventor of virtual memory
> (German title: F.-R. Güntsch: /Logischer Entwurf eines digitalen
> Rechengeräts
> mit mehreren asynchron laufenden Trommeln und automatischem
> Schnellspeicherbetrieb./
> TU Berlin, 1957 (Dissertation)) - and: Fritz-Rudolf Güntsch later (in
> the 1960s)
> worked for Telefunken and was the designer of the TR 4 and the TR 440 :-)
>
> Kind regards
>
> Bernd
>
>
>
> Am 16.11.2020 um 20:46 schrieb Seymour J Metz:
> > Faster than Atlas?
> >
> >
> > --
> > Shmuel (Seymour J.) Metz
> > http://mason.gmu.edu/~smetz3
> >
> >
> > ____________
> > From: IBM Mainframe Discussion List  on
> behalf of Bernd Oppolzer 
> > Sent: Sunday, November 15, 2020 5:43 PM
> > To: IBM-MAIN@LISTSERV.UA.EDU
> > Subject: Re: Improve OMVS cp performance?
> >
> > Telefunken TR 4, designed 1958, first delivered in 1962. This predates
> > IBM/360
> > by at least 4 years. The fastest mainframe built in Europe at that time.
> > The internal code ("Zentralcode") was an 8-bit code using 256 characters.
> > Word structure, a word had 48 bits plus 2 tag bits (tagged architecture)
> > plus some parity bits, not seen by the programmer.
> > Mostly used with ALGOL; the TR 4 was "a Hardware implementation of ALGOL"
> > (quote from E.J. Dijkstra).
> > A word could hold up to six characters, but some later languages
> > (like Fortran) decided to store only 4 characters in one word,
> > to be more compatible with IBM Fortran.
> >
> > Kind regards
> >
> > Bernd
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>


-- 
Wayne V. Bickerdike

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Bernd Oppolzer]

Well, I didn't know much about Atlas until now, so I had to do some 
research;

very impressive, indeed. Atlas is about 3 to 5 times faster than the TR 4.
But in contrast to Atlas, the TR 4 was produced in a sort of small 
industrial series (35 machines);

Atlas seems to be a super computer with very few installations.

What strikes me most are the similarities:

- 48 bit words
- two 24 bit signed integers or six 8 bit-bytes or one 48 bit integer or 
floating point

- or one instruction (two on TR 4 / TR 440)
- 24 bit address space (only 16 on the TR 4, later 22 on the TR 440)
- 16 k words of core store (32 k max on TR 4, 192 k words on the TR 440, 
maybe more)
- 8 k words of read only memory for supervisor etc. (4 k on TR 4, 
nothing on TR 440,

was loaded at startup time)

Atlas was the first implementation of virtual memory,
but the first paper on virtual memory is from Fritz-Rudolf Güntsch from 
1957

https://de.wikipedia.org/wiki/Fritz-Rudolf_G%C3%BCntsch -
German wikipedia says: Güntsch is the inventor of virtual memory
(German title: F.-R. Güntsch: /Logischer Entwurf eines digitalen 
Rechengeräts
mit mehreren asynchron laufenden Trommeln und automatischem 
Schnellspeicherbetrieb./
TU Berlin, 1957 (Dissertation)) - and: Fritz-Rudolf Güntsch later (in 
the 1960s)

worked for Telefunken and was the designer of the TR 4 and the TR 440 :-)

Kind regards

Bernd



Am 16.11.2020 um 20:46 schrieb Seymour J Metz:

Faster than Atlas?


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3



From: IBM Mainframe Discussion List  on behalf of Bernd 
Oppolzer 
Sent: Sunday, November 15, 2020 5:43 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Telefunken TR 4, designed 1958, first delivered in 1962. This predates
IBM/360
by at least 4 years. The fastest mainframe built in Europe at that time.
The internal code ("Zentralcode") was an 8-bit code using 256 characters.
Word structure, a word had 48 bits plus 2 tag bits (tagged architecture)
plus some parity bits, not seen by the programmer.
Mostly used with ALGOL; the TR 4 was "a Hardware implementation of ALGOL"
(quote from E.J. Dijkstra).
A word could hold up to six characters, but some later languages
(like Fortran) decided to store only 4 characters in one word,
to be more compatible with IBM Fortran.

Kind regards

Bernd



--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[R.S.]

As a old computer hobbyist I've collected (very) few pictures of TR 4.
Wooden parts of cabinets, construction similar to contemporary 19" racks.
Nice machine, part of IT history.

--
Radoslaw Skorupka
Lodz, Poland





==

Jeśli nie jesteś adresatem tej wiadomości:

- powiadom nas o tym w mailu zwrotnym (dziękujemy!),
- usuń trwale tę wiadomość (i wszystkie kopie, które wydrukowałeś lub zapisałeś 
na dysku).
Wiadomość ta może zawierać chronione prawem informacje, które może wykorzystać 
tylko adresat.Przypominamy, że każdy, kto rozpowszechnia (kopiuje, rozprowadza) 
tę wiadomość lub podejmuje podobne działania, narusza prawo i może podlegać 
karze.

mBank S.A. z siedzibą w Warszawie, ul. Senatorska 18, 00-950 
Warszawa,www.mBank.pl, e-mail: kont...@mbank.pl. Sąd Rejonowy dla m. st. 
Warszawy XII Wydział Gospodarczy Krajowego Rejestru Sądowego, KRS 025237, 
NIP: 526-021-50-88. Kapitał zakładowy (opłacony w całości) według stanu na 
01.01.2020 r. wynosi 169.401.468 złotych.

If you are not the addressee of this message:

- let us know by replying to this e-mail (thank you!),
- delete this message permanently (including all the copies which you have 
printed out or saved).
This message may contain legally protected information, which may be used 
exclusively by the addressee.Please be reminded that anyone who disseminates 
(copies, distributes) this message or takes any similar action, violates the 
law and may be penalised.

mBank S.A. with its registered office in Warsaw, ul. Senatorska 18, 00-950 
Warszawa,www.mBank.pl, e-mail: kont...@mbank.pl. District Court for the Capital 
City of Warsaw, 12th Commercial Division of the National Court Register, KRS 
025237, NIP: 526-021-50-88. Fully paid-up share capital amounting to PLN 
169.401.468 as at 1 January 2020.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

That's still wrong:

http://bitsavers.org/pdf/ibm/7030/22-6530-2_7030RefMan.pdf#page=169


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3



From: IBM Mainframe Discussion List  on behalf of 
Mike Schwab 
Sent: Sunday, November 15, 2020 4:13 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Sorry.  First computer to use 8 bits per character.

On Sun, Nov 15, 2020 at 11:09 AM Seymour J Metz  wrote:
>
> > You have to remember that S/360 was the first 8 bit computer.
>
> What is the 7030, chopped liver?
>
>
> --
> Shmuel (Seymour J.) Metz
> http://mason.gmu.edu/~smetz3
>
>
> 
> From: IBM Mainframe Discussion List  on behalf of 
> Mike Schwab 
> Sent: Saturday, November 14, 2020 9:25 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> You have to remember that S/360 was the first 8 bit computer.  Prior
> computers used 4 bits for a digit and 6 bits for a character.  They
> designed EBCDIC to be easily converted for use with existing 7 track
> tape drives, printers, card and tape readers and punches.  There was a
> proposed ASCII code that was put on documentation but dropped for the
> 370 virtual memory bit in the PSW.
>
> On Sat, Nov 14, 2020 at 6:39 PM Seymour J Metz  wrote:
> >
> > I doubt that IBM custumers would have been happy with an 8-bit code page 
> > with only 128 valid code points. International considerations would still 
> > have forced IBM to device incompatible code pages for different countries.
> >
> > Obviously 8859 is another Tower of Babel; why do you think I described it 
> > as "a dollar short"?
> >
> > No,, IBM could not have implemented full Unicode, or even the full MLP, 
> > back in the 1960s. But it could certainly have implemented a basic subset 
> > for all customers and selected additional pages for international 
> > customers. Had Unicode and UTF-8 been around at the time, I'm certain that 
> > IBM would have gone that route.
> >
> >
> > --
> > Shmuel (Seymour J.) Metz
> > http://mason.gmu.edu/~smetz3
> >
> >
> > ____
> > From: IBM Mainframe Discussion List  on behalf of 
> > Paul Gilmartin <000433f07816-dmarc-requ...@listserv.ua.edu>
> > Sent: Saturday, November 14, 2020 6:22 PM
> > To: IBM-MAIN@LISTSERV.UA.EDU
> > Subject: Re: Improve OMVS cp performance?
> >
> > On Sat, 14 Nov 2020 23:00:00 +, Seymour J Metz wrote:
> >
> > >Because there was no standard 8-bit code at the time. IBM did push for an 
> > >8-bit ASCII,
> > >
> > That's not an obstacle.  DEC PDP-8 stored ASCII characters one per
> > 12-bit word.  IBM could have simply declared the top bit "reserved"
> > as they are so often wont to do.
> >
> > >but it never happened except for a mapping between octets and punch 
> > >combinations on cards. Had Unicode been around at the time they would 
> > >probably have jumped at it.
> > >
> > >ISO 8859 was a day late and a dollar short.
> > >
> > ISO-8859-* is afflicted with the same babel as EBCDIC code pages
> > because of the "*" you elided.
> >
> > UTF-8 is the norm nowadays because of a peculiar upward compatibility
> > with ASCII.  But the mebibytes and megahertz to support it came a day late.
> >
> > -- gil
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
>
>
> --
> Mike A Schwab, Springfield IL USA
> Where do Forest Rangers go to get away from it all?
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

> They only had 7 track tapes at the time.

Are you a betting man?


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3



From: IBM Mainframe Discussion List  on behalf of 
Mike Schwab 
Sent: Sunday, November 15, 2020 5:28 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

They only had 7 track tapes at the time.

9 track tapes introduced with S/360 in 1964.
https://secure-web.cisco.com/15XvqSwTkb8sH4cs6M51pKSQPk3d5UMx0-OZWOJ-qVN-Xi3Em6QAF-WxBpwlIxb4H-MIL_lx-Sjm3-fYErj8-_wb3XmyqHn8YwVkyLcFpqfg5HOxgP-3u6TV5WZoS9eSBARFIPSwxXNPiVInLz7T7XfYMW8GF4xFUHfMkxEQqaugiyRJI2b-7ozMXS67SLYZdCL_GML_68jxRtZS8n1wpWPPi4_gJBf7UrHScZXtCb_mTD6A0KCSIXj79GLDoX1HfombbmMh1rWbORrUkfDCfuqb3oudoIDruIhlxDkbAP_-7tY49ZzJiT37K2xxiMe3hr2vjakbV8KMdSRC_vYHFs6DTqTkK9PhyVHfnweIZwHbdul1nV5ajaPlYlmIHw915zOZMUqtYPHHOw_GoTnqH6YimQOMHeC53GnS4KHEiXoelc3T9eAJL_QjV-RFTKdkI/https%3A%2F%2Fen.wikipedia.org%2Fwiki%2F9_track_tape

On Sun, Nov 15, 2020 at 3:36 PM Paul Gilmartin
<000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
>
> On Sun, 15 Nov 2020 15:13:24 -0600, Mike Schwab wrote:
>
> >Sorry.  First computer to use 8 bits per character.
> >
> Wikipedia, not necessarily an authority, says:
> 
> https://secure-web.cisco.com/1nkIhum4wox25jkI5iyrdaNhYiLZ-wEKIr3K4FJU0KBpoOtcfywPRSnP9jzVOX-9S8QZjIml46GxRNS1b2fKnylfTLqYAXUVsy4GHz8fzHXPRAKuVy0q0Q-ha8KLERcqzewzfdpjYIlNEbkZQ8xYjzgsF9vD3-o9FFmMk-ZeQsGyfATz-ct5fo02aXF-g0sn7_8Ax67Ftiy2kjGEiZk3yDaEs75vcK08soi_H9HBWYSZfMwEgNQ5Chi0qaI-OEaMOGvf-fNfsdgO7cGmxGOyIGst1qyQOoBzS9V1CrFsevh-h3fKH6xVuolZBz2I9ylh_gOKwSoCLhkRxeb_KPNJNUXkPHactrTwT2naYwfJdOoMoksJuj1NYmtSqNw74vRX0d7Z3Gsf0FyPEnz-J_EIEafTVDrXhAJrT7AnH2MdZC9iUY1BW9ldH5qyOiC-gCu7f/https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FIBM_7030_Stretch#Data_formats
> Alphanumeric characters are variable length and can use any character 
> code of 8 bits or less.
>
> ... so, if true, 8 bits was at least possible.  Was it widely in "use"?
>
> >On Sun, Nov 15, 2020 at 11:09 AM Seymour J Metz wrote:
> >>
> >> > You have to remember that S/360 was the first 8 bit computer.
> >>
> >> What is the 7030, chopped liver?
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

Faster than Atlas?


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3



From: IBM Mainframe Discussion List  on behalf of 
Bernd Oppolzer 
Sent: Sunday, November 15, 2020 5:43 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Telefunken TR 4, designed 1958, first delivered in 1962. This predates
IBM/360
by at least 4 years. The fastest mainframe built in Europe at that time.
The internal code ("Zentralcode") was an 8-bit code using 256 characters.
Word structure, a word had 48 bits plus 2 tag bits (tagged architecture)
plus some parity bits, not seen by the programmer.
Mostly used with ALGOL; the TR 4 was "a Hardware implementation of ALGOL"
(quote from E.J. Dijkstra).
A word could hold up to six characters, but some later languages
(like Fortran) decided to store only 4 characters in one word,
to be more compatible with IBM Fortran.

Kind regards

Bernd


Am 15.11.2020 um 03:25 schrieb Mike Schwab:
> You have to remember that S/360 was the first 8 bit computer.  Prior
> computers used 4 bits for a digit and 6 bits for a character.  They
> designed EBCDIC to be easily converted for use with existing 7 track
> tape drives, printers, card and tape readers and punches.  There was a
> proposed ASCII code that was put on documentation but dropped for the
> 370 virtual memory bit in the PSW.
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

Did you ask him what he though the various distribution tapes were?  On 7-track 
tape you really don't have much choice what to use.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3



From: IBM Mainframe Discussion List  on behalf of 
Paul Gilmartin <000433f07816-dmarc-requ...@listserv.ua.edu>
Sent: Sunday, November 15, 2020 6:06 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

On Sun, 15 Nov 2020 16:28:02 -0600, Mike Schwab  wrote:

>They only had 7 track tapes at the time.
>
>9 track tapes introduced with S/360 in 1964.
>https://secure-web.cisco.com/1g3kTcng6DKLa2wZpjswahVbue_vj3uf2QkkhXiKJmCVFfYS7CwPJCOtzLp6sJGh2jwDBJtf-fQIAfhpg9JoyFjr-NP3GJ49DpgasRSaveY70zBAeUcG96cx9DjCCN7RLPVCYV0a5iunz5SDfQn5nUBPwdAzgTbqxMX_OnnnrbUL7OE0eJTsPrbRH4BSajmMRuFZmnPF6bjm9LjWZhRPwPeokTh9c_ZgD83Nr3egz_aDsyHEFvR8fWZl1pcJuZD_eSGb4o_-LVqZnSkkL72wa2225iWLTRU_R7Lmg5vfQNsInxpKROCItW2KofD7KGBQqfEte6pHmt7jYDvsPEWDAgyA4uUlhNRJ6lYNNRE6AzdSfLrBhnJIaQa1NyExfHakIOcVdnTX_c-_bwT1zHcIr9fhzdLOUcUwlKyrq_rn8c2c-w42RxSz2qvekUt_uRHN0/https%3A%2F%2Fen.wikipedia.org%2Fwiki%2F9_track_tape
>
Which was no obstacle.  There were two modes of I/O to 7-track tapes:
TRTCH=T, which mapped each tape frame of 6 data bits
plus 1 parity to one 8-bit byte.
TRTCH=C, which mapped the 24 data bits of 4 tape frames to 3 bytes

I know; I successfully used the latter.  I had to argue with ops who tried
to tell me that I shouldn't use that; it would corrupt mu data.  It didn't.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Bernd Oppolzer]

FWIW: the Telefunken TR4 indeed only supported half word addressing;
the half words being 24 bits; full words of 48 (plus some extra) bits 
had even adresses,
which also were the addresses of the left halfword; the right halfword 
had the address

of the left halfword plus one.

The TR 4 offspring TR 440 (which was ten times faster, a time sharing 
machine)
supported COBOL and later Multics-PL/1; and then there was the need to 
support

byte addressing, too. So some operations were added which worked
on so-called "Oktadenadressen", which were in fact halfword addresses
multiplied by 3 (you can imagine how the mapping of the byte addresses 
to the
halfword addresses worked, very similar to the IBM/360). The Telefunken 
TR 440
was developped from 1965 on and first delivered in 1969. So this was 
indeed after
IBM/360; but IIRC, the byte addressing added to the TR 440 was not 
inspired by the

360 machine - the COBOL compiler group asked for that.

I recently read some old documents about the German computer industry in 
the 1960s.
The IBM/360 announcement apparently came like a shock to Telefunken, but 
they
finally decided to keep direction, because some important customers 
asked for a

system to replace the TR 4 machine, which had been quite successful.
And German government supported and funded the development.

About 50 TR 440 machines were built, and they worked until the mid 1980s 
in Germany

(at universities). 20 Million Deutsche Marks for one machine :-)
It was no commercial success, but it motivated many people to enter the
computer business; it was much fun, after all :-)

I worked with the TR 440 at Stuttgart university from 1977 to 1981,
when I was a young student of computer science. The programming language
I used most was Pascal (and some Telefunken-ASSEMBLER).
From 1984 to 2001 ca.: Pascal/VS on IBM machines on VM/CMS :-)
Later 370-ASSEMBLER, PL/1 and C.
Today again: New Stanford Pascal - http://bernd-oppolzer.de/job9.htm

Kind regards

Bernd



Am 16.11.2020 um 09:28 schrieb Timothy Sipples:

Mike Schwab wrote:

You have to remember that S/360 was the first 8 bit computer.
[]
Sorry.  First computer to use 8 bits per character.

I see others have cited the IBM 7030 and Telefunken TR 4 as examples of
early computers that used (or at least were explicitly engineered to use)
8 bit character encoding. However, as far as I can tell both of those
machines were word addressable machines, and their word sizes were
different and much larger than their character sizes. Was there any
pre-System/360 example of a computer that stored characters in 8 bits
*and* offered 8 bit memory addressing? (Or 6 and 6, or 7 and 7?) For that
matter, are there any still extant digital computer processors that (only)
have word addressable memory and don't have 8 bit byte addressable memory?

History evidently judges that particular System/360 design decision as
wise or at least not unwise.

- - - - - - - - - -
Timothy Sipples
I.T. Architect Executive
Digital Asset & Other Industry Solutions
IBM Z & LinuxONE
- - - - - - - - - -
E-Mail: sipp...@sg.ibm.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

No, the 7030 was bit addressable. 

At the time of the S/360 announcement I thought that the decision to have a 
Model T "any byte size you want as long as it's 8" was a bad one, and I still 
think so. I also didn't like the 4 bit storage key and the 24 bit address. I 
thought that general registers might have been a good idea if there were 32, 
but that a maximum of base+index registers of 15 was too small.

What blindsided me was that IBM never supported the ASCII bit .


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3



From: IBM Mainframe Discussion List  on behalf of 
Timothy Sipples 
Sent: Monday, November 16, 2020 3:28 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Mike Schwab wrote:
>You have to remember that S/360 was the first 8 bit computer.
>[]
>Sorry.  First computer to use 8 bits per character.

I see others have cited the IBM 7030 and Telefunken TR 4 as examples of
early computers that used (or at least were explicitly engineered to use)
8 bit character encoding. However, as far as I can tell both of those
machines were word addressable machines, and their word sizes were
different and much larger than their character sizes. Was there any
pre-System/360 example of a computer that stored characters in 8 bits
*and* offered 8 bit memory addressing? (Or 6 and 6, or 7 and 7?) For that
matter, are there any still extant digital computer processors that (only)
have word addressable memory and don't have 8 bit byte addressable memory?

History evidently judges that particular System/360 design decision as
wise or at least not unwise.

- - - - - - - - - -
Timothy Sipples
I.T. Architect Executive
Digital Asset & Other Industry Solutions
IBM Z & LinuxONE
- - - - - - - - - -
E-Mail: sipp...@sg.ibm.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Timothy Sipples]

Mike Schwab wrote:
>You have to remember that S/360 was the first 8 bit computer.
>[]
>Sorry.  First computer to use 8 bits per character.

I see others have cited the IBM 7030 and Telefunken TR 4 as examples of 
early computers that used (or at least were explicitly engineered to use) 
8 bit character encoding. However, as far as I can tell both of those 
machines were word addressable machines, and their word sizes were 
different and much larger than their character sizes. Was there any 
pre-System/360 example of a computer that stored characters in 8 bits 
*and* offered 8 bit memory addressing? (Or 6 and 6, or 7 and 7?) For that 
matter, are there any still extant digital computer processors that (only) 
have word addressable memory and don't have 8 bit byte addressable memory?

History evidently judges that particular System/360 design decision as 
wise or at least not unwise.

- - - - - - - - - -
Timothy Sipples
I.T. Architect Executive
Digital Asset & Other Industry Solutions
IBM Z & LinuxONE
- - - - - - - - - -
E-Mail: sipp...@sg.ibm.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Sun, 15 Nov 2020 16:28:02 -0600, Mike Schwab  wrote:

>They only had 7 track tapes at the time.
>
>9 track tapes introduced with S/360 in 1964.
>https://en.wikipedia.org/wiki/9_track_tape
>
Which was no obstacle.  There were two modes of I/O to 7-track tapes:
TRTCH=T, which mapped each tape frame of 6 data bits
plus 1 parity to one 8-bit byte.
TRTCH=C, which mapped the 24 data bits of 4 tape frames to 3 bytes

I know; I successfully used the latter.  I had to argue with ops who tried
to tell me that I shouldn't use that; it would corrupt mu data.  It didn't.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Bernd Oppolzer]

Telefunken TR 4, designed 1958, first delivered in 1962. This predates 
IBM/360

by at least 4 years. The fastest mainframe built in Europe at that time.
The internal code ("Zentralcode") was an 8-bit code using 256 characters.
Word structure, a word had 48 bits plus 2 tag bits (tagged architecture)
plus some parity bits, not seen by the programmer.
Mostly used with ALGOL; the TR 4 was "a Hardware implementation of ALGOL"
(quote from E.J. Dijkstra).
A word could hold up to six characters, but some later languages
(like Fortran) decided to store only 4 characters in one word,
to be more compatible with IBM Fortran.

Kind regards

Bernd


Am 15.11.2020 um 03:25 schrieb Mike Schwab:

You have to remember that S/360 was the first 8 bit computer.  Prior
computers used 4 bits for a digit and 6 bits for a character.  They
designed EBCDIC to be easily converted for use with existing 7 track
tape drives, printers, card and tape readers and punches.  There was a
proposed ASCII code that was put on documentation but dropped for the
370 virtual memory bit in the PSW.



--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Mike Schwab]

They only had 7 track tapes at the time.

9 track tapes introduced with S/360 in 1964.
https://en.wikipedia.org/wiki/9_track_tape

On Sun, Nov 15, 2020 at 3:36 PM Paul Gilmartin
<000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
>
> On Sun, 15 Nov 2020 15:13:24 -0600, Mike Schwab wrote:
>
> >Sorry.  First computer to use 8 bits per character.
> >
> Wikipedia, not necessarily an authority, says:
> https://en.wikipedia.org/wiki/IBM_7030_Stretch#Data_formats
> Alphanumeric characters are variable length and can use any character 
> code of 8 bits or less.
>
> ... so, if true, 8 bits was at least possible.  Was it widely in "use"?
>
> >On Sun, Nov 15, 2020 at 11:09 AM Seymour J Metz wrote:
> >>
> >> > You have to remember that S/360 was the first 8 bit computer.
> >>
> >> What is the 7030, chopped liver?
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



-- 
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Sun, 15 Nov 2020 15:13:24 -0600, Mike Schwab wrote:

>Sorry.  First computer to use 8 bits per character.
>
Wikipedia, not necessarily an authority, says:
https://en.wikipedia.org/wiki/IBM_7030_Stretch#Data_formats
Alphanumeric characters are variable length and can use any character code 
of 8 bits or less.

... so, if true, 8 bits was at least possible.  Was it widely in "use"?

>On Sun, Nov 15, 2020 at 11:09 AM Seymour J Metz wrote:
>>
>> > You have to remember that S/360 was the first 8 bit computer.
>>
>> What is the 7030, chopped liver?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Mike Schwab]

Sorry.  First computer to use 8 bits per character.

On Sun, Nov 15, 2020 at 11:09 AM Seymour J Metz  wrote:
>
> > You have to remember that S/360 was the first 8 bit computer.
>
> What is the 7030, chopped liver?
>
>
> --
> Shmuel (Seymour J.) Metz
> http://mason.gmu.edu/~smetz3
>
>
> 
> From: IBM Mainframe Discussion List  on behalf of 
> Mike Schwab 
> Sent: Saturday, November 14, 2020 9:25 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> You have to remember that S/360 was the first 8 bit computer.  Prior
> computers used 4 bits for a digit and 6 bits for a character.  They
> designed EBCDIC to be easily converted for use with existing 7 track
> tape drives, printers, card and tape readers and punches.  There was a
> proposed ASCII code that was put on documentation but dropped for the
> 370 virtual memory bit in the PSW.
>
> On Sat, Nov 14, 2020 at 6:39 PM Seymour J Metz  wrote:
> >
> > I doubt that IBM custumers would have been happy with an 8-bit code page 
> > with only 128 valid code points. International considerations would still 
> > have forced IBM to device incompatible code pages for different countries.
> >
> > Obviously 8859 is another Tower of Babel; why do you think I described it 
> > as "a dollar short"?
> >
> > No,, IBM could not have implemented full Unicode, or even the full MLP, 
> > back in the 1960s. But it could certainly have implemented a basic subset 
> > for all customers and selected additional pages for international 
> > customers. Had Unicode and UTF-8 been around at the time, I'm certain that 
> > IBM would have gone that route.
> >
> >
> > --
> > Shmuel (Seymour J.) Metz
> > http://mason.gmu.edu/~smetz3
> >
> >
> > ____
> > From: IBM Mainframe Discussion List  on behalf of 
> > Paul Gilmartin <000433f07816-dmarc-requ...@listserv.ua.edu>
> > Sent: Saturday, November 14, 2020 6:22 PM
> > To: IBM-MAIN@LISTSERV.UA.EDU
> > Subject: Re: Improve OMVS cp performance?
> >
> > On Sat, 14 Nov 2020 23:00:00 +, Seymour J Metz wrote:
> >
> > >Because there was no standard 8-bit code at the time. IBM did push for an 
> > >8-bit ASCII,
> > >
> > That's not an obstacle.  DEC PDP-8 stored ASCII characters one per
> > 12-bit word.  IBM could have simply declared the top bit "reserved"
> > as they are so often wont to do.
> >
> > >but it never happened except for a mapping between octets and punch 
> > >combinations on cards. Had Unicode been around at the time they would 
> > >probably have jumped at it.
> > >
> > >ISO 8859 was a day late and a dollar short.
> > >
> > ISO-8859-* is afflicted with the same babel as EBCDIC code pages
> > because of the "*" you elided.
> >
> > UTF-8 is the norm nowadays because of a peculiar upward compatibility
> > with ASCII.  But the mebibytes and megahertz to support it came a day late.
> >
> > -- gil
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
>
>
> --
> Mike A Schwab, Springfield IL USA
> Where do Forest Rangers go to get away from it all?
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



-- 
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

> You have to remember that S/360 was the first 8 bit computer.  

What is the 7030, chopped liver?


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3



From: IBM Mainframe Discussion List  on behalf of 
Mike Schwab 
Sent: Saturday, November 14, 2020 9:25 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

You have to remember that S/360 was the first 8 bit computer.  Prior
computers used 4 bits for a digit and 6 bits for a character.  They
designed EBCDIC to be easily converted for use with existing 7 track
tape drives, printers, card and tape readers and punches.  There was a
proposed ASCII code that was put on documentation but dropped for the
370 virtual memory bit in the PSW.

On Sat, Nov 14, 2020 at 6:39 PM Seymour J Metz  wrote:
>
> I doubt that IBM custumers would have been happy with an 8-bit code page with 
> only 128 valid code points. International considerations would still have 
> forced IBM to device incompatible code pages for different countries.
>
> Obviously 8859 is another Tower of Babel; why do you think I described it as 
> "a dollar short"?
>
> No,, IBM could not have implemented full Unicode, or even the full MLP, back 
> in the 1960s. But it could certainly have implemented a basic subset for all 
> customers and selected additional pages for international customers. Had 
> Unicode and UTF-8 been around at the time, I'm certain that IBM would have 
> gone that route.
>
>
> --
> Shmuel (Seymour J.) Metz
> http://mason.gmu.edu/~smetz3
>
>
> 
> From: IBM Mainframe Discussion List  on behalf of 
> Paul Gilmartin <000433f07816-dmarc-requ...@listserv.ua.edu>
> Sent: Saturday, November 14, 2020 6:22 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> On Sat, 14 Nov 2020 23:00:00 +, Seymour J Metz wrote:
>
> >Because there was no standard 8-bit code at the time. IBM did push for an 
> >8-bit ASCII,
> >
> That's not an obstacle.  DEC PDP-8 stored ASCII characters one per
> 12-bit word.  IBM could have simply declared the top bit "reserved"
> as they are so often wont to do.
>
> >but it never happened except for a mapping between octets and punch 
> >combinations on cards. Had Unicode been around at the time they would 
> >probably have jumped at it.
> >
> >ISO 8859 was a day late and a dollar short.
> >
> ISO-8859-* is afflicted with the same babel as EBCDIC code pages
> because of the "*" you elided.
>
> UTF-8 is the norm nowadays because of a peculiar upward compatibility
> with ASCII.  But the mebibytes and megahertz to support it came a day late.
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



--
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Hobart Spitz]

CMS/TSO Pipes!!!

On Sat, 14 Nov 2020, 8:26 pm Mike Schwab,  wrote:

> You have to remember that S/360 was the first 8 bit computer.  Prior
> computers used 4 bits for a digit and 6 bits for a character.  They
> designed EBCDIC to be easily converted for use with existing 7 track
> tape drives, printers, card and tape readers and punches.  There was a
> proposed ASCII code that was put on documentation but dropped for the
> 370 virtual memory bit in the PSW.
>
> On Sat, Nov 14, 2020 at 6:39 PM Seymour J Metz  wrote:
> >
> > I doubt that IBM custumers would have been happy with an 8-bit code page
> with only 128 valid code points. International considerations would still
> have forced IBM to device incompatible code pages for different countries.
> >
> > Obviously 8859 is another Tower of Babel; why do you think I described
> it as "a dollar short"?
> >
> > No,, IBM could not have implemented full Unicode, or even the full MLP,
> back in the 1960s. But it could certainly have implemented a basic subset
> for all customers and selected additional pages for international
> customers. Had Unicode and UTF-8 been around at the time, I'm certain that
> IBM would have gone that route.
> >
> >
> > --
> > Shmuel (Seymour J.) Metz
> > http://mason.gmu.edu/~smetz3
> >
> >
> > 
> > From: IBM Mainframe Discussion List  on
> behalf of Paul Gilmartin <0000000433f07816-dmarc-requ...@listserv.ua.edu>
> > Sent: Saturday, November 14, 2020 6:22 PM
> > To: IBM-MAIN@LISTSERV.UA.EDU
> > Subject: Re: Improve OMVS cp performance?
> >
> > On Sat, 14 Nov 2020 23:00:00 +, Seymour J Metz wrote:
> >
> > >Because there was no standard 8-bit code at the time. IBM did push for
> an 8-bit ASCII,
> > >
> > That's not an obstacle.  DEC PDP-8 stored ASCII characters one per
> > 12-bit word.  IBM could have simply declared the top bit "reserved"
> > as they are so often wont to do.
> >
> > >but it never happened except for a mapping between octets and punch
> combinations on cards. Had Unicode been around at the time they would
> probably have jumped at it.
> > >
> > >ISO 8859 was a day late and a dollar short.
> > >
> > ISO-8859-* is afflicted with the same babel as EBCDIC code pages
> > because of the "*" you elided.
> >
> > UTF-8 is the norm nowadays because of a peculiar upward compatibility
> > with ASCII.  But the mebibytes and megahertz to support it came a day
> late.
> >
> > -- gil
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
>
>
> --
> Mike A Schwab, Springfield IL USA
> Where do Forest Rangers go to get away from it all?
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Mike Schwab]

You have to remember that S/360 was the first 8 bit computer.  Prior
computers used 4 bits for a digit and 6 bits for a character.  They
designed EBCDIC to be easily converted for use with existing 7 track
tape drives, printers, card and tape readers and punches.  There was a
proposed ASCII code that was put on documentation but dropped for the
370 virtual memory bit in the PSW.

On Sat, Nov 14, 2020 at 6:39 PM Seymour J Metz  wrote:
>
> I doubt that IBM custumers would have been happy with an 8-bit code page with 
> only 128 valid code points. International considerations would still have 
> forced IBM to device incompatible code pages for different countries.
>
> Obviously 8859 is another Tower of Babel; why do you think I described it as 
> "a dollar short"?
>
> No,, IBM could not have implemented full Unicode, or even the full MLP, back 
> in the 1960s. But it could certainly have implemented a basic subset for all 
> customers and selected additional pages for international customers. Had 
> Unicode and UTF-8 been around at the time, I'm certain that IBM would have 
> gone that route.
>
>
> --
> Shmuel (Seymour J.) Metz
> http://mason.gmu.edu/~smetz3
>
>
> 
> From: IBM Mainframe Discussion List  on behalf of 
> Paul Gilmartin <000433f07816-dmarc-requ...@listserv.ua.edu>
> Sent: Saturday, November 14, 2020 6:22 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> On Sat, 14 Nov 2020 23:00:00 +, Seymour J Metz wrote:
>
> >Because there was no standard 8-bit code at the time. IBM did push for an 
> >8-bit ASCII,
> >
> That's not an obstacle.  DEC PDP-8 stored ASCII characters one per
> 12-bit word.  IBM could have simply declared the top bit "reserved"
> as they are so often wont to do.
>
> >but it never happened except for a mapping between octets and punch 
> >combinations on cards. Had Unicode been around at the time they would 
> >probably have jumped at it.
> >
> >ISO 8859 was a day late and a dollar short.
> >
> ISO-8859-* is afflicted with the same babel as EBCDIC code pages
> because of the "*" you elided.
>
> UTF-8 is the norm nowadays because of a peculiar upward compatibility
> with ASCII.  But the mebibytes and megahertz to support it came a day late.
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



-- 
Mike A Schwab, Springfield IL USA
Where do Forest Rangers go to get away from it all?

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

I doubt that IBM custumers would have been happy with an 8-bit code page with 
only 128 valid code points. International considerations would still have 
forced IBM to device incompatible code pages for different countries.

Obviously 8859 is another Tower of Babel; why do you think I described it as "a 
dollar short"?

No,, IBM could not have implemented full Unicode, or even the full MLP, back in 
the 1960s. But it could certainly have implemented a basic subset for all 
customers and selected additional pages for international customers. Had 
Unicode and UTF-8 been around at the time, I'm certain that IBM would have gone 
that route.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3



From: IBM Mainframe Discussion List  on behalf of 
Paul Gilmartin <000433f07816-dmarc-requ...@listserv.ua.edu>
Sent: Saturday, November 14, 2020 6:22 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

On Sat, 14 Nov 2020 23:00:00 +, Seymour J Metz wrote:

>Because there was no standard 8-bit code at the time. IBM did push for an 
>8-bit ASCII,
>
That's not an obstacle.  DEC PDP-8 stored ASCII characters one per
12-bit word.  IBM could have simply declared the top bit "reserved"
as they are so often wont to do.

>but it never happened except for a mapping between octets and punch 
>combinations on cards. Had Unicode been around at the time they would probably 
>have jumped at it.
>
>ISO 8859 was a day late and a dollar short.
>
ISO-8859-* is afflicted with the same babel as EBCDIC code pages
because of the "*" you elided.

UTF-8 is the norm nowadays because of a peculiar upward compatibility
with ASCII.  But the mebibytes and megahertz to support it came a day late.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Sat, 14 Nov 2020 23:00:00 +, Seymour J Metz wrote:

>Because there was no standard 8-bit code at the time. IBM did push for an 
>8-bit ASCII, 
>
That's not an obstacle.  DEC PDP-8 stored ASCII characters one per
12-bit word.  IBM could have simply declared the top bit "reserved"
as they are so often wont to do.

>but it never happened except for a mapping between octets and punch 
>combinations on cards. Had Unicode been around at the time they would probably 
>have jumped at it.
>
>ISO 8859 was a day late and a dollar short.
>
ISO-8859-* is afflicted with the same babel as EBCDIC code pages
because of the "*" you elided.

UTF-8 is the norm nowadays because of a peculiar upward compatibility
with ASCII.  But the mebibytes and megahertz to support it came a day late.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

Because there was no standard 8-bit code at the time. IBM did push for an 8-bit 
ASCII, but it never happened except for a mapping between octets and punch 
combinations on cards. Had Unicode been around at the time they would probably 
have jumped at it.

ISO 8859 was a day late and a dollar short.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3



From: IBM Mainframe Discussion List  on behalf of 
Paul Gilmartin <000433f07816-dmarc-requ...@listserv.ua.edu>
Sent: Friday, November 13, 2020 10:26 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

On Sat, 14 Nov 2020 11:21:08 +1100, Andrew Rowley wrote:
>
>I admit I had never looked at the details of FILEDATA=RECORD. It appears
>that IBM has taken RFC 4506 and made a small change so their
>implementation is incompatible with anything that actually follows the RFC.
>
Is that:
RFC 4506   XDR: External Data Representation Standard   May 2006
???  Which section?

>Why would IBM do that? Implementing the RFC format seems reasonable.
>Implementing an incompatible variation, less so.
>
Because they can.  Why EBCDIC?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Andrew Rowley]

On 14/11/2020 2:26 pm, Paul Gilmartin wrote:


Is that:
 RFC 4506   XDR: External Data Representation Standard   May 2006
???  Which section?


Using Data Sets : JCL Parameters for Unix Files says:

"If you code FILEDATA=RECORD, the access method follows a portion of RFC 
4506, XDR, External Data Representation Standard. The records that 
FILEDATA=RECORD defines are either of these XDR types:

- Variable-length opaque data
- String
However the access method does not follow one characteristic of the XDR 
standard. The standard specifies that if the number of bytes of data is 
not a multiple of four, then the bytes are padded on the right with 
binary zeroes. That would require the access method to insert the bytes 
so that every record begins at an offset that is a multiple of four from 
the beginning of the UNIX file. The access method does not insert any 
padding."


Call me a pedant, but if the record length information is not where it 
is supposed to be IBM is not following RFC 4506.


Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Sat, 14 Nov 2020 13:52:15 -0600, Kirk Wolf wrote:
>
>> On 14/11/2020 12:54 am, Kirk Wolf wrote:
>>
>> > The new commands (getpds and putpds) use BPAM+BSAM ...
>
>If you were reading a UNIX file, you would need to know whether it had
>IBM-style RDWs, or l4 (which is compatible with FILEDATA=RECORD)
> 
One way you might know is from the tag information.  Empirically,
contrary to the doc, a RECORD tag overrides the default when a
file is allocated with JCL DD PATH=...
>
>-l rdw  has a length that includes the 4-byte RDW length, compatible with
>IBM
>
Close enough.  Using Data Sets mentions 1 byte reserved and 3 bytes length.
(Sounds like S/360 addressing conventions.)

>FYI - the "fromdsn" and "todsn" commands also have these options, but they
>read or write a single data set.
>
Do todsn and putpds respect the file tags if "-s source-codepage" or "-l4"
and "-b" and "-l" is omitted?

Does putpds have a facility to undo "getpds -l4"?

If getpds overwrites an existing file does its existing FILEDATA and CCSID
tags prevail over defaulted options to getpds?

Do command line options override file tags quietly or with warnings?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Kirk Wolf]

On Fri, Nov 13, 2020 at 3:35 PM Andrew Rowley 
wrote:

> On 14/11/2020 12:54 am, Kirk Wolf wrote:
>
> > The new commands (getpds and putpds) use BPAM+BSAM along with our
> existing
> > Co:Z record <-> stream processing framework to allow you to copy PDS
> > members to and from z/OS UNIX files.  Also included are extensive options
> > for ISPF stats processing.
> >
>
> Nice work. I have immediate uses for it.
>
> One question, I see the options:
>
> rdw
> preceed lines with a four byte IBM-style RDW, consisting of a two byte
> network order (big endian) length, followed by two bytes of zeros.
>
> l4
> preceed lines with a four byte network order (big endian) length of the
> record that follows. Note: Unlike the rdw option, this length value does
> not include the size of the length field.
>
> If you are looking at a file, how would you know whether the length
> field is included in the record length?
>

If you were reading a UNIX file, you would need to know whether it had
IBM-style RDWs, or l4 (which is compatible with FILEDATA=RECORD)


>
> The common practice seems to be that the length value in the RDW
> includes the length of the RDW itself. This is observed by other
> products e.g. FTP. Is there a reason to have the option to exclude the
> RDW length?
>
>
-l rdw  has a length that includes the 4-byte RDW length, compatible with
IBM

FYI - the "fromdsn" and "todsn" commands also have these options, but they
read or write a single data set.


> The whole RDW/record length concept causes a lot of confusion - even it
> seems in IBM. Requiring people to choose what type of length field will
> add to the confusion.
>
> Andrew Rowley
> Black Hill Software
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Sat, 14 Nov 2020 11:21:08 +1100, Andrew Rowley wrote:
>
>I admit I had never looked at the details of FILEDATA=RECORD. It appears
>that IBM has taken RFC 4506 and made a small change so their
>implementation is incompatible with anything that actually follows the RFC.
> 
Is that:
RFC 4506   XDR: External Data Representation Standard   May 2006
???  Which section?

>Why would IBM do that? Implementing the RFC format seems reasonable.
>Implementing an incompatible variation, less so.
> 
Because they can.  Why EBCDIC?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Andrew Rowley]

On 14/11/2020 10:16 am, Paul Gilmartin wrote:

Except for UNIX files.  From Using Data Sets:
Record Processing for UNIX Files

When a file is accessed as record-oriented (with FILEDATA=RECORD), your program 
does not see the record prefixes. The PUT macro adds a prefix to each record in 
the same format as with BSAM or QSAM. A GET macro removes the prefix. Each 
record prefix is mapped by the IGGRPFX macro. It is the following four bytes:

Offset  Length Symbol Description
01 RPFX00  Reserved.
13 RPFXLLL Length of record that follows this prefix.
(FSVO "same".)


I admit I had never looked at the details of FILEDATA=RECORD. It appears 
that IBM has taken RFC 4506 and made a small change so their 
implementation is incompatible with anything that actually follows the RFC.


Why would IBM do that? Implementing the RFC format seems reasonable. 
Implementing an incompatible variation, less so.


Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Sat, 14 Nov 2020 08:34:15 +1100, Andrew Rowley wrote:

>On 14/11/2020 12:54 am, Kirk Wolf wrote:
>
>> The new commands (getpds and putpds) use BPAM+BSAM along with our existing
>> Co:Z record <-> stream processing framework to allow you to copy PDS
>> members to and from z/OS UNIX files.  Also included are extensive options
>> for ISPF stats processing.
>>
>
>Nice work. I have immediate uses for it.
>
>One question, I see the options:
>
>rdw
>preceed lines with a four byte IBM-style RDW, consisting of a two byte
>network order (big endian) length, followed by two bytes of zeros.
>
>l4
>preceed lines with a four byte network order (big endian) length of the
>record that follows. Note: Unlike the rdw option, this length value does
>not include the size of the length field.
>
>If you are looking at a file, how would you know whether the length
>field is included in the record length?
>
>The common practice seems to be that the length value in the RDW
>includes the length of the RDW itself. This is observed by other
>products e.g. FTP. Is there a reason to have the option to exclude the
>RDW length?
>
Except for UNIX files.  From Using Data Sets:
Record Processing for UNIX Files

When a file is accessed as record-oriented (with FILEDATA=RECORD), your program 
does not see the record prefixes. The PUT macro adds a prefix to each record in 
the same format as with BSAM or QSAM. A GET macro removes the prefix. Each 
record prefix is mapped by the IGGRPFX macro. It is the following four bytes:

Offset  Length Symbol Description
01 RPFX00  Reserved.
13 RPFXLLL Length of record that follows this prefix.
(FSVO "same".)

>The whole RDW/record length concept causes a lot of confusion - even it
>seems in IBM. Requiring people to choose what type of length field will
>add to the confusion.
>
The wonderful thing about standards is that you get to pick from
among several the one you want!

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Andrew Rowley]

On 14/11/2020 12:54 am, Kirk Wolf wrote:


The new commands (getpds and putpds) use BPAM+BSAM along with our existing
Co:Z record <-> stream processing framework to allow you to copy PDS
members to and from z/OS UNIX files.  Also included are extensive options
for ISPF stats processing.



Nice work. I have immediate uses for it.

One question, I see the options:

rdw
preceed lines with a four byte IBM-style RDW, consisting of a two byte 
network order (big endian) length, followed by two bytes of zeros.


l4
preceed lines with a four byte network order (big endian) length of the 
record that follows. Note: Unlike the rdw option, this length value does 
not include the size of the length field.


If you are looking at a file, how would you know whether the length 
field is included in the record length?


The common practice seems to be that the length value in the RDW 
includes the length of the RDW itself. This is observed by other 
products e.g. FTP. Is there a reason to have the option to exclude the 
RDW length?


The whole RDW/record length concept causes a lot of confusion - even it 
seems in IBM. Requiring people to choose what type of length field will 
add to the confusion.


Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Fri, 13 Nov 2020 11:00:29 -0600, Kirk Wolf wrote:
>
>Are these requests for features that you would actually use?
>
Honestly, no.  My employer avoided ISV and FOSS extensions lest
we leak dependencies into customer-facing code.

>re: the line separator option.   These are the same as all of the other
>Co:Z tools have been for more than 10 years.   It includes support for both
>RDWs and 4-byte big endian length.
>
Does that antedate the availability of FILEDATA=RECORD?

I suspect that to achieve the performance you report you must use
raw UNIX I/O with RYO rather than relying on BSAM/QSAM to
allocated UNIX files to perform the conversions.

Thanks,
gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Kirk Wolf]

Gil,

Are these requests for features that you would actually use?

re: the line separator option.   These are the same as all of the other
Co:Z tools have been for more than 10 years.   It includes support for both
RDWs and 4-byte big endian length.



On Fri, Nov 13, 2020 at 10:41 AM Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:

> On Fri, 13 Nov 2020 07:54:22 -0600, Kirk Wolf wrote:
> >
> >For more information:
> >https://dovetail.com/docs/zos-utilities/dsp-ref_getpds.html
> >https://dovetail.com/docs/zos-utilities/dsp-ref_putpds.html
> >https://dovetail.com/docs/cozinstall/changes.html
> >Co:Z is available free under our Community License
> >.
> >>
> Impressive!  But I can still add suggestions:
>
> //DDNAME:  format?
>
> -M option, as for /bin/cp to translate "_.-"  (Oops!  Option name
> conflict!)
>
> upp[ercase]=y[es]|n[o]
> file member names uppercased? (default=no: lower case)
> Also "ASIS" for putpds in case UNIX directory contains foobar,
> FooBar, and FOOBAR?  (Does upp=yes for getpds actually mean
> ASIS?)
>
> -l line-separator
> nl | cr | lf | crlf | crnl
> How about the equivalent of FILEDATA(RECORD) as supported by JCL?
> "crnl"?  "lfcr"?  Damn IBM for conflating LF and NL!  (I fear ISVs started
> it!)
>
> Long member names?  I understand PDSE supports long aliases (is that
> only for program objects?)  I could envision storing long UNIX member
> names with generated member names and the original UNIX names
> as aliases.  Are they simply ignored?  Warning?  Likewise subdirectories?
>
> Should multiply-linked files be stored as aliases?  Which name wins?
> (How does ISPF behave when an alias or a member having an alias
> is edited?)  Should getpds render aliases as (symbolic?) links?
>
> >> >FWIW: It's a pity that the IBM C library doesn't have any support for
> >> BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
> agree:
> >> >
> >>
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Fri, 13 Nov 2020 07:54:22 -0600, Kirk Wolf wrote:
>
>For more information:
>https://dovetail.com/docs/zos-utilities/dsp-ref_getpds.html
>https://dovetail.com/docs/zos-utilities/dsp-ref_putpds.html
>https://dovetail.com/docs/cozinstall/changes.html
>Co:Z is available free under our Community License
>.
>>
Impressive!  But I can still add suggestions:

//DDNAME:  format?

-M option, as for /bin/cp to translate "_.-"  (Oops!  Option name conflict!)

upp[ercase]=y[es]|n[o]
file member names uppercased? (default=no: lower case)
Also "ASIS" for putpds in case UNIX directory contains foobar,
FooBar, and FOOBAR?  (Does upp=yes for getpds actually mean
ASIS?)

-l line-separator
nl | cr | lf | crlf | crnl
How about the equivalent of FILEDATA(RECORD) as supported by JCL?
"crnl"?  "lfcr"?  Damn IBM for conflating LF and NL!  (I fear ISVs started it!)

Long member names?  I understand PDSE supports long aliases (is that
only for program objects?)  I could envision storing long UNIX member
names with generated member names and the original UNIX names
as aliases.  Are they simply ignored?  Warning?  Likewise subdirectories?

Should multiply-linked files be stored as aliases?  Which name wins?
(How does ISPF behave when an alias or a member having an alias
is edited?)  Should getpds render aliases as (symbolic?) links?

>> >FWIW: It's a pity that the IBM C library doesn't have any support for
>> BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you agree:
>> >
>> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Kirk Wolf]

I wanted to update this thread with a bit of news.   We have had the
enjoyable experience of working with Lionel and Henri (please check out:
http://zigi.rocks) on requirements for some new shell commands for the Co:Z
Toolkit V6.2.0, which we released this week.

The new commands (getpds and putpds) use BPAM+BSAM along with our existing
Co:Z record <-> stream processing framework to allow you to copy PDS
members to and from z/OS UNIX files.  Also included are extensive options
for ISPF stats processing.

The performance (when compared to "/bin/cp") is better than we could have
expected.  The following example copies all of the members of SYS1.MACLIB
to text files in the current directory:

$ getpds //sys1.maclib .
getpds(SYS1.MACLIB)[N]: 2015 members/2435218 records/194817440 bytes read;
194011101 bytes written in 1.791 seconds (103.307
MBytes/sec).

For more information:
https://dovetail.com/docs/zos-utilities/dsp-ref_getpds.html
https://dovetail.com/docs/zos-utilities/dsp-ref_putpds.html
https://dovetail.com/docs/cozinstall/changes.html
Co:Z is available free under our Community License
.

Kirk Wolf
http://dovetail.com


On Mon, Jun 22, 2020 at 9:30 AM Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:

> On Wed, 17 Jun 2020 07:14:39 -0500, Lionel B Dyck wrote:
>
> >Kirk - thank you for the ideas.
> >
> >What I'm doing is in the ZIGI (see https://zigi.rocks) where I need to
> copy PDS members to/from USS so that Git can manage them. With small
> projects this isn't an issue but with larger projects it could take enough
> time for you to go to lunch ☹
> >
> >Btw. I voted your RFE.
> >
> I notice that replies this thread have focused on Classic PDS as the
> performance culprit.  Has someone tried the benchmarks:
>
> cp UNIX -> UNIX  vs.  cp PDS -> PDS
>
> ... and compared performance?  I suspect Classic OS was optimized
> for a few large data sets; UNIX for many small files.
>
> Might there be an argument here for maintaining the data where it
> works best and omitting the copying?
>
> >-Original Message-
> >From: Kirk Wolf
> >Sent: Wednesday, June 17, 2020 7:03 AM
> >
> >FWIW: It's a pity that the IBM C library doesn't have any support for
> BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
> >agree:
> >
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Wed, 17 Jun 2020 07:14:39 -0500, Lionel B Dyck wrote:

>Kirk - thank you for the ideas.
>
>What I'm doing is in the ZIGI (see https://zigi.rocks) where I need to copy 
>PDS members to/from USS so that Git can manage them. With small projects this 
>isn't an issue but with larger projects it could take enough time for you to 
>go to lunch ☹
>
>Btw. I voted your RFE.
> 
I notice that replies this thread have focused on Classic PDS as the
performance culprit.  Has someone tried the benchmarks:

cp UNIX -> UNIX  vs.  cp PDS -> PDS

... and compared performance?  I suspect Classic OS was optimized
for a few large data sets; UNIX for many small files.

Might there be an argument here for maintaining the data where it
works best and omitting the copying?

>-Original Message-
>From: Kirk Wolf
>Sent: Wednesday, June 17, 2020 7:03 AM
>
>FWIW: It's a pity that the IBM C library doesn't have any support for 
>BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
>agree:
>https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Frank Swarbrick]

I'll see if I can carve out some time to put this together.  If I do I'll 
probably post it here before making an official request, as people here are 
much smarter than me and will probably provide good feedback.  Any initial 
thoughts from anyone here are also welcome!


From: IBM Mainframe Discussion List  on behalf of 
Seymour J Metz 
Sent: Wednesday, June 17, 2020 6:24 PM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

I think that it is worth an RFE if you can put together a compelling business 
case. Be sure to cover any functionality that's important to you, e.g.,

 Program Objects?
 PDS(E) directory data
 Performance for multiple members in the same PDS(E)
 Record formats
 Character set issues
 Line ending conventions, e.g., CRLF, LF, NEL, NL


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 6:53 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Do you think its worth an RFE?



From: IBM Mainframe Discussion List  on behalf of 
Seymour J Metz 
Sent: Wednesday, June 17, 2020 4:52 PM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

I think that technically it would be a piece of cake, although it would take 
some careful design to make it efficient and  transparent.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 6:34 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I mean technical effort on IBM's part to implement it.

It seems to me that, along with some mapping/naming rules and sensible 
defaults, in addition to Lionel's use case it could also be useful for FTP 
clients, so non-z/OS users can have "understandable" access to MVS data sets 
w/o having to even know that's what they are doing.


From: IBM Mainframe Discussion List  on behalf of 
Seymour J Metz 
Sent: Wednesday, June 17, 2020 12:42 PM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Technical effort or administrative effort. I suspect that it will be a lot more 
involved to get it approved than to actually implement it.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 1:12 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I wonder what kind of effort it might be for z/OS to support Unix path names as 
aliases/links to MVS legacy data sets.  Probably a lot of work, but it seems 
like it would be quite useful for situations such as this where the Unix 
application only supports Unix paths.

Just a wild thought.  Not opening an RFE for it or anything, unless someone can 
say it sounds at all reasonable.

By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the terms 
"legacy data sets" or "MVS data sets" saying "the z/OS side".  It's all z/OS!  🙂


From: IBM Mainframe Discussion List  on behalf of 
Lionel B Dyck 
Sent: Wednesday, June 17, 2020 6:14 AM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see 
https://secure-web.cisco.com/16y9woZ-KKNGioIsmrX3l0VloGx62ZmJbaIDUMxNEu3X4_DmZ6nR-qUkWT-lIJna_FSvyQDSqjJHfEE6JVKyGupXrO9tZC1YO3pkDcHdunPFAORyGUqh2yJwHFSDhnEN1TtNm5g8EvDfoNzNsauNWFq4Y_5InaggC3Djt9nYuC7fv4BlPnY08D6jHmqOOticix9GjXAoL9A2DFzRItsD7RyJt7tNFtjfWZFZu3E-ycsOQlpWifUJvx4QAe6GSVyJinMUnvQ0cG7veolIcRj9KdGLoHSjf0diK1UVO0q3LRSzwcLSv9K5aNecTApJaWO0YizryqgYg3UqT4cDPbHG-sLS4b_8_CV6WTBJFipgSWaH1SnA6EsjjHognkaFW_J9WuXnPRq9b-odk_z1k3sQt4Li6oWKIpRIBAyK2b2RvwvBW2Vq0v_9ybXpgNqjKaNAI/https%3A%2F%2Fzigi.rocks)
 where I need to copy PDS members to/from USS so that Git can manage them. With 
small projects this isn't an issue but with larger projects it could take 
enough time for you to go to lunch ☹

Btw. I voted your RFE.


Lionel B. Dyck <
Website: 
https://secure-web.cisco.com/1M2R-g6A5gGoKDKQxyFiYYV7WXr9OdUyu6Ixa3zjpJ6CJbQmAW8FtYkxjXgLOeIDnvjGXI0lOmtzYcfyzGiCcRXXoTJiT7jNfLgtZaozhB1snDKIsuSqxrUFNz5eHiU9FuFPR_nAf45mLn3swUsAfJB77d53cb7b4d1kHMZxbp69FYPo4AbO7fSOmun_rPhg-ArrddqzghsnvKXAWOl88mKZ5TLED7NhEjdb1G3D2OauWeZmmrYS7YNoaApm9ci2dFr9POiD9mLXAhZEOO5HJImkuCdd3O0jGChFv7l7yIQnE1_ZXQ--KqkeAjWI7XPrhKZnpDex2AYCsAZ0f-4tNYyL1XagaW7uBJxH8OFBFFApall8Wssfm

Re: Improve OMVS cp performance?

[John McKown]

On Wed, Jun 17, 2020 at 12:12 PM Frank Swarbrick <
frank.swarbr...@outlook.com> wrote:

> I wonder what kind of effort it might be for z/OS to support Unix path
> names as aliases/links to MVS legacy data sets.  Probably a lot of work,
> but it seems like it would be quite useful for situations such as this
> where the Unix application only supports Unix paths.
>
> Just a wild thought.  Not opening an RFE for it or anything, unless
> someone can say it sounds at all reasonable.
>
> By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the
> terms "legacy data sets" or "MVS data sets" saying "the z/OS side".  It's
> all z/OS!  🙂
>
>
Supposedly, you can use the NFS server on z/OS to export a PDS or up to
some node of a DSN which can then mount into a UNIX filesystem, on the same
or other system using the NFS client of that system. But I never got it to
work on my very old z/OS 1.12 system. NFS might just be a bit too
complicated for my old brain.

-- 
People in sleeping bags are the soft tacos of the bear world.
Maranatha! <><
John McKown

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Andrew Rowley]

On 18/06/2020 8:01 pm, David Crayford wrote:
Interesting! The Java program probably is probably much faster because 
it runs on a full capacity zIIP. At my shop we run an enterprise class 
machine and I don't see the same results. It's very difficult to 
measure Java vs native when the gcp's also run full capacity.


Can you share some of your SMF reports?



The GCPs are full capacity, or whatever we get on the Dallas RDP system. 
It doesn't appear to be CPU bound at all. I think cp takes twice as long 
because it opens and closes the file twice for every member vs once in 
the Java program.


One thing I am convinced of is that open and close in the unix 
filesystem is much faster than open and close of MVS datasets. Even more 
so when you factor in enqueues.


I'll see if I can put together some SMF reports.

--
Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[David Crayford]

Interesting! The Java program probably is probably much faster because 
it runs on a full capacity zIIP. At my shop we run an enterprise class 
machine and I don't see the same results. It's very difficult to measure 
Java vs native when the gcp's also run full capacity.


Can you share some of your SMF reports?

On 2020-06-18 7:42 AM, Andrew Rowley wrote:

On 18/06/2020 12:24 am, Kirk Wolf wrote:

Lionel,
I wasn't thinking of using the "all members" form of cp - that seems 
like
it should be *much* better, although it would depend on how cp works 
under

the covers - evidence indicates that it just loops and does
alloc/open/close/free on each member.    If only the cp authors had 
better

C library support for PDSs ;-)

I'm curious - how much time did you save by preallocating the PDS?

Kirk Wolf
http://dovetail.com

Preallocating the PDS gave me about a 12x speed improvement. Here is a 
Rexx shell script rxalloc to do the allocation (I couldn't figure out 
a way to do bpxwdyn from the shell):


/* rexx */
parse arg dataset
call bpxwdyn "alloc da("|| dataset || ") old msg(2) rtddn(ddname)"
say ddname

and a shell script to test:

#!/bin/sh
export _BPX_SHAREAS=YES
./rxalloc SYS1.MACLIB
/bin/cp -T -U -S a=.txt "//'SYS1.MACLIB'" /home/andrewr/temp

The individual member copies with progress indicator in Zigi also 
seems to have significant overhead. Approximate copy speeds on my 
system were:

Zigi: 1 member/second
cp, without preallocation: 8 members/second
cp, with preallocation: 100 members/second

SMF also suggests cp for some reason opens and closes the PDS twice 
for each member. I wrote a small Java program to perform the copy 
using the JZOS classes, this coped about 200 members/second.




--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

I think that it is worth an RFE if you can put together a compelling business 
case. Be sure to cover any functionality that's important to you, e.g.,

 Program Objects?
 PDS(E) directory data
 Performance for multiple members in the same PDS(E)
 Record formats
 Character set issues
 Line ending conventions, e.g., CRLF, LF, NEL, NL


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 6:53 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Do you think its worth an RFE?



From: IBM Mainframe Discussion List  on behalf of 
Seymour J Metz 
Sent: Wednesday, June 17, 2020 4:52 PM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

I think that technically it would be a piece of cake, although it would take 
some careful design to make it efficient and  transparent.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 6:34 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I mean technical effort on IBM's part to implement it.

It seems to me that, along with some mapping/naming rules and sensible 
defaults, in addition to Lionel's use case it could also be useful for FTP 
clients, so non-z/OS users can have "understandable" access to MVS data sets 
w/o having to even know that's what they are doing.


From: IBM Mainframe Discussion List  on behalf of 
Seymour J Metz 
Sent: Wednesday, June 17, 2020 12:42 PM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Technical effort or administrative effort. I suspect that it will be a lot more 
involved to get it approved than to actually implement it.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 1:12 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I wonder what kind of effort it might be for z/OS to support Unix path names as 
aliases/links to MVS legacy data sets.  Probably a lot of work, but it seems 
like it would be quite useful for situations such as this where the Unix 
application only supports Unix paths.

Just a wild thought.  Not opening an RFE for it or anything, unless someone can 
say it sounds at all reasonable.

By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the terms 
"legacy data sets" or "MVS data sets" saying "the z/OS side".  It's all z/OS!  🙂


From: IBM Mainframe Discussion List  on behalf of 
Lionel B Dyck 
Sent: Wednesday, June 17, 2020 6:14 AM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see 
https://secure-web.cisco.com/16y9woZ-KKNGioIsmrX3l0VloGx62ZmJbaIDUMxNEu3X4_DmZ6nR-qUkWT-lIJna_FSvyQDSqjJHfEE6JVKyGupXrO9tZC1YO3pkDcHdunPFAORyGUqh2yJwHFSDhnEN1TtNm5g8EvDfoNzNsauNWFq4Y_5InaggC3Djt9nYuC7fv4BlPnY08D6jHmqOOticix9GjXAoL9A2DFzRItsD7RyJt7tNFtjfWZFZu3E-ycsOQlpWifUJvx4QAe6GSVyJinMUnvQ0cG7veolIcRj9KdGLoHSjf0diK1UVO0q3LRSzwcLSv9K5aNecTApJaWO0YizryqgYg3UqT4cDPbHG-sLS4b_8_CV6WTBJFipgSWaH1SnA6EsjjHognkaFW_J9WuXnPRq9b-odk_z1k3sQt4Li6oWKIpRIBAyK2b2RvwvBW2Vq0v_9ybXpgNqjKaNAI/https%3A%2F%2Fzigi.rocks)
 where I need to copy PDS members to/from USS so that Git can manage them. With 
small projects this isn't an issue but with larger projects it could take 
enough time for you to go to lunch ☹

Btw. I voted your RFE.


Lionel B. Dyck <
Website: 
https://secure-web.cisco.com/1M2R-g6A5gGoKDKQxyFiYYV7WXr9OdUyu6Ixa3zjpJ6CJbQmAW8FtYkxjXgLOeIDnvjGXI0lOmtzYcfyzGiCcRXXoTJiT7jNfLgtZaozhB1snDKIsuSqxrUFNz5eHiU9FuFPR_nAf45mLn3swUsAfJB77d53cb7b4d1kHMZxbp69FYPo4AbO7fSOmun_rPhg-ArrddqzghsnvKXAWOl88mKZ5TLED7NhEjdb1G3D2OauWeZmmrYS7YNoaApm9ci2dFr9POiD9mLXAhZEOO5HJImkuCdd3O0jGChFv7l7yIQnE1_ZXQ--KqkeAjWI7XPrhKZnpDex2AYCsAZ0f-4tNYyL1XagaW7uBJxH8OFBFFApall8WssfmP94dlBt_13DdsDtCrFzObRE8x5b5008xoJkZ08eOE8HskcXulsVEDlDkwgBNRyy-qwYCMUyXsKHV/https%3A%2F%2Fwww.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more d

Re: Improve OMVS cp performance?

[Andrew Rowley]

On 18/06/2020 12:24 am, Kirk Wolf wrote:

Lionel,
I wasn't thinking of using the "all members" form of cp - that seems like
it should be *much* better, although it would depend on how cp works under
the covers - evidence indicates that it just loops and does
alloc/open/close/free on each member.If only the cp authors had better
C library support for PDSs ;-)

I'm curious - how much time did you save by preallocating the PDS?

Kirk Wolf
http://dovetail.com

Preallocating the PDS gave me about a 12x speed improvement. Here is a 
Rexx shell script rxalloc to do the allocation (I couldn't figure out a 
way to do bpxwdyn from the shell):


/* rexx */
parse arg dataset
call bpxwdyn "alloc da("|| dataset || ") old msg(2) rtddn(ddname)"
say ddname

and a shell script to test:

#!/bin/sh
export _BPX_SHAREAS=YES
./rxalloc SYS1.MACLIB
/bin/cp -T -U -S a=.txt "//'SYS1.MACLIB'" /home/andrewr/temp

The individual member copies with progress indicator in Zigi also seems 
to have significant overhead. Approximate copy speeds on my system were:

Zigi: 1 member/second
cp, without preallocation: 8 members/second
cp, with preallocation: 100 members/second

SMF also suggests cp for some reason opens and closes the PDS twice for 
each member. I wrote a small Java program to perform the copy using the 
JZOS classes, this coped about 200 members/second.


--
Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Thu, 18 Jun 2020 08:52:52 +1000, Andrew Rowley wrote:

>On 18/06/2020 1:43 am, Paul Gilmartin wrote:
>>> 
(Concerning preallocating the PDS):
>>
>> I'm mystified that it made a difference since Lionel never used the allocate
>> "dd" in the call to "cp".  Only SVC 99 knows.
>
>I was doing some testing on this last week. The overhead appears to be
>obtaining and freeing the enqueue on the PDS for each member. If you
>already have a SHR enqueue this is bypassed. The cp command needs to run
>in the same address space that obtained the enqueue.
>
Aha!  Highly plausible explanation.

If it improves performance significantly, should it be incorporated
into "cp"?  RFE?

Or documented in a Usage Note?  RCF?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Frank Swarbrick]

Do you think its worth an RFE?



From: IBM Mainframe Discussion List  on behalf of 
Seymour J Metz 
Sent: Wednesday, June 17, 2020 4:52 PM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

I think that technically it would be a piece of cake, although it would take 
some careful design to make it efficient and  transparent.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 6:34 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I mean technical effort on IBM's part to implement it.

It seems to me that, along with some mapping/naming rules and sensible 
defaults, in addition to Lionel's use case it could also be useful for FTP 
clients, so non-z/OS users can have "understandable" access to MVS data sets 
w/o having to even know that's what they are doing.


From: IBM Mainframe Discussion List  on behalf of 
Seymour J Metz 
Sent: Wednesday, June 17, 2020 12:42 PM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Technical effort or administrative effort. I suspect that it will be a lot more 
involved to get it approved than to actually implement it.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 1:12 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I wonder what kind of effort it might be for z/OS to support Unix path names as 
aliases/links to MVS legacy data sets.  Probably a lot of work, but it seems 
like it would be quite useful for situations such as this where the Unix 
application only supports Unix paths.

Just a wild thought.  Not opening an RFE for it or anything, unless someone can 
say it sounds at all reasonable.

By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the terms 
"legacy data sets" or "MVS data sets" saying "the z/OS side".  It's all z/OS!  🙂


From: IBM Mainframe Discussion List  on behalf of 
Lionel B Dyck 
Sent: Wednesday, June 17, 2020 6:14 AM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see 
https://secure-web.cisco.com/16y9woZ-KKNGioIsmrX3l0VloGx62ZmJbaIDUMxNEu3X4_DmZ6nR-qUkWT-lIJna_FSvyQDSqjJHfEE6JVKyGupXrO9tZC1YO3pkDcHdunPFAORyGUqh2yJwHFSDhnEN1TtNm5g8EvDfoNzNsauNWFq4Y_5InaggC3Djt9nYuC7fv4BlPnY08D6jHmqOOticix9GjXAoL9A2DFzRItsD7RyJt7tNFtjfWZFZu3E-ycsOQlpWifUJvx4QAe6GSVyJinMUnvQ0cG7veolIcRj9KdGLoHSjf0diK1UVO0q3LRSzwcLSv9K5aNecTApJaWO0YizryqgYg3UqT4cDPbHG-sLS4b_8_CV6WTBJFipgSWaH1SnA6EsjjHognkaFW_J9WuXnPRq9b-odk_z1k3sQt4Li6oWKIpRIBAyK2b2RvwvBW2Vq0v_9ybXpgNqjKaNAI/https%3A%2F%2Fzigi.rocks)
 where I need to copy PDS members to/from USS so that Git can manage them. With 
small projects this isn't an issue but with larger projects it could take 
enough time for you to go to lunch ☹

Btw. I voted your RFE.


Lionel B. Dyck <
Website: 
https://secure-web.cisco.com/1M2R-g6A5gGoKDKQxyFiYYV7WXr9OdUyu6Ixa3zjpJ6CJbQmAW8FtYkxjXgLOeIDnvjGXI0lOmtzYcfyzGiCcRXXoTJiT7jNfLgtZaozhB1snDKIsuSqxrUFNz5eHiU9FuFPR_nAf45mLn3swUsAfJB77d53cb7b4d1kHMZxbp69FYPo4AbO7fSOmun_rPhg-ArrddqzghsnvKXAWOl88mKZ5TLED7NhEjdb1G3D2OauWeZmmrYS7YNoaApm9ci2dFr9POiD9mLXAhZEOO5HJImkuCdd3O0jGChFv7l7yIQnE1_ZXQ--KqkeAjWI7XPrhKZnpDex2AYCsAZ0f-4tNYyL1XagaW7uBJxH8OFBFFApall8WssfmP94dlBt_13DdsDtCrFzObRE8x5b5008xoJkZ08eOE8HskcXulsVEDlDkwgBNRyy-qwYCMUyXsKHV/https%3A%2F%2Fwww.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each member, 
but you might get some improvement by allocating a DD to the PDSE and then 
passing  //DD(member) to cp, so as to avoid allocation each time.
 If you do this, then you will also know for sure if you are using local spawn 
(_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new AS was 
forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the sc

Re: Improve OMVS cp performance?

[Andrew Rowley]

On 18/06/2020 1:43 am, Paul Gilmartin wrote:

On Wed, 17 Jun 2020 09:24:58 -0500, Kirk Wolf wrote:

I wasn't thinking of using the "all members" form of cp ...
I'm curious - how much time did you save by preallocating the PDS?


I'm mystified that it made a difference since Lionel never used the allocate
"dd" in the call to "cp".  Only SVC 99 knows.
I was doing some testing on this last week. The overhead appears to be 
obtaining and freeing the enqueue on the PDS for each member. If you 
already have a SHR enqueue this is bypassed. The cp command needs to run 
in the same address space that obtained the enqueue.


--
Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

I think that technically it would be a piece of cake, although it would take 
some careful design to make it efficient and  transparent.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 6:34 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I mean technical effort on IBM's part to implement it.

It seems to me that, along with some mapping/naming rules and sensible 
defaults, in addition to Lionel's use case it could also be useful for FTP 
clients, so non-z/OS users can have "understandable" access to MVS data sets 
w/o having to even know that's what they are doing.


From: IBM Mainframe Discussion List  on behalf of 
Seymour J Metz 
Sent: Wednesday, June 17, 2020 12:42 PM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Technical effort or administrative effort. I suspect that it will be a lot more 
involved to get it approved than to actually implement it.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 1:12 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I wonder what kind of effort it might be for z/OS to support Unix path names as 
aliases/links to MVS legacy data sets.  Probably a lot of work, but it seems 
like it would be quite useful for situations such as this where the Unix 
application only supports Unix paths.

Just a wild thought.  Not opening an RFE for it or anything, unless someone can 
say it sounds at all reasonable.

By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the terms 
"legacy data sets" or "MVS data sets" saying "the z/OS side".  It's all z/OS!  🙂


From: IBM Mainframe Discussion List  on behalf of 
Lionel B Dyck 
Sent: Wednesday, June 17, 2020 6:14 AM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see 
https://secure-web.cisco.com/16y9woZ-KKNGioIsmrX3l0VloGx62ZmJbaIDUMxNEu3X4_DmZ6nR-qUkWT-lIJna_FSvyQDSqjJHfEE6JVKyGupXrO9tZC1YO3pkDcHdunPFAORyGUqh2yJwHFSDhnEN1TtNm5g8EvDfoNzNsauNWFq4Y_5InaggC3Djt9nYuC7fv4BlPnY08D6jHmqOOticix9GjXAoL9A2DFzRItsD7RyJt7tNFtjfWZFZu3E-ycsOQlpWifUJvx4QAe6GSVyJinMUnvQ0cG7veolIcRj9KdGLoHSjf0diK1UVO0q3LRSzwcLSv9K5aNecTApJaWO0YizryqgYg3UqT4cDPbHG-sLS4b_8_CV6WTBJFipgSWaH1SnA6EsjjHognkaFW_J9WuXnPRq9b-odk_z1k3sQt4Li6oWKIpRIBAyK2b2RvwvBW2Vq0v_9ybXpgNqjKaNAI/https%3A%2F%2Fzigi.rocks)
 where I need to copy PDS members to/from USS so that Git can manage them. With 
small projects this isn't an issue but with larger projects it could take 
enough time for you to go to lunch ☹

Btw. I voted your RFE.


Lionel B. Dyck <
Website: 
https://secure-web.cisco.com/1M2R-g6A5gGoKDKQxyFiYYV7WXr9OdUyu6Ixa3zjpJ6CJbQmAW8FtYkxjXgLOeIDnvjGXI0lOmtzYcfyzGiCcRXXoTJiT7jNfLgtZaozhB1snDKIsuSqxrUFNz5eHiU9FuFPR_nAf45mLn3swUsAfJB77d53cb7b4d1kHMZxbp69FYPo4AbO7fSOmun_rPhg-ArrddqzghsnvKXAWOl88mKZ5TLED7NhEjdb1G3D2OauWeZmmrYS7YNoaApm9ci2dFr9POiD9mLXAhZEOO5HJImkuCdd3O0jGChFv7l7yIQnE1_ZXQ--KqkeAjWI7XPrhKZnpDex2AYCsAZ0f-4tNYyL1XagaW7uBJxH8OFBFFApall8WssfmP94dlBt_13DdsDtCrFzObRE8x5b5008xoJkZ08eOE8HskcXulsVEDlDkwgBNRyy-qwYCMUyXsKHV/https%3A%2F%2Fwww.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each member, 
but you might get some improvement by allocating a DD to the PDSE and then 
passing  //DD(member) to cp, so as to avoid allocation each time.
 If you do this, then you will also know for sure if you are using local spawn 
(_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new AS was 
forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the script could do the cp for each member.   The hope is
that since cp is also a shell "built-in" you might avoid spawning processes for 
each one.

- the "best" performance possible would be writing your own BPAM code that
also does

Re: Improve OMVS cp performance?

[Frank Swarbrick]

I mean technical effort on IBM's part to implement it.

It seems to me that, along with some mapping/naming rules and sensible 
defaults, in addition to Lionel's use case it could also be useful for FTP 
clients, so non-z/OS users can have "understandable" access to MVS data sets 
w/o having to even know that's what they are doing.


From: IBM Mainframe Discussion List  on behalf of 
Seymour J Metz 
Sent: Wednesday, June 17, 2020 12:42 PM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Technical effort or administrative effort. I suspect that it will be a lot more 
involved to get it approved than to actually implement it.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 1:12 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I wonder what kind of effort it might be for z/OS to support Unix path names as 
aliases/links to MVS legacy data sets.  Probably a lot of work, but it seems 
like it would be quite useful for situations such as this where the Unix 
application only supports Unix paths.

Just a wild thought.  Not opening an RFE for it or anything, unless someone can 
say it sounds at all reasonable.

By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the terms 
"legacy data sets" or "MVS data sets" saying "the z/OS side".  It's all z/OS!  🙂


From: IBM Mainframe Discussion List  on behalf of 
Lionel B Dyck 
Sent: Wednesday, June 17, 2020 6:14 AM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see 
https://secure-web.cisco.com/16y9woZ-KKNGioIsmrX3l0VloGx62ZmJbaIDUMxNEu3X4_DmZ6nR-qUkWT-lIJna_FSvyQDSqjJHfEE6JVKyGupXrO9tZC1YO3pkDcHdunPFAORyGUqh2yJwHFSDhnEN1TtNm5g8EvDfoNzNsauNWFq4Y_5InaggC3Djt9nYuC7fv4BlPnY08D6jHmqOOticix9GjXAoL9A2DFzRItsD7RyJt7tNFtjfWZFZu3E-ycsOQlpWifUJvx4QAe6GSVyJinMUnvQ0cG7veolIcRj9KdGLoHSjf0diK1UVO0q3LRSzwcLSv9K5aNecTApJaWO0YizryqgYg3UqT4cDPbHG-sLS4b_8_CV6WTBJFipgSWaH1SnA6EsjjHognkaFW_J9WuXnPRq9b-odk_z1k3sQt4Li6oWKIpRIBAyK2b2RvwvBW2Vq0v_9ybXpgNqjKaNAI/https%3A%2F%2Fzigi.rocks)
 where I need to copy PDS members to/from USS so that Git can manage them. With 
small projects this isn't an issue but with larger projects it could take 
enough time for you to go to lunch ☹

Btw. I voted your RFE.


Lionel B. Dyck <
Website: 
https://secure-web.cisco.com/1M2R-g6A5gGoKDKQxyFiYYV7WXr9OdUyu6Ixa3zjpJ6CJbQmAW8FtYkxjXgLOeIDnvjGXI0lOmtzYcfyzGiCcRXXoTJiT7jNfLgtZaozhB1snDKIsuSqxrUFNz5eHiU9FuFPR_nAf45mLn3swUsAfJB77d53cb7b4d1kHMZxbp69FYPo4AbO7fSOmun_rPhg-ArrddqzghsnvKXAWOl88mKZ5TLED7NhEjdb1G3D2OauWeZmmrYS7YNoaApm9ci2dFr9POiD9mLXAhZEOO5HJImkuCdd3O0jGChFv7l7yIQnE1_ZXQ--KqkeAjWI7XPrhKZnpDex2AYCsAZ0f-4tNYyL1XagaW7uBJxH8OFBFFApall8WssfmP94dlBt_13DdsDtCrFzObRE8x5b5008xoJkZ08eOE8HskcXulsVEDlDkwgBNRyy-qwYCMUyXsKHV/https%3A%2F%2Fwww.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each member, 
but you might get some improvement by allocating a DD to the PDSE and then 
passing  //DD(member) to cp, so as to avoid allocation each time.
 If you do this, then you will also know for sure if you are using local spawn 
(_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new AS was 
forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the script could do the cp for each member.   The hope is
that since cp is also a shell "built-in" you might avoid spawning processes for 
each one.

- the "best" performance possible would be writing your own BPAM code that
also does the Unix fileio.   Assembler is fine, but I would use C/++ for
everything except the low level BPAM I/O routines, since I would probably use 
buffered filestreams for the Unix files.

FWIW: It's a pity that the IBM C library doesn't have any support for 
BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
agree:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811



On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  w

Re: Improve OMVS cp performance?

[Kirk Wolf]

And don't forget "Muggeridge's Law" :-)

On Wed, Jun 17, 2020 at 12:32 PM Jesse 1 Robinson 
wrote:

> Malcom Muggeridge was famous for characterizing UK and US as being
> *separated* by a common language.
>
> .
> .
> J.O.Skip Robinson
> Southern California Edison Company
> Electric Dragon Team Paddler
> SHARE MVS Program Co-Manager
> 323-715-0595 Mobile
> 626-543-6132 Office ⇐=== NEW
> robin...@sce.com
>
> -Original Message-
> From: IBM Mainframe Discussion List  On Behalf
> Of Lionel B Dyck
> Sent: Wednesday, June 17, 2020 10:23 AM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: (External):Re: Improve OMVS cp performance?
>
> CAUTION EXTERNAL EMAIL
>
> Frank - true it is all z/OS but that is like saying it's all Europe - in
> this case z/OS speaks one language and OMVS another - they just share the
> continent and unlike England and the US they don't share a common language
> 😊
>
>
> Lionel B. Dyck <
> Website: https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is what
> you are, reputation merely what others think you are." - John Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On Behalf
> Of Frank Swarbrick
> Sent: Wednesday, June 17, 2020 12:12 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> I wonder what kind of effort it might be for z/OS to support Unix path
> names as aliases/links to MVS legacy data sets.  Probably a lot of work,
> but it seems like it would be quite useful for situations such as this
> where the Unix application only supports Unix paths.
>
> Just a wild thought.  Not opening an RFE for it or anything, unless
> someone can say it sounds at all reasonable.
>
> By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the
> terms "legacy data sets" or "MVS data sets" saying "the z/OS side".  It's
> all z/OS!  🙂
>
> 
> From: IBM Mainframe Discussion List  on behalf
> of Lionel B Dyck 
> Sent: Wednesday, June 17, 2020 6:14 AM
> To: IBM-MAIN@LISTSERV.UA.EDU 
> Subject: Re: Improve OMVS cp performance?
>
> Kirk - thank you for the ideas.
>
> What I'm doing is in the ZIGI (see https://zigi.rocks) where I need to
> copy PDS members to/from USS so that Git can manage them. With small
> projects this isn't an issue but with larger projects it could take enough
> time for you to go to lunch ☹
>
> Btw. I voted your RFE.
>
>
> Lionel B. Dyck <
> Website: https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is what
> you are, reputation merely what others think you are." - John Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On Behalf
> Of Kirk Wolf
> Sent: Wednesday, June 17, 2020 7:03 AM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> Hi Lionel,
>
> Can you provide any more detail on how you are invoking "cp" ?
>
> - With cp, there won't be any way to avoid opening the PDSE for each
> member, but you might get some improvement by allocating a DD to the PDSE
> and then passing  //DD(member) to cp, so as to avoid allocation each time.
>  If you do this, then you will also know for sure if you are using local
> spawn (_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new
> AS was forked.
>
> - The other issue would be the cost of spawning a Unix process for each
> member, even if local spawned.   I haven't tested this, but you might write
> a shell script that is passed the DD as arg and member names as lines to
> stdin.   Then the script could do the cp for each member.   The hope is
> that since cp is also a shell "built-in" you might avoid spawning
> processes for each one.
>
> - the "best" performance possible would be writing your own BPAM code that
> also does the Unix fileio.   Assembler is fine, but I would use C/++ for
> everything except the low level BPAM I/O routines, since I would probably
> use buffered filestreams for the Unix files.
>
> FWIW: It's a pity that the IBM C library doesn't have any support for
> BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
> agree:
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811
>
>
>
> On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  wrote:
>
> > " What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"
> >
> > On the z/OS side is a PDS(E).
> >
> > Thanks
> >
> >

Re: Improve OMVS cp performance?

[Seymour J Metz]

Technical effort or administrative effort. I suspect that it will be a lot more 
involved to get it approved than to actually implement it.


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Frank Swarbrick [frank.swarbr...@outlook.com]
Sent: Wednesday, June 17, 2020 1:12 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I wonder what kind of effort it might be for z/OS to support Unix path names as 
aliases/links to MVS legacy data sets.  Probably a lot of work, but it seems 
like it would be quite useful for situations such as this where the Unix 
application only supports Unix paths.

Just a wild thought.  Not opening an RFE for it or anything, unless someone can 
say it sounds at all reasonable.

By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the terms 
"legacy data sets" or "MVS data sets" saying "the z/OS side".  It's all z/OS!  🙂


From: IBM Mainframe Discussion List  on behalf of 
Lionel B Dyck 
Sent: Wednesday, June 17, 2020 6:14 AM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see 
https://secure-web.cisco.com/16y9woZ-KKNGioIsmrX3l0VloGx62ZmJbaIDUMxNEu3X4_DmZ6nR-qUkWT-lIJna_FSvyQDSqjJHfEE6JVKyGupXrO9tZC1YO3pkDcHdunPFAORyGUqh2yJwHFSDhnEN1TtNm5g8EvDfoNzNsauNWFq4Y_5InaggC3Djt9nYuC7fv4BlPnY08D6jHmqOOticix9GjXAoL9A2DFzRItsD7RyJt7tNFtjfWZFZu3E-ycsOQlpWifUJvx4QAe6GSVyJinMUnvQ0cG7veolIcRj9KdGLoHSjf0diK1UVO0q3LRSzwcLSv9K5aNecTApJaWO0YizryqgYg3UqT4cDPbHG-sLS4b_8_CV6WTBJFipgSWaH1SnA6EsjjHognkaFW_J9WuXnPRq9b-odk_z1k3sQt4Li6oWKIpRIBAyK2b2RvwvBW2Vq0v_9ybXpgNqjKaNAI/https%3A%2F%2Fzigi.rocks)
 where I need to copy PDS members to/from USS so that Git can manage them. With 
small projects this isn't an issue but with larger projects it could take 
enough time for you to go to lunch ☹

Btw. I voted your RFE.


Lionel B. Dyck <
Website: 
https://secure-web.cisco.com/1M2R-g6A5gGoKDKQxyFiYYV7WXr9OdUyu6Ixa3zjpJ6CJbQmAW8FtYkxjXgLOeIDnvjGXI0lOmtzYcfyzGiCcRXXoTJiT7jNfLgtZaozhB1snDKIsuSqxrUFNz5eHiU9FuFPR_nAf45mLn3swUsAfJB77d53cb7b4d1kHMZxbp69FYPo4AbO7fSOmun_rPhg-ArrddqzghsnvKXAWOl88mKZ5TLED7NhEjdb1G3D2OauWeZmmrYS7YNoaApm9ci2dFr9POiD9mLXAhZEOO5HJImkuCdd3O0jGChFv7l7yIQnE1_ZXQ--KqkeAjWI7XPrhKZnpDex2AYCsAZ0f-4tNYyL1XagaW7uBJxH8OFBFFApall8WssfmP94dlBt_13DdsDtCrFzObRE8x5b5008xoJkZ08eOE8HskcXulsVEDlDkwgBNRyy-qwYCMUyXsKHV/https%3A%2F%2Fwww.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each member, 
but you might get some improvement by allocating a DD to the PDSE and then 
passing  //DD(member) to cp, so as to avoid allocation each time.
 If you do this, then you will also know for sure if you are using local spawn 
(_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new AS was 
forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the script could do the cp for each member.   The hope is
that since cp is also a shell "built-in" you might avoid spawning processes for 
each one.

- the "best" performance possible would be writing your own BPAM code that
also does the Unix fileio.   Assembler is fine, but I would use C/++ for
everything except the low level BPAM I/O routines, since I would probably use 
buffered filestreams for the Unix files.

FWIW: It's a pity that the IBM C library doesn't have any support for 
BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
agree:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811



On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  wrote:

> " What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"
>
> On the z/OS side is a PDS(E).
>
> Thanks
>
>
> Lionel B. Dyck <
> Website: 
> https://secure-web.cisco.com/1M2R-g6A5gGoKDKQxyFiYYV7WXr9OdUyu6Ixa3zjpJ6CJbQmAW8FtYkxjXgLOeIDnvjGXI0lOmtzYcfyzGiCcRXXoTJiT7jNfLgtZaozhB1snDKIsuSqxrUFNz5eHiU9FuFPR_nAf45mLn3swUsAfJB77d53cb7b4d1kHMZxbp69FYPo4AbO7fSOmun_rPhg-ArrddqzghsnvKXAWOl88mKZ5TLED7NhEjdb1G3D2OauWeZmmrYS7YNoaApm9ci2dFr9POiD9mLXAhZEOO5HJImkuCdd3O0jGChFv7l7yIQnE1_ZXQ--KqkeAjWI7XPrhKZnpDex2AYCsA

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Wed, 17 Jun 2020 17:12:11 +, Frank Swarbrick wrote:

>I wonder what kind of effort it might be for z/OS to support Unix path names 
>as aliases/links to MVS legacy data sets.  Probably a lot of work, but it 
>seems like it would be quite useful for situations such as this where the Unix 
>application only supports Unix paths.
> 
They exist as "external links", but only for program objects in LINKLIST, I 
believe.

>Just a wild thought.  Not opening an RFE for it or anything, unless someone 
>can say it sounds at all reasonable.
>
>By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the terms 
>"legacy data sets" or "MVS data sets" saying "the z/OS side".  It's all z/OS!  
>🙂
> 
+1
(I think it began as "MVS is UNIX.")

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Wed, 17 Jun 2020 11:07:58 -0500, Kirk Wolf wrote:

>On Wed, Jun 17, 2020 at 10:43 AM Paul Gilmartin wrote:
>
>> On Wed, 17 Jun 2020 09:24:58 -0500, Kirk Wolf wrote:
>>
>If you have a reusable allocation, then the cost is less than a new
>allocation.   open/close for each member would still be expensive for a PDS
>with tons of members.   A BLDL,loop: POINT,READ*  is what you really want
>IMO.
> 
Your RFE says, "I voted."

https://encrypted-tbn0.gstatic.com/images?q=tbn%3AANd9GcSTjEUCD7ng9dOs60AHyiHyR5eh0qq5UdeUDzi11H9nG6Y67Jd_&usqp=CAU
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811

Damit!  I believe ISPF LM services can do this, but it's zFS-ignorant.
It would still be nice to read each member into a compound symbol
then SYSCALL WRITEFILE.

Dammit!  LM SERVICES is compound-symbol-ignorant.

>Another fun thing to try is IEBUPDTE - I once played around with a REXX
>wrapper and then piped the control and data into it via DD:SYSIN pointing
>to /dev/fd0 from a shell script.  It's *really* fast as expected, but has
>issues wrt the record format and content of the data that could mess it up.
> 
Where's Shell Archive when you need it?:
https://en.wikipedia.org/wiki/Shar

>BTW: If it was as fast as it should be, Lionel probably doesn't need a
>"progress" popup :-)
>
IIRC, there's one place ISPF shows a progress popup.  Perhaps in DDLIST M?

OTOH, searching a log ISPF pauses every 9,999 lines and asks the
user whether to continue.  Why not an interruptible progress popup?
TSO/VTAM limitations, I suppose.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: [EXTERNAL] Re: Improve OMVS cp performance?

[Horne, Jim]

G.B. Shaw was even more famous for it

Jim Horne

Malcom Muggeridge was famous for characterizing UK and US as being *separated* 
by a common language.


NOTICE: All information in and attached to the e-mails below may be 
proprietary, confidential, privileged and otherwise protected from improper or 
erroneous disclosure. If you are not the sender's intended recipient, you are 
not authorized to intercept, read, print, retain, copy, forward, or disseminate 
this message. If you have erroneously received this communication, please 
notify the sender immediately by phone (704-758-1000) or by e-mail and destroy 
all copies of this message electronic, paper, or otherwise. By transmitting 
documents via this email: Users, Customers, Suppliers and Vendors collectively 
acknowledge and agree the transmittal of information via email is voluntary, is 
offered as a convenience, and is not a secured method of communication; Not to 
transmit any payment information E.G. credit card, debit card, checking 
account, wire transfer information, passwords, or sensitive and personal 
information E.G. Driver's license, DOB, social security, or any other 
information the user wishes to remain confidential; To transmit only 
non-confidential information such as plans, pictures and drawings and to assume 
all risk and liability for and indemnify Lowe's from any claims, losses or 
damages that may arise from the transmittal of documents or including 
non-confidential information in the body of an email transmittal. Thank you.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Jesse 1 Robinson]

Malcom Muggeridge was famous for characterizing UK and US as being *separated* 
by a common language. 

.
.
J.O.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler 
SHARE MVS Program Co-Manager
323-715-0595 Mobile
626-543-6132 Office ⇐=== NEW
robin...@sce.com

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Lionel B Dyck
Sent: Wednesday, June 17, 2020 10:23 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: (External):Re: Improve OMVS cp performance?

CAUTION EXTERNAL EMAIL

Frank - true it is all z/OS but that is like saying it's all Europe - in this 
case z/OS speaks one language and OMVS another - they just share the continent 
and unlike England and the US they don't share a common language 😊


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Frank Swarbrick
Sent: Wednesday, June 17, 2020 12:12 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I wonder what kind of effort it might be for z/OS to support Unix path names as 
aliases/links to MVS legacy data sets.  Probably a lot of work, but it seems 
like it would be quite useful for situations such as this where the Unix 
application only supports Unix paths.

Just a wild thought.  Not opening an RFE for it or anything, unless someone can 
say it sounds at all reasonable.

By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the terms 
"legacy data sets" or "MVS data sets" saying "the z/OS side".  It's all z/OS!  🙂


From: IBM Mainframe Discussion List  on behalf of 
Lionel B Dyck 
Sent: Wednesday, June 17, 2020 6:14 AM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see https://zigi.rocks) where I need to copy PDS 
members to/from USS so that Git can manage them. With small projects this isn't 
an issue but with larger projects it could take enough time for you to go to 
lunch ☹

Btw. I voted your RFE.


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each member, 
but you might get some improvement by allocating a DD to the PDSE and then 
passing  //DD(member) to cp, so as to avoid allocation each time.
 If you do this, then you will also know for sure if you are using local spawn 
(_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new AS was 
forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the script could do the cp for each member.   The hope is
that since cp is also a shell "built-in" you might avoid spawning processes for 
each one.

- the "best" performance possible would be writing your own BPAM code that
also does the Unix fileio.   Assembler is fine, but I would use C/++ for
everything except the low level BPAM I/O routines, since I would probably use 
buffered filestreams for the Unix files.

FWIW: It's a pity that the IBM C library doesn't have any support for 
BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
agree:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811



On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  wrote:

> " What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"
>
> On the z/OS side is a PDS(E).
>
> Thanks
>
>
> Lionel B. Dyck <
> Website: https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is 
> what you are, reputation merely what others think you are." - John 
> Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On 
> Behalf Of Paul Gilmartin
> Sent: Tuesday, June 16, 2020 9:23 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> On Tue, 16 Jun 2020 20:34:59 -0500, Lionel B Dyck wrote:
>
> >Any suggestions on how to speed up cp copying multiple file to and 
> >from
> z/OS?
> >
> >I ga

Re: Improve OMVS cp performance?

[Lionel B Dyck]

Frank - true it is all z/OS but that is like saying it's all Europe - in this 
case z/OS speaks one language and OMVS another - they just share the continent 
and unlike England and the US they don't share a common language 😊


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Frank Swarbrick
Sent: Wednesday, June 17, 2020 12:12 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

I wonder what kind of effort it might be for z/OS to support Unix path names as 
aliases/links to MVS legacy data sets.  Probably a lot of work, but it seems 
like it would be quite useful for situations such as this where the Unix 
application only supports Unix paths.

Just a wild thought.  Not opening an RFE for it or anything, unless someone can 
say it sounds at all reasonable.

By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the terms 
"legacy data sets" or "MVS data sets" saying "the z/OS side".  It's all z/OS!  🙂


From: IBM Mainframe Discussion List  on behalf of 
Lionel B Dyck 
Sent: Wednesday, June 17, 2020 6:14 AM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see https://zigi.rocks) where I need to copy PDS 
members to/from USS so that Git can manage them. With small projects this isn't 
an issue but with larger projects it could take enough time for you to go to 
lunch ☹

Btw. I voted your RFE.


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each member, 
but you might get some improvement by allocating a DD to the PDSE and then 
passing  //DD(member) to cp, so as to avoid allocation each time.
 If you do this, then you will also know for sure if you are using local spawn 
(_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new AS was 
forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the script could do the cp for each member.   The hope is
that since cp is also a shell "built-in" you might avoid spawning processes for 
each one.

- the "best" performance possible would be writing your own BPAM code that
also does the Unix fileio.   Assembler is fine, but I would use C/++ for
everything except the low level BPAM I/O routines, since I would probably use 
buffered filestreams for the Unix files.

FWIW: It's a pity that the IBM C library doesn't have any support for 
BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
agree:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811



On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  wrote:

> " What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"
>
> On the z/OS side is a PDS(E).
>
> Thanks
>
>
> Lionel B. Dyck <
> Website: https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is 
> what you are, reputation merely what others think you are." - John 
> Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On 
> Behalf Of Paul Gilmartin
> Sent: Tuesday, June 16, 2020 9:23 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> On Tue, 16 Jun 2020 20:34:59 -0500, Lionel B Dyck wrote:
>
> >Any suggestions on how to speed up cp copying multiple file to and 
> >from
> z/OS?
> >
> >I gave SHAREAS=YES. Anything else?  Can I control buffers or ?
> >
> What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?
>
> I might suggest as an extreme measure Rexx under ISPF using ADDRESS 
> SYSCALL I/O for OMVS and LM services for Classic.
> SYSCALL READFILE/WRITEFILE are available for text files only.
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive acce

Re: Improve OMVS cp performance?

[Frank Swarbrick]

I wonder what kind of effort it might be for z/OS to support Unix path names as 
aliases/links to MVS legacy data sets.  Probably a lot of work, but it seems 
like it would be quite useful for situations such as this where the Unix 
application only supports Unix paths.

Just a wild thought.  Not opening an RFE for it or anything, unless someone can 
say it sounds at all reasonable.

By the way, z/OS Unix is z/OS, as some like to say.  I prefer to use the terms 
"legacy data sets" or "MVS data sets" saying "the z/OS side".  It's all z/OS!  🙂


From: IBM Mainframe Discussion List  on behalf of 
Lionel B Dyck 
Sent: Wednesday, June 17, 2020 6:14 AM
To: IBM-MAIN@LISTSERV.UA.EDU 
Subject: Re: Improve OMVS cp performance?

Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see https://zigi.rocks) where I need to copy PDS 
members to/from USS so that Git can manage them. With small projects this isn't 
an issue but with larger projects it could take enough time for you to go to 
lunch ☹

Btw. I voted your RFE.


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each member, 
but you might get some improvement by allocating a DD to the PDSE and then 
passing  //DD(member) to cp, so as to avoid allocation each time.
 If you do this, then you will also know for sure if you are using local spawn 
(_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new AS was 
forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the script could do the cp for each member.   The hope is
that since cp is also a shell "built-in" you might avoid spawning processes for 
each one.

- the "best" performance possible would be writing your own BPAM code that
also does the Unix fileio.   Assembler is fine, but I would use C/++ for
everything except the low level BPAM I/O routines, since I would probably use 
buffered filestreams for the Unix files.

FWIW: It's a pity that the IBM C library doesn't have any support for 
BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
agree:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811



On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  wrote:

> " What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"
>
> On the z/OS side is a PDS(E).
>
> Thanks
>
>
> Lionel B. Dyck <
> Website: https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is
> what you are, reputation merely what others think you are." - John
> Wooden
>
> -----Original Message-
> From: IBM Mainframe Discussion List  On
> Behalf Of Paul Gilmartin
> Sent: Tuesday, June 16, 2020 9:23 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> On Tue, 16 Jun 2020 20:34:59 -0500, Lionel B Dyck wrote:
>
> >Any suggestions on how to speed up cp copying multiple file to and
> >from
> z/OS?
> >
> >I gave SHAREAS=YES. Anything else?  Can I control buffers or ?
> >
> What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?
>
> I might suggest as an extreme measure Rexx under ISPF using ADDRESS
> SYSCALL I/O for OMVS and LM services for Classic.
> SYSCALL READFILE/WRITEFILE are available for text files only.
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Lionel B Dyck]

XMIT might be but the goal is to be as git compliant as possible with zigi 
being just a git frontend (aka IDE) that provides the interface to z/OS 
datasets for git.  Thus using XMIT format would defeat the purpose and violate 
git in many ways 😊


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Seymour J Metz
Sent: Wednesday, June 17, 2020 12:07 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

It might be more efficient to use XMIT format. Is there a REXX function package 
available for unpacking members?


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Kirk Wolf [k...@wolf-associates.com]
Sent: Wednesday, June 17, 2020 12:07 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

On Wed, Jun 17, 2020 at 10:43 AM Paul Gilmartin < 
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:

> On Wed, 17 Jun 2020 09:24:58 -0500, Kirk Wolf wrote:
> >
> >I wasn't thinking of using the "all members" form of cp ...
> >I'm curious - how much time did you save by preallocating the PDS?
> >
> I'm mystified that it made a difference since Lionel never used the 
> allocate "dd" in the call to "cp".  Only SVC 99 knows.
>
>
If you have a reusable allocation, then the cost is less than a new
allocation.   open/close for each member would still be expensive for a PDS
with tons of members.   A BLDL,loop: POINT,READ*  is what you really want
IMO.


> >I would think that you might actually want ISPF-style enqueues.
> >
> I thought later of OGETX/OPUTX, which piggybacks on ISPF.
>
> Good point - that would definitely be something to try.

Another fun thing to try is IEBUPDTE - I once played around with a REXX wrapper 
and then piped the control and data into it via DD:SYSIN pointing to /dev/fd0 
from a shell script.  It's *really* fast as expected, but has issues wrt the 
record format and content of the data that could mess it up.

BTW: If it was as fast as it should be, Lionel probably doesn't need a 
"progress" popup :-)

-- Kirk

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Seymour J Metz]

It might be more efficient to use XMIT format. Is there a REXX function package 
available for unpacking members?


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
Kirk Wolf [k...@wolf-associates.com]
Sent: Wednesday, June 17, 2020 12:07 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

On Wed, Jun 17, 2020 at 10:43 AM Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:

> On Wed, 17 Jun 2020 09:24:58 -0500, Kirk Wolf wrote:
> >
> >I wasn't thinking of using the "all members" form of cp ...
> >I'm curious - how much time did you save by preallocating the PDS?
> >
> I'm mystified that it made a difference since Lionel never used the
> allocate
> "dd" in the call to "cp".  Only SVC 99 knows.
>
>
If you have a reusable allocation, then the cost is less than a new
allocation.   open/close for each member would still be expensive for a PDS
with tons of members.   A BLDL,loop: POINT,READ*  is what you really want
IMO.


> >I would think that you might actually want ISPF-style enqueues.
> >
> I thought later of OGETX/OPUTX, which piggybacks on ISPF.
>
> Good point - that would definitely be something to try.

Another fun thing to try is IEBUPDTE - I once played around with a REXX
wrapper and then piped the control and data into it via DD:SYSIN pointing
to /dev/fd0 from a shell script.  It's *really* fast as expected, but has
issues wrt the record format and content of the data that could mess it up.

BTW: If it was as fast as it should be, Lionel probably doesn't need a
"progress" popup :-)

-- Kirk

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Kirk Wolf]

On Wed, Jun 17, 2020 at 10:43 AM Paul Gilmartin <
000433f07816-dmarc-requ...@listserv.ua.edu> wrote:

> On Wed, 17 Jun 2020 09:24:58 -0500, Kirk Wolf wrote:
> >
> >I wasn't thinking of using the "all members" form of cp ...
> >I'm curious - how much time did you save by preallocating the PDS?
> >
> I'm mystified that it made a difference since Lionel never used the
> allocate
> "dd" in the call to "cp".  Only SVC 99 knows.
>
>
If you have a reusable allocation, then the cost is less than a new
allocation.   open/close for each member would still be expensive for a PDS
with tons of members.   A BLDL,loop: POINT,READ*  is what you really want
IMO.


> >I would think that you might actually want ISPF-style enqueues.
> >
> I thought later of OGETX/OPUTX, which piggybacks on ISPF.
>
> Good point - that would definitely be something to try.

Another fun thing to try is IEBUPDTE - I once played around with a REXX
wrapper and then piped the control and data into it via DD:SYSIN pointing
to /dev/fd0 from a shell script.  It's *really* fast as expected, but has
issues wrt the record format and content of the data that could mess it up.

BTW: If it was as fast as it should be, Lionel probably doesn't need a
"progress" popup :-)

-- Kirk

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Wed, 17 Jun 2020 09:24:58 -0500, Kirk Wolf wrote:
>
>I wasn't thinking of using the "all members" form of cp ...
>I'm curious - how much time did you save by preallocating the PDS?
> 
I'm mystified that it made a difference since Lionel never used the allocate
"dd" in the call to "cp".  Only SVC 99 knows.

>I would think that you might actually want ISPF-style enqueues.
>
I thought later of OGETX/OPUTX, which piggybacks on ISPF.

>On Wed, Jun 17, 2020 at 7:22 AM Lionel B Dyck wrote:
>
>> Kirk - just allocating the dataset prior to the cp was faster - and that
>> was without passing the //DD.
>>
>> Alloc f(dd) shr reuse ds('my.pds')
>> Bpxwunix - cp -v -S a=.txt "//'my.pds'" .
>> Free f(dd)

>> -Original Message-
>> From: Kirk Wolf
>> Sent: Wednesday, June 17, 2020 7:03 AM
>>
>> - With cp, there won't be any way to avoid opening the PDSE for each
>> member, but you might get some improvement by allocating a DD to the PDSE
>> and then passing  //DD(member) to cp, ...
>> 
I believe that construct is not available in z/OS C RTL, but various ISVs
provide it. 
...
>> FWIW: It's a pity that the IBM C library doesn't have any support for
>> BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
>> agree:
>> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Kirk Wolf]

Lionel,
I wasn't thinking of using the "all members" form of cp - that seems like
it should be *much* better, although it would depend on how cp works under
the covers - evidence indicates that it just loops and does
alloc/open/close/free on each member.If only the cp authors had better
C library support for PDSs ;-)

I'm curious - how much time did you save by preallocating the PDS?

Kirk Wolf
http://dovetail.com

PS> I wonder what serialization you are getting for the target PDS is your
example?
Not so important if only PDSEs I guess :-)
I would think that you might actually want ISPF-style enqueues.

On Wed, Jun 17, 2020 at 7:22 AM Lionel B Dyck  wrote:

> Kirk - just allocating the dataset prior to the cp was faster - and that
> was without passing the //DD.
>
> Alloc f(dd) shr reuse ds('my.pds')
> Bpxwunix - cp -v -S a=.txt "//'my.pds'" .
> Free f(dd)
>
> Appreciate your suggestion
>
>
> Lionel B. Dyck <
> Website: https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is what
> you are, reputation merely what others think you are." - John Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On Behalf
> Of Kirk Wolf
> Sent: Wednesday, June 17, 2020 7:03 AM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> Hi Lionel,
>
> Can you provide any more detail on how you are invoking "cp" ?
>
> - With cp, there won't be any way to avoid opening the PDSE for each
> member, but you might get some improvement by allocating a DD to the PDSE
> and then passing  //DD(member) to cp, so as to avoid allocation each time.
>  If you do this, then you will also know for sure if you are using local
> spawn (_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new
> AS was forked.
>
> - The other issue would be the cost of spawning a Unix process for each
> member, even if local spawned.   I haven't tested this, but you might write
> a shell script that is passed the DD as arg and member names as lines to
> stdin.   Then the script could do the cp for each member.   The hope is
> that since cp is also a shell "built-in" you might avoid spawning
> processes for each one.
>
> - the "best" performance possible would be writing your own BPAM code that
> also does the Unix fileio.   Assembler is fine, but I would use C/++ for
> everything except the low level BPAM I/O routines, since I would probably
> use buffered filestreams for the Unix files.
>
> FWIW: It's a pity that the IBM C library doesn't have any support for
> BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
> agree:
> https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811
>
>
>
> On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  wrote:
>
> > " What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"
> >
> > On the z/OS side is a PDS(E).
> >
> > Thanks
> >
> >
> > Lionel B. Dyck <
> > Website: https://www.lbdsoftware.com
> >
> > "Worry more about your character than your reputation.  Character is
> > what you are, reputation merely what others think you are." - John
> > Wooden
> >
> > -Original Message-
> > From: IBM Mainframe Discussion List  On
> > Behalf Of Paul Gilmartin
> > Sent: Tuesday, June 16, 2020 9:23 PM
> > To: IBM-MAIN@LISTSERV.UA.EDU
> > Subject: Re: Improve OMVS cp performance?
> >
> > On Tue, 16 Jun 2020 20:34:59 -0500, Lionel B Dyck wrote:
> >
> > >Any suggestions on how to speed up cp copying multiple file to and
> > >from
> > z/OS?
> > >
> > >I gave SHAREAS=YES. Anything else?  Can I control buffers or ?
> > >
> > What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?
> >
> > I might suggest as an extreme measure Rexx under ISPF using ADDRESS
> > SYSCALL I/O for OMVS and LM services for Classic.
> > SYSCALL READFILE/WRITEFILE are available for text files only.
> >
> > -- gil
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions, send
> > email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions, send
> > email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
> >
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send email
> to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[David Crayford]

Kirk is spot on (as usual). You need a library like this one 
https://support.sas.com/documentation/onlinedoc/ccompiler/doc750/html/lr2/z2mvsbri.htm


On 2020-06-17 8:21 PM, Lionel B Dyck wrote:

Kirk - just allocating the dataset prior to the cp was faster - and that was 
without passing the //DD.

Alloc f(dd) shr reuse ds('my.pds')
Bpxwunix - cp -v -S a=.txt "//'my.pds'" .
Free f(dd)

Appreciate your suggestion


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you are, 
reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each member, 
but you might get some improvement by allocating a DD to the PDSE and then 
passing  //DD(member) to cp, so as to avoid allocation each time.
  If you do this, then you will also know for sure if you are using local spawn 
(_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new AS was 
forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the script could do the cp for each member.   The hope is
that since cp is also a shell "built-in" you might avoid spawning processes for 
each one.

- the "best" performance possible would be writing your own BPAM code that
also does the Unix fileio.   Assembler is fine, but I would use C/++ for
everything except the low level BPAM I/O routines, since I would probably use 
buffered filestreams for the Unix files.

FWIW: It's a pity that the IBM C library doesn't have any support for 
BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
agree:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811



On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  wrote:


" What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"

On the z/OS side is a PDS(E).

Thanks


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is
what you are, reputation merely what others think you are." - John
Wooden

-Original Message-
From: IBM Mainframe Discussion List  On
Behalf Of Paul Gilmartin
Sent: Tuesday, June 16, 2020 9:23 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

On Tue, 16 Jun 2020 20:34:59 -0500, Lionel B Dyck wrote:


Any suggestions on how to speed up cp copying multiple file to and
from

z/OS?

I gave SHAREAS=YES. Anything else?  Can I control buffers or ?


What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?

I might suggest as an extreme measure Rexx under ISPF using ADDRESS
SYSCALL I/O for OMVS and LM services for Classic.
SYSCALL READFILE/WRITEFILE are available for text files only.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions, send
email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions, send
email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Lionel B Dyck]

Kirk - just allocating the dataset prior to the cp was faster - and that was 
without passing the //DD.

Alloc f(dd) shr reuse ds('my.pds')
Bpxwunix - cp -v -S a=.txt "//'my.pds'" .
Free f(dd)

Appreciate your suggestion


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each member, 
but you might get some improvement by allocating a DD to the PDSE and then 
passing  //DD(member) to cp, so as to avoid allocation each time.
 If you do this, then you will also know for sure if you are using local spawn 
(_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new AS was 
forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the script could do the cp for each member.   The hope is
that since cp is also a shell "built-in" you might avoid spawning processes for 
each one.

- the "best" performance possible would be writing your own BPAM code that
also does the Unix fileio.   Assembler is fine, but I would use C/++ for
everything except the low level BPAM I/O routines, since I would probably use 
buffered filestreams for the Unix files.

FWIW: It's a pity that the IBM C library doesn't have any support for 
BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
agree:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811



On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  wrote:

> " What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"
>
> On the z/OS side is a PDS(E).
>
> Thanks
>
>
> Lionel B. Dyck <
> Website: https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is 
> what you are, reputation merely what others think you are." - John 
> Wooden
>
> -Original Message-----
> From: IBM Mainframe Discussion List  On 
> Behalf Of Paul Gilmartin
> Sent: Tuesday, June 16, 2020 9:23 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> On Tue, 16 Jun 2020 20:34:59 -0500, Lionel B Dyck wrote:
>
> >Any suggestions on how to speed up cp copying multiple file to and 
> >from
> z/OS?
> >
> >I gave SHAREAS=YES. Anything else?  Can I control buffers or ?
> >
> What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?
>
> I might suggest as an extreme measure Rexx under ISPF using ADDRESS 
> SYSCALL I/O for OMVS and LM services for Classic.
> SYSCALL READFILE/WRITEFILE are available for text files only.
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send 
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send 
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Lionel B Dyck]

Kirk - thank you for the ideas.

What I'm doing is in the ZIGI (see https://zigi.rocks) where I need to copy PDS 
members to/from USS so that Git can manage them. With small projects this isn't 
an issue but with larger projects it could take enough time for you to go to 
lunch ☹

Btw. I voted your RFE.


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Kirk Wolf
Sent: Wednesday, June 17, 2020 7:03 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each member, 
but you might get some improvement by allocating a DD to the PDSE and then 
passing  //DD(member) to cp, so as to avoid allocation each time.
 If you do this, then you will also know for sure if you are using local spawn 
(_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new AS was 
forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the script could do the cp for each member.   The hope is
that since cp is also a shell "built-in" you might avoid spawning processes for 
each one.

- the "best" performance possible would be writing your own BPAM code that
also does the Unix fileio.   Assembler is fine, but I would use C/++ for
everything except the low level BPAM I/O routines, since I would probably use 
buffered filestreams for the Unix files.

FWIW: It's a pity that the IBM C library doesn't have any support for 
BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
agree:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811



On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  wrote:

> " What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"
>
> On the z/OS side is a PDS(E).
>
> Thanks
>
>
> Lionel B. Dyck <
> Website: https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is 
> what you are, reputation merely what others think you are." - John 
> Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On 
> Behalf Of Paul Gilmartin
> Sent: Tuesday, June 16, 2020 9:23 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> On Tue, 16 Jun 2020 20:34:59 -0500, Lionel B Dyck wrote:
>
> >Any suggestions on how to speed up cp copying multiple file to and 
> >from
> z/OS?
> >
> >I gave SHAREAS=YES. Anything else?  Can I control buffers or ?
> >
> What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?
>
> I might suggest as an extreme measure Rexx under ISPF using ADDRESS 
> SYSCALL I/O for OMVS and LM services for Classic.
> SYSCALL READFILE/WRITEFILE are available for text files only.
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send 
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send 
> email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Kirk Wolf]

Hi Lionel,

Can you provide any more detail on how you are invoking "cp" ?

- With cp, there won't be any way to avoid opening the PDSE for each
member, but you might get some improvement by allocating a DD to the PDSE
and then passing  //DD(member) to cp, so as to avoid allocation each time.
 If you do this, then you will also know for sure if you are using local
spawn (_BPX_SHAREAS=YES), since otherwise the DD won't be visible if a new
AS was forked.

- The other issue would be the cost of spawning a Unix process for each
member, even if local spawned.   I haven't tested this, but you might write
a shell script that is passed the DD as arg and member names as lines to
stdin.   Then the script could do the cp for each member.   The hope is
that since cp is also a shell "built-in" you might avoid spawning processes
for each one.

- the "best" performance possible would be writing your own BPAM code that
also does the Unix fileio.   Assembler is fine, but I would use C/++ for
everything except the low level BPAM I/O routines, since I would probably
use buffered filestreams for the Unix files.

FWIW: It's a pity that the IBM C library doesn't have any support for
BLDL/NOTE/POINT processing of PDS/Es -- see my old RFE and vote if you
agree:
https://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe&CR_ID=80811



On Wed, Jun 17, 2020 at 5:30 AM Lionel B Dyck  wrote:

> " What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"
>
> On the z/OS side is a PDS(E).
>
> Thanks
>
>
> Lionel B. Dyck <
> Website: https://www.lbdsoftware.com
>
> "Worry more about your character than your reputation.  Character is what
> you are, reputation merely what others think you are." - John Wooden
>
> -Original Message-
> From: IBM Mainframe Discussion List  On Behalf
> Of Paul Gilmartin
> Sent: Tuesday, June 16, 2020 9:23 PM
> To: IBM-MAIN@LISTSERV.UA.EDU
> Subject: Re: Improve OMVS cp performance?
>
> On Tue, 16 Jun 2020 20:34:59 -0500, Lionel B Dyck wrote:
>
> >Any suggestions on how to speed up cp copying multiple file to and from
> z/OS?
> >
> >I gave SHAREAS=YES. Anything else?  Can I control buffers or ?
> >
> What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?
>
> I might suggest as an extreme measure Rexx under ISPF using ADDRESS
> SYSCALL I/O for OMVS and LM services for Classic.
> SYSCALL READFILE/WRITEFILE are available for text files only.
>
> -- gil
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions, send email
> to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Lionel B Dyck]

" What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?"

On the z/OS side is a PDS(E).

Thanks


Lionel B. Dyck <
Website: https://www.lbdsoftware.com

"Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden

-Original Message-
From: IBM Mainframe Discussion List  On Behalf Of 
Paul Gilmartin
Sent: Tuesday, June 16, 2020 9:23 PM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: Improve OMVS cp performance?

On Tue, 16 Jun 2020 20:34:59 -0500, Lionel B Dyck wrote:

>Any suggestions on how to speed up cp copying multiple file to and from z/OS?
>
>I gave SHAREAS=YES. Anything else?  Can I control buffers or ?
> 
What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?

I might suggest as an extreme measure Rexx under ISPF using ADDRESS SYSCALL I/O 
for OMVS and LM services for Classic.
SYSCALL READFILE/WRITEFILE are available for text files only.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email to 
lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Improve OMVS cp performance?

[Paul Gilmartin]

On Tue, 16 Jun 2020 20:34:59 -0500, Lionel B Dyck wrote:

>Any suggestions on how to speed up cp copying multiple file to and from z/OS?
>
>I gave SHAREAS=YES. Anything else?  Can I control buffers or ?
> 
What's on the non-OMVS side?  a  PDS(E)?  Multiple PS data sets?

I might suggest as an extreme measure Rexx under ISPF using
ADDRESS SYSCALL I/O for OMVS and LM services for Classic.
SYSCALL READFILE/WRITEFILE are available for text files only.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Improve OMVS cp performance?

[Lionel B Dyck]

Any suggestions on how to speed up cp copying multiple file to and from z/OS?

I gave SHAREAS=YES. Anything else?  Can I control buffers or ?

Lionel B. Dyck <
Website: www.lbdsoftware.com
Sent from my iPhone 8+

Worry more about your character than your reputation.  Character is what you 
are, reputation merely what others think you are." - John Wooden 
--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN