Re: IBM SSI Function Codes 16 and 17

[Jerry Callen]

See: https://github.com/CBTTape/290

It contains an implementation of subsystem datasets.

The documentation you WANT is the Washington Systems Center Technical Bulletin 
GG66-3131-00, which,  unfortunately, seems to have become Pure Unobtainium.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: casting with XL C\C++ compiler

[Jerry Callen]

> Anyway read the link and give the compiler a try!

I have, and it works - very well! If you haven't tried this compiler, you 
should. There are some limitations, but if you like the 64-bit LE programming 
model, this compiler is a win.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Dr. John Ehrman

[Jerry Callen]

I came across this in "The Archives" (aka "the huge pile of crap in my home 
office") and
it brought back very fond memories of John. 

https://photos.app.goo.gl/Ksqfw4MzFSgfG5g97

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COW for fork() is disappearing in z/OS 2.4

[Jerry Callen]

David Crayford says...

> ...It would be fantastic to have bash as the default shell for z/OS but that
> ain't gonna happen anytime soon :)

FWIW: Here's how I tend to work on z/OS.

* I leave my default login shell (in the OMVS segment) as /bin/sh.
* I have my .profile/.bash_profile/.bashrc files set up to switch me to bash
   immediately (though my files are set up to work with /bin/sh , just in case).
* I usually write shell scripts with a #!/bin/sh shebang (giving up some bash
   features in scripts).

It works for me. I get bash interactive behavior, but my scripts don't require 
bash.
Yes, a compromise, but one I consider acceptable. The Rocket bash port has
some advantages in a mixed-encoding environment (_BPXK_AUTOCVT=ON).

Since I run nearly all my shells in an emacs buffer anyway, shell interactive 
behavior
isn't something I usually care much about anyway. :-)

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COW for fork() is disappearing in z/OS 2.4

[Jerry Callen]

Kirk Wolf wrote:

> Unfortunately, bash is pretty complicated software, and updating it to do
> this is not at all easy because of the difference in semantics between
> fork()/exec() and spawn().

Amen, brethren! :-)

Interestingly, it looks like there is exactly ONE call to fork() in the bash 
source.

But -- it's in a function that gets called all over the place, with large 
amounts of
setup and post-fork processing, on both sides (parent/child). It might be 
possible to
substitute spawn in some cases but, as you noted, it wouldn't be easy.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: COW for fork() is disappearing in z/OS 2.4

[Jerry Callen]

> The Percent29 at the end breaks the URL. Is your e-mail software doing that 
> automatically?

I created the post directly from the listserv's web interface - no email 
software involved. I *did* have a closing paren at the end of the URL, since I 
was in the middle of a parenthesized phrase; it seems that somehow got rolled 
into the URL.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

COW for fork() is disappearing in z/OS 2.4

[Jerry Callen]

According to the "Summary of changes for z/OS UNIX System Services Planning for 
Version 2 Release 4 (V2R4)" (see 
https://www.ibm.com/support/knowledgecenter/SSLTBW_2.4.0/com.ibm.zos.v2r4.bpxb200/bpxb2soc24.htm),
 copy on write for fork() has been removed from z/OS. It will be interesting to 
benchmark the performance of fork() between 2.3 and 2.4 on a variety of 
processes.

I wonder why this was done? (Hi, Peter Relson -- hint, hint...)

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: USS script : how to test that a MVS Dataset exists ?

[Jerry Callen]

> So I cannot use the "tso" Unix command...

Doh. 

Never mind.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: USS script : how to test that a MVS Dataset exists ?

[Jerry Callen]

(Another "too late" reply. That'll teach me to go on vacation.)

This script illustrates how to do it, but I'm not sure it would work early in 
the IPL sequence (since it needs to run TSO commands):

#!/bin/sh

if /bin/tsocmd "listd '$1'" >/dev/null 2>/dev/null ; then
  echo "$1 exists"
else
  echo "$1 does not exist"
fi

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: getting a list of FMIDS and the product description

[Jerry Callen]

(A day late and a dollar short...)

My nifty little USS SMP query tool can get you the necessary bits of 
information from the CSI, 
but there is Some Assembly Required to pull it together into a nice report 
(which I will not do  
here, but if I had to, would use some combination of sed and awk). 

That tool is here:  https://github.com/zorts/smpapi

Here are the queries and their output (on my system).

-- Jerry

smp -z global -e product -s prodid,vrm,description

PRODUCT|GLOBAL||DESCRIPTION|z/OS
PRODUCT|GLOBAL||PRODID|5650-ZOS
PRODUCT|GLOBAL||VRM|02.03.00
PRODUCT|GLOBAL||DESCRIPTION|IBM z/OS Connect Enterprise Edition V3
PRODUCT|GLOBAL||PRODID|5655-CE3
PRODUCT|GLOBAL||VRM|03.00.00
PRODUCT|GLOBAL||DESCRIPTION|IBM 64-bit SDK for z/OS, Java 2 Technology Edition, 
V8
PRODUCT|GLOBAL||PRODID|5655-DGH
PRODUCT|GLOBAL||VRM|08.00.00
PRODUCT|GLOBAL||DESCRIPTION|VISUAL AGE FOR JAVA
PRODUCT|GLOBAL||PRODID|5655-JAV
PRODUCT|GLOBAL||VRM|02.00.00
PRODUCT|GLOBAL||DESCRIPTION|Print Services Facility for z/OS
PRODUCT|GLOBAL||PRODID|5655-M32
PRODUCT|GLOBAL||VRM|04.05.00
PRODUCT|GLOBAL||DESCRIPTION|IBM Encryption Facility for z/OS
PRODUCT|GLOBAL||PRODID|5655-P97
PRODUCT|GLOBAL||VRM|01.02.00
PRODUCT|GLOBAL||DESCRIPTION|IBM Fault Analyzer for z/OS V14
PRODUCT|GLOBAL||PRODID|5655-Q41
PRODUCT|GLOBAL||VRM|14.01.00
PRODUCT|GLOBAL||DESCRIPTION|IBM File Manager for z/OS V14
PRODUCT|GLOBAL||PRODID|5655-Q42
PRODUCT|GLOBAL||VRM|14.01.00
PRODUCT|GLOBAL||DESCRIPTION|IBM Developer for z Debug Edition V14
PRODUCT|GLOBAL||PRODID|5655-Q50
PRODUCT|GLOBAL||VRM|14.01.00
PRODUCT|GLOBAL||DESCRIPTION|DITTO/ESA FOR MVS
PRODUCT|GLOBAL||PRODID|5655-103
PRODUCT|GLOBAL||VRM|01.03.00
PRODUCT|GLOBAL||DESCRIPTION|GDDM IVU
PRODUCT|GLOBAL||PRODID|5668-723
PRODUCT|GLOBAL||VRM|01.01.03
PRODUCT|GLOBAL||DESCRIPTION|GDDM IMD
PRODUCT|GLOBAL||PRODID|5668-801
PRODUCT|GLOBAL||VRM|02.01.03
PRODUCT|GLOBAL||DESCRIPTION|TELEPROCESSING NETWORK SIMULATOR
PRODUCT|GLOBAL||PRODID|5688-121
PRODUCT|GLOBAL||VRM|03.05.00
PRODUCT|GLOBAL||DESCRIPTION|C/370 LIBRARY
PRODUCT|GLOBAL||PRODID|5688-188
PRODUCT|GLOBAL||VRM|02.02.00
PRODUCT|GLOBAL||DESCRIPTION|IBM PAGE PRINTER FORMATTING AID/370
PRODUCT|GLOBAL||PRODID|5688-190
PRODUCT|GLOBAL||VRM|01.01.00
PRODUCT|GLOBAL||DESCRIPTION|OVERLAY GENERATION LANGUAGE/370
PRODUCT|GLOBAL||PRODID|5688-191
PRODUCT|GLOBAL||VRM|01.01.00
PRODUCT|GLOBAL||DESCRIPTION|IBM Compiler for REXX/370
PRODUCT|GLOBAL||PRODID|5695-013
PRODUCT|GLOBAL||VRM|01.04.00
PRODUCT|GLOBAL||DESCRIPTION|IBM Library for REXX/370
PRODUCT|GLOBAL||PRODID|5695-014
PRODUCT|GLOBAL||VRM|01.04.00
PRODUCT|GLOBAL||DESCRIPTION|IBM High Level Assembler for MVS & VM & VSE
PRODUCT|GLOBAL||PRODID|5696-234
PRODUCT|GLOBAL||VRM|01.06.00
PRODUCT|GLOBAL||DESCRIPTION|Content Manager OnDemand Premium Feature Pack for 
z/OS
PRODUCT|GLOBAL||PRODID|5697-CM1
PRODUCT|GLOBAL||VRM|10.01.00

smp -z global -e feature -s product,fmid

FEATURE|GLOBAL|DZOBB230|FMID|HCM1G10
FEATURE|GLOBAL|DZOBB230|FMID|HGD3201
FEATURE|GLOBAL|DZOBB230|FMID|HJS77B0
FEATURE|GLOBAL|DZOBB230|FMID|HLB77B0
FEATURE|GLOBAL|DZOBB230|FMID|HMOS705
FEATURE|GLOBAL|DZOBB230|FMID|HNET7B0
FEATURE|GLOBAL|DZOBB230|FMID|HOPI7B0
FEATURE|GLOBAL|DZOBB230|FMID|HQX77B0
FEATURE|GLOBAL|DZOBB230|FMID|HRF77B0
FEATURE|GLOBAL|DZOBB230|FMID|HRM77B0
FEATURE|GLOBAL|DZOBB230|FMID|HSM1N00
FEATURE|GLOBAL|DZOBB230|FMID|H24P111
FEATURE|GLOBAL|DZOBB230|FMID|JBD6201
FEATURE|GLOBAL|DZOBB230|FMID|JBD6202
FEATURE|GLOBAL|DZOBB230|FMID|JMQ416A
FEATURE|GLOBAL|DZOBB230|PRODUCT|5650-ZOS,02.03.00
FEATURE|GLOBAL|ICLIB220|FMID|HCLB201
FEATURE|GLOBAL|ICLIB220|FMID|HDL1202
FEATURE|GLOBAL|ICLIB220|FMID|JCLB203
FEATURE|GLOBAL|ICLIB220|FMID|JCLB204
FEATURE|GLOBAL|ICLIB220|FMID|JDL1212
FEATURE|GLOBAL|ICLIB220|FMID|JDL1214
FEATURE|GLOBAL|ICLIB220|PRODUCT|5688-188,02.02.00
FEATURE|GLOBAL|IDBTBE1A|FMID|HADRE10
FEATURE|GLOBAL|IDBTBE1A|FMID|HVWR180
FEATURE|GLOBAL|IDBTBE1A|FMID|H09F210
FEATURE|GLOBAL|IDBTBE1A|PRODUCT|5655-Q50,14.01.00
FEATURE|GLOBAL|IDITE130|FMID|H0GB310
FEATURE|GLOBAL|IDITE130|PRODUCT|5655-103,01.03.00
FEATURE|GLOBAL|IEFAD120|FMID|HCF773D
FEATURE|GLOBAL|IEFAD120|PRODUCT|5655-P97,01.02.00
FEATURE|GLOBAL|IEFAS120|FMID|HCF7740
FEATURE|GLOBAL|IEFAS120|PRODUCT|5655-P97,01.02.00
FEATURE|GLOBAL|IEHDAA10|FMID|HQN4450
FEATURE|GLOBAL|IEHDAA10|PRODUCT|5697-CM1,10.01.00
FEATURE|GLOBAL|IFANBE10|FMID|HADQE10
FEATURE|GLOBAL|IFANBE10|FMID|HVWR180
FEATURE|GLOBAL|IFANBE10|PRODUCT|5655-Q41,14.01.00
FEATURE|GLOBAL|IGDMD213|FMID|HGD3202
FEATURE|GLOBAL|IGDMD213|PRODUCT|5668-801,02.01.03
FEATURE|GLOBAL|IGDVE113|FMID|HGD3204
FEATURE|GLOBAL|IGDVE113|PRODUCT|5668-723,01.01.03
FEATURE|GLOBAL|IHLAS160|FMID|HMQ4160
FEATURE|GLOBAL|IHLAS160|PRODUCT|5696-234,01.06.00
FEATURE|GLOBAL|IHLAT160|FMID|JMQ416A
FEATURE|GLOBAL|IHLAT160|PRODUCT|5696-234,01.06.00
FEATURE|GLOBAL|IISZB800|FMID|HJVB800
FEATURE|GLOBAL|IISZB800|PRODUCT|5655-DGH,08.00.00
FEATURE|GLOBAL|IIZEB300|FMID|HZC3000
FEATURE|GLOBAL|IIZEB300|FMID|JZC3002
FEATURE|GLOBAL|IIZEB300|PRODUCT|5655-CE3,03.00.00
FEATURE|GLOBAL|IOGLE110|FMID|HVRL100

What's the intent of the "LOAD AND ZERO RIGHTMOST BYTE" instruction?

[Jerry Callen]

Is it Friday yet?

The z13 (I think) added this puppy:

=

LOAD AND ZERO RIGHTMOST BYTE

The second operand, with the rightmost byte set to zero, is placed at the 
first-operand location. For LZRF, the first and second operands are 32 bits, 
and for LZRG, the first and second operands are 64 bits.

The displacement is treated as a 20-bit signed binary integer.

It is unpredictable whether an access exception is recognized for the rightmost 
byte of the second operand.

=

Why?

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: RFE to expand Unix path length for DYNALLOC beyond 255 characters

[Jerry Callen]

On Tue, 7 May 2019 14:24:26 -0500, Paul Gilmartin wrote:

> At least support, for consistency:
>
> https://www.ibm.com/support/knowledgecenter/en/SSLTBW_2.3.0/com.ibm.zos.v2r3.bpxa400/pathname.htm
>
> ... A path name can be up to 1023 characters long, including
> all directory names, file names, and separating slashes. ...

Yes, although I can tell you that this is the limit imposed by Windows
(I think), and for npm, even that has proven to be a problem. Deeply
nested Java class paths also bump into this.

I don't really understand why there should need for any limit at all.
Hasn't anyone heard of malloc? :-)

> The enhancement should apply to JCL allocation as well as DYNALLOC.

Hmm, yet another code path. Though it might well be that fixing it in
DYNALLOC would fix it for JCL as well.

> Does any SVC 99 TU exceed 256 characters?  Is this a relic of the
> MVC instruction?

Dollars to donuts that's the reason.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

RFE to expand Unix path length for DYNALLOC beyond 255 characters

[Jerry Callen]

I recently discovered that the maximum path length for dynamic
allocation key 8017 (Unix PATH name) is 255 characters, in spite of
the fact that the text unit length field is 16 bits. While most "human
generated" path lengths will not be that long, software generated paths
can easily exceed that. Anyone who has used npm on Windows knows
exactly what I'm talking about, and npm is now being used on z/OS
(with node.js).

I have opened an RFE to expand this length; please consider voting for it.

http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=132716

Thanks!

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Newbie SMP/E questions

[Jerry Callen]

> The other option is to see if the ISPF environment has the ISPF SMP/E panels 
> available.

If you are coming from a Unix background, are comfortable with the USS command 
line and scripting, and have the xlc compiler available (a lot of "ifs"...), 
you might want to try this:

https://github.com/zorts/smpapi

It's a little C program that uses the SMP query API to dig information out of 
the SMPCSI. I personally find it easier to use than the ISPF panels.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: RFE for Posix semaphores

[Jerry Callen]

Timothy Sipples  wrote:

 > Would it also make sense to ask for automatic release/cleanup of stale 
 > semaphores?

Someone could certainly ask for a z/OS-specific variation on the XSI API.

That doesn't eliminate the need for a standards-conforming implementation of 
Posix semaphores.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: RFE for Posix semaphores

[Jerry Callen]

Stackoverflow offers a nice little comparison between XSI and Posix smaphores 
here:

https://stackoverflow.com/questions/11058045/differences-between-semaphore-h-and-sys-sem-h

Stevens & Rago, in "Advanced Programming in the Unix Environment", section 15.8:

http://www.apuebook.com/toc3e.html

provide these comments:

1. A semaphore is not simply a single non-negative value. Instead, we have to 
define a semaphore as a set of one or more semaphore values. When we create the 
semaphore, we specify the number of values in the set.

2. The creation of a semaphore (semget) is independent of its initialization 
(semctl). This is a fatal flaw, since we cannot atomically creste a new 
semaphore set and initialize all the values in the set.

3. Since all forms of XSI IPC remain in existence even when no process is using 
them, we have to worry about a program that terminates without releasing the 
semaphores it has allocated. The undo feature that we describe later is 
supposed to handle this.

Unix System Services Planning, in the section "Managing Interprocess 
Communication (IPC)":

https://www.ibm.com/support/knowledgecenter/SSLTBW_2.3.0/com.ibm.zos.v2r3.bpxb200/ipcm.htm

notes that:

Users can invoke applications that create IPC resources and wait for IPC 
resources. IPC resources are not automatically released when a process is ended 
or a user logs off. Therefore, it is possible that an IPC user might need 
assistance to do either of the following:

 * Remove an IPC resource using the shell's ipcrm command

 * Remove an IPC resource using the shell's ipcrm command to release a user 
from an IPC wait state

This last reference just hints at another problem: semaphores are a limited 
resource, and if they are not properly cleaned up, they can be exhausted. The 
"Troubleshooting" page for IBM's implementation of node.js refers to this:

https://www.ibm.com/support/knowledgecenter/en/SSTRRS_6.0.0/com.ibm.nodejs.zos.v6.doc/troubleshoot.htm

Node.js uses z/OS UNIX System Services semaphores. If the Node.js runtime ends 
unexpectedly, stale semaphores might be left on the system, potentially 
resulting in a lack of available semaphores for invoking more Node.js 
instances. Currently, you must manually clean up stale semaphores, by using the 
ipcrm command. Too many stale semaphores results in the following node assert 
or error:

CEE3204S The system detected a protection exception (System Completion 
Code=0C4).
 From entry point _ZN4node6AssertEPA4_KPKc at compile unit offset 
 +25F3AC86 at entry offset +00BE at address
 25F3AC86.

I'll add this description of the "limitations" of XSI semaphores to the RFE.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: RFE for Posix semaphores

[Jerry Callen]

Timothy Sipples  writes:

> Does this article help?
> https://www.ibm.com/developerworks/library/l-semaphore/index.html

Nope. That's pretty much what I'm doing as a workaround. I want the real thing.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

RFE for Posix semaphores

[Jerry Callen]

See: http://www.ibm.com/developerworks/rfe/execute?use_case=viewRfe_ID=129007

AFAIK, z/OS only supports XSI semaphores, not Posix semaphores. It's possible 
to emulate Posix semaphores, but with restrictions and messiness. 

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Bring back the IBM Journal of R!

[Jerry Callen]

From another thread:

> Many of these techniques have been documented in the IBM Journal of Research 
> and Development.
> Unfortunately, a few years ago IBM decided to hide that behind a paywall.

This has always seemed to me like a colossal marketing failure. There is just 
no better showcase for IBM's technical chops than this publication.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

How can I list the contents of a ADRDSSU DUMP dataset?

[Jerry Callen]

I'd like to be able to list the names of the files present in a "logical dump" 
dataset produced by ADRDSSU. AFAIK, there isn't a command to ADRDSSU to do that 
(seems like an odd omission...). 

The format of the dump dataset is thoroughly described:

https://www.ibm.com/support/knowledgecenter/SSLTBW_2.3.0/com.ibm.zos.v2r3.adru000/fmtdds.htm

But I really don't want to have to write the code to grovel over the dataset 
and format this stuff if someone else has already done it. :-)

I looked on the CBT Tape; no joy.

Any suggestions?

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How call BPXMTEXT from Posix STC

[Jerry Callen]

OK, a few clues. 

bpxmtext is, as others have pointed out, a rexx exec. In at least some cases it 
turns around and calls another rexx exec, edcmtext. That, in turn winds up 
looking at this module: CEE.SCEERUN(EDCEJR). That, finally, contains message 
text. The rexx is (to me, a non-rexx programmer) somewhat inscrutable. Just 
groveling over a binary copy of CEE.SCEERUN(EDCEJR) suggests that it has some 
sort of index table at the front, followed by the actual message texts. 

And that's all I have time for today (all of 5 minutes...). Maybe in a few more 
days I can dig deeper.

Would an API have been SO hard? G

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How call BPXMTEXT from Posix STC

[Jerry Callen]

> Based on advice on another thread I am now calling errno2() on certain
> socket errors. I get back an integer that I dutifully display in hex:
> 
> XXX0123I errno2 = x'12345678'. Look up using BPXMTEXT.
> 
> That message kind of offends me. When I get a basic library error I use
> strerror() to convert it to text. I would like to convert errno2() to text,
> rather than telling customers to go look it up for themselves.

Likewise.

A while back I went looking for some way to convert errno2 to a message without 
calling BPXMTEXT 
and got nowhere before I had to get back to Real Work (tm). It seems to me that 
either:

* strerror() just ought to do it, or
* there should be a dead-simple API.

I guess this calls for an enhancement request...

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Zowe for systems programmer ?

[Jerry Callen]

On Sun, Sep 2, 2018, 9:56 AM Bruce Armstrong  wrote:

> If you know what ISPF is and are happy with it you may not be the
> target market for Zowe. Zowe is primarily targeted for the "next gen"
> sysprog who does not have 30+ years of learning the the nuances of
> z/OS and ISPF.

Or - maybe you are the target market, but don't yet know it.

I *DO* have many years of experience with z/OS -- I learned to program in PL/I 
and assembler on OS/MVT in the early 1970s and was a systems programmer through 
the mid 1980s. I've worked on more modern z/OS systems for the past 4 years.

I also have many years of experience with both Unix and Windows systems. It's 
that experience that makes me so excited about the potential of Zowe. It's 
providing a framework for bringing highly-evolved system management tools of 
all kinds to z/OS. This will ultimately make the systems programmer's job 
easier. That's not a bad thing.

It's not just that the "next gen" programmers need to learn z/OS -- it's time 
for established z/OS programmers to start adopting tools and practices from 
other platforms, too.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Zowe for systems programmer ?

[Jerry Callen]

Andrew Rowley wrote:

On 31/08/2018 3:05 AM, Jerry Callen wrote:

> Everyone has to follow the convention, and on z/OS they LARGELY do.

(Emphasis added)  

I rest my case. :-)

> Source control is not a better solution, it is a solution
> to a slightly different problem.

Fair enough.

> When using source control you STILL need to make sure that 2 people 
> are not updating the same file at the same time - it is just the
> window that is smaller.

On z/OS you could solve that with DISP=OLD (though that's not practical
in all situations). I would argue that the right solution is to lock
your critical datasets down tightly with SAF and automated procedures
such that direct, uncontrolled updates become a firing offense.

> When I have worked at larger sites, there might be 5-10 systems
> programmers with changes scheduled for a weekend IPL. When the IPL was
> confirmed, typically there were multiple people who needed to update
> the same members of SYS1.PARMLIB. We did have manual processes to
> coordinate updates (typically they were all funneled through a
> designated person) but without that offline manual process it would be
> likely that there were multiple people trying to update the same file
> at the same time.

This is precisely the situation where you want source control, with code
review and a single controlled update. As others have noted, the "pull
request" idiom used by systems like GitHub and BitBucket are ideally
suited to this situation. Everyone puts their needed changes on a branch,
the branches are merged, humans review the result, and ONE person or
automated process deploys the change.

> When I heard about git for z/OS my first question was can it handle
> z/OS datasets like SYS1.PARMLIB, answer: no.

Sure it can, just not in the obvious manner. You have to be willing
to stop treating the PDS as the "repository of record" and instead
treat it as a deployed resource. Your git server becomes the canonical
reference, and you update the PDS (via an automated process) when
changes are merged into the "master" branch by that server. 

> We are where we are - it is important that existing functions continue 
> to work as expected. So, please, make Zowe edit compatible with ISPF 
> edit serialization.

I think the proposals by Matt Hogstrom and Kirk Wolf solve the same
problem in a better way. I don't think Zowe should perpetuate a
practice that, IMO, is actually broken. 

> Do other platforms really use source control for everything? How many 
> unix systems have you encountered where /etc is under source control, 
> people have their own copies and merge changes into the real /etc? Any?

Not to air dirty laundry, but some places I've worked have done this,
others not. :-/ In any case, it's not unheard of, and becoming the norm.

What I keep coming back to is this: we now have better tools for
system management. Why WOULDN'T we use them? It won't happen
overnight, but this is surely the direction we should be heading.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Zowe for systems programmer ?

[Jerry Callen]

> Question for y’all.  I would like to do a diff on the modified file and 
> present the user with the changes.
> They could REPLACE the file with the editor contents, MERGE them which would 
> be something they 
> would do after a review of the diff, ABANDON them or save them somewhere 
> else.  Does this strategy 
> sound acceptable ?

Sounds completely awesome.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Zowe for systems programmer ?

[Jerry Callen]

On Aug 29, 2018, at 8:20 PM, Andrew Rowley  wrote:
> 
> On 30/08/2018 12:16 AM, Jerry Callen wrote:
>> The whole idea of holding a lock on a file while a human being slowly edits 
>> it is so 1960s.
>> 
>> Since at least the mid 1970s, editors like emacs have loaded the file for 
>> editing and noted the timestamp. 
>> When the user attempts to save the file. the timestamp is checked again, and 
>> if it changed, the user is asked what to do.

> I always thought this was a result of being unable to effectively prevent 
> simultaneous edits, not a feature.

Of course it's possible to prevent simultaneous edits, at least to the extent 
that you claim ISPF does. ISPF doesn't REALLY prevent simultaneous edits; it 
relies on a convention, and you have to hope that everyone follows the 
convention. That's the issue that started this thread. In Unix that's 
traditionally been done with a lock directory playing the role of an ENQ, but 
it can be done. It's just that -- no one does, because source control is a 
better solution.

> The user is asked what to do - do you want to throw away your changes or the 
> other guy's changes? 
> The answer is always the other guy's changes, right?

No. It's trivial to save my copy of the file with another name, load the other 
guy's changes, and merge them. Of course, that's because I use emacs. Your 
mileage (with less capable editors) may vary.

[Discussion of merging and its perils elided]

Here's the dirty little secret: simultaneous edits of the exact same file (in 
non-z/OS contexts) are uncommon. I'll go out on a limb and bet that even in 
z/OS, simultaneous edits of even the exact same file are uncommon. If you're 
not using source control system, then, yes, preventing those rare occurrences 
is a big deal.

What IS common is simultaneous edits to COPIES of the same file (in user 
sandboxes), and that's why merge tools exist. 

But - why not use a source control system and prohibit direct editing-in-place 
of critical things libraries like PARMLIBs, PROCLIBs, etc? You already have 
many system libraries under SMP control; no direct changes to, say, 
SYS1.LINKLIB. You COULD put everything else under SMP, but the effort to do so 
is just too high. This is where modern source control systems come in - 
control, with minimal pain.

I keep hammering on this because I think it really MATTERS. If you're a systems 
programmer, you ought to be at the head of line when it comes to adopting tools 
that allow you to bring consistency and order to system maintenance. 

It's early days for Zowe. The crucial first step was to get a framework in 
place into which. Zowe builds on the back-end framework of ZOSMF and similar 
services and provides a modern, responsive web UI for those services. What 
comes next is, to some extent, up to all of us. Remember when OS/360 was open 
source, and brought us things like HASP? Zowe is a return to those roots.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Zowe for systems programmer ?

[Jerry Callen]

The whole idea of holding a lock on a file while a human being slowly edits it 
is so 1960s. 

Since at least the mid 1970s, editors like emacs have loaded the file for 
editing and noted the timestamp. When the user attempts to save the file. the 
timestamp is checked again, and if it changed, the user is asked what to do.

And, of course, if you use a distributed source control system like git, 
handling merge conflicts is a built-in and normal part of the process.

It's very easy to imagine a zowe plugin that backs a source PDS with a git repo 
on a server (GitHub/BitBucket) and integrates editing with the whole git 
ecosystem of branches and pull requests. 

I say again: z/OS systems are too complex and mission-critical for us to 
continue using outdated tooling and engineering processes to maintain them.



--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Zowe for systems programmer ?

[Jerry Callen]

On Fri, 24 Aug 2018 07:20:24 -0500, Tom Marchant wrote:

>> via the Zowe command line you can stay in the laptop context
>> and Zowe will fetch a data set behind the scene, allow it to
>> be edited by your favorite editor and save back to z/OS
>> transparently.

> Will it do so while providing integrity so that two users can't
> edit the same data set (or member) at the same time, whether
> using Zowe or ISPF?

I would argue that shared files that might be updated by multiple
users simultaneously should be in a source control system anyway. If
the files MUST wind up in a PDS (parmlib members, anyone?), that
staging should happen in a controlled manner.

So - keep your text files in the Unix file system under the control of
git. ISPF will happily edit USS files -- no need to copy them to a
PDS.  You can keep your parmlib members in an off-mainframe server
(like an on-premises github or bitbucket server), have your changes
code-reviewed by your fellow sysprogs, committed, and pushed to the
parmlib via a deployment mechanism like Jenkins. All that stuff is
accessed via the browser.

You think that's crazy? It's how the off-mainframe world has been
working for the past decade, at least. With the proper tools, and the
willingness to learn to use them well, it improves productivity and
reduces errors.

IMO, it's time to start treating the mainframe like the complex,
mission-critical embedded system that it is, and use modern tools
to manage it.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Anyone here exprerienced in JSON parser (assembler)

[Jerry Callen]

So I can't help but wonder: what's the big picture here? WHY is it necessary to 
parse JSON from assembler? There is presumably a good reason that assembler is 
in the driver's seat here, instead of, say, C, C++, or Python (all viable 
languages on z/OS). 

Yes, it's certainly possible to write recursive assembler code, and possible to 
do it efficiently with a stack, but WHY? It would be easier in Metal C (if the 
environment dictates that), and VERY easy from LE C/C++ or Python. And for the 
bash scripters out there, there's a nifty little command line tool (the "jq" 
command) that will parse JSON and deliver it in a convenient form to a shell 
script. (Rocket has it ported to z/OS internally, but not yet available for 
download.)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: [EXTERNAL] Zowe install woes

[Jerry Callen]

NOTE: I'm not a Zowe developer, but I sit near some of them at Rocket
and have some understanding of its structure via osmosis. :-) This
should not be considered "authoritative."

There are several pieces to Zowe:

* A web client (code that runs in the browser: HTML, CSS and
  JavaScript). This is, as I understand it, all open source. Since it
  runs in the browser, there's not much choice. :-)

* A node.js server that provides some services itself (including
  serving the web content to the browser) and proxies to other
  services. This can run on z/OS or on other platforms. Since it's
  mostly just a proxy server, there are a bunch of other "back end"
  servers it can talk to:

  * A z/OS server that provides some of the "native" z/OS services.
  * The standard z/OS TN3270 portal (to provide 3270 emulation support).
  * The standard z/OS telnet portal (for USS native logins).
  * REST services provided by ZOSMF or anything else.

The web client code is designed to allow new components to be easily
plugged-in; those components can use REST or WebSockets to talk to
whatever backend pieces they need. The slick part is the "virtual
desktop" in which these plug-ins live.

Because Zowe includes a TN3270 client, it doesn't in any sense
prohibit access to traditional TSO/ISPF services. Similarly, it
provides a "VT" client, so it can talk to USS, as well. Each of
these is just a window on the virtual desktop. 

Plug-ins can be either open or closed source, especially on the
server side.

It's pretty nifty. :-)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Would SHARE kindly kick IBM in the ass for what the've done with their web content?

[Jerry Callen]

On Thu, 14 Jun 2018 11:27:47 -0500, Mike Schwab  wrote:

>Even if you have a z/OS 2.1 manual, some PTFs do add features or
>command or document changes to older versions, and they only update
>the newest manual.  Sometimes it is best to get the newer manual then
>check the document changes so you aren't referencing a feature only in
>the newest release.

OK, here's a constructive proposal: there should be a single HTML-based web 
tree that contains the doc for the MOST CURRENT VERSION of z/OS. Each page 
would then have a menu dropdown to allow you to select an earlier release. The 
URLs for the current pages would be kept STABLE, so that links from other pages 
won't go dead when future releases come out.

The web designers would use robots.txt or whatever to keep crawlers out of the 
back level doc, so that only the most current doc is indexed. The when you 
search for "IDCAMS REPRO", you get the hits to the LATEST doc, but can find the 
back-level stuff easily enough.

I agree that the PDFs are valuable (I use them all the time), but it's ALSO 
really handy to be able to embed a link to the EXACT SPOT n the documentation 
that I am referring to, and that requires a URL to a page, not to a PTF.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: PDSEGEN - 2 years old

[Jerry Callen]

So, I'm curious: What are people actually USING member generations for? 

For source code, it seems like a decent source control system is a better bet. 
Are member generations primarily a form of "oops recovery?"

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: CONTROVERSY! z/OS UNIX: is it an enhancement or a tool of the Devil?

[Jerry Callen]

On Fri, 11 May 2018 10:46:20 -0400, Hobart Spitz  wrote:

>Second to that are the deficient string and file models:
>
>   - There are separate techniques for processing text, on one hand, and
>   binary data on the other.  (z/OS, z/VM have no such requirement.)
>   - Using the wrong technique can break data structures badly, as can
>   inadvertent/erroneous embedding of binary data in text strings/files.
>   (z/OS, z/VM have not such problem.)
>   - One of *nix's most powerful features, piping, is crippled because most
>   filters are text oriented and cannot process binary data.  Dispatching is
>   mostly at the discretion of the operating system.  (Dispatching generally
>   deterministic, ands streams can be split and rejoined, because record
>   movement is generally defined.)
>   - When processing text strings or files, having to scan for the string
>   terminator or CR/LF, results in performance-killing working-set/cache
>   flooding.  (Pipelines processes records not characters, and unneeded
>   pages/cache-lines never have to be staged in.)
>
>The *nix string and file models were great for slow PDP-11s and the like,
>but make no sense on modern hardware.

This reflects a lack of understanding of Unix programming.

There are no "separate techniques for processing text ... and binary data", any 
more so than there are on MVS. A Unix file is a stream of bytes, period.

Many programs interpret that stream as a sequence of textual lines, delimited 
by newlines; this uniform representation makes possible the rich array of 
text-processing tools available in Unix, and enables files written on one 
platform to be usefully consumed on another platform. The notion that Unix 
pipelines are "crippled" by their text orientation reflects a lack of 
familiarity with Unix.

If you WANT to process binary data, it's easy to write programs that treat 
files as containing a sequence of records, either fixed length (think FB) or 
variable (think VB) with explicit record lengths.

Every program, on any platform, places restrictions on its inputs and outputs. 
z/OS programs are no different. In fact, the standard z/OS access methods very 
tightly couple programs to very specific dataset organizations and record 
formats.

Unix System Services has brought all of the powerful text processsing tools of 
Unix to z/OS. I can't see how that can be considered a bad thing.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: MF Application source control?

[Jerry Callen]

On Fri, 4 May 2018 13:00:55 +, Rob Scott  wrote:

 Dave
 
 As someone who is a complete dinosaur and used to source control
 systems like SCLM, it took a bit of getting used to for me.
 
 I found that once I abandoned PDS datasets and used z/OS unix
 file systems on z/OS to store the source I was working on, all
 things began to fall into place.
 
 For build processes that need PDS input/output, OPUTX/OGETX are
 your friends.
 
 Rob.

[Disclaimer: I work at Rocket Software, and have interacted
 a fair amount with Rob.]

Rob, if you're a dinosaur, the species must be "velociraptor". :-)

I'm in the Open Source Tools group at Rocket; our goal is to make
the z/OS Unix environment as tool-rich and developer-friendly as
other Unix platforms.

We (the Open Source Tools group) store all of our code in git, using
the Rocket z/OS git port on one side and either BitBucket (internal)
or GitHub (external) on the other. The .gitattributes support we
added to git helps ease the EBCDIC/ASCII impedance mismatch. As Rob
knows, there are still some twitchy spots (ISPF panels can present
problems).

I do nearly all of my work in emacs (shell buffers are a wonderful
thing...), with very ocassional visits to TSO/ISPF and batchland (I
*really* want to be able to run SMP from a shell script; I'll get
there eventually...). While I understand the appeal of tools like
IDz, I personally prefer the somewhat looser integration of emacs,
and use a fair number of browser-based tools for collaboration
(Jira, the Confluence wiki, BitBucket/GitHub). I guess that makes
me the Unix equivalent of a dinosaur...

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: IRS - 60-Year-Old IT System Failed on Tax Day Due to New Hardware (nextgov.com)

[Jerry Callen]

On Sun, 22 Apr 2018 18:29:38 -0400, Hobart Spitz  wrote:

>That's said, I can answer the question, for C/C++, as follows.  (I pose the
>question for Java, below.)
>
>*With the *nix/C record and string models, there are these issues:*
>
>   1. Errant/unexpected/unintended pieces of binary data in a text
>   file/string can break something.
>   2. Separate functions/methods/techniques must be used to manipulate text
>   files/strings versus binary files/string. You *must* know what you are
>   dealing with up front, and/or somehow code logic for both. (I'm not sure
>   the latter is possible in the general case.)
>   3. Even with *nix/C oriented machine instructions, the need to inspect
>   all characters up to a target point results in performance killing cache
>   flooding.
>   4. C++ does garbage collection resulting in "pauses" in forward
>   progress, and working set, caching, and CPU spikes, among other things.
>
>Let's call these attributes fragility, productivity, and efficiency,
>respectively, for the convenience convenience.  C has issue with these
>characteristics.

C and C++ do NOT have garbage collection; memory management is controlled by 
the programmer, with C++ providing a lot of assistance in the form of 
constructors and destructors.

String manipulation can be done in a variety of ways; C++, in particular, 
provides counted strings and byte arrays with no restrictions on content.

I've programmed in assembler, PL/I, COBOL and C/C++. If you are dealing with 
COBOL records that make extensive use of OCCURS DEPENDING ON, it probably make 
sense to stick with COBOL. Otherwise, it's quite easy to write robust, 
efficient code in C and C++ on z/OS.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Any C++ regex template class gotchas?

[Jerry Callen]

On Mon, 9 Apr 2018 10:28:34 -0400, Charles Mills  wrote:

>To clarify, by "works" you mean does NOT ABEND?

Oops, yes, I meant "does not ABEND".

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Any C++ regex template class gotchas?

[Jerry Callen]

Works for me, using exactly the code posted, so it looks like something 
peculiar to your maintenance level. I'd open a PMR.

make -k CHARSET=ebcdic
xlC -Wc,phaseid -qebcdic -q64 -qnocse -qgonum -qbitfield=signed 
-qtarget=zosv2r2 -qarch=10 -qstrict \
 -qfloat=ieee:nomaf -qlanglvl=extended0x -O2  -DNDEBUG -D__MV17195__ 
-D_XOPEN_SOURCE=600  -o test1 
test1.cpp
CCN(I) Product(5650-ZOS) Phase(CCNEOPTP) Level(D171026.Z2R2)
CCN(I) Product(5650-ZOS) Phase(CCNDRVR ) Level(D171026.Z2R2)
CCN(I) Product(5650-ZOS) Phase(CCNEPP  ) Level(D171026.Z2R2)
CCN(I) Product(5650-ZOS) Phase(CCNETBY ) Level(D171026.Z2R2)
CCN(I) Product(5650-ZOS) Phase(CCNECWI ) Level(D171026.Z2R2)
./test1
RegEx test 4/6/2018


FWIW, it's a good idea to post the compile and link lines along with the source 
code, as they can dramatically affect the outcome. Good luck!

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Waiting for IBM to step away from Laura Ingraham sponsorship

[Jerry Callen]

> Now that you have established you disapprove of the first and second 
> amendments, 
> which others do you have problems with? Why would you allow a high school 
> kids with
> a Nazi salute and Nazi style armbands make policy?

Well, that's about the shortest proof of Godwin's Law that I've ever seen...

We now return you to our regularly scheduled technical discussions, already in 
progress.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: GitHub setup for Z/os

[Jerry Callen]

There are detailed instructions on using the z/OS port of git with GitHub here:

https://forum.rocketsoftware.com/t/using-git-for-z-os-with-github/654


-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: z/OS "interactive computing" - AKA TSO/ISPF or UNIX shell

[Jerry Callen]

> If you want a break with 3270, why a Rube Goldberg with line mode. Make the 
> application an X11 client instead.
> 
> We'd all be better of if SUN's paradigm of the thin client (X11 server) had 
> caught on.

Um, well, maybe. 

You CAN run X11 clients on z/OS. But my personal experience is that X11 has a 
lot more network traffic than just a plain old full-duplex, character mode 
application (using curses or not). I use emacs on z/OS (and, usually, even on 
other platforms) in character mode, not as an X11 client. It's just more 
responsive, and adding X11 to emacs doesn't really buy you much.

Other posters in this thread have mentioned running emacs off-platform (there's 
a nice windows port) and then using things like ange-ftp and  remote shell 
sessions (via ssh) to access z/OS. I used to do that for remote access to Unix 
systems, and it's actually a very nice model. All the GUI interaction happens 
locally (making it fast), and you're not dependent on the level of emacs that 
happens to be installed on the remote server. 

But I've never been able to get that working with z/OS. (I also haven't tried 
all that hard). Maybe I should. BUt I'm going tohave to learn a lot more elisp 
to do that

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Can the CSI info alone produce a final list required for full resolution of researched PTFs?

[Jerry Callen]

>> Normally, the process of resolving all REQ/PREREQ/IFREQ requirements for
>> given PTF(s) and collecting all missing items that prevent resolution, up to 
>> the point where APPLY CHECK suggests that actual APPLY could be successful
>> could require several iterations.

> I have used GROUPEXTEND since being introduced and can't recall ever being
> successful in attaining full resolution with one pull.
> 
> It always took several pulls (iterations) to attain a full resolution.

It sounds to me as if the problem is that the SYSMODs identified by
GROUPEXTEND can have their own REQs, which are (of course) not in the
CSI until they get RECEIVEd.

Is anyone aware of an IBM-provided REST API to the entire SYSMOD
database? If you had that, you could presumably use that in
conjunction with the SMP "GIMAPI" to construct the full graph.

You could probably html-scrape content from Service Link, but that
seems hopelessly primitive.

I smell a nice science experiment brewing...

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: z/OS "interactive computing" - AKA TSO/ISPF or UNIX shell

[Jerry Callen]

I'm going to be an EXTREME outlier here.

Background: I learned computing on OS/360 thru MVS, first using cards, then 
TSO/ISPF. I jumped ship to Unix in the mid 80s and now I'm back on the 
mainframe, doing ports of open source software to z/OS (under USS) at Rocket 
Software.

I am logged into both USS (via ssh from PuTTY) and TSO/ISPF (via BlueZone) from 
a Windows laptop all day long. If I had a decent tool for accessing JES 
(there's no avoiding SDSF for the time being) from USS, I'd NEVER be in TSO.

I use emacs as my development environment. I don't call it an "editor" because 
it does so much more than edit text. In particular, the "shell buffer" feature 
is indispensible; think of TSO session manager, but on insane steroids. The USS 
port of emacs is ancient and creaky (though I dearly hope we can remedy that 
within the next year), and I will grant that emacs has a very stiff learning 
curve, but once you know it, it's unbelievably productive.

For source control, I use the Rocket port of git. Essentially all of our 
mainframe development is moving from other source control systems (SCLM, cvs, 
svn) to git; there are good open source tools for converting from cvs and svn 
that preserve all the history and branches.

For builds, I use whatever the open source project I'm currently working on 
uses, which is generally some variation on automake/autoconf/configure/make. 
The automake/autoconf situation on z/OS isn't yet what it wants to be. For my 
own projects, I just use raw make. I often create make files that work on both 
USS and Linux on Z (my go-to Unix when I need to use a tool not yet on USS).

In short: I treat z/OS as a Unix box. Nearly all of the compilers (COBOL, PL/I, 
C/C++, plus the assembler and binder) can be used from USS, on Unix files (no 
need to move source, maclibs, include files, etc. into a PDS). IBM has provided 
very good, albeit complex and tricky to use well, ASCII/EBCDIC "bimodal" 
encoding support to ease the encoding problem. IBM is actively porting newer 
languages (like JavaScript in node.js) to z/OS.

I can run TSO commands from the shell prompt (using, of course, the "tsocmd" 
command...) when I need to. I keep building tools to help insulate me from TSO 
and batch (like my SMP query interface at https://github.com/zorts/smpapi), and 
of course Rocket continues to release new and updated tools for free (though 
our bandwidth is limited...). The big remaining hole is JES queue access. I 
can, of course, submit jobs from USS, but getting the output in a nice, 
consumable manner remains a challenge; hence, my TSO session.

We have a cadre of younger developers who follow a similar path, though often 
using vim instead of emacs, and im some cases Windows-based editors (Eclipse, 
Webstorm, SlickEdit, etc.) and FTP.

Bear in mind that my first "real" editor was ISPF, which I used for years. Even 
with that history, I can't imagine using it for any serious editing at this 
point.

Slight diversion: Linux on Z is a VERY nice platform. I have rarely encountered 
any problems porting x86 Unix code to Linux on Z, and usually I don't have to; 
it's already a real, well-equipped Unix. Given hipersocket connectivity to 
z/OS, I think it's got potential to be a terrific alternative to USS. However, 
it's still just too weird for many shops: it requires a completely new set of 
system administration skills, its own LPAR or VM, and it just doesn't seem to 
getting much traction.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Interesting - NODE for z/OS

[Jerry Callen]

You don't have to build from source; you can get a downloadable SDK:

https://www.ibm.com/us-en/marketplace/sdk-nodejs-compiler-zos

There is a DeveloperWorks forum for node.js on IBM platforms (including z/OS 
and Linux on Z):
 
https://www.ibm.com/developerworks/community/groups/service/html/communitystart?communityUuid=39f4d6be-d936-4a36-8fbf-874bf4bde113

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Dr. John Ehrman

[Jerry Callen]

I knew John from my involvement with SHARE assembler committee, and through 
work with IBM's Federal Systems Division in the 1980s. He was a true gentleman, 
always kind, very smart, and an absolute pleasure to work with. I was delighted 
to have been able to chat with him (for the first time in years) at SHARE in 
Providence last year. He seemed very at peace.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Small SMP utility for USS (was "Include FMID, PTFID and APARID as eye catcher in each LMOD")

[Jerry Callen]

I've written a little Unix command that uses the SMP/E query API to dig 
information out of the CSI. The output is not especially human-friendly; it's 
intended for processing by scripts. You can get it from github: 
https://github.com/zorts/smpapi

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: git, z/OS and COBOL

[Jerry Callen]

Regarding line numbering:  git has the ability to run complementary "clean" and 
"smudge" filters to be used when moving source between the repo and the working 
directory; these might prove useful. See 
https://git-scm.com/book/en/v2/Customizing-Git-Git-Attributes for more 
information.

That said - I agree that if you CAN move away from line numbering, you SHOULD.

(Shameless plug...) BTW, if anyone has git questions specific to the z/OS port 
of git from Rocket, please ask them on Rocket's open source forum at 
https://forum.rocketsoftware.com/c/rocket-z-os-open-source-languages-tools. I 
(and other Rocket employees) monitor it regularly. 

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Passings

[Jerry Callen]

(A day late & a dollar short; I just noticed this.)

Dave (AKA "dammital" or "DBA") and I go WAY back; I met him as a freshman at 
Florida Technological University in 1972. He took me under his wing and 
mentored me, introducing me to APL (on an IBM 1130), formal language 
specification, compiler theory and so much more.  Dave got me my first 
post-college job (at Orange County Public Schools), where we were both system 
programmers (VS1 on a 370/145, then a 158). 

When I left central Florida in the late 70s Dave and I remained friends, 
corresponding first by good old snail mail and then via early email systems 
(notably BITNET). We would reconnect in person at SHARE and on my visits to 
Florida, where my family remained. We would go out out riding on Segways (he 
owned two), on walks around Winter Park and the massively expanded FTU campus 
(now University of Central Florida). For a while we shared a colo BSD server 
for our personal domains; Dave did most of the system admin stuff.

Dave was a licensed pilot, active in community theater (primarily as a stage 
tech), devoted husband and father of two delightful children.

I've never met a finer human being, and I miss him terribly.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

USS PFS Interface - 136 byte save area?

[Jerry Callen]

I'm writing a Physical File System (PFS) for USS, and I've come across an 
oddity in the interface documentation. The "Environment for PFS Operations" 
section of z/OS UNIX System Services File System Interface Reference says that, 
on entry to VFS function routines, R13 is "Save area address, of a 136-byte 
save area". I don't think there is such a beast. I'm guessing it means that I 
should treat this as a 144-byte F4SA, and not chain forward (using the "next" 
slot at +136 of the old save area). 

Has anyone else used this interface, and if so, can you confirm that this is 
how it's supposed to work? This is supervisor-state, key-zero code, some I'm 
eager to get it right the first time. :-) Thanks!

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: git on z/OS

[Jerry Callen]

On Wed, 10 Aug 2016 10:26:55 -0500, Paul Gilmartin  wrote 
(regarding porting git to z/OS):

>What would be the obstacles to building it in Enhanced ASCII mode?

For a tool like git that has to straddle character encoding worlds, enhanced 
ASCII support (EAS) brings only limited value. We're likely to explicitly 
convert data, and control tagging, for text moving into and out of the 
repository, rather than relying on EAS to Do The Right Thing. Current plans 
(subject to change, of course...) are to support checkout as ISO-8859-1, UTF-8 
or IBM-1047 (or conceivably ANY encoding supported by iconv) and respect tags 
on add/commit of new files.  As I said - it's a challenging problem. Git 
currently ignores encoding altogether, with the exception of line endings (the 
perennial CRLF/LF/CR/whatever problem...).

Given the complexity of the problem, I think IBM has done a fairly decent job 
with EAS. Most of the existing C/C++ tool chain (most notably, of course, xlc) 
do a pretty decent job of handling a mix of character encodings in its input 
files, and the bulk of the POSIX system and library interfaces support ASCII. 
See 
http://www.ibm.com/support/knowledgecenter/SSLTBW_2.2.0/com.ibm.zos.v2r2.bpxbd00/libfuncsupport.htm

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

git on z/OS

[Jerry Callen]

[Prompted by mentions of git on another thread]

Rocket is in the process of porting git to z/OS (on USS, of course). As others 
have noted, it's challenging, given the ASCII/EBCDIC issues. Nonetheless, we 
hope to have a public beta available by the end of the year; it will be 
announced on Rocket's open source forum at:

http://forum.rocketsoftware.com/c/rocket-z-os-open-source-languages-tools

At least initially (and, I suspect, forever...), it will only support USS 
files, not MVS datasets (that's a whole nother kettle 'o fish...).

Meanwhile: I use git with z/OS using the "large sledgehammer" approach. I keep 
my actual git repositories on Windows and use WinSCP to periodically copy the 
source back and forth between Windows and z/OS. Crude, but functional. Once on 
Windows, I can push/pull github repos. Look here for a repos containing z/OS 
source: 

https://github.com/zorts/pause_release

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Real purpose of hipersocket

[Jerry Callen]

On Wed, 6 Jul 2016 09:48:37 -0500, Parwez Hamid  
wrote:


>The other point to note about OSes when comparing SMC-D with HiperSockets, 
>SMC-D is only supported by z/OS whereas HiperSockets is supported by all OSes 
>that run on z Systems.

Right. :-( 

I am eagerly awaiting Linux on Z support of SMC-D. Since it already supports 
SMC-R, this would seem (to someone who doesn't have to implement it...) to be 
straightforward. 

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Real purpose of hipersocket

[Jerry Callen]

Z/OS 2.2 also provides SMC-D, which I am eager to get a chance to play with. 
All the convenience and familiarity of sockets, but with most of the TCP/IP 
overhead stripped out (since the underlying transport layer is reliable). It 
should have significantly better performance and less CPU consumption than 
hipersockets.  Information here: 
https://www-01.ibm.com/software/network/commserver/SMC-D/index.html

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: SUSPEND/RESUME is slower than WAIT/POST. PAUSE/RELEASE is slower than both

[Jerry Callen]

In the interest of full disclosure, the code is now available on github: 
https://github.com/zorts/pause_release

As noted way back in my original posting, the code is unauthorized, uses only 
TCBs, and ping-pongs between two tasks. I'm running this code on a moderately 
loaded z13, native z/OS 2.2 LPAR with many CPUs and zIIPs (though, since it's 
unauthorized, this code is only using CPUs).

I believe the ping-pong nature of the test explains why the impact of the 
"fast" wait/post code has relatively little impact on the measured performance; 
the inner loop is so small that an actual WAIT or POST is required for every 
iteration.

This is by no means "real world" code, and I would never call it a "benchmark". 
However, I do believe it correctly illustrates the difference in performance 
between various synchronization mechanisms in its admittedly limited set of 
test cases. 

It's great that this has stimulated the discussion it has; I've learned a lot.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: SUSPEND/RESUME is slower than WAIT/POST. PAUSE/RELEASE is slower than both.

[Jerry Callen]

Edward Jaffe wrote:

> Stated as simply as I can: If the POST bit is already on, why call the 
> WAIT service in the first place? All it's gonna do is immediately return.

Yes, that's clear. What always gets my knickers twisted is when you can RESET 
the ECB and be sure you don't lose an event.

I think I've convinced myself that, in this simple ping-pong case, just testing 
the post bit will work. Maybe I'll run THAT test, too. 

This is why I prefer data parallel programming to thread parallel programming; 
it makes my head hurt less.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: SUSPEND/RESUME is slower than WAIT/POST. PAUSE/RELEASE is slower than both.

[Jerry Callen]

Edward Jaffe wrote:

> And, in the absence of contention, WAIT/POST in professionally-developed 
> software becomes orders of magnitude faster since everyone knows to skip 
> the WAIT if the post bit is on and to attempt a "quick" POST of the ECB 
> (using CS) if the wait bit is off.

I understand the "quick POST" trick w/CS, but bypassing WAIT? I can see how 
that would work for something like waiting on I/O completion, but for trading 
control back and forth, I think it requires (for example) an auxiliary counter 
(and CDS to update both the ECB and the counter simultaneously). Am I missing 
something?

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: SUSPEND/RESUME is slower than WAIT/POST. PAUSE/RELEASE is slower than both.

[Jerry Callen]

...and here we go. The results differ somewhat from the previously posted 
results; I just ran these, and the machine is somewhat quieter than before.

Clearly the "fast POST" trick is worthwhile - though it didn't make as much 
difference as I expected.

Relative performance: total CPU
ECB   1.00
ECB No Fast Post  1.59
Pause/Resume  3.93
Transfer and Pause2.86
Transfer with separate Pause  4.05

Relative performance: elapsed
ECB   1.00
ECB No Fast Post  1.60
Pause/Resume  5.24
Transfer and Pause3.91
Transfer with separate Pause  5.35

--  Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: SUSPEND/RESUME is slower than WAIT/POST. PAUSE/RELEASE is slower than both.

[Jerry Callen]

Peter Relson wrote:

>>I agree the reference materials _imply_ that WAIT/POST are deprecated 
>>and that Pause/Release/Transfer should be used when possible. 
>If you have a concrete reference, please provide it. WAIT/POST is in no 
>way deprecated.

I wouldn't call it "deprecated", but this does suggest that the "transfer" 
feature of pause elements avoids dispatching overhead and therefore *MIGHT* be 
faster:

https://www.ibm.com/support/knowledgecenter/SSLTBW_2.1.0/com.ibm.zos.v2r1.ieaa800/pausrel.htm

"The Transfer service can both release a paused task or SRB and pass control 
directly to the released task or SRB. The Transfer service can also pause the 
task or SRB that calls the service. Thus, Transfer enables quick dispatches, 
saving the overhead of work search. It also allows two dispatchable units to 
trade control back and forth with minimal overhead."

That's actually the main reason I ran my test; I wanted to see if this really 
made a difference (in the context of my admittedly unrealistic, TCB-only test). 
That's why there are 4 test scenarios: wait/post, pause/release, transfer, and 
transfer/pause.

(And, as I've mentioned in another response, there should be a 5th: wait/post 
WITHOUT the "fast post" trick.)

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: SUSPEND/RESUME is slower than WAIT/POST. PAUSE/RELEASE is slower than both.

[Jerry Callen]

> And, in the absence of contention, WAIT/POST in professionally-developed 
> software becomes orders of magnitude faster since everyone knows to skip the 
> WAIT if the post bit is on and to attempt a "quick" POST of the ECB (using 
> CS) if the wait bit is off.



And my timing code did just that; it's such an inherent piece of the idiom that 
it didn't occur to me to disable the "quick POST" for this test.

So my timings are, arguably, completely wrong.

If I get a chance, I'll re-test.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: SUSPEND/RESUME is slower than WAIT/POST. PAUSE/RELEASE is slower than both.

[Jerry Callen]

Greg Dyck wrote:

> No one every said that Pause/Release was faster or consumed the same 
> amount of CPU as WAIT/POST, or that they should be used as a universal 
> replacement for WAIT/POST.  They should be used where they provide value.

The reason I wrote the test is that the documentation is unclear as to 
performance, and suggests that TRANSFER, in particular, might be faster than 
WAIT/POST. From the 2.2 Assembler Services Guide:

===
If you have, for example, an application that requires two or more tasks to 
trade control back and forth, these services provide efficient transfers of 
control.
[...]
The Transfer service can both release a paused task and pass control directly 
to the released task. The Transfer service can also pause the task that calls 
the service. Thus, Transfer enables quick dispatches, saving the overhead of 
work search. It also allows two tasks to trade control back and forth with 
minimal overhead.
===

Aside from earlier comments here in IBM-MAIN, I had no particular 
preconceptions regarding the performance, which is why I wrote the tests. Both 
services are easy to use, so there was no compelling advantage to either on 
that score.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: SUSPEND/RESUME is slower than WAIT/POST. PAUSE/RELEASE is slower than both.

[Jerry Callen]

(A very delayed follow-up on a thread from yesteryear...)

tl;dr: In unauthorized code, ECBs are much faster than pause elements.

I wrote a simple test program to compare the performance of WAIT/POST and pause 
elements. The program has two tasks and simply ping-pongs back and forth 
between them (no overlapped execution). Each task has a synchronization gadget, 
either an ECB or a pause element. The tasks use each other's synchronization 
gadget to just transfer control back and forth between each other in a loop. 
This is a pretty unrealistic test (in terms of being like anything "real" code 
would do), but it does illustrate the relative performance of the two 
synchronization methods.

Note that this unauthorized code, and tasks, not SRBs This means that all of 
the synchronization primitives used SVCs, not branch entry points. The code is 
64-bit C++ compiled with xlC using -q64 and thread_create() to create the 
tasks. The code is running on a z13, native LPAR, z/OS 2.2.

The mechanisms I tested are:

* ECBs: each task just alternates between a WAIT on its own ECB and a POST of 
the other task's ECB.
* Pause elements with Pause (iea4pse) and Release (iea4rls) taking the role of 
WAIT and POST.
* Pause elements using Transfer (iea4xfr) in "transfer and pause" mode (one 
call to do both).
* Pause elements using Transfer (iea4xfr) in "just transfer" mode followed by 
Pause (iea4pse).

Here are the results:

Relative performance: total CPU
ECB   1.0
Pause/Resume  4.30
Transfer and Pause3.25
Transfer with separate Pause  4.55

Relative performance: elapsed
ECB   1.0
Pause/Resume  6.06
Transfer and Pause4.75
Transfer with separate Pause  6.28

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

How well does z/OS handle large, but sparse, memory objects?

[Jerry Callen]

The 64-bit address space available to memory objects opens up a whole range of 
algorithms that use lots of VIRTUAL memory, but relatively modest amounts of 
REAL memory. The data is "clumped" into a relatively small number of pages; 
it's not like sprinkling a few bytes all over the object, which would drive the 
real storage utilization through the roof.  I've used such algorithms 
successfully on both Windows and Linux, and I'm wondering if there are issues 
peculiar to z/OS that would make this approach infeasible.

As an example: Does z/OS require that there be sufficient page space available 
to back all of the space requested for a large memory object? I might have an 
object whose virtual size is several terabytes, but requires only a few 
gigabytes of real storage. 

With the amount of memory available on machines like the z/13 this is a 
perfectly sensible way to use memory, but if z/OS insists on having enough page 
space available for the whole object, this is not going to work.

I've done some testing on both z/OS and Linux on Z, with good results, but 
that's on LPARs specifically designated for experimentation. What might I run 
into on a production system?

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

What was a 3314? (was: Whither VIO)

[Jerry Callen]

In the "Whither VIO" thread, J.O.Skip Robinson wrote:

>  In a previous life, we defined VIO (I believe) to device 3314 even though we 
> had none left on the floor

That's a device type I've never heard of, and the Google knows not of. Could 
this be a typo for "2314"?

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Whither VIO?

[Jerry Callen]

In a thread in a distant galaxy, J.O.Skip Robinson wrote:

> ...be aware that some customers (like us) did away with VIO a long time ago.

I've been away from MVS for a while. Did something better than VIO come along 
while I wasn't looking? Why would VIO have been vaporized?

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How expensive is AMODE switching?

[Jerry Callen]

Well, DUH! I'll look at the code and see if that's feasible.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How expensive is AMODE switching?

[Jerry Callen]

There is a short segment of 31-bit code that is generated on the fly by another 
piece of software that I need to call. I don't control that piece of software, 
and I'm putting the results of that computation into an above-the-bar chunk of 
memory for later processing. No I/O. 

Since it sounds as if the switching is fairly expensive, I'll see if there is 
some reasonable way to batch the calls, rather than making the 64->31->64 
switch for every item.

Thanks to all for the replies!

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

How expensive is AMODE switching?

[Jerry Callen]

Before I sit down and write a test program to time it, can anyone tell me how 
expensive the AMODE switching instructions are (SAM31/SAM64)? I have an 
AMODE-64 function that needs to execute a short segment of AMODE-31 code in a 
loop (I know this sounds wacky, but there is actually a reason why I just can't 
change the AMODE-31 code), and I'm wondering how painful the mode switching 
will be.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Callable service for DD check?

[Jerry Callen]

This is probably overkill, but a DYNALLOC "info" call can also do it, and can 
easily provide other useful information about the dataset at the same time (if 
you need to know more than just whether or not the DD is present).

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Is there a source for detailed, instruction-level performance info?

[Jerry Callen]

Jack J. Woehr wrote:

> Not sure how relevant that this is to mainframe programming, but years ago
> when I designed and executed with a team of nine a data-heavy server in
> Unix optimized for multiple cores, what we found was that reroutable queuing
> of data from one simplistic processing engine to the next (with reservoirs
> for data accumulation) got the most performance.

I'm not sure I grok this. Are you talking about hardware or software (or a 
combination)? I *think* what you are describing is something akin to what 
happens in a Unix pipeline or message queueing systems. Can you provide a 
reference? It sounds interesting, and similar to other "flow-based" programming 
systems (Volcano, IBM's DataStage EE, Expressor Software's parallel engine, 
various parallel database engines, etc.).

That said: in this case, the core algorithm is very small (under 100 
instructions), and already parallelized across multiple threads with limited 
interaction across threads (data parallel style). Pipeline parallelism isn't 
appropriate WITHIN this core, though the core could sensibly be used as a 
component in a larger pipeline-parallel job. I'm really looking to make this 
core as fast as possible.

The effect of other work in the system has been mentioned; I get that. The 
impact of the "hinting" instructions IBM has provided (PFD, BPP, NIAI) will 
obviously be affected by context switches. But IBM presumably provided them for 
a reason.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Is there a source for detailed, instruction-level performance info?

[Jerry Callen]

Charles Mills wrote:

> I would assume there is some sort of a compiler/hardware architecture
> liaison group within IBM. I would bet that if someone from that group were
> to put together a SHARE presentation called "Write Machine Code Like a
> Compiler -- How to Write the Fastest Code Possible for the z13 (z14,
> whatever)" that it would be a big hit.

I second that!

Bob Rogers (no longer at IBM...) ran a series of "How Do You Do What You Do 
When You're a (z10/z196/z13) CPU?" sessions over the years. But they were more 
focused on overall characteristics of the system, not on specifically how an 
assembly language programmer can best exploit the hardware. A session that 
presumes that background knowledge and really gets down into the weeds would be 
great. And maybe follow it up with a BOF for assembly weenies to address 
specific questions.

How about it, IBM? Surely there must be someone Poughkeepsie who wants to visit 
San Antonio in March? :-)

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Is there a source for detailed, instruction-level performance info?

[Jerry Callen]

Back from Christmas break to a lot of responses - thanks to all. Summarizing 
the responses thus far:

* Select and/or tune your algorithm first.

Check.

* Tuning for a specific machine is a bad idea because you'll have to retune for 
every new machine.

Check. This code is sufficiently critical that it's worth doing 
platform-specific tuning. Otherwise I wouldn't be coding in assembler anyway. 
And it's small enough to be manageable.

* The compilers are very good and can probably do better than you can.

Usually, I agree. On this small kernel, though, I'm already beating xlc and (by 
a lesser margin) gcc. Looking at the code they generate has been enlightening; 
gcc is especially aggressive on loop unrolling, and that's on my list of  
things to try.

* Modern machines are too complex to enable detailed timing formulae to be 
published.

I'm not really after detailed timing. I'm looking for implementation details of 
the same sort used by compiler writers to guide selection of instruction 
sequences, where the guiding principle is, "How can I avoid pipeline stalls?" 
As I noted, several SHARE presentations contain SOME of this information, which 
I've already benefited from, but I'm looking for more.

* Just write some timing loops and figure it out yourself.

I've been doing that, using mostly the algorithm itself plus small timing 
experiments (which can be deceiving since they aren't in the context of the 
rest of the algorithm). I've already managed to squeeze out about a 15% 
improvement over my initial code.

An example: which of these code sequences do you suppose runs faster?

 la rX,0(rI,rBase)   rX -> array[i]
 lg rY,0(,rX)rY = array[i]
 agsi 0(rX),1++array[i]
* Now do something with rY

vs:
 lg rY,0(rI,rBase)   rY = array[i]
 la rX,1(,rY)rX = rY + 1
 stg rX,0(rI,rBase)  effect is ++array[i]
* Now do something with rY

The first is substantially faster. I would have GUESSED that the second would 
be faster, since I need the value in rY anyway. (I'm in 64-bit mode, so using 
"LOAD ADDRESS for the increment is safe...)

* Suggestions regarding branch prediction.

Spot on, and I already did that, too. Luckily branch prediction is one of the 
few performance characteristics that's actually semi-architectural (see the 
description of "BRANCH PREDICTION PRELOAD" in the Principles of Operation). 
Consider these two code sequences:

loop ds 0h
*Compute a value in r1
 cgrjne r2,r1,loop   continue if not done
vs:

loop ds 0h
*Compute a value in r1
 cgrje r2,r1,loopexitexit if done
 j loop
loopexit ds 0h

The second sequence is MUCH faster - though I haven't yet tried using "BRANCH 
PREDICTION PRELOAD" to alter the default prediction.

* Cache issues

This kernel unfortunately has little (data) cache locality; it's inherent in 
the problem. What I HAVE found is that using "PREFETCH DATA" and specifying 
store intent before the first access DOES help a bit. I haven't fiddled with 
"NEXT INSTRUCTION ACCESS INTENT" yet; that seems like a potentially enormous 
rathole. Using MVCL isn't an option; my data elements are small, aligned 
multiples of doublewords, and I determined very early on that a load/store loop 
way outperforms both MVCL and MVC.

On the I-cache side: the kernel is very small (a few dozen instructions) and 
chugs along for a good long time once it's called, so it's undoubtedly in 
cache. Following David Bond's suggestion to align the tops of loops on 16-byte 
boundaries helped a bit.

* The usual IBM-MAIN drift, reminiscences of old machines/code/operating 
systems/management practices/etc.

Check. :-)

I'm still hoping that someone will reveal the existence of a document intended 
for compiler writers...

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Is there a source for detailed, instruction-level performance info?

[Jerry Callen]

I'm in the process of hand-tuning a small, performance critical algorithm on a 
Z13, and I'm hampered by the lack of detailed information on the 
instruction-level performance of the machine. Back in the day, IBM used to 
publish a "Functional Characteristics" manual for each CPU model that provided 
this information; those seem to have been discontinued. I'm looking for things 
like:

* Is register renaming used for GPRs and/or FPRs? (affects the need for loop 
unrolling)
* Are there any ways to bypass the L1 cache on moves of less than a page, when 
simply moving data without looking at it?
* Does LMG/STMG outperform a linear sequence of LG/STG? Under what 
circumstances?

At the very least, compiler maintainers need this information to select the 
right instruction sequences for each model. Does anyone know of a source for 
this information, other than writing test kernels and trying things out? I 
already have the SHARE presentations by David Bond and Bob Rogers, which 
contain some of this information, but it's not enough.

I was really excited by the addition of vector registers on the Z13 (yippee, an 
additional 512 bytes of high speed scratch storage!), but the load/store 
performance hasn't turned out to be what I had hoped for. I may well be using 
the facility "the wrong way"; having detailed implementation information would 
sure help.


-- Jerry Callen
   Rocket Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Is there a source for detailed, instruction-level performance info?

[Jerry Callen]

BTW - I applaud IBM's provision of the non-privileged ECAG instruction for 
obtaining cache characteristics. Here's the output of a little program I wrote 
to format the information it provides (on a Z13):

level 0: private
  data: line size=256, set associativity=8, total size=128K
  instruction: line size=256, set associativity=6, total size=96K

level 1: private
  data: line size=256, set associativity=8, total size=2048K
  instruction: line size=256, set associativity=8, total size=2048K

level 2: shared
  unified: line size=256, set associativity=16, total size=65536K

level 3: shared
  unified: line size=256, set associativity=30, total size=491520K

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: AW: PL/I and optional parameters

[Jerry Callen]

> I have a PL/I subroutine call that has 297 optional parameters.  That is the 
> function has a parameter list with up to 100 tuples of parameters.  The first 
> tuple is required, and tuple 2 through 100 are optional.

OK, I've been watching this thread with growing curiosity and I can't stand it 
any longer: What in the world are you doing with a subroutine with that many 
parameters? Is the code that calls this routine machine-generated? I can't 
imagine a human being coding such a call. 

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: C lang. embedded HLASM latest z/OS 2.1

[Jerry Callen]

> What I'm aiming at is to rewrite my CBT distributed UNIX HLASM programs as C 
> programs instead.
> Although I don't really know why I should. Just another of my oddities, I 
> guess. [...]  I doubt there
> are many z/OS people who like playing on the UNIX command line instead of in 
> ISPF.

Count me in! I use emacs in USS (yes, it exists for z/OS - see 
http://www-03.ibm.com/systems/z/os/zos/features/unix/bpxa1ty1.html), and I find 
that it enables me to be much more productive than I am with ISPF. I was fluent 
with ISPF back in the 80's, but many years of living in Unixland has perverted 
me. :-)

But you're right - we're probably a tiny minority. And I'm mostly working with 
data in the Unix file system, not z/OS datasets, which colors my perspective.

As to rewriting HLASM code in C - if you mostly just groveling through control 
blocks and using non-authorized services, why not? It'll make the code more 
accessible by others, in many cases it's just as efficient, and - it's fun!

-- Jerry Callen
   Rocket Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Setting up Sublime Text to edit PDS members

[Jerry Callen]

IBM Explorer for z/OS can edit pretty much any sequential file, either native 
z/OS or USS, but in the generic Eclipse editor. Eclipse is endlessly 
customizable, so there may well be language-sensitive editing modes floating 
around for it.

I've made (feeble) efforts to get emacs to do it, without success. My current 
lame hack is to suck the file into Eclipse, cut'n'paste into an emacs buffer 
edit, then cut'n'paset back and save. Blech.

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

3380 was actually FBA?

[Jerry Callen]

In another thread, l...@garlic.com wrote:

  ... but then if MVS had FBA support wouldn't have needed to do 3380 as CKD 
(even tho inherently it was FBA underneath) ...

I didn't know that.

Was that the first (and/or last?) IBM SLED to be inherently FBA under the hood? 
Where were the smarts for that implemented, in the control unit, or the drive 
itself?

-- Jerry

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN