Re: How can I tell the zFS files being used by my application

[Andrew Rowley]

On 20/05/2021 6:25 pm, Colin Paice wrote:

Are there any magic options to say "load a jar - just once ...  not once
for each thread", or to tell Java to load from the shared classes cache?
It feels like an old (non java) problem of having a non reentrant load
module, so it was loaded from disk every time, instead of reusing the
in-memory copy.



I doubt that Java is loading the classes multiple times. I think the 
class cache is more for sharing between different instances of the JVM.


I suspect that there is something else in the jar that is being accessed 
that is not cached. I see that jar contains a number of properties files 
e.g.

/com/ibm/eserver/zos/racf/nls/IRRExceptionMessages.properties
I wonder whether they could be what is being repeatedly loaded?

Andrew Rowley

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Colin Paice]

Timothy,
Thanks for these links.  Are these fixes available in the ACDC downloads?

I was told that shared classes should make startup faster. I've been
playing with it and have noticed a few interesting things.

   1. I think the default shared classes cache size is 60MB...  Ive made it
   100MB and it is 93% full ! it would be good if  appropriate size was given
   in z/OSMF
   2. When I start z/OSMF a second time, the SMF 92-11 records show over
   1000 jar files are being opened, and some are being opened more than once,
   for example /usr/include/java_classes/IRRRacf.jar is being opened 68 times!
  1. Shared classes are being used, as I can display the file
  information in the shared cache
  2. It looks like jar files are >not< being loaded from the shared
  classes data space and they are being loaded from the file system.

Are there any magic options to say "load a jar - just once ...  not once
for each thread", or to tell Java to load from the shared classes cache?
It feels like an old (non java) problem of having a non reentrant load
module, so it was loaded from disk every time, instead of reusing the
in-memory copy.

Colin


On Thu, 20 May 2021 at 06:00, Timothy Sipples  wrote:

> If you're experiencing long z/OS Management Facility startup times then
> please check that you have installed the PTFs for several APARs. You might
> consider disabling the z/OSMF help feature (if you can safely live without
> it) since it represents a substantial fraction of startup time/energy.
> Details are available here:
>
>
> https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/jing-hua-jiang1/2020/10/30/make-your-zosmf-server-starts-significantly-faster
>
> https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/zhi-li1/2020/08/10/tailor-your-zosmf-server-for-better-performance-in
>
> - - - - - - - - - -
> Timothy Sipples
> I.T. Architect Executive
> Digital Asset & Other Industry Solutions
> IBM Z & LinuxONE
> - - - - - - - - - -
> E-Mail: sipp...@sg.ibm.com
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Timothy Sipples]

If you're experiencing long z/OS Management Facility startup times then 
please check that you have installed the PTFs for several APARs. You might 
consider disabling the z/OSMF help feature (if you can safely live without 
it) since it represents a substantial fraction of startup time/energy. 
Details are available here:

https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/jing-hua-jiang1/2020/10/30/make-your-zosmf-server-starts-significantly-faster
https://community.ibm.com/community/user/ibmz-and-linuxone/blogs/zhi-li1/2020/08/10/tailor-your-zosmf-server-for-better-performance-in

- - - - - - - - - -
Timothy Sipples
I.T. Architect Executive
Digital Asset & Other Industry Solutions
IBM Z & LinuxONE
- - - - - - - - - -
E-Mail: sipp...@sg.ibm.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[David Crayford]

On 19/05/2021 3:46 pm, Andrew Rowley wrote:

On 19/05/2021 12:27 pm, David Crayford wrote:

Then why not write your product in C++? It would be a lot easier to 
process SMF records using structs then having to write get methods in 
Java. Contrary to what some people believe C++ is a very modern 
language with the benefit that it's not garbage collected. We chose 
Java because the value proposition of offloading to
zIIP processors is what customers want. Some of my ex-colleagues have 
spent the last two years zIIP enabling SMF reporting products that 
are written in PL/X. In the age of analytics where customers want 
near real time analytics feeds zIIP offloading isn't a nice to have 
any more.




I'm not arguing the benefits of zIIP, it's a valuable bonus, but 
running on zIIP is not the primary aim of the product.


I started in C++ on Windows, but found that C# is easier with no 
noticeable reduction in performance, then ported the mapping to Java 
so it can run on z and other platforms. The mapping of the records 
that structs would provide is a small part of the work, there is a lot 
of extra code making it easy to use.


The aim of the Java code is to provide an implementation that people 
can use for custom and ad-hoc SMF reporting. I think it is easier to 
program against a Java API than C++. You can also develop and test on 
Windows/Linux and run the code unchanged on z/OS.


OK, I get it now. Your product is Java library. Looks very nice BTW. And 
yes, the customer may not even have a C++ compiler (or skills) and Java 
is free.





You can see an example of the type of reporting that is possible here:
https://github.com/BlackHillSoftware/easysmf-samples/blob/master/reports/src/main/java/com/smfreports/type30/BeforeAfterProgramStatistics.java 



It shows how SMF record sections, date/time values, CPU time etc. is 
accessed. I think that sort of report would be much more complex to 
create in HLASM or Rexx, or even using C++ structs.


Sweet! It's uncanny that I have something very similar but with 
different names such as com.rocketsoft.smf.io.RecordReader, 
com.rocketsoft.smf.io.LogstreamReader, com.rocketsoft.smf.io.CidbReader 
etc. The first two just wrap Jzos classes.
If you come from a C# background you should check out Kotlin. Although 
your Java library would work well with Kotlin and I could swap the 
Optional's for elvis operators.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Andrew Rowley]

On 19/05/2021 12:27 pm, David Crayford wrote:

Then why not write your product in C++? It would be a lot easier to 
process SMF records using structs then having to write get methods in 
Java. Contrary to what some people believe C++ is a very modern 
language with the benefit that it's not garbage collected. We chose 
Java because the value proposition of offloading to
zIIP processors is what customers want. Some of my ex-colleagues have 
spent the last two years zIIP enabling SMF reporting products that are 
written in PL/X. In the age of analytics where customers want near 
real time analytics feeds zIIP offloading isn't a nice to have any more.




I'm not arguing the benefits of zIIP, it's a valuable bonus, but running 
on zIIP is not the primary aim of the product.


I started in C++ on Windows, but found that C# is easier with no 
noticeable reduction in performance, then ported the mapping to Java so 
it can run on z and other platforms. The mapping of the records that 
structs would provide is a small part of the work, there is a lot of 
extra code making it easy to use.


The aim of the Java code is to provide an implementation that people can 
use for custom and ad-hoc SMF reporting. I think it is easier to program 
against a Java API than C++. You can also develop and test on 
Windows/Linux and run the code unchanged on z/OS.


You can see an example of the type of reporting that is possible here:
https://github.com/BlackHillSoftware/easysmf-samples/blob/master/reports/src/main/java/com/smfreports/type30/BeforeAfterProgramStatistics.java

It shows how SMF record sections, date/time values, CPU time etc. is 
accessed. I think that sort of report would be much more complex to 
create in HLASM or Rexx, or even using C++ structs.


--
Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[David Crayford]

On 19/05/2021 8:14 am, Andrew Rowley wrote:


That's unfortunate. It's the same where I work. We run a enterprise 
class z15 and the zIIP normalized times in the SMF30s match GCP 
times. What you really want is an image where you can compare code 
running on a zIIP to a sub-capacity GCP
as that's the compelling reason why running your Java product is 
better than REXX or home rolled HLASM code.


I don't think that matters. I want to make it perform as fast as 
possible, and for that purpose it is helpful if the engines are the 
same speed as it's easy to compare. Beyond that, if you run it on a 
slower engine it will run slower, on a faster engine it will run 
faster. The system is never going to truly replicate a customer system 
anyway because one of the major performance factors is competition 
with other work running on the system, and I am never going to 
simulate a real production workload.


You can calculate zIIP normalization times using the RMF 79 records 
using the same formula as SDSF.




I don't see zIIP as a primary reason for SMF reporting in Java. It is 
a bonus. The compelling reason for Java is better performance than 
Rexx (for large quantities of data) and much easier and more powerful 
programming than both Rexx and HLASM. You can create reports in Java 
that are impracticable in Rexx or HLASM.


I was originally doing SMF reporting in C# on Windows, and the ease of 
processing data and writing reports made me decide it was worth 
porting to Java (a similar language) so it could be used on z/OS. 
Object oriented languages with classes representing record types are a 
VERY good fit for SMF data.


Then why not write your product in C++? It would be a lot easier to 
process SMF records using structs then having to write get methods in 
Java. Contrary to what some people believe C++ is a very modern language 
with the benefit that it's not garbage collected. We chose Java because 
the value proposition of offloading to
zIIP processors is what customers want. Some of my ex-colleagues have 
spent the last two years zIIP enabling SMF reporting products that are 
written in PL/X. In the age of analytics where customers want near real 
time analytics feeds zIIP offloading isn't a nice to have any more.




--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Andrew Rowley]

On 18/05/2021 9:21 pm, David Crayford wrote:

That's unfortunate. It's the same where I work. We run a enterprise 
class z15 and the zIIP normalized times in the SMF30s match GCP times. 
What you really want is an image where you can compare code running on 
a zIIP to a sub-capacity GCP
as that's the compelling reason why running your Java product is 
better than REXX or home rolled HLASM code.


I don't think that matters. I want to make it perform as fast as 
possible, and for that purpose it is helpful if the engines are the same 
speed as it's easy to compare. Beyond that, if you run it on a slower 
engine it will run slower, on a faster engine it will run faster. The 
system is never going to truly replicate a customer system anyway 
because one of the major performance factors is competition with other 
work running on the system, and I am never going to simulate a real 
production workload.


I don't see zIIP as a primary reason for SMF reporting in Java. It is a 
bonus. The compelling reason for Java is better performance than Rexx 
(for large quantities of data) and much easier and more powerful 
programming than both Rexx and HLASM. You can create reports in Java 
that are impracticable in Rexx or HLASM.


I was originally doing SMF reporting in C# on Windows, and the ease of 
processing data and writing reports made me decide it was worth porting 
to Java (a similar language) so it could be used on z/OS. Object 
oriented languages with classes representing record types are a VERY 
good fit for SMF data.


--
Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Sebastian Welton]

On Fri, 14 May 2021 13:03:17 -0400, Matt Hogstrom  wrote:

>It takes z/OSMF about an hour to fully initialize on my ZD … same behavior 
>(lots of SIOs and CPU)  IBM says its fixing this but its been quite a while.  
>Its Java related.
>
>Matt Hogstrom
>m...@hogstrom.org

Wow, that's a long time. I have it running on a laptop and only takes between 
10-12 minutes before I can get in and work with it. I used it to deploy zCX on  
the system with no problems and now have Zowe running (that also takes some 
time starting). There were serious problems with zOSMF starting on these 
systems but it has got better in fact it's even mentioned here:

https://www.ibm.com/docs/en/zos/2.4.0?topic=configurations-setting-kcindex-parameter

Sebastian

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Seymour J Metz]

Flagship? I thought that was Linux on Z.

Why doesn't z/OS have, e.g., ANSI REXX, NTP, OOREXX, various compilers 
ubiquitous in the *ix world? Why does STP cost extra?


--
Shmuel (Seymour J.) Metz
http://mason.gmu.edu/~smetz3


From: IBM Mainframe Discussion List [IBM-MAIN@LISTSERV.UA.EDU] on behalf of 
David Crayford [dcrayf...@gmail.com]
Sent: Tuesday, May 18, 2021 5:12 AM
To: IBM-MAIN@LISTSERV.UA.EDU
Subject: Re: How can I tell the zFS files being used by my application

On 18/05/2021 1:52 am, Kirk Wolf wrote:
>
> - good low-overhead (sampling) profiling tools for the z/OS JVM don't
> exist (maybe this has changed?).   IMO this was a huge barrier for
> Java on z/OS.

Unfortunately, othing has changed :( z/OS only partially supports the
JVMTI implementation. Hopefully, now IBM/OpenJ9 is open source the Java
tooling landscape will improve. RedHat supports Flight Record in their
OpenJDK distros
and IMO z/OS should at the very least have the same. I say again, Java
on z/OS needs a lot more love then it's currently getting. z/OS is
supposed to be the flagship IBM OS. We've started to introduce Kotlin to
modernize our tooling as IBM are too slow updating
Java versions on z/OS. So far it's been a success. Kotlin adds some
runtime overhead due to additional boilerplate code but it's well worth
it. Null safety, no checked exceptions and type safe builders alone are
a good enough reason to start transitioning, albeit slowly.


>
> - Sometimes CPU costs don't matter where time to market or
> functionality are more important or where the application doesn't run
> that much.

Agreed. And I suppose that was Andrew's point. If his Java product can
run quick enough as a batch utility then happy days. If it runs like a
stallion on a zIIP :two-thumbs-up:

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[David Crayford]

On 18/05/2021 6:35 pm, Andrew Rowley wrote:

On 18/05/2021 5:21 pm, David Crayford wrote:


Do they have a zIIP on the RDP system?

Yes, although the regular CPs are the same speed. Having the zIIP 
means I can get figures for zIIP, CP and zIIP on CP.


That's unfortunate. It's the same where I work. We run a enterprise 
class z15 and the zIIP normalized times in the SMF30s match GCP times. 
What you really want is an image where you can compare code running on a 
zIIP to a sub-capacity GCP
as that's the compelling reason why running your Java product is better 
than REXX or home rolled HLASM code. Most small/medium sized customers 
run sub-capacity so the GCPs are way below zIIP speeds. That's your 
market, right?




Are you talking about CMF 110 records? If so we've both worked on 
similar stuff. One of our products is a CICS Performance Analyzer. 
It's written in HLASM. A couple of years ago the CTO at the time was 
interested in using Java to process
SMF records and me to port the HLASM code to Java. When we ran it on 
our z13s the Java code smashed the HLASM. The reason was because the 
Jave was running on a full capacity zIIP and the HLASM code on a 
sub-capacity GP.


BTW, what kind of compression do you mean?


Yes, the CMF 110 records. I decompress the RLE compression in the 
record in Java. That means that the data can be processed on a PC if 
desired. (I even processed CICS SMF data on a Raspberry Pi to prove it 
could be done!)


I'm happy to use the standard operating system facilities to 
decompress data from the SMF logstreams. Sometimes it would be nice to 
get the compressed data e.g. if I'm just going to compress it again to 
transfer somewhere else, but for reporting the uncompressed data is 
what I want.


You can't do that if you use the JZos Zlogstream class to read the 
logstream and zEDC is in play. It's not too difficult to reverse 
engineer if you can work out the headers and how to interpret the raw 
zlib payload.





Andrew Rowley

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Andrew Rowley]

On 18/05/2021 5:21 pm, David Crayford wrote:


Do they have a zIIP on the RDP system?

Yes, although the regular CPs are the same speed. Having the zIIP means 
I can get figures for zIIP, CP and zIIP on CP.


Are you talking about CMF 110 records? If so we've both worked on 
similar stuff. One of our products is a CICS Performance Analyzer. 
It's written in HLASM. A couple of years ago the CTO at the time was 
interested in using Java to process
SMF records and me to port the HLASM code to Java. When we ran it on 
our z13s the Java code smashed the HLASM. The reason was because the 
Jave was running on a full capacity zIIP and the HLASM code on a 
sub-capacity GP.


BTW, what kind of compression do you mean?


Yes, the CMF 110 records. I decompress the RLE compression in the record 
in Java. That means that the data can be processed on a PC if desired. 
(I even processed CICS SMF data on a Raspberry Pi to prove it could be 
done!)


I'm happy to use the standard operating system facilities to decompress 
data from the SMF logstreams. Sometimes it would be nice to get the 
compressed data e.g. if I'm just going to compress it again to transfer 
somewhere else, but for reporting the uncompressed data is what I want.


Andrew Rowley

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[David Crayford]

Sorry about the typos. OpenJDK Flight Record was GA from Java 9 onwards. 
So fingers crossed we will see that when the Java 11 is finally 
available on z/OS.


On 18/05/2021 5:12 pm, David Crayford wrote:
Unfortunately, othing has changed :( z/OS only partially supports the 
JVMTI implementation. Hopefully, now IBM/OpenJ9 is open source the 
Java tooling landscape will improve. RedHat supports Flight Record in 
their OpenJDK distros


Unfortunately, *nothing has changed z/OS only partially supports the 
JVMTI implementation. Hopefully, now IBM/OpenJ9 is open source the Java 
tooling landscape will improve. RedHat supports Flight *Recorder in 
their OpenJDK distros


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[David Crayford]

On 18/05/2021 1:52 am, Kirk Wolf wrote:


- good low-overhead (sampling) profiling tools for the z/OS JVM don't 
exist (maybe this has changed?).   IMO this was a huge barrier for 
Java on z/OS.


Unfortunately, othing has changed :( z/OS only partially supports the 
JVMTI implementation. Hopefully, now IBM/OpenJ9 is open source the Java 
tooling landscape will improve. RedHat supports Flight Record in their 
OpenJDK distros
and IMO z/OS should at the very least have the same. I say again, Java 
on z/OS needs a lot more love then it's currently getting. z/OS is 
supposed to be the flagship IBM OS. We've started to introduce Kotlin to 
modernize our tooling as IBM are too slow updating
Java versions on z/OS. So far it's been a success. Kotlin adds some 
runtime overhead due to additional boilerplate code but it's well worth 
it. Null safety, no checked exceptions and type safe builders alone are 
a good enough reason to start transitioning, albeit slowly.





- Sometimes CPU costs don't matter where time to market or 
functionality are more important or where the application doesn't run 
that much. 


Agreed. And I suppose that was Andrew's point. If his Java product can 
run quick enough as a batch utility then happy days. If it runs like a 
stallion on a zIIP :two-thumbs-up:


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[David Crayford]

On 18/05/2021 1:32 pm, Andrew Rowley wrote:

On 16/05/2021 12:59 pm, David Crayford wrote:


And there is a good chance that you could introduce unacceptable 
software charges. I've worked on performance reporting tools for over 
two decades and in my experience customers measure everything, as any 
spike in workload can be a disaster in terms of cost. Java is a 
brilliant addition to z/OS and
I use it a lot but I'm more than aware of the caveats of writing z/OS 
products that require it.


WAS and Tomcat are containers for applications, using them to judge 
Java performance is like using a CICS address space to judge COBOL.


My Java experience has mostly been writing programs to run in batch 
under JZOS - I haven't touched WAS or Tomcat yet.


Java seems to have good performance once it gets going, and the 
facilities it provides (Collections etc.) mean that you can often 
choose more efficient algorithms and e.g. eliminate sorting of input 
data.


Agreed. Java is a fast horse that needs of a couple of laps to warm up.




Some of the Java programs I have written, with performance 
measurements (using the Dallas RDP system):


Do they have a zIIP on the RDP system?




- A program to do gzip compression of z/OS datasets for data transfer 
to & from z/OS, including correct handling of variable length z/OS 
records. CPU usage was very similar to the Rocket gzip port (which is 
presumably C code). Performance was much better than TERSE. This is 
without zEDC. I am working to get access to zEDC which should benefit 
both Java and gzip, and leave TERSE even further behind.


That's interesting. Java ZipOuputStream uses the native zlib library 
under the covers. There may be some overhead for crossing the JNI 
boundary but if you're processing records at a time not much.



A program to summarize CICS transaction data, producing a CSV file 
with totals by applid by transaction by minute.


Reporting 12 million transactions, 6.5GB CICS compressed/42GB 
uncompressed took 71 CPU seconds, including uncompressing the CICS SMF 
data in Java.


Are you talking about CMF 110 records? If so we've both worked on 
similar stuff. One of our products is a CICS Performance Analyzer. It's 
written in HLASM. A couple of years ago the CTO at the time was 
interested in using Java to process
SMF records and me to port the HLASM code to Java. When we ran it on our 
z13s the Java code smashed the HLASM. The reason was because the Jave 
was running on a full capacity zIIP and the HLASM code on a sub-capacity GP.


BTW, what kind of compression do you mean? There are two types of 
possible compression. There is the RLE compression of the 110 which is 
quite easy to decode and if you're processing SMF logstrams that use 
zEDC you will need to uncompress the blocks. The block structures are
not published but we reversed engineered it years ago so I implemented 
that in Java too.





You might say these CPU figures are high, but I think considering the 
work being done they are not too bad. I'm not sure they are any more 
than e.g. some SMP/E apply jobs. Interestingly, the CPU time on z/OS 
usually tracks closely with the run time on my PC.


If you are creating a whole application in Java, you probably need 
zIIP(s). But I would not let the lack of a zIIP stop you from using 
Java for utility and reporting programs. Java on z/OS is very viable 
for this sort of work.


Fair point. I'm not arguing that Java isn't viable on z/OS. In fact I 
think it's a brilliant platform. When you consider the amount of REXX 
utilities floating about on z/OS that process SMF data it's much better 
to do it in Java. Even if it's not on a zIIP. REXX is the biggest CPU 
hog of all.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Andrew Rowley]

On 15/05/2021 3:03 am, Matt Hogstrom wrote:

It takes z/OSMF about an hour to fully initialize on my ZD … same behavior 
(lots of SIOs and CPU)  IBM says its fixing this but its been quite a while.  Its 
Java related.



I did some investigation a while back using type 92 SMF records to try 
to figure out why z/OSMF was doing about 90,000,000 EXCPs during startup 
on my system.


I found that there were many files (mostly doc.zip) where the same file 
was opened and closed hundreds or even thousands of times during startup.


Despite this, z/OSMF startup on my system still only took around 1 
minute. I have a hypothesis that this might be because the filesystem is 
mounted read only - perhaps if the filesystem is read/write last access 
times are updated for each open/close, which means write I/Os instead of 
cached read I/Os.


Files are e.g.
ation/helps/eclipse/plugins/com.ibm.cpomessages.help.doc/doc.zip
(name is truncated)

Is your filesystem read/write? Does it help if it is read only?

I think IBM has made some improvements since I did the investigation. 
Looking now z/OSMF has 29,000,000 EXCP instead of 90,000,000. I probably 
should repeat it and see what has changed.


--
Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Andrew Rowley]

On 16/05/2021 12:59 pm, David Crayford wrote:


And there is a good chance that you could introduce unacceptable 
software charges. I've worked on performance reporting tools for over 
two decades and in my experience customers measure everything, as any 
spike in workload can be a disaster in terms of cost. Java is a 
brilliant addition to z/OS and
I use it a lot but I'm more than aware of the caveats of writing z/OS 
products that require it.


WAS and Tomcat are containers for applications, using them to judge Java 
performance is like using a CICS address space to judge COBOL.


My Java experience has mostly been writing programs to run in batch 
under JZOS - I haven't touched WAS or Tomcat yet.


I concede that starting the JVM is a relatively expensive operation, so 
you don’t want to do it frequently for short running programs. But for 
larger jobs it is not so significant. Java seems to have good 
performance once it gets going, and the facilities it provides 
(Collections etc.) mean that you can often choose more efficient 
algorithms and e.g. eliminate sorting of input data.


Some of the Java programs I have written, with performance measurements 
(using the Dallas RDP system):


- A program to do gzip compression of z/OS datasets for data transfer to 
& from z/OS, including correct handling of variable length z/OS records. 
CPU usage was very similar to the Rocket gzip port (which is presumably 
C code). Performance was much better than TERSE. This is without zEDC. I 
am working to get access to zEDC which should benefit both Java and 
gzip, and leave TERSE even further behind.


- A program to scan SMF data for duplicate records, reporting duplicates 
and distinguishing between occasional duplicates, all data for a time 
period duplicated, or specific record types included multiple times.


In my testing this took 31 CPU seconds to scan 3.4 million records 
(9.4GB) and find 34,000 duplicated records.


- A program to summarize CICS transaction data, producing a CSV file 
with totals by applid by transaction by minute.


Reporting 12 million transactions, 6.5GB CICS compressed/42GB 
uncompressed took 71 CPU seconds, including uncompressing the CICS SMF 
data in Java.


You might say these CPU figures are high, but I think considering the 
work being done they are not too bad. I'm not sure they are any more 
than e.g. some SMP/E apply jobs. Interestingly, the CPU time on z/OS 
usually tracks closely with the run time on my PC.


If you are creating a whole application in Java, you probably need 
zIIP(s). But I would not let the lack of a zIIP stop you from using Java 
for utility and reporting programs. Java on z/OS is very viable for this 
sort of work.



--
Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Graham Harris]

SMF 92 subtype 11 is usually useful in showing USS file activity, which can
tend to reveal if anything 'odd' is going on with file based activity.
I/O is also seen when network activity is involved, but I have not found
92's very useful in determining what is going on in that arena, but others
may have had success.
Note that if you are not already collecting 92's, some subtypes can
generate enormous amounts of SMF records, so we choose to NOTYPE on
92(10,16,17)

On Mon, 17 May 2021 at 22:31, David Crayford  wrote:

> On 18/05/2021 2:05 am, Kirk Wolf wrote:
> > Sorry, I meant to add that agree with David.   If you have big Java
> > apps that use significant CPU, you really need to run them on zIIP or
> > the costs will be intolerable.
> >
> > In the particular case of the large Australian customer, I wonder why
> > they didn't see the issue coming early on?
>
>
> Good question. It's my understanding that they did indeed plan for
> growth but the workload surpassed capacity planning. The application was
> a customer facing mobile application and mobile workloads have shocked a
> lot of mainframe customers over the last decade.
> There is a ratio of the maximum number of zIIPs tied to the number of
> GCPs. This used to be 1:1 but I believe it is now 2:1. IIRC, they also
> added a caching layer to optimize the application.
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[David Crayford]

On 18/05/2021 2:05 am, Kirk Wolf wrote:
Sorry, I meant to add that agree with David.   If you have big Java 
apps that use significant CPU, you really need to run them on zIIP or 
the costs will be intolerable.


In the particular case of the large Australian customer, I wonder why 
they didn't see the issue coming early on? 



Good question. It's my understanding that they did indeed plan for 
growth but the workload surpassed capacity planning. The application was 
a customer facing mobile application and mobile workloads have shocked a 
lot of mainframe customers over the last decade.
There is a ratio of the maximum number of zIIPs tied to the number of 
GCPs. This used to be 1:1 but I believe it is now 2:1. IIRC, they also 
added a caching layer to optimize the application.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Kirk Wolf]

Sorry, I meant to add that agree with David.   If you have big Java apps 
that use significant CPU, you really need to run them on zIIP or the 
costs will be intolerable.


In the particular case of the large Australian customer, I wonder why 
they didn't see the issue coming early on?


Kirk Wolf
Dovetailed Technologies
http://dovetail.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Kirk Wolf]

We haven't done much with Java in the last few years, but our experience 
with mission-critical production Java apps on z/OS (going back almost 20 
years) is a little mixed.


- JVM startup is rather expensive, especially the full JITing of classes.

- the more classes you pull in (like huge class libraries), the more 
costs can get out of hand


- most Java programmers don't worry about what things cost (at all :-)

- long running apps that are well written and focus on performance can 
run very well, even when compared to traditional MVS compiled languages 
or even assembler -  Not based on low level language metrics, but by 
exploiting algorithms and class libraries for data structures, caching, 
etc.


- good low-overhead (sampling) profiling tools for the z/OS JVM don't 
exist (maybe this has changed?).   IMO this was a huge barrier for Java 
on z/OS.


- Sometimes CPU costs don't matter where time to market or functionality 
are more important or where the application doesn't run that much.



Kirk Wolf
Dovetailed Technologies
http://dovetail.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Tom Brennan]

4) z15 System Recovery Boost (which I guess the sales folks thought 
would be fun to call SRB) can use zIIP's for general CP work around IPL 
time.


On 5/15/2021 6:31 PM, Andrew Rowley wrote:

On 15/05/2021 3:33 pm, David Crayford wrote:

I don't believe that is true. zIIP engines give you 3 things:

1) Savings in software cost.
2) Full speed CPUs, if your general purpose CPs are not full speed
3) Very often, they are less utilized than the general purpose CPs so 
you spend less time waiting for dispatch.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[David Crayford]

On 16/05/2021 9:31 am, Andrew Rowley wrote:

On 15/05/2021 3:33 pm, David Crayford wrote:

Java is not viable on z/OS without zIIP engines.


I don't believe that is true. zIIP engines give you 3 things:



Answered below.



1) Savings in software cost.


Imperative. One of our customers, a large Austalian bank, recently 
undertook a significant application modernization project where they 
moved all of their web application servers into Websphere Liberty 
servers running in CICS.
This was a real win/win as they could decommission distributed servers 
and middleware such as CTG and last but not least the management would 
be fully under the mainframe team. They started to run into problems 
when there was not
enough zIIP capacity in production for the increased workload. They were 
already at the zIIP limit as it's pegged to the number of GCPs. Adding 
more GCPs would have blown the TCO and the project
would have failed. Their MSU software charges increased to unacceptable 
levels when the Java workloads would spill over onto the GCPs. I heard 
that they resolved this problem after negotiating with IBM who was a partner
on the project. Without zIIPs this kind of project would not be viable. 
With zIIPs it was touch and go. Capacity planning for zIIPs is critical.




2) Full speed CPUs, if your general purpose CPs are not full speed


The majority of our customers don't run at full capacity. IIRC, most are 
subcapacity.



3) Very often, they are less utilized than the general purpose CPs so 
you spend less time waiting for dispatch.


And very often that is not the case in a production environment. I would 
argue that it would be sensible for a shop to plan for what kind of zIIP 
capacity they need.



#3 may be significant, but nothing runs well if it isn't being 
dispatched - that isn't a Java problem.


The other factor that gives Java a bad reputation is that it allows 
you to bring applications to z/OS that were written with the 
assumption that they were the main thing running on the system and 
there is CPU to spare. This isn't how most people run their z/OS 
systems. I suspect this is the root of your Tomcat problem, but that 
is a Tomcat issue not a Java issue.



The problem was that the customer was running our Tomcat server on a 
kneecapped subcapacity LPAR with no zIIP. Tomcat is lightweight. Imagine 
if they were running a fully blown JEE stack like WAS! Almost all the 
open source Java software I use runs brilliantly on z/OS. I'm currently 
working on a project that uses Spring Boot and we have introduced 
Kotlin. I wouldn't advise customers  to run it on z/OS if they don't 
have zIIPs. One of our products (OMEGAMON for JVM on z/OS ) shows the 
low-level details of the JVM. Even a hello world program runs about 8 
threads, 3 for JIT, 1 for OMR,  2 for GC etc. This all requires 
significantly more resources than the equivalent C++ code.





Java will run as well as any other work on general purpose CPs - but 
you lose the software cost savings.


And there is a good chance that you could introduce unacceptable 
software charges. I've worked on performance reporting tools for over 
two decades and in my experience customers measure everything, as any 
spike in workload can be a disaster in terms of cost. Java is a 
brilliant addition to z/OS and
I use it a lot but I'm more than aware of the caveats of writing z/OS 
products that require it. Most of the time this isn't a problem as 
customers have zIIPs. Projects like Zowe have Java application servers 
in the stack. Maybe IBM will include a free zIIP in future hardware models.





Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Andrew Rowley]

On 15/05/2021 3:33 pm, David Crayford wrote:

Java is not viable on z/OS without zIIP engines.


I don't believe that is true. zIIP engines give you 3 things:

1) Savings in software cost.
2) Full speed CPUs, if your general purpose CPs are not full speed
3) Very often, they are less utilized than the general purpose CPs so 
you spend less time waiting for dispatch.


#3 may be significant, but nothing runs well if it isn't being 
dispatched - that isn't a Java problem.


The other factor that gives Java a bad reputation is that it allows you 
to bring applications to z/OS that were written with the assumption that 
they were the main thing running on the system and there is CPU to 
spare. This isn't how most people run their z/OS systems. I suspect this 
is the root of your Tomcat problem, but that is a Tomcat issue not a 
Java issue.


Java will run as well as any other work on general purpose CPs - but you 
lose the software cost savings.


Andrew Rowley
Black Hill Software

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[David Crayford]

On 15/05/2021 1:03 am, Matt Hogstrom wrote:

It takes z/OSMF about an hour to fully initialize on my ZD … same behavior 
(lots of SIOs and CPU)  IBM says its fixing this but its been quite a while.  Its 
Java related.



And how much does ZD cost? We had customer complain that our Apache 
Tomcat server was taking 15 minutes to start on one LPAR and 15 seconds 
on another. Turns out the didn't have a ZIIP enabled

on the first LPAR. Java is not viable on z/OS without zIIP engines.




Matt Hogstrom
m...@hogstrom.org
+1-919-656-0564
PGP Key: 0x90ECB270
Facebook   LinkedIn 
  Twitter 

“It may be cognitive, but, it ain’t intuitive."
— Hogstrom


On May 14, 2021, at 1:01 PM, Colin Paice  wrote:

Im using z/OSMF on my little zPDT system, and it is doing about 50,000 SIO
a second,
when it is doing "nothing"

How canI tell what it is doing?  I expect most of these files are in the
zFS file system, but I dont know the best way of finding out what is going
on.
I do not have tools like Strobe or APA,and I am familiar with dumps etc

Colin

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[David Crayford]

This is usually due to Websphere scanning the dropins folder for 
updates. z/OS doesn't have edge triggered kernel hooks like inotify on 
Linux so has to use brute force. It's not just I/O it consumes a lot of CPU.


You can disable it in server.xml


locationMonitoringEnabled="false" updateTrigger="mbean" pollingRate="10s"/>


On 15/05/2021 1:01 am, Colin Paice wrote:

Im using z/OSMF on my little zPDT system, and it is doing about 50,000 SIO
a second,
when it is doing "nothing"

How canI tell what it is doing?  I expect most of these files are in the
zFS file system, but I dont know the best way of finding out what is going
on.
I do not have tools like Strobe or APA,and I am familiar with dumps etc

Colin

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Chris Hoelscher]

Tell them what?

Chris Hoelscher
Lead Sys DBA 
IBM Global Technical Services on assignmemt to Humana Inc.
T 502.476.2538  or 502.407.7266

The information transmitted is intended only for the person or entity to which 
it is addressed
and may contain CONFIDENTIAL material.  If you receive this 
material/information in error,
please contact the sender and delete or destroy the material/information.

Humana Inc. and its subsidiaries comply with applicable Federal civil rights 
laws and
do not discriminate on the basis of race, color, national origin, ancestry, 
age, disability, sex,
marital status, gender, sexual orientation, gender identity, or religion. 
Humana Inc. and its subsidiaries do not
exclude people or treat them differently because of race, color, national 
origin, ancestry, age,
disability, sex, marital status, gender, sexual orientation, gender identity, 
or religion.

English: ATTENTION: If you do not speak English, language assistance services, 
free
of charge, are available to you. Call 1‐877‐320‐1235 (TTY: 711).

Español (Spanish): ATENCIÓN: Si habla español, tiene a su disposición servicios
gratuitos de asistencia lingüística. Llame al 1‐877‐320‐1235 (TTY: 711).

繁體中文(Chinese):注意:如果您使用繁體中文,您可以免費獲得語言援助
服務。請致電 1‐877‐320‐1235 (TTY: 711)。

Kreyòl Ayisyen (Haitian Creole): ATANSION: Si w pale Kreyòl Ayisyen, gen sèvis 
èd
pou lang ki disponib gratis pou ou. Rele 1‐877‐320‐1235 (TTY: 711).

Polski (Polish): UWAGA: Jeżeli mówisz po polsku, możesz skorzystać z bezpłatnej
pomocy językowej. Zadzwoń pod numer 1‐877‐320‐1235 (TTY: 711).

한국어 (Korean): 주의: 한국어를 사용하시는 경우, 언어 지원 서비스를 무료로
이용하실 수 있습니다. 1‐877‐320‐1235 (TTY: 711)번으로 전화해 주십시오.


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Colin Paice]

Matt,

One problem with the Slow start up is the first time it has to
compile/cache things - the second time it is faster.  Ive blogged on saving
the cached stuff across restarts.
I hear one of the problems is to do with the checking access to files, not
Java as such.
Z/OSMF had been up and I had been using ISPF through it ... I then jumped
to TSO and saw the high IO rate in SDSF.
Is there a C_run_ time/LE trace/Other which would give me some hints?
Colin

On Fri, 14 May 2021 at 18:03, Matt Hogstrom  wrote:

> It takes z/OSMF about an hour to fully initialize on my ZD … same
> behavior (lots of SIOs and CPU)  IBM says its fixing this but its been
> quite a while.  Its Java related.
>
> Matt Hogstrom
> m...@hogstrom.org
> +1-919-656-0564
> PGP Key: 0x90ECB270
> Facebook   LinkedIn <
> https://linkedin/in/mhogstrom>  Twitter 
>
> “It may be cognitive, but, it ain’t intuitive."
> — Hogstrom
>
> > On May 14, 2021, at 1:01 PM, Colin Paice  wrote:
> >
> > Im using z/OSMF on my little zPDT system, and it is doing about 50,000
> SIO
> > a second,
> > when it is doing "nothing"
> >
> > How canI tell what it is doing?  I expect most of these files are in the
> > zFS file system, but I dont know the best way of finding out what is
> going
> > on.
> > I do not have tools like Strobe or APA,and I am familiar with dumps etc
> >
> > Colin
> >
> > --
> > For IBM-MAIN subscribe / signoff / archive access instructions,
> > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: How can I tell the zFS files being used by my application

[Matt Hogstrom]

It takes z/OSMF about an hour to fully initialize on my ZD … same behavior 
(lots of SIOs and CPU)  IBM says its fixing this but its been quite a while.  
Its Java related.

Matt Hogstrom
m...@hogstrom.org
+1-919-656-0564
PGP Key: 0x90ECB270
Facebook   LinkedIn 
  Twitter 

“It may be cognitive, but, it ain’t intuitive."
— Hogstrom

> On May 14, 2021, at 1:01 PM, Colin Paice  wrote:
> 
> Im using z/OSMF on my little zPDT system, and it is doing about 50,000 SIO
> a second,
> when it is doing "nothing"
> 
> How canI tell what it is doing?  I expect most of these files are in the
> zFS file system, but I dont know the best way of finding out what is going
> on.
> I do not have tools like Strobe or APA,and I am familiar with dumps etc
> 
> Colin
> 
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN