Re: Why sort (was Microprocessor Optimization Primer)

David, I haven't seen benchmarks yet, just a bunch of wall time comparisons. We have terabytes of memory on our Z13 that hasn't been deployed yet, I'm cynical but our guys have to start to run interference. On Thu, Apr 7, 2016 at 9:46 PM, David Crayford wrote: > On

Re: Why sort (was Microprocessor Optimization Primer)

@LISTSERV.UA.EDU Subject: (External):Re: Why sort (was Microprocessor Optimization Primer) On 04/06/2016 07:01 AM, Andrew Rowley wrote: > On 05/04/2016 01:20 AM, Tom Marchant wrote: >> On Mon, 4 Apr 2016 16:45:37 +1000, Andrew Rowley wrote: >> >>> A Hashmap potentially allows

Re: Why sort (was Microprocessor Optimization Primer)

...hey Wayne. Mitch Mccluhan mitc...@aol.com On Thursday, April 7, 2016 Wayne Bickerdike wrote: I'm slightly gobsmacked that this discussion is needed. I guess the forest is lost in the trees. I can recommend "Principles of Program Design" by Michael Jackson c. 1975. Of

Re: Why sort (was Microprocessor Optimization Primer)

On 7/04/2016 7:56 PM, John McKown wrote: On Thu, Apr 7, 2016 at 5:59 AM, Wayne Bickerdike wrote: I'm slightly gobsmacked that this discussion is needed. I guess the forest is lost in the trees. I can recommend "Principles of Program Design" by Michael Jackson c. 1975. Of

Re: Why sort (was Microprocessor Optimization Primer)

On Thu, Apr 7, 2016 at 5:59 AM, Wayne Bickerdike wrote: > I'm slightly gobsmacked that this discussion is needed. I guess the forest > is lost in the trees. > > I can recommend "Principles of Program Design" by Michael Jackson c. 1975. > > Of greater concern is the implication

Re: Why sort (was Microprocessor Optimization Primer)

On 7/04/2016 6:59 PM, Wayne Bickerdike wrote: I'm slightly gobsmacked that this discussion is needed. I guess the forest is lost in the trees. I can recommend "Principles of Program Design" by Michael Jackson c. 1975. Of greater concern is the implication that Oracle on AIX outperforms DB2 on

Re: Why sort (was Microprocessor Optimization Primer)

I'm slightly gobsmacked that this discussion is needed. I guess the forest is lost in the trees. I can recommend "Principles of Program Design" by Michael Jackson c. 1975. Of greater concern is the implication that Oracle on AIX outperforms DB2 on z/OS at our shop. Surely not :( On Thu, Apr 7,

Re: Why sort (was Microprocessor Optimization Primer)

On 04/06/2016 07:01 AM, Andrew Rowley wrote: > On 05/04/2016 01:20 AM, Tom Marchant wrote: >> On Mon, 4 Apr 2016 16:45:37 +1000, Andrew Rowley wrote: >> >>> A Hashmap potentially allows you to read sequentially and match records >>> between files, without caring about the order. >> Can you please

Re: Why sort (was Microprocessor Optimization Primer)

On Wed, Apr 6, 2016 at 7:01 AM, Andrew Rowley wrote: > On 05/04/2016 01:20 AM, Tom Marchant wrote: > >> On Mon, 4 Apr 2016 16:45:37 +1000, Andrew Rowley wrote: >> >> A Hashmap potentially allows you to read sequentially and match records >>> between files, without

Re: Why sort (was Microprocessor Optimization Primer)

On 05/04/2016 01:20 AM, Tom Marchant wrote: On Mon, 4 Apr 2016 16:45:37 +1000, Andrew Rowley wrote: A Hashmap potentially allows you to read sequentially and match records between files, without caring about the order. Can you please explain what you mean by this? Are you talking about using

Re: Microprocessor Optimization Primer

Rule 3 says there are events, some explicit (BCR 15,0) and some implicit (interrupts), that force conceptual completion to occur, over and above those required by rule number 1. These rules are for storage consistency, especially between multiple processors. Registers are slightly different

Re: Microprocessor Optimization Primer

On Mon, 4 Apr 2016 09:30:47 -0700, Greg Dyck wrote: > >1) ... >The results of one instruction are placed in storage after the results >of all preceding instructions have been placed in storage and before >any results of the succeeding instructions are stored, as observed by >other CPUs and by

Re: Microprocessor Optimization Primer

David Crayford wrote: How do I find out how much time is spent in JIT overhead compared to actual processing? What tools do you use to profile your JVM apps? Last year I looked into JinsightLive for IBM System z and the IBM Health Center for Java for an cpu and runtime increase a client was

Re: Microprocessor Optimization Primer

Operand fetches and stores must appear to occur in proper order. So is it saying that the machine obeys Rule 1? I will bet dollars to donuts that that is not the case, that the machine performs read operations before logically preceding write operations, and that a program can be run to

Re: Why sort (was Microprocessor Optimization Primer)

On Mon, 4 Apr 2016 16:45:37 +1000, Andrew Rowley wrote: >A Hashmap potentially allows you to read sequentially and match records >between files, without caring about the order. Can you please explain what you mean by this? Are you talking about using the hashmap to determine which record to

Re: Microprocessor Optimization Primer

note that test was on both 360/67 and 360/65 machines and was atomic. I've commented before about charlie invented compare (chosen because CAS are his initials) while doing fine-grain multiprocessor locking working on CP67 (360/67 precursor to vm370) at the science center.

Re: Why sort (was Microprocessor Optimization Primer)

On 4/04/2016 11:25, David Betten wrote: First the idea of loading all the data into a large hashmap to do the sort tends to eliminate one very important thing and that's overlap. Essentially, you read the entire input, conduct your massive hashsort, and then write the output with no overlap of

Re: Why sort (was Microprocessor Optimization Primer)

On 2016-04-03, at 19:25, David Betten wrote: > First of all, full disclaimer that I was in DFSORT development for about 8 > years so I might be biased. But I just want to share a few thoughts. > > First the idea of loading all the data into a large hashmap to do the sort > tends to eliminate

Re: Why sort (was Microprocessor Optimization Primer)

On 4/3/2016 6:21 PM, Ed Jaffe wrote: DFSORT, Syncsort, etc. use the CPC/UPT hardware instructions to implement the fastest sort on the platform. Typo. Of course, I meant to write CFC/UPT... :-[ -- Edward E Jaffe Phoenix Software International, Inc 831 Parkview Drive North El Segundo, CA 90245

Re: Why sort (was Microprocessor Optimization Primer)

[image: Mic Drop] On Apr 3, 2016 20:53, "David Crayford" wrote: > > On 4/04/2016 7:41 AM, John McKown wrote: >> >> >> ​I'm not an application programmer. But I can just imagine the looks of >> astonishment and the "talk", if I were to write a COBOL program which does >> a

Re: Why sort (was Microprocessor Optimization Primer)

On 4/04/2016 7:41 AM, John McKown wrote: On Sun, Apr 3, 2016 at 6:00 PM, Andrew Rowley wrote: On 3/04/2016 22:43, David Crayford wrote: Good question! Sort can be utilised for other purposes than sorting, it can be used as an I/O engine. DFSORT (or Syncsort)

Re: Microprocessor Optimization Primer

I'm behind reading this epic thread, but I sent a question related to it to Bill Collier, instructor I had at IBM in 1968 while in full-time training for OS/360 System Design Department. He runs http://www.mpdiag.com and is biographied at http://www.bestweb.net/~collier -- so is valuable

Re: Why sort (was Microprocessor Optimization Primer)

Discussion List <IBM-MAIN@LISTSERV.UA.EDU> wrote on 04/03/2016 07:28:39 PM: > From: Andrew Rowley <and...@blackhillsoftware.com> > To: IBM-MAIN@LISTSERV.UA.EDU > Date: 04/03/2016 07:32 PM > Subject: Re: Why sort (was Microprocessor Optimization Primer) > Sent by: IBM Mai

Re: Why sort (was Microprocessor Optimization Primer)

On 4/3/2016 4:28 PM, Andrew Rowley wrote: DFSORT seems to be most useful where you need to sort more data than can be processed in storage - but I'm wondering how often that really needs to be done. I'm not so interested in utilities and databases calling it under the covers - more in

Re: Why sort (was Microprocessor Optimization Primer)

On Sun, Apr 3, 2016 at 6:28 PM, Andrew Rowley wrote: > The reason I like Java on Z so much is I got used to using Hashtable in > C#, then tried to use Rexx stems to do the same thing. (It was semi > successful but I always felt like it was very fragile due to the

Re: Why sort (was Microprocessor Optimization Primer)

On Sun, Apr 3, 2016 at 6:00 PM, Andrew Rowley wrote: > On 3/04/2016 22:43, David Crayford wrote: > >> Good question! Sort can be utilised for other purposes than sorting, it >> can be used as an I/O engine. DFSORT (or Syncsort) implements bespoke >> highly optinized

Re: Why sort (was Microprocessor Optimization Primer)

The reason I like Java on Z so much is I got used to using Hashtable in C#, then tried to use Rexx stems to do the same thing. (It was semi successful but I always felt like it was very fragile due to the potential for unexpected values etc. for the stems.) Then I found Java had real hash

Re: Why sort (was Microprocessor Optimization Primer)

On 3/04/2016 22:43, David Crayford wrote: Good question! Sort can be utilised for other purposes than sorting, it can be used as an I/O engine. DFSORT (or Syncsort) implements bespoke highly optinized I/O using low-level programming interfaces such as chained EXCPs which are significantly

Re: Why sort (was Microprocessor Optimization Primer)

On 3 April 2016 at 02:50, Andrew Rowley wrote: > One question that puzzles me (maybe it's my lack of an application > programming background): Why is sort used so much on z/OS? As others have pointed out, sort on z/OS (whether IBM's or other vendors') can be used

Re: Why sort (was Microprocessor Optimization Primer)

-MAIN@LISTSERV.UA.EDU Subject: Re: Why sort (was Microprocessor Optimization Primer) I used to work for the late great Security Pacific, at the time the largest bank based in Los Angeles. When DFSORT was a pimply-faced teenager, some of us sysprogs were invited to Santa Teresa to meet

Re: Why sort (was Microprocessor Optimization Primer)

03, 2016 6:52 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: (External):Re: Why sort (was Microprocessor Optimization Primer) Along with the other reasons outlined by others, it significantly improves bulk processing, I shy away from the term batch because that has come to have a bad connotation. When

Re: Why sort (was Microprocessor Optimization Primer)

Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On Behalf Of Andrew Rowley Sent: Sunday, April 03, 2016 2:51 AM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Why sort (was Microprocessor Optimization Primer) On 02/04/2016 10:09 PM, David Crayford wrote: > IBM switched the magic bit to offload the J

Re: Why sort (was Microprocessor Optimization Primer)

On 3/04/2016 2:50 PM, Andrew Rowley wrote: On 02/04/2016 10:09 PM, David Crayford wrote: IBM switched the magic bit to offload the JZOS JNI C/C++ workload to a zIIP so they could do the same for DFSORT. A well engineered library could handle the callbacks so the client just reads records like a

Re: Why sort (was Microprocessor Optimization Primer)

On Sun, Apr 3, 2016 at 1:50 AM, Andrew Rowley wrote: > On 02/04/2016 10:09 PM, David Crayford wrote: > >> IBM switched the magic bit to offload the JZOS JNI C/C++ workload to a >> zIIP so they could do the same for DFSORT. A well engineered library >> could handle

Why sort (was Microprocessor Optimization Primer)

On 02/04/2016 10:09 PM, David Crayford wrote: IBM switched the magic bit to offload the JZOS JNI C/C++ workload to a zIIP so they could do the same for DFSORT. A well engineered library could handle the callbacks so the client just reads records like a normal API. That would certainly push Java

Re: Microprocessor Optimization Primer

On 2/04/2016 11:29 PM, Kirk Wolf wrote: You are right again about the licensing problem with the JZOS tools on non-zOS platforms - IBM unfortunately blew it when it comes to using the portable parts of JZOS on other platforms. You can certainly download the required JZOS jar from developerworks

Re: Microprocessor Optimization Primer

On 3/31/2016 10:26 PM, Andrew Rowley wrote: Yes, the Java program used a little more than double the CPU of the Assembler program reading 8GB of SMF data - but more than half was on zIIP so there was actually less CP time. I need to do some tests to work out how that varies with different

Re: Microprocessor Optimization Primer

.com/developerworks/mydeveloperworks/blogs/MartinPacker > > > > From: David Crayford <dcrayf...@gmail.com> > To: IBM-MAIN@LISTSERV.UA.EDU > Date: 02/04/2016 11:35 > Subject:Re: Microprocessor Optimization Primer > Sent by:IBM Mainframe Discussion Lis

Re: Microprocessor Optimization Primer

> > I looked at the RecordGenerator, but I have my own process to do a similar > thing. It's actually a bit more complicated because the DSECTS don't > contain the full data type information. I am converting STCK values to > java.time.ZonedDateTime, SMF date fields to java.time.LocalDate, CPU time

Re: Microprocessor Optimization Primer

On 2/04/2016 6:59 PM, Martin Packer wrote: But, yes, DFSORT to do the I/O and much of the processing, choreographed by JZOS, is certainly an idea worth pursuing. NOTE: Not much zIIP in that, if any. IBM switched the magic bit to offload the JZOS JNI C/C++ workload to a zIIP so they could do

Re: Microprocessor Optimization Primer

Re: Microprocessor Optimization Primer Sent by:IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> On 2/04/2016 5:21 PM, Martin Packer wrote: > Stray thought, Kirk (and yes made public)... > > Maybe JZOS should have a way of mapping com.ibm.jzos.fields to DFSORT &

Re: Microprocessor Optimization Primer

e: 01/04/2016 22:48 Subject: Re: Microprocessor Optimization Primer Sent by:IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> I'm curious, Andrew - Have you looked at the com.ibm.jzos.fields package for field mapping of records? These can be made to use fancy su

Re: Microprocessor Optimization Primer

IBM-MAIN@LISTSERV.UA.EDU Date: 01/04/2016 22:48 Subject: Re: Microprocessor Optimization Primer Sent by:IBM Mainframe Discussion List <IBM-MAIN@LISTSERV.UA.EDU> I'm curious, Andrew - Have you looked at the com.ibm.jzos.fields package for field mapping of records? These can be m

Re: Microprocessor Optimization Primer

On 2/04/2016 7:06 AM, Ed Jaffe wrote: Is there a noticeable difference using different browsers, Chrome, Firefox, Safari etc? All of those you mentioned are about the same given adequate workstation memory (our development/test machines have 16GB RAM). The real outlier is M$ IE, both in

Re: Microprocessor Optimization Primer

Hi Kirk, I have had a look at the fields package, and it's on my list to compare performance. I am mostly doing my own conversions because one of the objectives is to allow SMF processing on other platforms i.e. Windows, Linux if desired and I'm not sure what are the licensing implications of

Re: Microprocessor Optimization Primer

On 4/1/2016 8:16 AM, David Crayford wrote: That's fast! How are you measuring that, browser javascript toolkits? These call clock response time measurements come from our own (admittedly very simplistic) javascript code. We measure the clock time from request to response. For the averages,

Re: Microprocessor Optimization Primer

Agreed - there are high fixed costs to running a JVM. With a large codebase, there are also high warm-up costs for JITing the code up to full speed. Think of this as compiling your code every time you run it :-) But - there are many types of applications where these costs don't matter at all.

Re: Microprocessor Optimization Primer

I'm curious, Andrew - Have you looked at the com.ibm.jzos.fields package for field mapping of records? These can be made to use fancy support in the JVM for inline JIT replacement of primitive access with machine instructions. There is also some tools on the JZOS developerWorks site - the JZOS

Re: Microprocessor Optimization Primer

On 1/04/2016 10:59 PM, Ed Jaffe wrote: On 3/31/2016 7:47 PM, Andrew Rowley wrote: Call me a masochist, but I'm comparing Java batch to assembler. My target is to get the Java CPU less JVM startup overhead down to the same as assembler for the same report. I may not get there, but I will

Re: Microprocessor Optimization Primer

On 3/31/2016 7:47 PM, Andrew Rowley wrote: Call me a masochist, but I'm comparing Java batch to assembler. My target is to get the Java CPU less JVM startup overhead down to the same as assembler for the same report. I may not get there, but I will hopefully get close. I feel your pain! ;)

Re: Microprocessor Optimization Primer

C stdio is pretty close to assembler. They must have improved it because it used to be poor! The extra overhead is probably the linkage overhead of calling the LE runtime. Step Termination Statistics Program Name QSAMIOC hh:mm:ss.th Step Name QIOT

Re: Microprocessor Optimization Primer

Time to break out the 30s. +0234 SMF30CAS... Processor Accounting Section +0236 SMF30TFL... 8000SMF30CPT... 0.23SMF30CPS... 1.87SMF30ICU... 0 SMF30ISB... 0 +0248 SMF30JVU... 0 SMF30IVU... 0 SMF30JVA... 0 SMF30IVA... 0

Re: Microprocessor Optimization Primer

On 1/04/2016 4:03 PM, Andrew Rowley wrote: Thanks for those numbers. I'm guessing that zIIP time isn't in the Total CPU Time, so we are comparing user+sys in the second with Total CPU in the first? So it looks like Java overhead is about ~40% in this case? Not too dissimilar to what I found,

Re: Microprocessor Optimization Primer

Thanks for those numbers. I'm guessing that zIIP time isn't in the Total CPU Time, so we are comparing user+sys in the second with Total CPU in the first? So it looks like Java overhead is about ~40% in this case? Not too dissimilar to what I found, depending on whether Java did any

Re: Microprocessor Optimization Primer

On 1/04/2016 1:26 PM, Andrew Rowley wrote: On 1/04/2016 15:54, David Crayford wrote: Good luck with that mate! I've had conversations with the authors of JZOS about RecordReader and it uses all the tricks bar going down to EXCPs. There are also smarts to mitigate JNI overhead. If you write

Re: Microprocessor Optimization Primer

On 1/04/2016 15:54, David Crayford wrote: Good luck with that mate! I've had conversations with the authors of JZOS about RecordReader and it uses all the tricks bar going down to EXCPs. There are also smarts to mitigate JNI overhead. If you write your own JNI code it won't be zIIP eligible?

Re: Microprocessor Optimization Primer

> On 1 Apr 2016, at 10:47 AM, Andrew Rowley > wrote: > >> On 1/04/2016 13:01, David Crayford wrote: >> I guess you're using JZOS RecordReader? From what I know of the >> implementation it's micro-optimized and it's as good as it gets for Java. > > Yes. > >> You

Re: Microprocessor Optimization Primer

On 1/04/2016 13:01, David Crayford wrote: I guess you're using JZOS RecordReader? From what I know of the implementation it's micro-optimized and it's as good as it gets for Java. Yes. You can't compare a COBOL batch job to a Java batch job. The COBOL program will use 200K of memory and use

Re: Microprocessor Optimization Primer

On 1/04/2016 8:13 AM, Andrew Rowley wrote: I do know that the effects of allocation patterns are very different for GC languages and languages like C/C++. In C++ allocating and freeing short lived objects is expensive because the memory needs to be tracked. In GC languages short lived

Re: Microprocessor Optimization Primer

On 31/03/2016 23:34, David Crayford wrote: Thanks for the info Andrew. We use the default GC options and our workloads are both long running agents and batch jobs. I really do want to learn a lot more about how to tune the JVM for each specific workload. The SMF 30 records are useless. How do

Re: Microprocessor Optimization Primer

On 31/03/2016 5:47 PM, Andrew Rowley wrote: On 31/03/2016 07:01 PM, David Crayford wrote: When a stop the world event happens watch those zIIPs spike! I've been amused watching the RMF zIIP metrics in Grafana. Which garbage collection option do you use? Have you played around with

Re: Microprocessor Optimization Primer

Charles, I nj want to echo your comment about the IBM PDF mentioned, it's outstanding , thanks to the IBMers also..I am always learning. Regards, Scott On Thursday, March 31, 2016, Andrew Rowley wrote: > On 31/03/2016 07:01 PM, David Crayford wrote: > >> >>

Re: Microprocessor Optimization Primer

On 31/03/2016 07:01 PM, David Crayford wrote: When a stop the world event happens watch those zIIPs spike! I've been amused watching the RMF zIIP metrics in Grafana. Which garbage collection option do you use? Have you played around with different GC options? I have found it can make a big

Re: Microprocessor Optimization Primer

On 31/03/2016 12:39 PM, Ed Jaffe wrote: On 3/30/2016 5:19 PM, David Crayford wrote: Reading that book reinforced my opinion that I would rather leave micro optimizations to a compiler. How can you optimize for hardware that hasn't been invented yet? In 5 years time I can just recompile my

Re: Microprocessor Optimization Primer

Charles > > -Original Message- > From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On > Behalf Of David Crayford > Sent: Wednesday, March 30, 2016 5:20 PM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: Re: Microprocessor Optimization Primer > > Readi

Re: Microprocessor Optimization Primer

On 3/30/2016 5:19 PM, David Crayford wrote: Reading that book reinforced my opinion that I would rather leave micro optimizations to a compiler. How can you optimize for hardware that hasn't been invented yet? In 5 years time I can just recompile my code for the new architecture in assembly

Re: Microprocessor Optimization Primer

: Microprocessor Optimization Primer Reading that book reinforced my opinion that I would rather leave micro optimizations to a compiler. How can you optimize for hardware that hasn't been invented yet? In 5 years time I can just recompile my code for the new architecture in assembly you need

Re: Microprocessor Optimization Primer

ssage- > From: IBM Mainframe Discussion List [mailto:IBM-MAIN@LISTSERV.UA.EDU] On > Behalf Of Jim Mulder > Sent: Wednesday, March 30, 2016 12:35 PM > To: IBM-MAIN@LISTSERV.UA.EDU > Subject: Microprocessor Optimization Primer > > This may be of interest to some of you. > &g

Re: Microprocessor Optimization Primer

Optimization Primer This may be of interest to some of you. https://www.ibm.com/developerworks/community/forums/html/topic?id=5cf34211-c8e6-4747-a8c2-f8ff7379150b Jim Mulder z/OS System Test IBM Corp. Poughkeepsie, NY -- This message and any attachments are intended only for the use

Re: Microprocessor Optimization Primer

30, 2016 12:35 PM To: IBM-MAIN@LISTSERV.UA.EDU Subject: Microprocessor Optimization Primer This may be of interest to some of you. https://www.ibm.com/developerworks/community/forums/html/topic?id=5cf34211-c 8e6-4747-a8c2-f8ff7379150b ---

Microprocessor Optimization Primer

This may be of interest to some of you. https://www.ibm.com/developerworks/community/forums/html/topic?id=5cf34211-c8e6-4747-a8c2-f8ff7379150b Jim Mulder z/OS System Test IBM Corp. Poughkeepsie, NY -- For IBM-MAIN