Re: Interface to query length of storage allocated with CEEGTST LE service

2016-07-23 Thread Bernd Oppolzer 

Besides all the things that Bill mentioned, which as far as I see can
be activated by LE run time options, there is one additional technique
which requires some input into the ENV string (environment), which is
also a runtime parameter; you can replace the normal LE heap manager
by the so called "heap manager for memory checks"; its module name
is CEL4MCHK.

This heap manager does some bookkeeping of all storage allocation requests
together with the call stack that was active at the time of the allocation
and informs you at the end of the process (when the LE enclave stop) about
all storage areas that have been allocated but not freed. This can be 
very helpful

if your application consumes storage over time and gets in trouble because
storage is never freed (for example in an online region or in a stored 
procedure
which stays resident etc.; but even batch programs with large input may 
have

a problem).

I detected and solved a lot of storage problems in applications at several
customers' sites of mine, using this technique. You will need some 
program that

compresses the generated reports and collects the (many) allocations which
have the same origin and counts them, so that you are able to recognize
the critical places where the allocations come from. I've done this, 
using a

REXX program which captures the output from CEL4MCHK in some large stems
and then does some analyzing on them (which areas remain "unfreed" most 
often,

which have the largest sizes etc.).

See the book "LE Vendor Interfaces" for details; there is a (short) 
description
of CEL4MCHK and how to use it. If you need more information (or 
consulting),

contact me offline.

Like with the other techniques: don't use this in production; it will 
significantly
slow you application (of course) because of the bookkeeping activities 
at every

single storage allocation.

Very important: I discovered recently, that the parameter ALL31(ON) is 
needed
in the LE parms, otherwise the LE enclave will start looping already at 
startup

(this was a problem which I first had to solve, before I could use CEL4MCHK
at my actual customer ... no hint in the manuals, but with ALL31(ON), all
worked well).

Kind regards

Bernd


Am 23.07.2016 um 17:14 schrieb Bill Woodger:

Here's the link I failed to include, the suggested reading: 
http://www.ibm.com/support/knowledgecenter/SSLTBW_2.1.0/com.ibm.zos.v2r1.ceeam00/clsthp.htm

An overview, but, for the willing reader, follow the references and go as deep 
as you want.

LE can report on storage used, and storage (like heap) definitions can be tuned 
from the reports. You don't (you can, occasionally, but probably mostly not at 
all) run in Production with the reporting on, due to overheads. It is an LE 
run-time option, so no recompiles or anything needed.

You can use the report to make the initial heap allocation sufficient for the 
enclave. Then you only get one OS request to get the storage, and LE managing 
the storage thus acquired. When servicing more complex memory requirements 
(non-batch, large amounts of memory, need for release) the definitions can be 
more complex.

If you have some heap which gets corrupted, you can get LE to check everything 
as it goes along, and tell you as soon as something goes awry. Effective way of 
finding out around about where it gets tangled. You do this only when you know 
you already have a problem. It heavily impacts performance of heap usage, 
checking each reference made to heap storage to confirm all is still consistent 
for LE. Again LE run-time option.

Don't use all this type of stuff regularly, just because you can. These things 
are not proactive monitoring tools, but analysis (storage report) and 
determination (heap busted) tools. Know what is available, so you know what to 
use when/if needed.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Interface to query length of storage allocated with CEEGTST LE service

2016-07-23 Thread Bill Woodger 
Here's the link I failed to include, the suggested reading: 
http://www.ibm.com/support/knowledgecenter/SSLTBW_2.1.0/com.ibm.zos.v2r1.ceeam00/clsthp.htm

An overview, but, for the willing reader, follow the references and go as deep 
as you want.

LE can report on storage used, and storage (like heap) definitions can be tuned 
from the reports. You don't (you can, occasionally, but probably mostly not at 
all) run in Production with the reporting on, due to overheads. It is an LE 
run-time option, so no recompiles or anything needed.

You can use the report to make the initial heap allocation sufficient for the 
enclave. Then you only get one OS request to get the storage, and LE managing 
the storage thus acquired. When servicing more complex memory requirements 
(non-batch, large amounts of memory, need for release) the definitions can be 
more complex.

If you have some heap which gets corrupted, you can get LE to check everything 
as it goes along, and tell you as soon as something goes awry. Effective way of 
finding out around about where it gets tangled. You do this only when you know 
you already have a problem. It heavily impacts performance of heap usage, 
checking each reference made to heap storage to confirm all is still consistent 
for LE. Again LE run-time option.

Don't use all this type of stuff regularly, just because you can. These things 
are not proactive monitoring tools, but analysis (storage report) and 
determination (heap busted) tools. Know what is available, so you know what to 
use when/if needed.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

AW: Interface to query length of storage allocated with CEEGTST LE service

2016-07-23 Thread Peter Hunkeler 
Bill,thanks for your answer. I concur with everything you wrote. I don't know 
what they intended to do with this if it was available. I doubt they want to do 
their own storage management. I seem to remember they talked about "statistics" 
or "bookkeeping", but I'll have to ask for more details.
Anyway, I will not suggest to learn this information by scanning undocumented 
control blocks. It is too dangerous for production use. They will need to keep 
that information, if realLy required, in their own data. They already keep all 
kinds of things around.


--
Peter Hunkeler


 Von: Bill Woodger  An:   
IBM-MAIN@LISTSERV.UA.EDU Betreff: Interface to query length of storage 
allocated with CEEGTST LE service Datum: 23.07.16, 12:47

 
There is some evidence that the "control information" is stored in the heap 
itself (see the Usage Notes for "CEEFRST—Free heap storage")  and by 
implication associated directly with the storage allocated. However, it is not 
documented, and working it out through inspection would be pointless beyond 
simple interest, as you know. 
 
I still can't work out *why* it is wanted. I think some detail on that would 
help. Generally it won't. be. Specifically it could be 
 
See if they can read through this and explain LE heaps back to you correctly. 
There may be assumptions being made that "storage management" of some type is 
needed, when probably it isn't. 
 
Note, you could ask for a very small amount of storage and if new storage has 
to be obtained from the OS, an entire "increment" will arrive. So what? Other 
heap storage requests can be served from that. If there is (genuine) concern 
about freeing storage (to the enclave or to the OS) it can be done, and even 
(using FILL) an increment which could otherwise be unavailable due to 
"fragmentation" can be dealt with. If necessary. 
 
For performance *don't release storage to the OS that may be needed again*. 
Unless you really need to. Really need to, means really need to, not just "I 
think it would be good to do that, because I always do that in my smartphone 
app". Don't apply non-z/OS concepts to z/OS. Unless they happen to work, but 
you have to know. You have to know your application, and how z/OS, LE, and 
whatever else, are doing things. 
 
If the code is all C/C++, there is the chance you write your own, presumably 
optimised, heap management - see Vendor Heap Management. You can't do 
everything there, and it is not a way to manage what LE sets up (you have to do 
everything yourself) so doesn't answer the original question by providing some 
"map", but if they really, really need to manager their own storage, it is a 
possibility. 
 
-- 
For IBM-MAIN subscribe / signoff / archive access instructions, 
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Interface to query length of storage allocated with CEEGTST LE service

2016-07-23 Thread Bernd Oppolzer 

I've documented this some time ago for my own use and
for the use of a customer of mine, and it is not very complicated.
But I don't recall the details at the moment.

I wrote a routine, too, that writes a report of all allocated areas,
if you give one valid address to it; that is possible, because all areas
are chained together in some kind of chains of tree structures;
there are different chains for ANYHEAP, BELOWHEAP etc. -
IIRC, the anchors of these chains are in the CEECAA, so maybe
they can even be found without a valid address.

One problem is: there are different heap managers, and you can write
one of your own; for example CEL4MCHK is one that IBM provides
to check for potential memory leak. This one and how to use it is
documented in the Vendor Interface book ... and: maybe other heap
managers have other logic. So I can only describe the NORMAL
LE heap manager as it is at that particular moment (or: as it was in 2013,
because that's when I did this examination).

Contact me offline, if you really need this information
(because it will need some time to find the documents)

Kind regards

Bernd


Am 20.07.2016 um 22:14 schrieb Peter Hunkeler:

When using LE service CEEGTST to allocate a piece of heap storage, the length 
is provided by the caller of the service and the address of the storage is 
returned by the service. To free that storage CEEFRST is called. Only the 
addressof the area, as returned by CEEGTST, has to be passed to the service. LE 
remembers the length of the area.

Is there an interface or any other documented way to get the length of an area 
at a given address?


--
Peter Hunkeler




--
Peter Hunkeler

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN



--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Interface to query length of storage allocated with CEEGTST LE service

2016-07-23 Thread Bill Woodger 
There is some evidence that the "control information" is stored in the heap 
itself (see the Usage Notes for "CEEFRST—Free heap storage")  and by 
implication associated directly with the storage allocated. However, it is not 
documented, and working it out through inspection would be pointless beyond 
simple interest, as you know.

I still can't work out *why* it is wanted. I think some detail on that would 
help. Generally it won't. be. Specifically it could be

See if they can read through this and explain LE heaps back to you correctly. 
There may be assumptions being made that "storage management" of some type is 
needed, when probably it isn't.

Note, you could ask for a very small amount of storage and if new storage has 
to be obtained from the OS, an entire "increment" will arrive. So what? Other 
heap storage requests can be served from that. If there is (genuine) concern 
about freeing storage (to the enclave or to the OS) it can be done, and even 
(using FILL) an increment which could otherwise be unavailable due to 
"fragmentation" can be dealt with. If necessary.

For performance *don't release storage to the OS that may be needed again*. 
Unless you really need to. Really need to, means really need to, not just "I 
think it would be good to do that, because I always do that in my smartphone 
app". Don't apply non-z/OS concepts to z/OS. Unless they happen to work, but 
you have to know. You have to know your application, and how z/OS, LE, and 
whatever else, are doing things.

If the code is all C/C++, there is the chance you write your own, presumably 
optimised, heap management - see Vendor Heap Management. You can't do 
everything there, and it is not a way to manage what LE sets up (you have to do 
everything yourself) so doesn't answer the original question by providing some 
"map", but if they really, really need to manager their own storage, it is a 
possibility.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: AW: Re: Interface to query length of storage allocated with CEEGTST LE service

2016-07-22 Thread Steve Smith 
As far as memory (storage) references go, "Above" and "Below" can easily be
perceived and defended either way, although *I* would usually take "Above"
to mean higher addresses (cf. "above-the-line').  By context, I guess the
opposite was meant, probably by projection of a dump or most memory
displays.  Anyway, the terms, they are ambiguous.

I think "Before" and "After" are much clearer.

sas

On Fri, Jul 22, 2016 at 4:35 PM, J R  wrote:

> Of my vast collection of laptops, I use only the most recent two.  One
> started life under W7, the other under W8.  Of course, MS eventually
> cajoled me into "upgrading" them both to W10!  Their scrolling-by-gesture
> works in opposite directions and I have yet to bother fixing one to match
> the other.  Consequently, it always takes a few swipes to figure which way
> to scroll.
>
> So, it's not just Apple; MS has blindly followed, but not yet
> standardized.  :-(
>
> Sent from my iPhone
>
> > On Jul 22, 2016, at 13:23, Paul Gilmartin <
> 000433f07816-dmarc-requ...@listserv.ua.edu> wrote:
> >
> > Somewhat like ISPF scrolling.  A couple releases ago, Apple reversed the
> conventions
> > of scrolling-by-gesture in OS X (optionally, but default) to be
> compatible with iOS.
> >
> > But still, I wondered if it arose from a desire to page-align the base
> of the area.
> >
> > -- gil
> >
>
> --
> For IBM-MAIN subscribe / signoff / archive access instructions,
> send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
>



-- 
sas

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: AW: Re: Interface to query length of storage allocated with CEEGTST LE service

2016-07-22 Thread Paul Gilmartin 
On Fri, 22 Jul 2016 11:59:33 -0500, Victor Gil wrote:

>No, subtract 8-16 bytes from the area address to get to the header...
>
>I guess my usage of the words "top" and "ABOVE" could be misleading - this is 
>how we were drawing memory maps back in the 70s, so that they grow DOWN from 
>the staring location
> 
And Bill Waite teaching a CS class circa 1972 would sketch a memory map on the
chalkboard and describe it, saying "Up" and pointing to the floor; "Down" and 
pointing
to the ceiling.  Algebra vs. geometry.

Somewhat like ISPF scrolling.  A couple releases ago, Apple reversed the 
conventions
of scrolling-by-gesture in OS X (optionally, but default) to be compatible with 
iOS.

But still, I wondered if it arose from a desire to page-align the base of the 
area.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: AW: Re: Interface to query length of storage allocated with CEEGTST LE service

2016-07-22 Thread Victor Gil 
No, subtract 8-16 bytes from the area address to get to the header...

I guess my usage of the words "top" and "ABOVE" could be misleading - this is 
how we were drawing memory maps back in the 70s, so that they grow DOWN from 
the staring location


On Fri, 22 Jul 2016 08:53:54 -0500, Victor Gil wrote:

>I am guessing the length may be stored somewhere in the vicinity of the area 
>top address, say within 8-16 bytes ABOVE it.
> 
So to get to it you just add the length of the area to the address of the area?

>At least this is how CICS handles GETMAIN requests which also don't require 
>length on the FREEMAINs.
>
>Of course, if the convention is not documented it may change at any time ...

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: AW: Re: Interface to query length of storage allocated with CEEGTST LE service

2016-07-22 Thread Paul Gilmartin 
On Fri, 22 Jul 2016 08:53:54 -0500, Victor Gil wrote:

>I am guessing the length may be stored somewhere in the vicinity of the area 
>top address, say within 8-16 bytes ABOVE it.
> 
So to get to it you just add the length of the area to the address of the area?

>At least this is how CICS handles GETMAIN requests which also don't require 
>length on the FREEMAINs.
>
>Of course, if the convention is not documented it may change at any time ...

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: AW: Re: Interface to query length of storage allocated with CEEGTST LE service

2016-07-22 Thread Victor Gil 
I am guessing the length may be stored somewhere in the vicinity of the area 
top address, say within 8-16 bytes ABOVE it.

At least this is how CICS handles GETMAIN requests which also don't require 
length on the FREEMAINs.

Of course, if the convention is not documented it may change at any time ...

-Victor-

=
> I very much doubt it.


Yeah, me too. But I thought I might be missing something and better ask. 


>I'm curious; what will you do with this info if it is available? 
 

I'm asking because I have been asked by our middle war guys. Our applications 
are required to use our own middle ware service routines for certain functions 
instead of HLL or LE or CICS services directly. (This is not part of the 
discussion.)


They wanted to know if there is a possibility to retrieve the length somehow 
instead of remembering it somewhere in own data structures. (I don't know more 
details about the why and what for.)


Thanks for the hint regarding LE possibly rounding up. Since I do not know 
details what they intend to do, I cannot say if this rouding is important for 
them. But I'll pass it on.




Thanks


-- 
Peter Hunkeler

 


--
Peter Hunkeler 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

AW: Re: Interface to query length of storage allocated with CEEGTST LE service

2016-07-20 Thread Peter Hunkeler 


> I very much doubt it.


Yeah, me too. But I thought I might be missing something and better ask.


>I'm curious; what will you do with this info if it is available?


I'm asking because I have been asked by our middle war guys. Our applications 
are required to use our own middle ware service routines for certain functions 
instead of HLL or LE or CICS services directly. (This is not part of the 
discussion.)


They wanted to know if there is a possibility to retrieve the length somehow 
instead of remembering it somewhere in own data structures. (I don't know more 
details about the why and what for.)


Thanks for the hint regarding LE possibly rounding up. Since I do not know 
details what they intend to do, I cannot say if this rouding is important for 
them. But I'll pass it on.




Thanks


--
Peter Hunkeler




--
Peter Hunkeler

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Interface to query length of storage allocated with CEEGTST LE service

2016-07-20 Thread Paul Gilmartin 
On Wed, 20 Jul 2016 19:34:21 -0400, Tony Harminc wrote:

>On 20 July 2016 at 16:14, Peter Hunkeler  wrote:
>
>> Is there an interface or any other documented way to get the length of an 
>> area at a given address?
>
>I very much doubt it. I don't see anything in the LE Vendor Interfaces
>book. 
>
Mac OS has one, but that's irrelevant; not portable.

>I'm curious; what will you do with this info if it is available?
>
One might use it to decide whether a realloc() is necessary before a strcat().

W3.org uses a couple utilities, StrAllocCopy() and StrAllocCat() to somewhat
automate such management of string storage.

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Re: Interface to query length of storage allocated with CEEGTST LE service

2016-07-20 Thread Tony Harminc 
On 20 July 2016 at 16:14, Peter Hunkeler  wrote:
> When using LE service CEEGTST to allocate a piece of heap storage, the length 
> is provided by the caller of the service and the address of the storage is 
> returned by the service. To free that storage CEEFRST is called. Only the 
> addressof the area, as returned by CEEGTST, has to be passed to the service. 
> LE remembers the length of the area.
>
> Is there an interface or any other documented way to get the length of an 
> area at a given address?

I very much doubt it. I don't see anything in the LE Vendor Interfaces
book. But in any case, would you expect to always see the same length
as what you originally asked for? I believe in all languages supported
by LE and having the notion of "heap", there is nothing to say the
amount allocated cannot exceed the amount asked for. I imagine this is
exactly what will happen; the amount will be rounded up to the nearest
cell size both to avoid fragmentation and to avoid having to keep two
lengths around.

I'm curious; what will you do with this info if it is available?

Tony H.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN

Interface to query length of storage allocated with CEEGTST LE service

2016-07-20 Thread Peter Hunkeler 
When using LE service CEEGTST to allocate a piece of heap storage, the length 
is provided by the caller of the service and the address of the storage is 
returned by the service. To free that storage CEEFRST is called. Only the 
addressof the area, as returned by CEEGTST, has to be passed to the service. LE 
remembers the length of the area.

Is there an interface or any other documented way to get the length of an area 
at a given address?


--
Peter Hunkeler




--
Peter Hunkeler

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN