Re: Safepoint Bean?

[Kirk Pepperdine]

Hi Tony,

A side effect of modularization is that it’s breaking more diagnostic tooling. 
Oh well….

Kind regards,
Kirk


> On Oct 14, 2019, at 8:25 AM, Tony Printezis  wrote:
> 
> Is jvmstat a public / supported API? The jdk.internal.jvmstat module doesn’t 
> seem to be exporting anything publicly (and it also has “internal” in its 
> name).
> 
> Tony
> 
> 
> —
> Tony Printezis | @TonyPrintezis | tprinte...@twitter.com 
> 
> 
> 
> On October 11, 2019 at 11:10:18 PM, Yasumasa Suenaga (suen...@oss.nttdata.com 
> ) wrote:
> 
>> Hi, 
>> 
>> AFAIK the API for them does not provided, but we can use reader class for 
>> hsperfdata 
>> in jdk.internal.jvmstat module. 
>> Examples are available on my GitHub: 
>> 
>> https://github.com/YaSuenag/perfreader 
>>  
>> 
>> You can get safepoint statistics via sun.rt.safepoint* in hsperfdata. 
>> 
>> 
>> Yasumasa 
>> 
>> 
>> On 2019/10/12 10:30, Hohensee, Paul wrote: 
>> > I don’t know of any. Also, it appears that there are no uses of any of the 
>> > HotspotRuntimeMBean methods in the JDK, so it could actually be removed! 
>> > If you want to add its methods to a public interface, I’d create 
>> > com.sun.management.RuntimeMXBean by analogy to c.s.m.ThreadMXBean and use 
>> > the supported/enabled approach of *ThreadAllocatedBytes*. Needs a CSR, of 
>> > course. 
>> >  
>> > Paul 
>> >  
>> > *From: *serviceability-dev > > > on behalf of Tony 
>> > Printezis mailto:tprinte...@twitter.com>> 
>> > *Date: *Friday, October 11, 2019 at 1:45 PM 
>> > *To: *"serviceability-dev@openjdk.java.net 
>> > " 
>> > > > > 
>> > *Subject: *Safepoint Bean? 
>> >  
>> > Hi there, 
>> >  
>> > Is there a standard MBean (similar to GarbageCollectorMXBean), or other 
>> > mechanism, that can be used to get safepoint statistics from Java (count, 
>> > time, etc.)? I know it’s possible to get that info from 
>> > sun.management.HotspotRuntime.java, but I assume this is not a publicly 
>> > accessible API any more? Is there a standard alternative? 
>> >  
>> > Thanks, 
>> >  
>> > Tony 
>> >  
>> > — 
>> >  
>> > Tony Printezis | @TonyPrintezis | tprinte...@twitter.com 
>> >  > > > 
>> > 

Re: Safepoint Bean?

[Kirk Pepperdine]

Hi Tony,

I think delivering this information, available via logging, could be useful. Be 
happy to assist where I can.

Kind regards,
Kirk


> On Oct 14, 2019, at 11:23 AM, Tony Printezis  wrote:
> 
> Hi Mandy,
> 
> Thanks for the response! I hope you’re well!
> 
> We’d like to be able to get safepoint stats from within the JVM so we can 
> post them to our observability system. Would creating a Bean (a la 
> GarbageCollectorMXBean) that exposes this info make sense? I’d be happy to 
> work on it if there’s interest. Would you recommend an alternative method?
> 
> Tony
> 
> 
> —
> Tony Printezis | @TonyPrintezis | tprinte...@twitter.com 
> 
> 
> 
> On October 14, 2019 at 12:49:57 PM, Mandy Chung (mandy.ch...@oracle.com 
> ) wrote:
> 
>> jdk.internal.jvmstat is JDK internal unsupported API.  jstat and jcmd 
>> monitors the JVM statistics that you can use but I think they don't show the 
>> safepoint counters by default.
>> 
>> Mandy
>> 
>> On 10/14/19 8:25 AM, Tony Printezis wrote:
>>> Is jvmstat a public / supported API? The jdk.internal.jvmstat module 
>>> doesn’t seem to be exporting anything publicly (and it also has “internal” 
>>> in its name).
>>> 
>>> Tony
>>> 
>>> 
>>> —
>>> Tony Printezis | @TonyPrintezis | tprinte...@twitter.com 
>>> 
>>> 
>>> 
>>> On October 11, 2019 at 11:10:18 PM, Yasumasa Suenaga 
>>> (suen...@oss.nttdata.com ) wrote:
>>> 
 Hi,
 
 AFAIK the API for them does not provided, but we can use reader class for 
 hsperfdata
 in jdk.internal.jvmstat module.
 Examples are available on my GitHub:
 
 https://github.com/YaSuenag/perfreader 
 
 
 You can get safepoint statistics via sun.rt.safepoint* in hsperfdata.
 
 
 Yasumasa
 
 
 On 2019/10/12 10:30, Hohensee, Paul wrote:
 > I don’t know of any. Also, it appears that there are no uses of any of 
 > the HotspotRuntimeMBean methods in the JDK, so it could actually be 
 > removed! If you want to add its methods to a public interface, I’d 
 > create com.sun.management.RuntimeMXBean by analogy to c.s.m.ThreadMXBean 
 > and use the supported/enabled approach of *ThreadAllocatedBytes*. Needs 
 > a CSR, of course.
 >
 > Paul
 >
 > *From: *serviceability-dev >>> > > on behalf of Tony 
 > Printezis mailto:tprinte...@twitter.com>>
 > *Date: *Friday, October 11, 2019 at 1:45 PM
 > *To: *"serviceability-dev@openjdk.java.net 
 > " 
 > >>> > >
 > *Subject: *Safepoint Bean?
 >
 > Hi there,
 >
 > Is there a standard MBean (similar to GarbageCollectorMXBean), or other 
 > mechanism, that can be used to get safepoint statistics from Java 
 > (count, time, etc.)? I know it’s possible to get that info from 
 > sun.management.HotspotRuntime.java, but I assume this is not a publicly 
 > accessible API any more? Is there a standard alternative?
 >
 > Thanks,
 >
 > Tony
 >
 > —
 >
 > Tony Printezis | @TonyPrintezis | tprinte...@twitter.com 
 >   >
 >

Re: RFR(XS) 8230674 Heap dumps should exclude dormant CDS archived objects of unloaded classes

[Kirk Pepperdine]

Hi, 

Might I add a diagnostic twist to this request. It is sometimes useful to try 
to determine where unreferenced objects live in heap because this can help you 
solve questions of nepotism.

Kind regards,
Kirk


> On Sep 6, 2019, at 4:06 PM, Jiangli Zhou  wrote:
> 
> On Fri, Sep 6, 2019 at 3:17 PM Ioi Lam  wrote:
>> 
>> On 9/6/19 11:48 AM, Jiangli Zhou wrote:
>>> On Fri, Sep 6, 2019 at 9:43 AM Ioi Lam  wrote:
 
 
 On 9/5/19 11:11 PM, David Holmes wrote:
> On 6/09/2019 1:39 pm, Ioi Lam wrote:
>> On 9/5/19 8:18 PM, David Holmes wrote:
>>> Hi Ioi,
>>> 
>>> On 6/09/2019 12:27 pm, Ioi Lam wrote:
 https://bugs.openjdk.java.net/browse/JDK-8230674
 http://cr.openjdk.java.net/~iklam/jdk14/8230674-heap-dump-exclude-dormant-oops.v01
 
 
 Please review this small fix:
 
 When CDS is in use, archived objects are memory-mapped into the
 heap (currently G1GC only). These objects are partitioned into
 "subgraphs". Some of these subgraphs may not be loaded (e.g., those
 related to jdk.internal.math.FDBigInteger) at the time a heap dump is
 requested. >
 When a subgraph is not loaded, some of the objects in this subgraph
 may belong to a class that's not yet loaded.
 
 The bug happens when such an "dormant" object is dumped, but its class
 is not dumped because the class is not in the system dictionary.
 
 There is already code in DumperSupport::dump_instance() that tries
 to handle dormant objects, but it needs to be extended to cover
 arrays, as well as and references from non-dormant object/arrays to
 dormant ones.
>>> I have to confess I did not pay any attention to the CDS archived
>>> objects work, so I don't have a firm grasp of how you have
>>> implemented things. But I'm wondering how can you have a reference
>>> to a dormant object from a non-dormant one? Shouldn't the act of
>>> becoming non-dormant automatically cause the subgraph from that
>>> object to also become non-dormant? Or do you have "read barriers" to
>>> perform the changes on demand?
>>> 
 Ah -- my bug title is not correct.
 
 I changed the bug title (and this e-mail subject) to
 
 Heap dumps should exclude dormant CDS archived objects **of unloaded
 classes**
 
 During the heap dump, we scan all objects in the heap, regardless of
 reachability. There's no way to decide reachability in
 HeapObjectDumper::do_object(), unless we perform an actual GC.
 
 But it's OK to include unreachable objects in the heap dump. (I guess
 it's useful to see how much garbage you have in the heap. There's an
 option to run a collection before dumping the heap.)
 
 There are 2 kinds of unreachable objects -- garbage: those that were
 once reachable but no longer, dormant: the archived objects that have
 never been reachable.
>>> Currently Java object archiving framework only supports one
>>> directional state change: dormant -> live. An archived object can
>>> become a live object from dormant state, but it cannot go back to the
>>> dormant state. Need to investigate thoroughly for all cases before the
>>> 'live -> dormant' transition can be supported. All objects in the
>>> 'Open' archive heap region are associated with the builtin class
>>> loaders and their classes are not unloaded. The existing static fields
>>> for archiving within the JDK classes are selected and the associated
>>> objects do not become garbage once 'installed'.
>>> 
 Anyway, it's OK to dump dormant objects as long as their class has been
 loaded. The problem happens only when we dump a dormant object who class
 is not yet loaded (Eclipase MAT get confused when it sees an object
 whose class ID is invalid).
>>> Yes. That's a scenario needs to be handled for a tool that iterates
>>> the Java heap. A dormant object in the 'Open' archive heap region may
>>> have a 'invalid' klass since the klass may not be loaded yet at the
>>> moment.
>>> 
>>> Your webrev looks reasonable to me on high level pending information
>>> for following questions. Can you please give more details on the
>>> dormant objects referenced from the arrays? What specific arrays are
>>> those?
>> Hi Jiangli,
>> 
>> Thanks for the review. I add the following code:
>> 
>>   // [id]* elements
>>   for (int index = 0; index < length; index++) {
>> oop o = array->obj_at(index);
 
>> if (o != NULL && mask_dormant_archived_object(o) == NULL) {
>>   ResourceMark rm;
>>   tty->print_cr("%s array contains %s object",
>> array->klass()->external_name(), o->klass()->external_name());
>> }
 
>> o = mask_dormant_archived_object(o);
>> writer->write_objectID(o);
>>   }
>> 
>> and the output is:
>> 
>> $ java -cp .  -XX:+HeapDumpAfterFullGC HelloGC
>> Dumping heap to java_pid20956.hprof ...
>> 

Re: JFR thread sampling mechanism

[Kirk Pepperdine]

Hi Gil,

I would support an improvement in sampling as there is an obvious bias which 
allows me to write benchmarks where JFR completely misses things it should 
find. That said, I’m not sure that waking a thread up every 10ms is a great 
idea as it is very disruptive to Linux thread scheduling. I’d very much like to 
experiment with lower sampling rates.

Kind regards,
Kirk


> On Jun 30, 2019, at 9:20 AM, Gil Tene  wrote:
> 
> I would like to discuss a potential improvement to the JFR thread
> sampling mechanism, and would like to see if the change we'd
> propose has already been considered in the past.
> 
> I believe that the current thread sampling mechanism (mostly via
> hotspot/share/jfr/periodic/sampling/jfrThreadSampler.cpp) can be
> summarized as: A control thread wakes up periodically (e.g. 100
> times per second) and in each period chooses a number (e.g. 5)
> threads to sample (by rotating through the overall list of threads)
> only if they are "in java", and a number (e.g. 1) threads (by
> separately rotating through the overall list of threads) to sample
> "only if it is in native". For each thread targeted to sample, the
> control thread suspends the target thread (e.g. for linux this is
> done by preparing a suspend request a sending a SIGUSR2 to
> make the thread deal with it), takes a stacktrace of the suspended
> thread, adds the stacktrace to JfrStackTraceRepository, and
> resumes the thread (e.g. on linux resumption involves setting up
> a resume request and again sending a SIGUSR2 to the thread to
> get it to handle it and resume).
> 
> We've been contemplating a change to make thread sampling use
> Posix timers instead, such that each thread would use a separate
> timer, and threads would receive signals based on their CPU
> consumption (the timer, e.g. created with timer_create(2), would
> be clocked by the thread CPU time of their associated threads,
> and signal their threads when that CPU time reaches a level
> [of e.g. 10 msec]). The signal handler will then perform the
> stacktrace sampling locally on the  thread, and deliver the
> stacktrace to JfrStackTraceRepository (either directly or by
> enqueing through an intermediary).
> 
> There are multiple potential benefits that may arise from switching
> to such a scheme, including significant reduction of sampling cost,
> improvement of density and focus of samples (fewer lost samples,
> ensuring that enough activity in a given thread will end up leading
> to a sample for that thread, etc.), and, potentially, an ability to
> (with additional  changes) better account for time spent "outside
> of java" in e.g. native and runtime code.
> 
> Has this (using thread-cpu-time-based posix timer sampling) been
> considered before?
> 

Re: OperatingSystemMXBean unaware of container memory limits

[Kirk Pepperdine]

Hi Mario,

I completely agree, the OS Bean should be consistent with the memory. That said 
I favor introducing new MXBeans to achieve the desired effects. If there is no 
container than the container MXBean could simply not be instantiated.

I’m looking for more visibility especially for the purposes of supporting 
diagnostic tooling. If visibility is shut down it’s just makes it that much 
harder to perform any reasonable level of high level diagnostics.

Kind regards,
Kirk


> On Jun 21, 2019, at 5:27 AM, Mario Torre  
> wrote:
> 
> Hi Kirk,
> 
> I think I understand what you mean, however then the OS Bean should be 
> consistent regarding CPU information as well.
> 
> I think I remember why this was fixed the way it is now was because of 
> incorrect behavior during GC configuration, or something along those lines, 
> so I guess we would need two APIs after all to give tooling a way to sneak 
> into the actual hardware properties.
> 
> I would guess, however, that from the point of view of a containerised VM its 
> OS is the container not the bare metal (or virtualized metal perhaps), so 
> tooling would need to check specifically for a separate bean.
> 
> That could be indicated via a property in the OS Bean (if it’s not the case 
> already).
> 
> Nevertheless, I think consistency in the OS Bean should be achieved.
> 
> Cheers,
> Mario 
> 
> On Fri 21. Jun 2019 at 13:23, Kirk Pepperdine  <mailto:kirk.pepperd...@gmail.com>> wrote:
> Hi Mario,
> 
> I don’t believe the MBean returns the wrong information. Is it not that the 
> calls by-pass the container? Would it not be more appropriate to add a 
> container aware mean? From a tooling perspective it’s a mistake to completely 
> seal the JVM away from the hardware as it makes certain diagnostics more 
> difficult to perform.
> 
> Kind regards,
> Kirk
> 
> 
>> On Jun 21, 2019, at 6:06 AM, Mario Torre > <mailto:neugens.limasoftw...@gmail.com>> wrote:
>> 
>> The way I understood the bug report is a two fold approach, i.e. fix the os 
>> bean and *possibly* add a container one or extend it to add more data.
>> 
>> I agree with you and Andrew that the current OS Bean returns wrong 
>> information, this should be fixed in any case I think.
>> 
>> Bob, do you have some design ideas to share regarding the new interface? I 
>> think this may be an interesting project to pick up, even for students 
>> wanting to write a thesis, as it seems time is a limiting factor here for 
>> you to work on that?
>> 
>> Cheers,
>> Mario
>> 
>> On Fri 21. Jun 2019 at 10:53, Severin Gehwolf > <mailto:sgehw...@redhat.com>> wrote:
>> Hi Bob,
>> 
>> On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
>> > Hi Andrew,
>> > 
>> > I am aware of the limitations of the OperatingSystemMXBean and was
>> > hoping to address these limitations during the implementation of
>> > https://bugs.openjdk.java.net/browse/JDK-8199944 
>> > <https://bugs.openjdk.java.net/browse/JDK-8199944>.
>> > 
>> > It would be helpful if you feel this is important to add some votes
>> > to this issue.
>> 
>> It seems strange that the getAvailableProcessors() returns the
>> container limit, while the memory limits are for the physical host. If
>> anything, shouldn't they agree (both physical host or both container
>> limits)?
>> 
>> When I briefly looked into it initially it seems to be a side-effect of
>> what is being used by the JDK code implementation-wise. IIRC
>> getAvailableProcessors() uses a runtime call. Memory reporting has it's
>> own logic[1], thus not reporting the container limit.
>> 
>> This seems weird from a consistency perspective. Thoughts?
>> 
>> If I understand you correctly, you are arguing that container reporting
>> should go into its own MX bean. On the other hand, CPU reporting for
>> the OS MX bean is container aware already. That makes me believe we
>> should rather make this consistent before evaluating a new MX bean.
>> 
>> Thanks,
>> Severin
>> 
>> [1] 
>> http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365
>>  
>> <http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365>
>> 
>> 
>> > Bob.
>> > 
>> > 
>> > > On Jun 20, 2019, at 9:43 AM, Andrew Azores > > > <mailto:aazo...@redhat.com>>
>> > > wrote:
>> > > 
>> > > Hi all,
&

Re: OperatingSystemMXBean unaware of container memory limits

[Kirk Pepperdine]

Hi Bob,

100% isolate reduces visibility and from a diagnostic perspective there are a 
number of issues that can only be seen if you have access to the host 
information.

Also one of the things that gets forgotten is how this change will affect or 
break tooling. I would suggest a better solution would be to introduce a new 
MXBean that was “container aware”.

Kind regards,
Kikr Pepperdine


> On Jun 21, 2019, at 5:40 AM, Bob Vandette  wrote:
> 
> I agree with you Mario.  When I originally designed the 
> jdk.internal.platform.Metrics
> API, I was considering providing both a Host and Container Metrics 
> implementation.
> The problem is that the primary goal of containers is to provide isolation 
> (hiding access to
> the host) and it would be very difficult to provide reliable forms of both 
> metrics from within
> a container.  The Linux kernel does leak some host metric information but not 
> everything.
> This is why I decided to focus on the container configuration and metric 
> data.  If Host
> metrics are of interest you can always just run Java outside of a container.
> 
> Bob.
> 
>> On Jun 21, 2019, at 8:27 AM, Mario Torre > <mailto:neugens.limasoftw...@gmail.com>> wrote:
>> 
>> Hi Kirk,
>> 
>> I think I understand what you mean, however then the OS Bean should be 
>> consistent regarding CPU information as well.
>> 
>> I think I remember why this was fixed the way it is now was because of 
>> incorrect behavior during GC configuration, or something along those lines, 
>> so I guess we would need two APIs after all to give tooling a way to sneak 
>> into the actual hardware properties.
>> 
>> I would guess, however, that from the point of view of a containerised VM 
>> its OS is the container not the bare metal (or virtualized metal perhaps), 
>> so tooling would need to check specifically for a separate bean.
>> 
>> That could be indicated via a property in the OS Bean (if it’s not the case 
>> already).
>> 
>> Nevertheless, I think consistency in the OS Bean should be achieved.
>> 
>> Cheers,
>> Mario 
>> 
>> On Fri 21. Jun 2019 at 13:23, Kirk Pepperdine > <mailto:kirk.pepperd...@gmail.com>> wrote:
>> Hi Mario,
>> 
>> I don’t believe the MBean returns the wrong information. Is it not that the 
>> calls by-pass the container? Would it not be more appropriate to add a 
>> container aware mean? From a tooling perspective it’s a mistake to 
>> completely seal the JVM away from the hardware as it makes certain 
>> diagnostics more difficult to perform.
>> 
>> Kind regards,
>> Kirk
>> 
>> 
>>> On Jun 21, 2019, at 6:06 AM, Mario Torre >> <mailto:neugens.limasoftw...@gmail.com>> wrote:
>>> 
>>> The way I understood the bug report is a two fold approach, i.e. fix the os 
>>> bean and *possibly* add a container one or extend it to add more data.
>>> 
>>> I agree with you and Andrew that the current OS Bean returns wrong 
>>> information, this should be fixed in any case I think.
>>> 
>>> Bob, do you have some design ideas to share regarding the new interface? I 
>>> think this may be an interesting project to pick up, even for students 
>>> wanting to write a thesis, as it seems time is a limiting factor here for 
>>> you to work on that?
>>> 
>>> Cheers,
>>> Mario
>>> 
>>> On Fri 21. Jun 2019 at 10:53, Severin Gehwolf >> <mailto:sgehw...@redhat.com>> wrote:
>>> Hi Bob,
>>> 
>>> On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
>>> > Hi Andrew,
>>> > 
>>> > I am aware of the limitations of the OperatingSystemMXBean and was
>>> > hoping to address these limitations during the implementation of
>>> > https://bugs.openjdk.java.net/browse/JDK-8199944 
>>> > <https://bugs.openjdk.java.net/browse/JDK-8199944>.
>>> > 
>>> > It would be helpful if you feel this is important to add some votes
>>> > to this issue.
>>> 
>>> It seems strange that the getAvailableProcessors() returns the
>>> container limit, while the memory limits are for the physical host. If
>>> anything, shouldn't they agree (both physical host or both container
>>> limits)?
>>> 
>>> When I briefly looked into it initially it seems to be a side-effect of
>>> what is being used by the JDK code implementation-wise. IIRC
>>> getAvailableProcessors() uses a runtime call. Memory reporting has it's
>>> own logic[1], thus not reporting the conta

Re: OperatingSystemMXBean unaware of container memory limits

[Kirk Pepperdine]

Hi Mario,

I don’t believe the MBean returns the wrong information. Is it not that the 
calls by-pass the container? Would it not be more appropriate to add a 
container aware mean? From a tooling perspective it’s a mistake to completely 
seal the JVM away from the hardware as it makes certain diagnostics more 
difficult to perform.

Kind regards,
Kirk


> On Jun 21, 2019, at 6:06 AM, Mario Torre  
> wrote:
> 
> The way I understood the bug report is a two fold approach, i.e. fix the os 
> bean and *possibly* add a container one or extend it to add more data.
> 
> I agree with you and Andrew that the current OS Bean returns wrong 
> information, this should be fixed in any case I think.
> 
> Bob, do you have some design ideas to share regarding the new interface? I 
> think this may be an interesting project to pick up, even for students 
> wanting to write a thesis, as it seems time is a limiting factor here for you 
> to work on that?
> 
> Cheers,
> Mario
> 
> On Fri 21. Jun 2019 at 10:53, Severin Gehwolf  > wrote:
> Hi Bob,
> 
> On Thu, 2019-06-20 at 10:16 -0400, Bob Vandette wrote:
> > Hi Andrew,
> > 
> > I am aware of the limitations of the OperatingSystemMXBean and was
> > hoping to address these limitations during the implementation of
> > https://bugs.openjdk.java.net/browse/JDK-8199944 
> > .
> > 
> > It would be helpful if you feel this is important to add some votes
> > to this issue.
> 
> It seems strange that the getAvailableProcessors() returns the
> container limit, while the memory limits are for the physical host. If
> anything, shouldn't they agree (both physical host or both container
> limits)?
> 
> When I briefly looked into it initially it seems to be a side-effect of
> what is being used by the JDK code implementation-wise. IIRC
> getAvailableProcessors() uses a runtime call. Memory reporting has it's
> own logic[1], thus not reporting the container limit.
> 
> This seems weird from a consistency perspective. Thoughts?
> 
> If I understand you correctly, you are arguing that container reporting
> should go into its own MX bean. On the other hand, CPU reporting for
> the OS MX bean is container aware already. That makes me believe we
> should rather make this consistent before evaluating a new MX bean.
> 
> Thanks,
> Severin
> 
> [1] 
> http://hg.openjdk.java.net/jdk/jdk/file/1c242c2d037f/src/jdk.management/unix/native/libmanagement_ext/OperatingSystemImpl.c#l365
>  
> 
> 
> 
> > Bob.
> > 
> > 
> > > On Jun 20, 2019, at 9:43 AM, Andrew Azores  > > >
> > > wrote:
> > > 
> > > Hi all,
> > > 
> > > Apologies if this is not the most appropriate list, in which case
> > > please direct me where to go.
> > > 
> > > I've noticed a surprising result from the
> > > com.sun.management.OperatingSystemMXBean implementation when
> > > running in
> > > a containerized (specifically, using Docker on Linux) environment.
> > > The
> > > bean appears to be container-aware for processors, in that running
> > > with
> > > Docker option `--cpus 1.0` for example, on a multicore system, will
> > > cause both java.lang.Runtime#availableProcessors and
> > > java.lang.management.OperatingSystemMXBean#getAvailableProcessors /
> > > com.sun.management.OperatingSystemMXBean#getAvailableProcessors to
> > > return 1. However, the Docker option `--memory 100M` (or any other
> > > limit value) is not reflected in the value returned by
> > > com.sun.management.OperatingSystemMXBeam#getTotalPhysicalMemorySize
> > > ,
> > > and instead the returned value is still the total physical memory
> > > of
> > > the host machine - of which only a small portion may actually be
> > > available to the "Operating System" of the JVM. Similarly for the
> > > methods regarding free physical memory, total swap, and free swap.
> > > 
> > > I have attached a patch which adds a small reproducer to the
> > > existing
> > > MemoryAwareness test.
> > > 
> > > This seems like a bug to me, since if the imposed container limit
> > > on
> > > processors as a resource is included as part of the "Operating
> > > System"
> > > resource reporting, then surely memory resources should be reported
> > > the
> > > same way. As I said, I found the current behaviour quite
> > > surprising.
> > > 
> > > -- 
> > > Andrew Azores
> > > Software Engineer, OpenJDK Team
> > > Red Hat
> > > 
> > 
> > 
> 
> -- 
> pgp key: http://subkeys.pgp.net/  PGP Key ID: 
> 80F240CF
> Fingerprint: BA39 9666 94EC 8B73 27FA  FC7C 4086 63E3 80F2 40CF
> 
> Java Champion - Blog: http://neugens.wordpress.com 
>  - Twitter: @neugens
> Proud GNU Classpath developer: http://www.classpath.org/ 
> 
> OpenJDK: http://openjdk.java.net/projects/caciocavallo/ 
> 

Re: Proposal: Always-on Statistical History

[Kirk Pepperdine]

This was true up until Oracle open sourced it (JDK 11).

If JFR is the framework around which we decide to get these types of metrics 
from the JVM (in addition to JMX),  then I think that we (the community) should 
continue to build out JFR adding in those metrics that are not already captured.

Kind regards,
Kirk Pepperdine


> On Nov 15, 2018, at 9:07 AM, Simon Roberts  
> wrote:
> 
> I don't begin to claim to know the politics, legalities, boundaries of JFR 
> license conditionsm and so forth" but:
> 
> Java Flight Recorder requires a commercial license for use in production."
> 
> Whereas, this as I understand is the *open* jdk list. So, I for one would 
> feel hard done by if your view prevailed and only the paying clients got 
> access to a valuable feature.
> 
> 
> On Thu, Nov 15, 2018 at 9:40 AM Roger Riggs  <mailto:roger.ri...@oracle.com>> wrote:
> Hi,
> 
> This looks like it has significant overlap with JFR.
> I don't think we want to start building in multiple mechanisms to keep 
> tabs on a running VM.
> 
> $.02, Roger
> 
> 
> On 11/14/2018 04:27 PM, Thomas Stüfe wrote:
> > Hi Bernd,
> >
> > On Wed, Nov 14, 2018 at 10:07 PM Bernd Eckenfels  > <mailto:e...@zusammenkunft.net>> wrote:
> >> Looks good Thomas,
> > thanks!
> >
> >> what would be the typical memory usage with the Default Settings?
> > ~ 80 Kb. Its very small.
> >
> >> Does the downsampling support min/max style rollups?
> > Not sure what you mean. Do you mean does it preserve peaks? Not yet,
> > such a feature would have to be added.
> >
> > Right now, downsampling is very primitive for performance reasons. For
> > snapshot values like heap size etc we just throw away the samples, so
> > you loose temporary peaks. For counter-like values-over-time (e.g.
> > number of pages swapped in etc), they just refer then to a larger time
> > span.
> >
> > Best Regards, Thomas
> >
> >>
> >>
> >> --
> >> http://bernd.eckenfels.net <http://bernd.eckenfels.net/>
> >>
> >>
> >>
> >> Von: Thomas Stüfe
> >> Gesendet: Mittwoch, 14. November 2018 16:29
> >> An: serviceability-dev@openjdk.java.net 
> >> <mailto:serviceability-dev@openjdk.java.net> 
> >> serviceability-dev@openjdk.java.net 
> >> <mailto:serviceability-dev@openjdk.java.net>
> >> Betreff: Proposal: Always-on Statistical History
> >>
> >>
> >>
> >> Hi all,
> >>
> >>
> >>
> >> We have that feature in our port which we would like to contribute,
> >>
> >> and I would like to gauge opinions.
> >>
> >>
> >>
> >> First off, I am not sure which list is correct. This is more of a
> >>
> >> serviceability issue, but implementation wise it fit hs-runtime
> >>
> >> better. I'll start with serviceability, but feel free crosspost if
> >>
> >> needed.
> >>
> >>
> >>
> >> Second, I am aware that this may require a JEP. If necessary and the
> >>
> >> feedback is positive, I will draft one.
> >>
> >>
> >>
> >> 
> >>
> >>
> >>
> >> In our port we have something called "Statistics History". Basically
> >>
> >> this is a rolling history, spanning up to 10 days, of a number of key
> >>
> >> values. Key values range from JVM specifics like heap size, metaspace
> >>
> >> size, number of threads etc, to platform specifics like memory
> >>
> >> footprint, cpu load, io- and swapping activity etc.
> >>
> >>
> >>
> >> A periodic tasks collects those values, in - by default - 15 second
> >>
> >> intervals. They are then fed into a FIFO. FIFO spans 10 days. To save
> >>
> >> memory that FIFO is downsampled in two steps, so we have the last n
> >>
> >> hours in high resolution and the last n days in low resolution (of
> >>
> >> course all these parameters are configurable).
> >>
> >>
> >>
> >> The history report can be triggered via jcmd, and also could get
> >>
> >> printed in the hs.err file (open for debate).
> >>
> >>
> >>
> >> ---
> >>
> >>
> >>
> >> Here some examples of how the whole thing looks like:
> >>
> >>
> >>
> >> http://cr.openjdk.java.net/~stuefe/webrevs/stathist/examples/s

Re: Proposal: Always-on Statistical History

[Kirk Pepperdine]

Hi,

I agree, this could be very useful…

— Kirk


> On Nov 14, 2018, at 10:29 AM, Simon Roberts  
> wrote:
> 
> I would say this could be pretty useful. It's almost like a 
> platform-independent, process specific vmstat, with JVM extras. Given the 
> existence of jps, this seems to fit in that ecosystem well. I find myself 
> having to work with windows just rarely enough that I'd have to look up how 
> to get this info on that host every time.
> $0.02
> 
> 
> On Wed, Nov 14, 2018 at 7:57 AM Thomas Stüfe  > wrote:
> Hi all,
> 
> We have that feature in our port which we would like to contribute,
> and I would like to gauge opinions.
> 
> First off, I am not sure which list is correct. This is more of a
> serviceability issue, but implementation wise it fit hs-runtime
> better. I'll start with serviceability, but feel free crosspost if
> needed.
> 
> Second, I am aware that this may require a JEP. If necessary and the
> feedback is positive, I will draft one.
> 
> 
> 
> In our port we have something called "Statistics History". Basically
> this is a rolling history, spanning up to 10 days, of a number of key
> values. Key values range from JVM specifics like heap size, metaspace
> size, number of threads etc, to platform specifics like memory
> footprint, cpu load, io- and swapping activity etc.
> 
> A periodic tasks collects those values, in - by default - 15 second
> intervals. They are then fed into a FIFO. FIFO spans 10 days. To save
> memory that FIFO is downsampled in two steps, so we have the last n
> hours in high resolution and the last n days in low resolution (of
> course all these parameters are configurable).
> 
> The history report can be triggered via jcmd, and also could get
> printed in the hs.err file (open for debate).
> 
> ---
> 
> Here some examples of how the whole thing looks like:
> 
> http://cr.openjdk.java.net/~stuefe/webrevs/stathist/examples/stathist-volker.txt
>  
> 
> 
> http://cr.openjdk.java.net/~stuefe/webrevs/stathist/examples/stathist-s390x.txt
>  
> 
> 
> ---
> 
> This feature has been really popular with our support folk over the
> years. Be it that the VM is starved for resources by the OS, that we
> have some slow- or fast developing leak situation etc: these values
> are a first and easy way to get a first stab at a situation, before we
> start more expensive analysis.
> 
> The explicit design goal of this history was to be very cheap - cheap
> enough to be *always on* and getting forgotten. It is, in our port,
> enabled by default. That way, if a problem occurs at a customer site,
> we immediately see developments spanning the last 10 days, without
> having to reproduce the issue.
> 
> It is also robust enough to be usable during error reporting without
> endangering the error reporting process or falsifying the picture.
> 
> I am aware that this crosses over into JFR territory. But this feature
> does not attempt to replace JFR, it is intended instead a cheap always
> on first stop historical overview.
> 
> --
> 
> I have a patch which can be applied atop of jdk12:
> 
> http://cr.openjdk.java.net/~stuefe/webrevs/stathist/stathist.patch 
> 
> 
> It works, passes our nightlies and no regressions are shown in dapapo
> benchmarks.
> 
> Please tell me what you think. Given enough interest, I will attempt
> to contribute (drafting a JEP if necessary.)
> 
> Thanks and Kind Regards,
> 
> Thomas
> 
> 
> -- 
> Simon Roberts
> (303) 249 3613
> 

Re: hprof format question

[Kirk Pepperdine]

Hi Simon,

I’ve also started a small project to try and solve the we need to look at very 
large heap problem. My solution is to load the data into Neo4J. You can find 
the project on my GitHub account.

So, I believe I’ve taken the same tactic in just abandoning the segment for the 
moment. It would be useful to sort that out but I listed it as a future…

Kind regards,
Kirk
L
> On Oct 31, 2018, at 4:07 AM, Simon Roberts  
> wrote:
> 
> Hi all, I'm hoping this is the correct list for a question on the hprof file 
> format (1.0.2)?
> 
> I found this information: 
> http://hg.openjdk.java.net/jdk6/jdk6/jdk/raw-file/tip/src/share/demo/jvmti/hprof/manual.html
>  
> 
> 
> and have been working on a small project to read these files. (Yes, I know 
> that NetBeans/VisualVM and Eclipse both have such libraries, and a number of 
> other tools have been derived from those, but so far as I can tell, they all 
> are fundamentally built on the notion of fully decoding everything, and 
> creating memory representations of the entire heap. I want to pull out only 
> certain pieces of information--specifically object counts by class--from a 
> large, ~20Gb, dump file, and those tools just give up the ghost on my 
> systems.)
> 
> Anyway, my code reads the file pretty well so far, except that the file I 
> want to analyze seems to contradict the specifications of the document 
> mentioned above. Specifically, after processing about five HEAP_DUMP_SEGMENTS 
> with around 1.5 million sub-records in each, I come across some 
> ROOT_JNI_LOCAL records. The first 15 follow the format specified in the above 
> document (one 8 byte "ID" and two four byte values.) But the 16th omits the 
> two four byte records (well, it might simply have more, but visual analysis 
> shows that after the 8 byte ID, I have a new block tag, and a believable 
> structure. I've actually noticed that several of the record types defined in 
> this "group" seem to diverge from the paper I mentioned.
> 
> My solution is that if my parser trips, it abandons that HEAP_DUMP_SEGMENT 
> from that point forward. It doesn't seem to matter much, since I was looking 
> for object data, and it appears that all of that has already been handled. 
> However, clearly this is not ideal!
> 
> Is there any more detailed, newer, better, information? Or anything else I 
> should know to pursue this tool (or indeed a simple object frequency by 
> classname result) from an hprof 1.0.2 format file?
> 
> (And yes, I'm pursuing a putative memory leak :)
> 
> Thanks for any input (including "dude, this is the wrong list!")
> Cheers,
> Simon
> 
> 
> 
> -- 
> Simon Roberts
> (303) 249 3613
> 
> 
> 
> -- 
> Simon Roberts
> (303) 249 3613
> 

Re: Kerberos authentication for JMX?

[Kirk Pepperdine]

Hi Peter,

This is an issue for prod environments that is becoming bigger as clusters 
become bigger and bigger. I believe the answer to your issues and others 
related to the reliance of RMI has been proven by a project call Jolokia 
(https://jolokia.org ) which uses REST. At issue is that 
Jolokia is *not* a drop in JMXConnector replacement meaning you can’t use 
standard client tooling and this unfortunately compromises Jolokia’s 
usefulness. There is a JEP (http://openjdk.java.net/jeps/8171311 
) for providing a REST adaptor that 
unfortunately also misses the mark in that it’s not a JMXConnector. I’m not 
sure *why* these efforts have seemingly avoided the obvious solution which 
would be an REST based implementation of the JMXConnector interface as I 
believe that would be about the same about of work and would allow everyone to 
continue to use already available tooling. I have the task to prototype my own 
implementation running 2rd on my todo list right after I get my heap dump 
analysis tooling functional. So, yes, this is a real issue and I hope a 
discussion will lead to a more scalable solution.

Kind regards,
Kirk

> On Jun 11, 2018, at 4:14 PM, Péter Gergely Horváth 
>  wrote:
> 
> Hi All,
> 
> I have been working with Big Data for a while and I have seen that a number 
> of the components have started to have their own custom baked solutions 
> (minimalistic Web UIs) for basic management operations, like showing metrics, 
> debugging etc instead of using JMX. 
> 
> I have the feeling that getting JMX working for dozens of different Java 
> services within a large cluster is an overly tough task, especially if you do 
> not want to make compromises around security. For me it seems, that at the 
> moment there is a gap between what the JDK offers regarding JMX 
> monitoring/management and what people would need in a real world setting to 
> use it effectively in an easy and secure way.
> 
> I am wondering if it would be possible to implement a Kerberos-based 
> authentication mechanism for JMX, allowing all services of a cluster to 
> authenticate JMX clients against a centrally managed Kerberos service, that 
> would also be officially supported by VisualVM so as to give an easy-to-use 
> user interface.
> 
> 
> Based on my understanding, this could either be a new protocol implementation 
> or assuming JDK-8171311: REST APIs for JMX gets done, an additional feature 
> around there to support GSS Negotiate/SPNEGO based authentication.
> 
> Could you please share your thoughts on this? Would anyone be interested to 
> sponsor this topic? 
> 
> Thanks,
> Peter
> 
> 
> 

Re: RFR(s): 8203343: VM.{metaspace|classloaders|classhierarchy...} jcmd should show invocation targets for Generated{Method|Constructor}AccessorImpl classes

[Kirk Pepperdine]

> On Jun 6, 2018, at 6:05 PM, Thomas Stüfe  wrote:
> 
> Dear all,
> 
> may I please have feedback and if possible reviews for this small addition:


I can see the need to visualize this but the output looks easily parsable so it 
all looks good from my perspective.

Not an official review
— Kirk

Re: RFR(S): 8200720: Print additional information in thread dump (times, allocated bytes etc.)

[Kirk Pepperdine]

> On Jun 4, 2018, at 8:24 PM, Chris Plummer  wrote:
> 
> Hi Gunter,
> 
> Just restating what I said with my initial review:
> 
>> The output you are adding is all useful. I think the question is (and I'd 
>> like to see a few people chime in on this for this review) is whether or not 
>> all of it is the appropriate for a thread's stack dump. For example, 
>> defined_classes is on the fringe of what I would call generally useful info 
>> in a stack dump. Sure, there might be that rare case when you need it, but 
>> how often compared to other useful info maintained on a per thread basis. 
>> How many other bits of useful info are not being printed in favor of 
>> defined_classes? It seems you have more in the queue. How many? I'm worried 
>> about how cluttered the stack dumps will get. Maybe we should add some sort 
>> of verbose thread dumping option. Just a thought. 
> Nothing has changed w.r.t. my opinions above. I would still like to know what 
> others think about the added details.

Thread dumps are a some what useful diagnostic tool.. what reduces their 
usefulness is the expense of acquiring them. I’d be hesitant to add to the 
cost. That said, this information can be useful that I fail to see how in the 
context of a thread dump. I’m wondering if a new diagnostic command would be in 
order to capture some of this information.

Kind regards,
Kirk

Re: jcmd - executing multiple commands at the same safepoint?

[Kirk Pepperdine]

> On May 10, 2018, at 11:26 AM, Thomas Stüfe <thomas.stu...@gmail.com> wrote:
> 
> On Thu, May 10, 2018 at 9:13 AM, Kirk Pepperdine
> <kirk.pepperd...@gmail.com> wrote:
>> The stacking at the safe point would be a huge win. Right now thread dump 
>> consistency can really confuse people as the tooling will show two threads 
>> owning the same lock at seemingly the same time. Reality, it’s just that 
>> people didn’t realize you get a safe point for each thread therefor there is 
>> motion in the system as you’re collecting the data.
> 
> I am a bit confused. What tooling are you talking about?

jstack. I’ve not seen it with jcmd. But I often see 2 threads holding the same 
lock at the “same time” which is of course non-sense. I can dig one up for you 
if you’re still confused.


> 
>> 
>> As an aside, it’s amazing how many dev’s aren’t aware of jcmd. Just 
>> yesterday after my session at Devoxx I had someone ask me about using jfr 
>> from the command line, many in that session had not seen jcmd before. The 
>> feedback was, wow, this is very useful and I wished I had of known about it 
>> earlier.
> 
> Yes, jcmd is quite useful. I also really like the simple design, which
> is by default up- and downward compatible (so you can talk to any VM,
> new, old, it should not matter). That is just good design. We - Sap -
> work to extend jcmd, to help our support folks. E.g. the whole new
> VM.metaspace command chain.

And simple it is….well done!!!

— Kirk

Re: jcmd - executing multiple commands at the same safepoint?

[Kirk Pepperdine]

The stacking at the safe point would be a huge win. Right now thread dump 
consistency can really confuse people as the tooling will show two threads 
owning the same lock at seemingly the same time. Reality, it’s just that people 
didn’t realize you get a safe point for each thread therefor there is motion in 
the system as you’re collecting the data.

As an aside, it’s amazing how many dev’s aren’t aware of jcmd. Just yesterday 
after my session at Devoxx I had someone ask me about using jfr from the 
command line, many in that session had not seen jcmd before. The feedback was, 
wow, this is very useful and I wished I had of known about it earlier.

— Kirk

> On May 10, 2018, at 7:50 AM, Thomas Stüfe <thomas.stu...@gmail.com> wrote:
> 
> Hi Kirk,
> 
> On Thu, May 10, 2018 at 8:07 AM, Kirk Pepperdine
> <kirk.pepperd...@gmail.com> wrote:
>> 
>>> On May 10, 2018, at 7:04 AM, Thomas Stüfe <thomas.stu...@gmail.com> wrote:
>>> 
>>> Oh... maybe. You are right. And I see commas are not good either since
>>> they are used in sub options (e.g. GC.class_stats).
>>> 
>>> Alternative ideas could be using brackets surrounding the
>>> command-and-options (e.g. '{''}'):
>>> 
>>> jcmd  { GC.class_stats columns=InstBytes,KlassBytes }  { VM.metaspace
>>> show-loaders }
>> 
>> are exposed braces safe in shells?
>>> 
>>> I think technically we would not even need delimiters, since the
>>> commands are as keywords limiting enough. E.g. one could just specify:
>>> 
>>> jcmd  GC.class_stats columns=InstBytes,KlassBytes VM.metaspace show-loaders
>> 
>> I’d vote for this form. Simple..
> 
> I think so too.
> 
> Note how it would influence the form of jcmd command files:
> 
> 
> 
> would execute two commands sequentially at different safepoints
> 
>  
> would execute two commands at the same safepoint
> 
> Arguably though, if we can stack commands at the command line, the
> jcmd file gets less important.
> 
> I'll play around a bit and wait what others say.
> 
> Thanks, Thomas
> 
>> 
>> — Kirk
>> 
>>> 
>>> and that would be for jcmd too. It is more for the benefit of the user.
>>> 
>>> On Thu, May 10, 2018 at 7:54 AM, Kirk Pepperdine
>>> <kirk.pepperd...@gmail.com> wrote:
>>>> Awesome idea! Would the semicolon would be an issue for shell scripts?
>>>>> On May 10, 2018, at 6:52 AM, Thomas Stüfe <thomas.stu...@gmail.com> wrote:
>>>>> 
>>>>> Hi all,
>>>>> 
>>>>> just asking around for opinions.
>>>>> 
>>>>> I am playing with the idea of bundling diagnostic commands to have
>>>>> them executed at the same safepoint, in order to get results
>>>>> consistent with each other.  E.g. a heapdump and a corresponding
>>>>> metaspace statistic.
>>>>> 
>>>>> Syntax wise, I think one could specify multiple commands in jcmd
>>>>> separated by comma or semicola:
>>>>> 
>>>>> jcmd   ;   
>>>>> 
>>>>> What do you think, would that be a useful addition?
>>>>> 
>>>>> Thanks, Thomas
>>>> 
>> 

Re: jcmd - executing multiple commands at the same safepoint?

[Kirk Pepperdine]

> On May 10, 2018, at 7:04 AM, Thomas Stüfe <thomas.stu...@gmail.com> wrote:
> 
> Oh... maybe. You are right. And I see commas are not good either since
> they are used in sub options (e.g. GC.class_stats).
> 
> Alternative ideas could be using brackets surrounding the
> command-and-options (e.g. '{''}'):
> 
> jcmd  { GC.class_stats columns=InstBytes,KlassBytes }  { VM.metaspace
> show-loaders }

are exposed braces safe in shells?
> 
> I think technically we would not even need delimiters, since the
> commands are as keywords limiting enough. E.g. one could just specify:
> 
> jcmd  GC.class_stats columns=InstBytes,KlassBytes VM.metaspace show-loaders

I’d vote for this form. Simple..

— Kirk

> 
> and that would be for jcmd too. It is more for the benefit of the user.
> 
> On Thu, May 10, 2018 at 7:54 AM, Kirk Pepperdine
> <kirk.pepperd...@gmail.com> wrote:
>> Awesome idea! Would the semicolon would be an issue for shell scripts?
>>> On May 10, 2018, at 6:52 AM, Thomas Stüfe <thomas.stu...@gmail.com> wrote:
>>> 
>>> Hi all,
>>> 
>>> just asking around for opinions.
>>> 
>>> I am playing with the idea of bundling diagnostic commands to have
>>> them executed at the same safepoint, in order to get results
>>> consistent with each other.  E.g. a heapdump and a corresponding
>>> metaspace statistic.
>>> 
>>> Syntax wise, I think one could specify multiple commands in jcmd
>>> separated by comma or semicola:
>>> 
>>> jcmd   ;   
>>> 
>>> What do you think, would that be a useful addition?
>>> 
>>> Thanks, Thomas
>> 

Re: jcmd - executing multiple commands at the same safepoint?

[Kirk Pepperdine]

Awesome idea! Would the semicolon would be an issue for shell scripts?
> On May 10, 2018, at 6:52 AM, Thomas Stüfe  wrote:
> 
> Hi all,
> 
> just asking around for opinions.
> 
> I am playing with the idea of bundling diagnostic commands to have
> them executed at the same safepoint, in order to get results
> consistent with each other.  E.g. a heapdump and a corresponding
> metaspace statistic.
> 
> Syntax wise, I think one could specify multiple commands in jcmd
> separated by comma or semicola:
> 
> jcmd   ;   
> 
> What do you think, would that be a useful addition?
> 
> Thanks, Thomas

Re: RFR (S): 8195115: G1 Old Gen MemoryPool CollectionUsage.used values don't reflect mixed GC results

[Kirk Pepperdine]

Hi Paul,

Is this is targeted for 11? It seems like a positive useful change.


My opinion is that if you’re going to break the tool chain.. make sure the 
change is well worth the pain and get in everything that you want or makes 
sense or what ever. Just make sure it’s worth the pain. It’s been far too often 
that this hasn’t been the case. 

As for this change, bundling values together hides information. This often 
leaves you either guessing as to what the underlying constraint is or more 
commonly, one responds to poor performance by expanding the heap size until the 
underlying supporting data structure is accidently at the a size it needed to 
be. A poor way of dealing with the issue at hand. Much better would be to 
understand which resource is stressed and only tune that bit leaving the other 
stuff alone meaning more information here is better then less. As the moment I 
calculate tenured size but subtracting young from total. The answer is of 
course incorrect in that is muxs in other stuff (reserved) but it’s much better 
to perform analytics on an incorrect tenured than a noisy total heap value. 
This It would be even better if the breakout was more precise and this work 
seems to provide a better breakout.

As an aside, a number of years ago with I was working on Crays, CRI published a 
not that their processors could overflow under certain conditions and they 
wanted to redesign the CPU around the IEEE standard to prevent/correct for 
this. The resounding response from the Cray users was NO. Reason being, these 
researchers had years of accumulated data and results and techniques that had 
been adjusted for the bugs in the Cray CPU. Everyone knew they were there… 
everyone knew that IEEE was better and a sensible fix to make except when you 
added in the disruption that would have been caused by the loss of years and 
years of very valuable historical data…

Kind regards,
Kirk




> On Jan 31, 2018, at 2:30 AM, Hohensee, Paul  wrote:
> 
> It’s true that my patch doesn’t completely solve the larger problem, but it 
> fixes the most immediately important part of it, particularly for JDK8 where 
> current expected behavior is entrenched. If we’re going to fix the larger 
> problem, imo we should file another bug/rfe to do it. I’d be happy to fix 
> that one too, once we have a spec.
> 
> What do you think of my suggestions? To summarize:
> 
> - Keep the existing memory pools and add humongous and archive pools.
> - Make the archive pool part of the old gen, and generalize it to all 
> collectors.
> - Split humongous regions off from the old gen pool into their own pool. The 
> old gen and humongous pools are disjoint region sets.
> - Keep the existing “G1 Young Generation” and “G1 Old Generation” collectors 
> and their associated memory pools (net of this patch). We add the humongous 
> pool to them.
> - Add “G1 Full” as an alias of the existing “G1 Old Generation” collector.
> - Add the “G1 Young”, “G1 Mixed” and “G1 Concurrent Cycle” collectors.
> - Set the G1 old gen memory pool max size to –Xmx, the archive space max size 
> to whatever it is, and the rest of the G1 memory pool max sizes to -1 == 
> undefined, as now.
> 
> The resulting memory pools:
> 
> “G1 Eden Space”
> “G1 Survivor Space”
> “G1 Old Gen”
> “G1 Humongous Space”
> “Archive Space”
> 
> The resulting collectors and their memory pools:
> 
> “G1 Young Generation” (the existing young/mixed collector), “G1 Old 
> Generation”/”G1 Full”, “G1 Mixed”
> - “G1 Eden Space”
> - “G1 Survivor Space”
> - “G1 Old Gen”
> - “G1 Humongous Space”
> “G1 Young”
> - “G1 Eden Space”
> - “G1 Survivor Space”
> - “G1 Humongous Space”
> “G1 Concurrent Cycle”
> - “G1 Old Gen”
> - “G1 Humongous Space”
> 
> I’m not religious about the names, but I like my suggestions. :)
> 
> The significant addition to my previous email, and an incompatible change, is 
> splitting humongous regions off from the old gen pool. This means that apps 
> that specifically monitor old gen occupancy will no longer see humongous 
> regions. Monitoring apps that just add up info about all a collector’s pools 
> won’t see a difference. I may be corrected (by Kirk, perhaps), but imo it’s 
> not as bad a compatibility issue as one might think, because the type of app 
> that uses a lot of humongous regions isn’t all that common. E.g., apps that 
> cache data in the heap in the form of large compressed arrays, and apps with 
> large hashmap bucket list arrays. The heaps such apps use are very often 
> large enough to use 32mb regions, hence need really big objects to actually 
> allocate humongous regions.
> 
> Thanks,
> 
> Paul
> 
> On 1/30/18, 5:51 AM, "Erik Helin"  wrote:
> 
>On 01/30/2018 03:07 AM, Hohensee, Paul wrote:
>> That’s one reviewer who’s ok with a short term patch. Anyone else? And, 
>> any reviewers for said short term patch? :)
> 
>Well, the patch is not really complete as it is. The problem is the 
>

Re: RFR (S): 8195115: G1 Old Gen MemoryPool CollectionUsage.used values don't reflect mixed GC results

[Kirk Pepperdine]

> On Jan 29, 2018, at 5:27 PM, Hohensee, Paul <hohen...@amazon.com> wrote:
> 
> A name change would affect Amazon’s heap monitoring, and thus I expect it 
> would affect other users as well.

I can name a number of tools that would be disrupted by this type of change. 
Additionally tooling would be complicated by the need to support both the old 
and new versions. The current names have been with us for years and they are 
well known, well documented and well understood. Given the level of disruption 
this change is likely to cause IMHO you’d need a very very good reason to want 
to make it.
>  
> As long as there are gc-specific memory pools, we’re going to need to be able 
> to identify them, and right now that’s done via name. All the mxbeans are 
> identified by name, so that’s a general design principle. The only way I can 
> think of to get rid of name dependency would be to figure out what abstract 
> metrics users want to monitor and implement them for all collectors. 
> HeapUsage (instantaneous occupancy) is one, CollectionUsage (long-lived 
> occupancy) is another, both of these for the entire heap, not just particular 
> memory pools. That said, imo there will always be a demand for the ability to 
> get collector and memory pool specific details, so I don’t see a way to get 
> around providing named entities.

Agreed…tuning strategies are implementation dependent and sensitive to specific 
versions. One is always going to need to know this information.

Kind regards,
Kirk Pepperdine

Re: RFR: JDK-8180709: java -javaagent:agent.jar with run-time that does not contain java.instrument prints confusing error

[Kirk Pepperdine]

Hi,

> On Dec 18, 2017, at 9:11 PM, gary.ad...@oracle.com wrote:
> 
> The only legitimate reason libinstrument would be missing is if 
> java.instrument was not included.
> Any other speculation about why libinstrument was not found is hard to 
> explain. e.g. until
> the vm is initialized we can not explicitly say whether java.instrument 
> module is present or not.

This is the only legitimate reason that is imaginable. I’d suggest that rather 
than overreaching you just state what you tried to do and that it failed.

Kind regards,
Kirk Pepperdine

> 
> 
> On 12/18/17 2:39 PM, Chris Plummer wrote:
>> Hi Gary,
>> 
>> Overall looks good. There's one minor change you might wanted to consider. 
>> It's possible the library failed to load for reasons other than 
>> java.instrument not being included. I see the error output already includes 
>> the error message from the failed dll_open() call, so why not just suggest 
>> that it *may* be due to java.instrument missing rather than implying that is 
>> known to be the reason why.
>> 
>> thanks,
>> 
>> Chris
>> 
>> On 12/18/17 6:38 AM, Gary Adams wrote:
>>> Here's a simple fix to include additional information when the instrument 
>>> library is
>>> missing from the jre and the command line requests -javaagent.
>>> 
>>>  Issue: https://bugs.openjdk.java.net/browse/JDK-8180709
>>>  Webrev: http://cr.openjdk.java.net/~gadams/8180709/webrev.00/
>>> 
>>> 
>> 
>> 
> 

Re: jmx-dev JEP review : JDK-8171311 - REST APIs for JMX

[Kirk Pepperdine]

> On Sep 12, 2017, at 12:44 PM, Erik Gahlin <erik.gah...@oracle.com> wrote:
> 
> I guess there are two use cases:
> 
> 1) Simple interoperability with other languages.
> 2) A drop in replacement for RMI
> 
> Can a JMX connector be written that support both use cases? I don't know, but 
> if not it could be that we need both a connector and an adapter.

I think if you were to extend JMXConnector to wrap the REST API you might be 
able to expose both. But I’m not sure it would be a great solution. I think a 
second JEP would be a better option.

— Kirk
 
> 
> Erik
> 
>> Hi Kirk,
>> 
>> I guess the term 'connector' here is loosely applied. When I say connector, 
>> I mean the connector that provides implementation of the package below,
>> 
>> https://docs.oracle.com/javase/8/docs/api/javax/management/remote/package-summary.html
>>  
>> <https://docs.oracle.com/javase/8/docs/api/javax/management/remote/package-summary.html>
>> RMIConnector is one implementation of above connector. 
>> 
>> On Tuesday 12 September 2017 12:56 PM, Kirk Pepperdine wrote:
>>> Hi Harsha,
>>> 
>>> From Chapter 5 of the JMX documentation. "Many different implementations of 
>>> connectors are possible. In particular, there are many possibilities for 
>>> the protocol used to communicate over a connection between client and 
>>> server.”
>>> 
>>> It goes on in the Generic Connector section under "User-Defined Protocols” 
>>> to say; "While the RMI connector must be present in every implementation of 
>>> the JMX Remote API, you can also implement a connector based on a protocol 
>>> that is not defined in the JMX Remote API standard. A typical example of 
>>> this is a connector based on a protocol that uses HTTP/S. Other protocols 
>>> are also possible. The JMX specification describes how to implement a 
>>> connector based on a user-defined protocol.”
>>> 
>>> Unless I’m missing something, this all suggests that there is nothing 
>>> inherently wrong is using REST behind a JMXConnector.
>> I hope above should clarify what I refer to when I say JMXConnector. In that 
>> sense, REST APIs alone cannot work as connector. In fact, it stands parallel 
>> to connector, as in it directly wraps the MBeanServer and does not wrap any 
>> JMXConnector. The JEP has detailed information about where the REST adapter 
>> sits in the JMX architecture. 
>> 
>> Are you suggesting that we implement a JMXConnector that works over REST?
>>> 
>>> As written this JEP pretty much looks like Jolokia. Jolokia is a great 
>>> project and as such I fail to see the benefits of simply duplicating it. 
>>> I’d also argue that the usefulness of that project has been some what muted 
>>> because it is not a drop in replacement for the standard RMI connector 
>>> meaning that one has to modify an entire tool chain just to make use of it. 
>>> However, creating a REST based JMXConnector would be immediately useful.
>>> As an aside, Jus last week I started on a JMXConnector that uses a shared 
>>> memory segment for communications. Of course this implementation would only 
>>> be available for local communications but it offers some advantages over 
>>> using a socket based protocol, even if that comms is over local loopback.
>>> 
>>> Kind regards,
>>> Kirk Pepperdine
>> 
>> Thanks
>> Harsha
>>> 
>>> 
>>> 
>>>> On Sep 12, 2017, at 9:04 AM, Harsha Wardhana B < 
>>>> <mailto:harsha.wardhan...@oracle.com>harsha.wardhan...@oracle.com 
>>>> <mailto:harsha.wardhan...@oracle.com>> wrote:
>>>> 
>>>> Hi Kirk,
>>>> 
>>>> REST APIs work as an adapter and cannot work as a connector. To quote from 
>>>> the JEP,
>>>> 
>>>> 
>>>>> The REST adapter is a part of the Distributed services level. Connectors 
>>>>> mirror the interfaces of agent level services to remote clients, whereas 
>>>>> adapters transform agent level services to different protocol. The 
>>>>> proposed functionality will transform Agent level services to REST APIs, 
>>>>> hence the name "REST adapter".
>>>> A connector *must* adhere to the JMX remoting spec. REST APIs cannot 
>>>> adhere to that because they expose APIs via HTTP and not Java. Hence it is 
>>>> called an Adapter and not a connector. It can never work as a 'drop-in' 
>>>> replacement for 

Re: jmx-dev JEP review : JDK-8171311 - REST APIs for JMX

[Kirk Pepperdine]

Hi Harsha,


> On Sep 12, 2017, at 10:40 AM, Harsha Wardhana B 
> <harsha.wardhan...@oracle.com> wrote:
> 
> Hi Kirk,
> 
> I guess the term 'connector' here is loosely applied. When I say connector, I 
> mean the connector that provides implementation of the package below,
> 
> https://docs.oracle.com/javase/8/docs/api/javax/management/remote/package-summary.html
>  
> <https://docs.oracle.com/javase/8/docs/api/javax/management/remote/package-summary.html>
> RMIConnector is one implementation of above connector. 
> 

Yes, this is the precise definition that I’ve been referring to.
> 
> On Tuesday 12 September 2017 12:56 PM, Kirk Pepperdine wrote:
>> Hi Harsha,
>> 
>> From Chapter 5 of the JMX documentation. "Many different implementations of 
>> connectors are possible. In particular, there are many possibilities for the 
>> protocol used to communicate over a connection between client and server.”
>> 
>> It goes on in the Generic Connector section under "User-Defined Protocols” 
>> to say; "While the RMI connector must be present in every implementation of 
>> the JMX Remote API, you can also implement a connector based on a protocol 
>> that is not defined in the JMX Remote API standard. A typical example of 
>> this is a connector based on a protocol that uses HTTP/S. Other protocols 
>> are also possible. The JMX specification describes how to implement a 
>> connector based on a user-defined protocol.”
>> 
>> Unless I’m missing something, this all suggests that there is nothing 
>> inherently wrong is using REST behind a JMXConnector.
> I hope above should clarify what I refer to when I say JMXConnector. In that 
> sense, REST APIs alone cannot work as connector.

Indeed it cannot because they are not part of the JMXConnector API.

Ok, I reread and see I misunderstood the use cases that you’re trying to cover 
off. It seems you’re only goal is to duplicate Jolokia whereas I’m looking at a 
different use case. The primary use case I encounter is motivated by the 
inability of various sites to use JMX simply because of the operational 
restrictions that prevent them from using RMI. This JEP will help with that use 
case. That said, adding a JMXConnector with a RESTful JMXConnector would open 
up an entire JMX tool chain to them rather than have them have to define a new 
tool chain but this is outside the scope of this JEP.

> In fact, it stands parallel to connector, as in it directly wraps the 
> MBeanServer and does not wrap any JMXConnector. The JEP has detailed 
> information about where the REST adapter sits in the JMX architecture. 
> 
> Are you suggesting that we implement a JMXConnector that works over REST?

Yes, adding a JMXConnector with a RESTful JMXConnector would open up an entire 
JMX tool chain to them rather than have them have to define a new tool chain 
but this appears to be outside the scope of this JEP.

Kind regards,
Kirk

Re: jmx-dev JEP review : JDK-8171311 - REST APIs for JMX

[Kirk Pepperdine]

Hi Harsha,

From Chapter 5 of the JMX documentation. "Many different implementations of 
connectors are possible. In particular, there are many possibilities for the 
protocol used to communicate over a connection between client and server.”

It goes on in the Generic Connector section under "User-Defined Protocols” to 
say; "While the RMI connector must be present in every implementation of the 
JMX Remote API, you can also implement a connector based on a protocol that is 
not defined in the JMX Remote API standard. A typical example of this is a 
connector based on a protocol that uses HTTP/S. Other protocols are also 
possible. The JMX specification describes how to implement a connector based on 
a user-defined protocol.”

Unless I’m missing something, this all suggests that there is nothing 
inherently wrong is using REST behind a JMXConnector.

As written this JEP pretty much looks like Jolokia. Jolokia is a great project 
and as such I fail to see the benefits of simply duplicating it. I’d also argue 
that the usefulness of that project has been some what muted because it is not 
a drop in replacement for the standard RMI connector meaning that one has to 
modify an entire tool chain just to make use of it. However, creating a REST 
based JMXConnector would be immediately useful.

As an aside, Jus last week I started on a JMXConnector that uses a shared 
memory segment for communications. Of course this implementation would only be 
available for local communications but it offers some advantages over using a 
socket based protocol, even if that comms is over local loopback.

Kind regards,
Kirk Pepperdine



> On Sep 12, 2017, at 9:04 AM, Harsha Wardhana B <harsha.wardhan...@oracle.com> 
> wrote:
> 
> Hi Kirk,
> 
> REST APIs work as an adapter and cannot work as a connector. To quote from 
> the JEP,
> 
> 
>> The REST adapter is a part of the Distributed services level. Connectors 
>> mirror the interfaces of agent level services to remote clients, whereas 
>> adapters transform agent level services to different protocol. The proposed 
>> functionality will transform Agent level services to REST APIs, hence the 
>> name "REST adapter".
> A connector *must* adhere to the JMX remoting spec. REST APIs cannot adhere 
> to that because they expose APIs via HTTP and not Java. Hence it is called an 
> Adapter and not a connector. It can never work as a 'drop-in' replacement for 
> JMX/RMI Connector. Existing tools using using RMIConnector will have to be 
> modified to use REST APIs. 
> 
> The current JEP does not allow all of the CRUD operations on MBeans. In the 
> spirit of keeping the APIs language agnostic, only read/write is supported. 
> It is not possible to specify create/delete REST APIs for JMX without 
> incorporating language specific features. I would welcome discussions around 
> including create/delete APIs for MBeans. 
> In lieu of the above, as of now REST adapter cannot exist independently and 
> will have to live along-side RMIConnector. 
> -Harsha
> 
> On Monday 11 September 2017 09:05 PM, Kirk Pepperdine wrote:
>> Hi Harsha,
>> 
>> The only reason I mentioned Jolokia is that it’s a project that usefulness 
>> is some what limited because it is *not* a compliment JMX connector and as 
>> such cannot be used as a straight drop-in replacement for the current RMI 
>> based connector. Is your plan here to make it a fully compliant connector so 
>> that we could configure tooling such as the MBean viewers in jConsole and 
>> VisualVM (or JMC for that matter) to use a restful connector instead of an 
>> RMI based connector? IMHO, doing so would represent a huge win as I know of 
>> quite a few projects that cannot or will not use JMX because of it’s 
>> reliance on RMI.
>> 
>> Consolidating all of the options under a single flag looks like another 
>> interesting win.
>> 
>> Kind regards,
>> Kirk
>> 
>> 
>> 
>>> On Sep 11, 2017, at 4:08 PM, Harsha Wardhana B 
>>> <harsha.wardhan...@oracle.com <mailto:harsha.wardhan...@oracle.com>> wrote:
>>> 
>>> Hi Erik,
>>> 
>>> On Monday 11 September 2017 07:14 PM, Erik Gahlin wrote:
>>>> Hi Harsha,
>>>> 
>>>> I haven't looked at Jolokia, or know what is the most reasonable approach 
>>>> in this case, but as a principle, I think we should strive for the best 
>>>> possible design rather than trying to be compatible with third party tools.
>>> Agreed. That will always be the first priority. That is the reason HTTP GET 
>>> interfaces will not be changed. I am undecided if the POST payloads need to 
>>> be changed (without compromising the REST de

Re: jmx-dev JEP review : JDK-8171311 - REST APIs for JMX

[Kirk Pepperdine]

> On Sep 11, 2017, at 10:30 PM, Martin Skarsaune <mar...@skarsaune.net> wrote:
> 
> I can agree to that. To be concrete, what does the JEP as it currently stands 
> offer over jolokia to be able to support JMXConnector? Could the client 
> interface and protocol be two separate concerns? 

The interface and the protocol clearly are two separate concerns. The interface 
in JMX is clearly defined and while the default protocol is RMI, it does not 
have to be RMI.

> 
> BTW: jolokia 2.0 will have support for JMX Notifications 
> https://ro14nd.de/jolokia-notifications 
> <https://ro14nd.de/jolokia-notifications>
Yes, my comment in the past has always been… it would be wonderful if jolokia 
fully supported JMXConnector but unfortunately it doesn’t which means you 
cannot easily use existing JMX clients with it. Other than that, it’s really 
useful in environments where using RMI is an issue. I think this where we can 
learn from jolokia.

Kind regards,
Kirk Pepperdine
  
> 
> Best Regards
> Martin Skarsaune 
> 
> man. 11. sep. 2017 kl. 21:55 skrev Kirk Pepperdine <kirk.pepperd...@gmail.com 
> <mailto:kirk.pepperd...@gmail.com>>:
>> On Sep 11, 2017, at 9:46 PM, Martin Skarsaune <mar...@skarsaune.net 
>> <mailto:mar...@skarsaune.net>> wrote:
>> 
>> Hi Harsha and Erik
>> 
>> I certainly understand the desire to design the API well. 
>> My point was just that when there is a mature battle-tested de-facto 
>> solution out in the wild,
> 
> I would agree that there are lessons to be learned from Jolokia. It is a 
> great/useful tool but it is not a JMXConnector. IMHO the REST layer should be 
> implemented as a JMXConnector. It is the implementation that has the ability 
> to integrate with widest set of exiting tooling. 
> 
> Kind regards,
> Kirk Pepperdine
> 
>> it would be a pity not to see potential for interoperability where the 
>> solutions are in fact really close. 
>> To illustrate where I'm coming from, I hacked the source of a plugin that is 
>> able to control the flight recorder via JMX , to adapt the POST payloads to 
>> this JEP. 
>> Assuming I understood it correctly the changes are quite small, but would 
>> the require a complete rewrite of all plugins, a layer of indirection or 
>> even worse a compatibility layer to use it. 
>> Note: I assumed the arguments are still an array and not an object?  ([] , 
>> not {}) ?
>> 
>> You can see an example of what changes would typically be needed here:
>> https://github.com/skarsaune/hawtio/commit/36ca65f495f05d20061b001fcc291ed3bc6e183f
>>  
>> <https://github.com/skarsaune/hawtio/commit/36ca65f495f05d20061b001fcc291ed3bc6e183f>
>> 
>> Cheers
>> 
>> Martin
>> 
>> 
>> man. 11. sep. 2017 kl. 17:36 skrev Kirk Pepperdine 
>> <kirk.pepperd...@gmail.com <mailto:kirk.pepperd...@gmail.com>>:
>> Hi Harsha,
>> 
>> The only reason I mentioned Jolokia is that it’s a project that usefulness 
>> is some what limited because it is *not* a compliment JMX connector and as 
>> such cannot be used as a straight drop-in replacement for the current RMI 
>> based connector. Is your plan here to make it a fully compliant connector so 
>> that we could configure tooling such as the MBean viewers in jConsole and 
>> VisualVM (or JMC for that matter) to use a restful connector instead of an 
>> RMI based connector? IMHO, doing so would represent a huge win as I know of 
>> quite a few projects that cannot or will not use JMX because of it’s 
>> reliance on RMI.
>> 
>> Consolidating all of the options under a single flag looks like another 
>> interesting win.
>> 
>> Kind regards,
>> Kirk
>> 
>> 
>> 
>>> On Sep 11, 2017, at 4:08 PM, Harsha Wardhana B 
>>> <harsha.wardhan...@oracle.com <mailto:harsha.wardhan...@oracle.com>> wrote:
>>> 
>>> Hi Erik,
>>> 
>>> On Monday 11 September 2017 07:14 PM, Erik Gahlin wrote:
>>>> Hi Harsha,
>>>> 
>>>> I haven't looked at Jolokia, or know what is the most reasonable approach 
>>>> in this case, but as a principle, I think we should strive for the best 
>>>> possible design rather than trying to be compatible with third party tools.
>>> Agreed. That will always be the first priority. That is the reason HTTP GET 
>>> interfaces will not be changed. I am undecided if the POST payloads need to 
>>> be changed (without compromising the REST design principles) to increase 
>>> adoption of this feature. 
>>>> 
>>>> How will the com

Re: jmx-dev JEP review : JDK-8171311 - REST APIs for JMX

[Kirk Pepperdine]

Hi Harsha,

The only reason I mentioned Jolokia is that it’s a project that usefulness is 
some what limited because it is *not* a compliment JMX connector and as such 
cannot be used as a straight drop-in replacement for the current RMI based 
connector. Is your plan here to make it a fully compliant connector so that we 
could configure tooling such as the MBean viewers in jConsole and VisualVM (or 
JMC for that matter) to use a restful connector instead of an RMI based 
connector? IMHO, doing so would represent a huge win as I know of quite a few 
projects that cannot or will not use JMX because of it’s reliance on RMI.

Consolidating all of the options under a single flag looks like another 
interesting win.

Kind regards,
Kirk



> On Sep 11, 2017, at 4:08 PM, Harsha Wardhana B <harsha.wardhan...@oracle.com> 
> wrote:
> 
> Hi Erik,
> 
> On Monday 11 September 2017 07:14 PM, Erik Gahlin wrote:
>> Hi Harsha,
>> 
>> I haven't looked at Jolokia, or know what is the most reasonable approach in 
>> this case, but as a principle, I think we should strive for the best 
>> possible design rather than trying to be compatible with third party tools.
> Agreed. That will always be the first priority. That is the reason HTTP GET 
> interfaces will not be changed. I am undecided if the POST payloads need to 
> be changed (without compromising the REST design principles) to increase 
> adoption of this feature. 
>> 
>> How will the command line look like to start the agent with the rest adapter?
>> 
>> In the past there have been discussions about adding syntactic sugar for 
>> -Dcom.sun.management, i.e.
>> 
>> -Xmanagement:ssl=false,port=7091,authenticate=false
>> 
>> instead of
>> 
>> -Dcom.sun.management.jmxremote.ssl=false 
>> -Dcom.sun.management.jmxremote.port=7091
>> -Dcom.sun.management.jmxremote.authenticate=false 
>> 
>> which is hard to remember, cumbersome to write and error prone since the 
>> parameters are not validated. If we are adding support for REST, it could 
>> perhaps be default, i.e.
>> 
>> -Xmanagement:ssl=false,authenticate=false,port=80
>> 
>> If you want to use JMX over RMI you would specify protocol:
>> 
>> -Xmanagement:ssl=false,port=7091,authenticate=false,protocol=rmi
> Yes. There is an enhancement request to add the -Xmanagemet:* syntatic sugar 
> for -Dcom.sun.management.jmxremote.* flags. REST adapter will use one of the 
> above flags though I haven't thought of the exact name for it yet. I will 
> update the JEP with the details of the flag shortly. 
>> 
>> Has there been any thoughts about JMX notifications?
> Notifications will not be supported in this JEP. 
> MBean Notifications are not a widely used feature and will not be supported 
> via the REST adapter.
>> 
>> I know it is outside the scope of the JEP, but I think we should take it 
>> into consideration when doing the design, so the functionality could be 
>> added on later without too much difficulty.
> Notifications can be added without modifying the current design too much. If 
> required, it will be worked upon via an enhancement request. 
>> 
>> Thanks
>> Erik
>> 
> Thanks
> Harsha
>>> Hi Martin,
>>> 
>>> In my opinion, the interfaces exposed by current JEP are lot closer to REST 
>>> style than the interfaces exposed by Jolokia. 
>>> For instance, HTTP GET by default should be used to read resources, but it 
>>> is made part of URL in Jolokia interfaces.
>>> 
>>> /read///
>>> 
>>> I would wait on opinions from more people before considering changing the 
>>> current interfaces.
>>> 
>>> Thanks
>>> -Harsha
>>> 
>>> On Wednesday 06 September 2017 11:40 AM, Martin Skarsaune wrote:
>>>> Hello
>>>> 
>>>> Would one at least consider adopting the same URL paths and payloads as 
>>>> Jolokia? This could make life a lot easier for third party tools that 
>>>> connect to it. 
>>>> 
>>>> Best Regards
>>>> 
>>>> Martin Skarsaune 
>>>> 
>>>> ons. 6. sep. 2017 kl. 07:04 skrev Harsha Wardhana B 
>>>> <harsha.wardhan...@oracle.com <mailto:harsha.wardhan...@oracle.com>>:
>>>> Hi Kirk,
>>>> 
>>>> Yes. Jolokia was considered and is listed as an alternative in the JEP.
>>>> 
>>>> Jolokia can serve as a viable alternative but can be bulky. We are looking 
>>>> for simple and lightweight solution.
>>>> 
>>>> -Harsha
>>>> 
>>>&g

Re: JEP review : JDK-8171311 - REST APIs for JMX

[Kirk Pepperdine]

Hi Harsha,

I wouldn’t consider Jolokia to be an alternative because as I suggested, it’s 
not a complete JMXConnector implementation and as such cannot be used with 
standard JMX clients such as the MBean viewer in JMC or VisualVM. In my 
opinion, it would be super if the product of the JEP could work as a drop-in 
replacement for the RMI-JMXConnctor that all standard JMX tooling currently 
uses.

Kind regards,
Kirk

> On Sep 6, 2017, at 7:03 AM, Harsha Wardhana B <harsha.wardhan...@oracle.com> 
> wrote:
> 
> Hi Kirk,
> 
> Yes. Jolokia was considered and is listed as an alternative in the JEP.
> 
> Jolokia can serve as a viable alternative but can be bulky. We are looking 
> for simple and lightweight solution.
> 
> -Harsha
> 
> On Wednesday 06 September 2017 10:21 AM, Kirk Pepperdine wrote:
>> Hi,
>> 
>> Have you run into this project? https://jolokia.org <https://jolokia.org/>. 
>> Unfortunately it’s not exactly a drop in replacement for the standard RMI 
>> based JMX connector but it’s not far off.
>> 
>> Kind regards,
>> Kirk
>> 
>>> On Sep 5, 2017, at 6:30 PM, Erik Gahlin <erik.gah...@oracle.com> 
>>> <mailto:erik.gah...@oracle.com> wrote:
>>> 
>>> Hi Harsha,
>>> 
>>> Looping in jmx-dev.
>>> 
>>>> byte[], short[], int[], float[], double[]
>>> Should long[] be included there as well?
>>> 
>>>> The REST adapter will come with a simple and lightweight JSON parser.
>>> Is this an internal parser or will it be exposed as an API?
>>> 
>>> If so, how does it relate to JEP 198: Light-Weight JSON API?
>>> http://openjdk.java.net/jeps/198 <http://openjdk.java.net/jeps/198>
>>> 
>>> Will com.sun.net.httpserver.HttpServer be used to serve the requests?
>>> 
>>> Thanks
>>> Erik
>>> 
>>>> Hi All,
>>>> 
>>>> Please review the JEP for REST APIs for JMX :
>>>>https://bugs.openjdk.java.net/browse/JDK-8171311 
>>>> <https://bugs.openjdk.java.net/browse/JDK-8171311>
>>>> 
>>>> The JEP aims at providing RESTful web interfaces to MBeans.
>>>> 
>>>> Access to MBeans registered in a MBeanServer running inside a JVM requires 
>>>> a Java client. Language-agnostic access to MBeans will require spawning a 
>>>> Java client which can be cumbersome. The proposed JEP allows MBeans to be 
>>>> accessed in a language/platform-independent, ubiquitous and seamless 
>>>> manner.
>>>> 
>>>> Thanks
>>>> -Harsha
>>>> 
>>>> 
> 

Re: JEP review : JDK-8171311 - REST APIs for JMX

[Kirk Pepperdine]

Hi,

Have you run into this project? https://jolokia.org. Unfortunately it’s not 
exactly a drop in replacement for the standard RMI based JMX connector but it’s 
not far off.

Kind regards,
Kirk

> On Sep 5, 2017, at 6:30 PM, Erik Gahlin  wrote:
> 
> Hi Harsha,
> 
> Looping in jmx-dev.
> 
> > byte[], short[], int[], float[], double[]
> 
> Should long[] be included there as well?
> 
> > The REST adapter will come with a simple and lightweight JSON parser.
> 
> Is this an internal parser or will it be exposed as an API?
> 
> If so, how does it relate to JEP 198: Light-Weight JSON API?
> http://openjdk.java.net/jeps/198
> 
> Will com.sun.net.httpserver.HttpServer be used to serve the requests?
> 
> Thanks
> Erik
> 
>> Hi All,
>> 
>> Please review the JEP for REST APIs for JMX :
>>https://bugs.openjdk.java.net/browse/JDK-8171311
>> 
>> The JEP aims at providing RESTful web interfaces to MBeans.
>> 
>> Access to MBeans registered in a MBeanServer running inside a JVM requires a 
>> Java client. Language-agnostic access to MBeans will require spawning a Java 
>> client which can be cumbersome. The proposed JEP allows MBeans to be 
>> accessed in a language/platform-independent, ubiquitous and seamless manner.
>> 
>> Thanks
>> -Harsha
>> 
>> 
> 

Re: output of jstack command

[Kirk Pepperdine]

Hi,

The format of thread dumps have been stable until 8 when it seems that we 
stopped caring about breaking tooling. That it changed in 9 is of no surprise. 
Again, pain we’re willing to accept if there is valuable information being 
added. 

Kind regards,
Kirk
> On May 25, 2017, at 6:01 PM, Ram Krishnan <ramkri...@gmail.com> wrote:
> 
> Hi Kirk, Daniel,
> 
> Many thanks for the immediate reply.
> 
> By type of thread, I meant GC thread vs normal thread. Looks like that 
> information is already there in the thread name.
> 
> Looks like in Java 9 the output of jstack is different, so the tools need to 
> change for Java 9.
> 
> Is it fair to count on a consistent format per Java release for jstack?
> 
> Thanks,
> Ramki
> 
> On Thu, May 25, 2017 at 8:07 AM, Kirk Pepperdine <kirk.pepperd...@gmail.com 
> <mailto:kirk.pepperd...@gmail.com>> wrote:
> Hi Ramki,
> 
> The source for jstack is in openJDK. Feel free to create your own copy of 
> jstack where you can output the information in any format he likes. If you 
> are suggesting that the existing format be changed do be aware that there are 
> many tools that expect the current format. These have been adjusted to a 
> change in format that was introduced with Java 8. I don’t see any reason why 
> the format shouldn’t include information that is currently missing and is 
> relevant. However I’d want to make sure that is is relevant and important 
> before breaking the tool chain once again.
> 
> I believe thread ids are already in the header. Certainly thread names are 
> there. Not sure what you mean by types of threads.
> 
> Kind regards,
> Kirk
> > On May 25, 2017, at 4:59 PM, Daniel D. Daugherty 
> > <daniel.daughe...@oracle.com <mailto:daniel.daughe...@oracle.com>> wrote:
> >
> > Adding serviceability-dev@... since jstack is a Serviceability tool.
> >
> > I believe jstack is experimental which means the output format can
> > change at any time...
> >
> > Dan
> >
> > On 5/25/17 8:35 AM, Ram Krishnan wrote:
> >> Hi,
> >>
> >> I would like to leverage the output of jstack command for extracting
> >> additional information about the type of threads, thread ids etc. Since I
> >> will be parsing the output, I need the precise format. Is there any
> >> documentation on jstack output format changes and the openjdk release(s)
> >> where the changes happened?
> >>
> >> ​Tha​nks in advance.
> >>
> >
> 
> 
> 
> 
> -- 
> Thanks, 
> Ramki

Re: output of jstack command

[Kirk Pepperdine]

Hi Ramki,

The source for jstack is in openJDK. Feel free to create your own copy of 
jstack where you can output the information in any format he likes. If you are 
suggesting that the existing format be changed do be aware that there are many 
tools that expect the current format. These have been adjusted to a change in 
format that was introduced with Java 8. I don’t see any reason why the format 
shouldn’t include information that is currently missing and is relevant. 
However I’d want to make sure that is is relevant and important before breaking 
the tool chain once again.

I believe thread ids are already in the header. Certainly thread names are 
there. Not sure what you mean by types of threads.

Kind regards,
Kirk
> On May 25, 2017, at 4:59 PM, Daniel D. Daugherty 
>  wrote:
> 
> Adding serviceability-dev@... since jstack is a Serviceability tool.
> 
> I believe jstack is experimental which means the output format can
> change at any time...
> 
> Dan
> 
> On 5/25/17 8:35 AM, Ram Krishnan wrote:
>> Hi,
>> 
>> I would like to leverage the output of jstack command for extracting
>> additional information about the type of threads, thread ids etc. Since I
>> will be parsing the output, I need the precise format. Is there any
>> documentation on jstack output format changes and the openjdk release(s)
>> where the changes happened?
>> 
>> ​Tha​nks in advance.
>> 
> 

Re: PRE-RFR: 8177154: Default configuration should disallow loading agents

[Kirk Pepperdine]

Hi Alan,

I have to agree with Mario here. I think a natural reaction would be to 
configure the JVM to opt-in by default. Otherwise you will break just about 
every APM/Monitoring tool out there. If opt-in is almost an automatic then you 
have to question the value of this change.

Kind regards,
Kirk

> On Mar 24, 2017, at 1:43 PM, Mario Torre  wrote:
> 
> On Fri, Mar 24, 2017 at 10:48 AM, Alan Bateman  
> wrote:
>> On 24/03/2017 09:34, Remi Forax wrote:
>> 
>>> :
>>> 
>>> One of strong feature that makes choosing Java on server is that you can
>>> attach, at runtime, a monitoring/troubleshooting tool when an event occurs,
>>> please do not send us back to the pre Java 6 world !
>>> 
>> This does not send us back to the pre JDK 6. The changes have no impact on
>> the troubleshooting tools and no impact on management tools that use the
>> attach API to start the JMX agent in the running VM. The change is simply to
>> make loading of arbitrary code with the attach API opt-in. On the server
>> then you the launch script can specify the proposed option and tools that
>> are loading agents will not see a difference.
> 
> Hello Alan,
> 
> This solution feels like a bit of a quick hack to me. Nevertheless, I
> think it should be opt-out, or at least with a configurable default. I
> can see the only reasonable use case where you don't want to have
> external code to plug in but still want to start with a default agent,
> for instance if you want to use byteman with the jvm, but not allow
> anything else to plug in at runtime (byteman or similar tools of
> course).
> 
> Cheers,
> Mario

Re: Low-Overhead Heap Profiling

[Kirk Pepperdine]

Hi Jeremy,

 
 The answer to that is not to use safepoint-biased execution profilers, I 
 think.  

Thank you for the advice. I’ve been advocating for a number of years that 
people understand the sampling bias that exists in all profilers. I also would 
consider safe point bias being the most egregious form or sampling possible. 
This is why I’m very happy to see Richard’s Honest profiler (using 
AsyncGetCallTrace) show up and also to have inspired Pierre Laporte to start 
work on a lock profiler project at our (un)conference (jCrete). As an aside, we 
hope jCrete can help inspire and support more efforts like this.

My desire here is to ensure that minimize the effects of sample bias on the 
results. In this regards, I like the approach you are proposing. My concern 
with Tony’s approach what that it would introduce a size based sampling bias. 
As I mentioned before, GC costs is one thing to consider however I’m more 
concerned with frequency related allocation costs and it’s effect on 
application throughputs.

Also, I think we have other techniques that can infer where an allocation has 
taken place be it in a tlab or even in tenured. I’m not sure that we have to 
profile for this type of information though if we can get it for almost free…..

Kind regards,
Kirk

Re: Low-Overhead Heap Profiling

[Kirk Pepperdine]

Hi Jeremy,

Sorry I wasn’t so clear, it’s not about collection, it’s about allocation. In 
this regard it’s not about about size, it’s about the frequency. People tend 
allocate small objects frequently and they will avoid allocating large objects 
frequently. The assumption is, large is expensive but small isn’t. These event 
will show up using execution profilers but given the safe-point bias of 
execution profilers and other factors, it’s often clearer to view this problem 
using memory profilers.

Kind regards,
Kirk

On Jun 25, 2015, at 7:34 PM, Jeremy Manson jeremyman...@google.com wrote:

 Why would allocation frequency be more damaging to performance?  Allocation 
 is cheap, and as long as they become dead before the YG collection, it costs 
 the same to collect one 1MB object as it does to collection 1000 1K objects.
 
 Jeremy
 
 On Wed, Jun 24, 2015 at 11:54 PM, Kirk Pepperdine kirk.pepperd...@gmail.com 
 wrote:
 
 But, seriously, why didn’t you like my proposal? It can do anything your 
 scheme can with fewer and simpler code changes. The only thing that it 
 cannot do is to sample based on object count (i.e., every 100 objects) 
 instead of based on object size (i.e., every 1MB of allocations). But I 
 think doing sampling based on size is the right approach here (IMHO).
 
 
 
 I would think that the size based sampling would create a size based bias in 
 your sampling. Since IME, it’s allocation frequency is more damaging to 
 performance, I’d prefer to see time boxed sampling
 
 Kind regards,
 Kirk Pepperdine
 
 

Re: Low-Overhead Heap Profiling

[Kirk Pepperdine]

Hi Tony,

 
 I would think that the size based sampling would create a size based bias in 
 your sampling. 
 
 
 That’s actually true. And this could be good (if you’re interested in what’s 
 filling up your eden, the larger objects might be of more interest) or bad 
 (if you want to get a general idea of what’s being allocated, the size bias 
 might make you miss some types of objects / allocation sites).
 
 

IME, if I see allocation rates  ~500MB/sec I know there are some significant 
wins I can make by tuning allocation hotspots. So, I’m interested in frequent 
allocations no matter what the size.
 
 Since IME, it’s allocation frequency is more damaging to performance, I’d 
 prefer to see time boxed sampling
 
 
 Do you mean “sample every X ms, say”?
 
 

I don’t find that sampling rate has to be that high to find hot allocation 
sites.

Kind regards,
Kirk Pepperdine

 
 Tony
 
 
 
 
 Kind regards,
 Kirk Pepperdine
 
 
 
 -
 
 Tony Printezis | JVM/GC Engineer / VM Team | Twitter
 
 @TonyPrintezis
 tprinte...@twitter.com

Re: Low-Overhead Heap Profiling

[Kirk Pepperdine]

 
 Do you mean “sample every X ms, say”?
 
 
 This is not impossible, but a little weird. 

Yes, you are right, this is weird and I don’t think is necessary for this type 
of profiling. Please ignore.

Regards,
Kirk

Re: Low-Overhead Heap Profiling

[Kirk Pepperdine]

Hi Vladimir,

The difference would be in the licensing.

Kind regards,
Kirk Pepperdine

On Jun 23, 2015, at 7:31 AM, Vladimir Voskresensky 
vladimir.voskresen...@oracle.com wrote:

 Hello Jeremy,
 
 If this is sampling, not tracing, then how is it different from the 
 low-overhead memory profiling provided by JFR [1].
 JFR samples per new TLAB allocation. It provides really very good picture and 
 I haven't seen overhead more than 2%.
 Btw, JFR also does not have false positives reported by instrumented 
 approaches for the cases when JIT was able to eliminate heap allocation.
 
 Thanks,
 Vladimir.
 [1] http://hirt.se/blog/?p=381
 
 On 22.06.2015 11:48, Jeremy Manson wrote:
 Hey folks,
 
 (cc'ing Aleksey and John, because I mentioned this to them at the JVMLS last 
 year, but I never followed up.)
 
 We have a patch at Google I've always wanted to contribute to OpenJDK, but I 
 always figured it would be unwanted.  I've recently been thinking that might 
 not be as true, though.  I thought I would ask if there is any interest / if 
 I should write a JEP / if I should just forget it.
 
 The basic problem is that there is no low-overhead supported way to figure 
 out where allocation hotspots are. That is, sets of stack traces where lots 
 of allocation / large allocations took place.
 
 What I had originally done (this was in ~2007) was use bytecode rewriting to 
 instrument allocation sites.  The instrumentation would call a Java method, 
 and the method would capture a stack trace.  To reduce overhead, there was a 
 per-thread counter that only took a stack trace once every N bytes 
 allocated, where N is a randomly chosen point in a probability distribution 
 that centered around ~512K.
 
 This was *way* too slow, and it didn't pick up allocations through JNI, so I 
 instrumented allocations at the VM level, and the overhead went away.  The 
 sampling is always turned on in our internal VMs, and a user can just query 
 an interface for a list of sampled stack traces.  The allocated stack traces 
 are held with weak refs, so you only get live samples.
 
 The additional overhead for allocations amounts to a subtraction, and an 
 occasional stack trace, which is usually a very, very small amount of our 
 CPU (although I had to do some jiggering in JDK8 to fix some performance 
 regressions).
 
 There really isn't another good way to do this with low overhead.  I was 
 wondering how the gruop would feel about our contributing it?
 
 Thoughts?
 
 Jeremy
 

Re: Low-Overhead Heap Profiling

[Kirk Pepperdine]

Hi Jeremy,

 
 It will also need a little polish to be available to the world.  Open design 
 questions for upstreaming are things like:
 
 - Should the interval between samples be configurable?

This could be helpful.

 
 - Should it *just* take a stack trace, or should the behavior be 
 configurable?  For example, we have a variant that allows it to invoke a 
 callback on allocation.  Do you want this?  Because it is being called during 
 allocation, the callback can't invoke JNI (because of the potential for a 
 safepoint), so it might be somewhat confusing to the user.

As long of these features don’t contribute to sampling bias I think they’d (at 
times) all be useful.


 
 - If the answer to the above is yes, should it be able to invoke *multiple* 
 callbacks with multiple intervals?  That could get very expensive and hairy.
 
 - Other than stack trace, what kind of information should the sampled data 
 contain?  Right now, the structure is:
 
 struct StackTraceData {
   ASGCT_CallTrace *trace;
   jint byte_size;
   jlong thread_id;
   const jbyte *name;
   jint name_length;
   jlong uid;
 };

If you could get age data that could be interesting as well.

Kind regards,
Kirk Pepperdine

Re: System.gc() causes for jcl (BufferPoolMXBean)

[Kirk Pepperdine]

 
 Back-off + gc is implementation specific and I don't think appropriate to add 
 to the BufferPoolMXBean. This type of tracking might be better served with 
 jvmstat counters. JFR isn't in OpenJDK but others here might be able to say 
 whether it could make use of the counters and updated instrumentation.

Certainly more visibility in the area can’t be harmful.

Regards,
Kirk

Re: More visibility into code cache churn

[Kirk Pepperdine]

Hi Ramki,

Anything to improve visibility of the CodeCache would be hugely appreciated!

Regards,
Kirk

On Jun 10, 2015, at 9:15 AM, Srinivas Ramakrishna ysr1...@gmail.com wrote:

 
 I filed https://bugs.openjdk.java.net/browse/JDK-8087107 and attached a patch 
 to the ticket that exposes some useful code cache stats as perf data counters:
 
 $ jcmd pid PerfCounter.print | grep -i sun\.ci\. 
 sun.ci.codeCacheCapacity=6291456 
 sun.ci.codeCacheMaxCapacity=6291456 
 sun.ci.codeCacheMethodsReclaimedNum=1030 
 sun.ci.codeCacheSweepsTotalNum=93 
 sun.ci.codeCacheSweepsTotalTimeMillis=63 
 sun.ci.codeCacheUsed=3386880 
 ... 
 
 At Twitter, we've found this useful for easy monitoring of code cache 
 activity, and would like to see this integrated into OpenJDK.
 
 thanks!
 -- ramki

Re: Updates to JEP-158: JVM Unified Logging are coming...

[Kirk Pepperdine]

Hi All,

I’m very happy for a digital solution. As for solving problems in JFR.. great 
but that is a commercial solution and thus doesn’t work for OpenJDK. Remember? 
Plus, with all due respect for JFR, not all of us agree that this is the best 
way to sort out performance issues ;-)

Regards,
Kirk

On Apr 17, 2015, at 3:16 PM, Mattis Castegren mattis.casteg...@oracle.com 
wrote:

 I guess it would be a possible later enhancement request to add a new logging 
 tag ”gc_compressed” for example that prints gc output in some easy to parse 
 format. It could be one line of comma separated values per GC for example 
 that you could import to other tools.
 
  
 
 But I agree with Fredrik, we should look at exactly what problems you try to 
 solve, and see if it instead can be solved in JFR. You could make a recording 
 that only gathers GC data, and you could run it for days and still get a 
 pretty small recording (at least smaller than what a corresponding gc log 
 would be). If you then still feel the need to import the data to some other 
 tool, maybe we should add that tool’s functionality to Mission Control?
 
  
 
 Kind Regards
 
 /Mattis
 
  
 
 From: Fredrik Arvidsson 
 Sent: den 17 april 2015 14:36
 To: Martijn Verburg
 Cc: serviceability-dev@openjdk.java.net serviceability-dev@openjdk.java.net; 
 hotspot-...@openjdk.java.net Source Developers
 Subject: Re: Updates to JEP-158: JVM Unified Logging are coming...
 
  
 
 Hi Martijn, and the others :)
 
 I'm glad you like what we do. We also think the tag based logging is 
 something good, and that it will make the logging in the JVM easier to 
 implement, configure and consume.
 
 Regarding 'binary / compressed' log format. We are not planning to add any 
 structured log format to the unified logging framework. It is outside the 
 scope of this JEP. The log data format is a simple string with some well 
 defined decorations. The logs are primarily meant to be read by humans.
 
 There will be log output extension points in the framework, but they will not 
 be accessible publicly and they will not support 3rd party extensions in 
 runtime. Even if the output extension API would be made fully accessible, the 
 data fed in to it would still only be a string, since thats what the 
 developers are adding in the code where the logging framework is called. I 
 guess one could add compression of log data as an extension, but I don't 
 think that is really what you are asking for.
 
 As you might know, Oracle offers low latency low intrusive structured logging 
 in its Java Flight Recorder product. We are not aiming in this JEP to overlap 
 that functionality more than absolutely needed.
 
 Cheers
 /Fredrik
 
 On 2015-04-17 12:12, Martijn Verburg wrote:
 
 Hi Fredrik, 
 
  
 
 Thanks from us (jClarity and some other orgs) for considering the tag based 
 system, we're really glad you've gone in this direction.  One quick bit of 
 feedback.
 
  
 
 Is it possible to add a binary / compressed logging format or an interface/ 
 API to plug one in for high performance Logging?  Human readable is nice for 
 looking at logs immediately, but for some of our customers they'd be looking 
 to offload that concern to another process/host.
 
 
 
 
 Cheers,
 Martijn
 
  
 
 On 17 April 2015 at 09:32, Fredrik Arvidsson HYPERLINK 
 mailto:fredrik.arvids...@oracle.com; \nfredrik.arvids...@oracle.com wrote:
 
 Hi
 
 We are planning to release an updated version of JEP-158 very soon. The work 
 to update this JEP has been going on for some time and we now feel we are 
 ready to present it to a wider audience.
 
 The JEP for JVM Unified Logging was created back in February 2012. The 
 requirements for the API and implementation were heavily influenced by the 
 logging functionality in JRockit JVM. The JEP was updated and refined in mid 
 2014 and eventually targeted for JDK 9 in late 2014:
 https://bugs.openjdk.java.net/browse/JDK-8046148
 
 During the review process of the JEP some concerns were voiced about the 
 chosen API and functionality but the conclusion was that the proposal would 
 work, and the JEP was targeted for JDK 9 and the implementation work was 
 started.
 
 Early 2015 during pre-reviews and presentations of the API and implementation 
 concerns were again raised about the formulation of the JEP, this time with 
 greater force. At this point we decided to take a time-out and investigate 
 our options going forward.
 
 The time-out led to a modified proposal that can be found here:
 HYPERLINK http://cr.openjdk.java.net/%7Efarvidsson/JEP-158/JEP-158.html; 
 \nhttp://cr.openjdk.java.net/~farvidsson/JEP-158/JEP-158.html
 and with some illustrations here:
 HYPERLINK 
 http://cr.openjdk.java.net/%7Efarvidsson/JEP-158/JEP-158_Illustration.png; 
 \nhttp://cr.openjdk.java.net/~farvidsson/JEP-158/JEP-158_Illustration.png
 
 So what have we changed, and why?
 In short we have switched from using an hierarchical log subject layout to 
 something that could be best described as 

Re: Updates to JEP-158: JVM Unified Logging are coming...

[Kirk Pepperdine]

On Apr 17, 2015, at 11:29 AM, Fredrik Arvidsson fredrik.arvids...@oracle.com 
wrote:

 Hi
 Ill try to reply in-line below.
 
 On 2015-04-17 10:27, Mattis Castegren wrote:
 Ok. One risk I see would be that you would forget base tags. For example, 
 if you write a lot of code in g1, it would be easy to log something as only 
 g1, not including the gc tag. For us in Sustaining, that could lead to 
 us having to use long verbose lines in order to get everything. This is 
 something we had in older versions of JRockit, where you had your favorite 
 verbose line that included tons of modules, like gc, yc, pausetime, 
 compaction etc.
 Of course there is a risk that people would forget to add a tag. But I can't 
 see that this risk is very high. Logging code has to be put through code 
 review just like any other change and we just have to pay extra attention to 
 this in the beginning i guess. Using the hierarchical scheme there is also a 
 risk that the logging topic is set to the wrong one.

Indeed!

 
 There are no recommendations right now on what tags to create or use in the 
 logging code. We will work with all groups trying to establish some 
 guidelines I guess to have tag names consistent.

My recommendation to teams has always been to give architectural guidelines or 
use requirements from operations to define logging. If you leave logging to dev 
it’s generally useful for dev and no one else.


 
 If we see a real need to have 'meta-tags' consisting of several other tags we 
 could introduce something like that I'm sure, but this is just something we 
 have discussed briefly.

JMS has a spec for managing this type of behavior.

 
 There will also be a way to list all existing 'tag combinations' used in the 
 code in runtime. This could maybe be used to spot the ones that are 
 potentially wrong?
 If the recommendation is to ALWAYS include gc if you log something to 
 g1, wouldn't it make sense to make g1 a subset of gc? Otherwise you 
 would have to copy boilerplate code to include all the tags in your current 
 area, like
 
 log (gc,g1,g1_init, ...)
 
 If you forget it in one place, and we want to gather full information from a 
 run, we would have to ask customers to run with 
 Xverbose:gc,g1,g1_init,g1_other_module,...
 Yes, this would be the way to handle missed tags in the code. I can't see 
 that there is another way really :(

Make them manageable so if you forget on the command line you can change them 
on the fly.

Regards,
Kirk



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: RFR: JDK-8068589: GCCause should distinguish jcmd GC.run from System.gc()

[Kirk Pepperdine]

Hi,

Not an official review but it’s a simple patch that looks good.

Kind regards,
Kirk

On Feb 2, 2015, at 4:51 PM, Yasumasa Suenaga yasue...@gmail.com wrote:

 Hi,
 
 I need more reviewer.
 Could you review it?
 
 http://cr.openjdk.java.net/~ysuenaga/JDK-8068589/webrev.02/
 
 Thanks,
 
 Yasumasa
 
 2015/01/28 17:24 Staffan Larsen staffan.lar...@oracle.com:
 Looks good!
 
 Thanks,
 /Staffan
 
  On 28 jan 2015, at 05:48, Yasumasa Suenaga yasue...@gmail.com wrote:
 
  Hi Staffan, Kirk,
 
  I agree to set Diagnostic Command to GCCause.
  So I applied it to new patch.
 
  http://cr.openjdk.java.net/~ysuenaga/JDK-8068589/webrev.02/
 
  Could you review it again?
 
 
  Thanks,
 
  Yasumasa
 
 
  On 2015/01/28 5:06, Kirk Pepperdine wrote:
  Hi Staffan,
 
 
  Anyway, it’s a record in a GC log so I don’t see the value of GC.run. 
  Certainly “DiagCmd or even Diagnostic Command” seems sufficient given 
  the context.
 
  Let’s go with “Diagnostic Command”, then.
 
  Thank you!
 
  Regards,
  Kirk
 
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: RFR: JDK-8068589: GCCause should distinguish jcmd GC.run from System.gc()

[Kirk Pepperdine]

On Jan 27, 2015, at 8:37 PM, Staffan Larsen staffan.lar...@oracle.com wrote:

 
 On 27 jan 2015, at 18:00, Kirk Pepperdine k...@kodewerk.com wrote:
 
 Hi,
 
 On Jan 27, 2015, at 1:22 PM, Yasumasa Suenaga yasue...@gmail.com wrote:
 
 Hi Staffan,
 
 Thank you for your comments.
 
 I've uploaded new webrev:
 http://cr.openjdk.java.net/~ysuenaga/JDK-8068589/webrev.01/
 
 I changed as below:
 - GCCause::_jcmd_gc_run - GCCause::_dcmd_gc_run
 - GCCause string: jcmd GC.run - GC.run DCmd”
 
 Can I suggest a shortened GCCause string to “DiagCmd”. At the very least I’d 
 prefer to not have the “.” in the string.
 
 I would have suggested “GC.run Diagnostic Command”, or “Diagnostic Command: 
 GC.run”. I don’t know what problems a ‘.’ causes. “System.gc()” already has 
 one.

Indeed it does and it and it has () and really wished neither were there. But 
unfortunately I missed the change from System to System.gc(). It complicates 
regex which may not seem like a big deal until you add in the entire GC log 
picture from 1.6.0 ’til 1.9.0 across all collectors and the possible flag 
combinations... it only adds (needlessly) to the mess.

Anyway, it’s a record in a GC log so I don’t see the value of GC.run. Certainly 
“DiagCmd or even Diagnostic Command” seems sufficient given the context.

Kind regards,
Kirk Pepperdine



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: RFR: JDK-8068589: GCCause should distinguish jcmd GC.run from System.gc()

[Kirk Pepperdine]

Hi Staffan,

 
 Anyway, it’s a record in a GC log so I don’t see the value of GC.run. 
 Certainly “DiagCmd or even Diagnostic Command” seems sufficient given the 
 context.
 
 Let’s go with “Diagnostic Command”, then.

Thank you!

Regards,
Kirk



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: RFR: JDK-8068589: GCCause should distinguish jcmd GC.run from System.gc()

[Kirk Pepperdine]

Hi,

On Jan 27, 2015, at 1:22 PM, Yasumasa Suenaga yasue...@gmail.com wrote:

 Hi Staffan,
 
 Thank you for your comments.
 
 I've uploaded new webrev:
 http://cr.openjdk.java.net/~ysuenaga/JDK-8068589/webrev.01/
 
 I changed as below:
   - GCCause::_jcmd_gc_run - GCCause::_dcmd_gc_run
   - GCCause string: jcmd GC.run - GC.run DCmd”

Can I suggest a shortened GCCause string to “DiagCmd”. At the very least I’d 
prefer to not have the “.” in the string.

Regards,
Kirk

 
 
 Could you review it?
 
 
 Thanks,
 
 Yasumasa
 
 
 On 2015/01/26 17:12, Staffan Larsen wrote:
 A bit of terminology here. ‘GC.run’ is called a ‘Diagnostic Command’. ‘jcmd’ 
 is one way to invoke a Diagnostic Command. Another way is via a JMX MBean. 
 With this in mind I think your references to ‘jcmd’ should be changed to 
 ‘diagnostic command’ (or an abbreviation thereof). For example: _jcmd_gc_run 
 - _dcmd_gc_run.
 
 Thanks,
 /Staffan
 
 On 25 jan 2015, at 14:15, Yasumasa Suenaga yasue...@gmail.com wrote:
 
 Hi all,
 
 GCCause which is printed in gc log is System.gc() when jcmd GC.run is 
 invoked.
 I think that GCCause which is caused by jcmd GC.run should be different 
 from System.gc() .
 
 I uploaded webrev for this enhancement:
 http://cr.openjdk.java.net/~ysuenaga/JDK-8068589/webrev.00/
 
 This patch prints jcmd GC.run to gc log when jcmd GC.run is invoked.
 
 
 Could you review it?
 
 
 Thanks,
 
 Yasumasa
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: RFR: JDK-8068589: GCCause should distinguish jcmd GC.run from System.gc()

[Kirk Pepperdine]

Hi Staffan,

There are other entry points in diagnosticCommand that will trigger a Full GC.

void ClassHistogramDCmd::execute(DCmdSource source, TRAPS) {
  VM_GC_HeapInspection heapop(output(),
  !_all.value() /* request full gc if false */);
  VMThread::execute(heapop);
}

and
void HeapDumpDCmd::execute(DCmdSource source, TRAPS) {
  // Request a full GC before heap dump if _all is false
  // This helps reduces the amount of unreachable objects in the dump
  // and makes it easier to browse.
  HeapDumper dumper(!_all.value() /* request GC if _all is false*/);
  int res = dumper.dump(_filename.value());
  if (res == 0) {
output()-print_cr(Heap dump file created);
  } else {
// heap dump failed
ResourceMark rm;
char* error = dumper.error_as_C_string();
if (error == NULL) {
  output()-print_cr(Dump failed - reason unknown);
} else {
  output()-print_cr(%s, error);
}
  }
}

For example. For consistency, how would you suggest these be handled.

Kind regards,
Kirk Pepperdine

On Jan 26, 2015, at 9:12 AM, Staffan Larsen staffan.lar...@oracle.com wrote:

 A bit of terminology here. ‘GC.run’ is called a ‘Diagnostic Command’. ‘jcmd’ 
 is one way to invoke a Diagnostic Command. Another way is via a JMX MBean. 
 With this in mind I think your references to ‘jcmd’ should be changed to 
 ‘diagnostic command’ (or an abbreviation thereof). For example: _jcmd_gc_run 
 - _dcmd_gc_run.
 
 Thanks,
 /Staffan
 
 On 25 jan 2015, at 14:15, Yasumasa Suenaga yasue...@gmail.com wrote:
 
 Hi all,
 
 GCCause which is printed in gc log is System.gc() when jcmd GC.run is 
 invoked. 
 I think that GCCause which is caused by jcmd GC.run should be different from 
 System.gc() . 
 
 I uploaded webrev for this enhancement:
 http://cr.openjdk.java.net/~ysuenaga/JDK-8068589/webrev.00/
 
 This patch prints jcmd GC.run to gc log when jcmd GC.run is invoked. 
 
 
 Could you review it?
 
 
 Thanks,
 
 Yasumasa
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: RFR: JDK-8068589: GCCause should distinguish jcmd GC.run from System.gc()

[Kirk Pepperdine]

Hi Yasumasa,

Thank you for your review of the code. Indeed you have filed this as a bug and 
it has been commented on as an enhancement, not a bug. As I have mentioned a 
single call to System.gc() makes no difference in any analytic that I performed 
on any application that I’ve encountered in the wild no matter what the cause. 
Other issues: it’s not currently clear to me that jcmd is the only entry point 
to this call and that would make the message misleading for those other paths. 
There are other entry points that can results in a full GC but they will remain 
labeled as is. I also see a downstream impact on tooling. Thus I don’t see a 
compelling argument for the value of this change thus I will argue towards not 
destabilizing downstream tooling.

That said, I’ve said my piece and have added enough spam to the list on this 
subject.

Kind regards,
Kirk Pepperdine


On Jan 26, 2015, at 4:53 AM, Yasumasa Suenaga yasue...@gmail.com wrote:

 Hi Kirk,
 
 I want to distinguish GC whether it is requested from internal (e.g. 
 System.gc()) or not.
 System.gc() is called in RMI as you said.
 
 In JVM implementation, entry point of GC is Universe::heap()-collect() 
 family. It is used by JVMTI force GC, heap inspection (e.g. jmap), etc. Thus 
 I guess that GCCause which is invoked from JVM should set valid GCCause which 
 is not System.gc().
 So I file this enhancement to JBS and create a patch.
 
 Thanks,
 
 Yasumasa
 
 2015/01/26 6:14 Kirk Pepperdine k...@kodewerk.com:
 Hi Yasumasa,
 
 
 
  I think the System.gc() call which is invoked by jcmd is special case and I 
  want to
  distinguish it.
  Programmer can call System.gc() from their code. But GC which is invoked by 
  jcmd is NOT explicitly call by programmer.
 
 System.gc() called for by RMI is also not explicitly called by the programmer 
 either. There are also other tools that will result in calls to System.gc() 
 also. Should we catalog them in GCCause also?
 
 
  Indeed, SystemGCDCmd will call System.gc() (same meaning).
  However, I think it has different meaning from System.gc() call from source 
  code.
 
 I’m not sure I understand the point. From an analytical POV, a single call to 
 System.gc() is pretty much meaningless. If it’s regular or there is a pattern 
 in the frequency of the calls then I’m interested. A call to System.gc() has 
 the same effect no matter who or why is responsible.
 
 My interest lies in keeping the GC logs as simple as possible. If there is 
 meaningful data to add so be it. That said, I’m not sure that this change 
 meets that bar.
 
 That said, I’m still concerned that the caller/callee division seems inside 
 out. But it’s Sunday so…
 
 Kind regards,
 Kirk Pepperdine
 
 
 
  Thanks,
 
  Yasumasa
 
 
  On 2015/01/25 22:56, Kirk Pepperdine wrote:
  Hi,
 
  IMHO, if a System.gc() is being called then the cause should be 
  System.gc(). If we start down the road of differentiating between the 
  various causes of calls to System.gc() this will turn in a nightmare!
 
  Indeed as I look at the patch it’s curious that it’s up to the caller to 
  determine is calls to System.gc() have been suppressed. I would have 
  expected the collect() call to make the decision as to should the call be 
  honored or not.
 
   void SystemGCDCmd::execute(DCmdSource source, TRAPS) {
 if (!DisableExplicitGC) {
  Universe::heap()-collect(GCCause::_jcmd_gc_run);
 } else {
   output()-print_cr(Explicit GC is disabled, no GC has been 
  performed.);
 }
 
 
 
  Kind regards,
  Kirk Pepperdine
 
  On Jan 25, 2015, at 2:15 PM, Yasumasa Suenaga yasue...@gmail.com 
  mailto:yasue...@gmail.com wrote:
 
  Hi all,
 
  GCCause which is printed in gc log is System.gc() when jcmd GC.run is 
  invoked.
  I think that GCCause which is caused by jcmd GC.run should be different 
  from System.gc() .
 
  I uploaded webrev for this enhancement:
  http://cr.openjdk.java.net/~ysuenaga/JDK-8068589/webrev.00/
 
  This patch prints jcmd GC.run to gc log when jcmd GC.run is invoked.
 
 
  Could you review it?
 
 
  Thanks,
 
  Yasumasa
 
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: RFR: JDK-8068589: GCCause should distinguish jcmd GC.run from System.gc()

[Kirk Pepperdine]

Hi,

IMHO, if a System.gc() is being called then the cause should be System.gc(). If 
we start down the road of differentiating between the various causes of calls 
to System.gc() this will turn in a nightmare!

Indeed as I look at the patch it’s curious that it’s up to the caller to 
determine is calls to System.gc() have been suppressed. I would have expected 
the collect() call to make the decision as to should the call be honored or not.

 void SystemGCDCmd::execute(DCmdSource source, TRAPS) {
   if (!DisableExplicitGC) {
Universe::heap()-collect(GCCause::_jcmd_gc_run);
   } else {
 output()-print_cr(Explicit GC is disabled, no GC has been performed.);
   }


Kind regards,
Kirk Pepperdine

On Jan 25, 2015, at 2:15 PM, Yasumasa Suenaga yasue...@gmail.com wrote:

 Hi all,
 
 GCCause which is printed in gc log is System.gc() when jcmd GC.run is 
 invoked. 
 I think that GCCause which is caused by jcmd GC.run should be different from 
 System.gc() . 
 
 I uploaded webrev for this enhancement:
 http://cr.openjdk.java.net/~ysuenaga/JDK-8068589/webrev.00/
 
 This patch prints jcmd GC.run to gc log when jcmd GC.run is invoked. 
 
 
 Could you review it?
 
 
 Thanks,
 
 Yasumasa



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: RFR: JDK-8068589: GCCause should distinguish jcmd GC.run from System.gc()

[Kirk Pepperdine]

Hi Yasumasa,


 
 I think the System.gc() call which is invoked by jcmd is special case and I 
 want to
 distinguish it.
 Programmer can call System.gc() from their code. But GC which is invoked by 
 jcmd is NOT explicitly call by programmer.

System.gc() called for by RMI is also not explicitly called by the programmer 
either. There are also other tools that will result in calls to System.gc() 
also. Should we catalog them in GCCause also?

 
 Indeed, SystemGCDCmd will call System.gc() (same meaning).
 However, I think it has different meaning from System.gc() call from source 
 code.

I’m not sure I understand the point. From an analytical POV, a single call to 
System.gc() is pretty much meaningless. If it’s regular or there is a pattern 
in the frequency of the calls then I’m interested. A call to System.gc() has 
the same effect no matter who or why is responsible. 

My interest lies in keeping the GC logs as simple as possible. If there is 
meaningful data to add so be it. That said, I’m not sure that this change meets 
that bar.

That said, I’m still concerned that the caller/callee division seems inside 
out. But it’s Sunday so…  

Kind regards,
Kirk Pepperdine

 
 
 Thanks,
 
 Yasumasa
 
 
 On 2015/01/25 22:56, Kirk Pepperdine wrote:
 Hi,
 
 IMHO, if a System.gc() is being called then the cause should be System.gc(). 
 If we start down the road of differentiating between the various causes of 
 calls to System.gc() this will turn in a nightmare!
 
 Indeed as I look at the patch it’s curious that it’s up to the caller to 
 determine is calls to System.gc() have been suppressed. I would have 
 expected the collect() call to make the decision as to should the call be 
 honored or not.
 
  void SystemGCDCmd::execute(DCmdSource source, TRAPS) {
if (!DisableExplicitGC) {
 Universe::heap()-collect(GCCause::_jcmd_gc_run);
} else {
  output()-print_cr(Explicit GC is disabled, no GC has been 
 performed.);
}
 
 
 
 Kind regards,
 Kirk Pepperdine
 
 On Jan 25, 2015, at 2:15 PM, Yasumasa Suenaga yasue...@gmail.com 
 mailto:yasue...@gmail.com wrote:
 
 Hi all,
 
 GCCause which is printed in gc log is System.gc() when jcmd GC.run is 
 invoked.
 I think that GCCause which is caused by jcmd GC.run should be different 
 from System.gc() .
 
 I uploaded webrev for this enhancement:
 http://cr.openjdk.java.net/~ysuenaga/JDK-8068589/webrev.00/
 
 This patch prints jcmd GC.run to gc log when jcmd GC.run is invoked.
 
 
 Could you review it?
 
 
 Thanks,
 
 Yasumasa
 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Need comments on JEP-158: Unified JVM Logging

[Kirk Pepperdine]

Hi Chris,

I too am bothered by the jumbled lines. However I don’t think that we need 
strict ordering. However I don’t believe it’s up to the logging framework to 
enforce order. If order is needed it should be up to the producer or the 
consumer of messages to deal with the issue of order. However I do believe 
these requirements should be struck from the document

Logging is performed at different levels: error, warning, info, debug, trace
Logging messages are in human-readable plain text
Messages can be “decorated” with level, component, thread id, timestamp
Enable/disable individual log messages by message id (for example by 
using__FILE__ / __LINE__)


IMHO, that is proposal specifies a taxonomy that is hierarchal and devoid of 
context in the domains in which it will be working is very limiting and over 
reaching. As I have stated in past discussion, feel that the mention of levels 
should be dropped. It it my view that at it’s core, logging is a messaging 
problem and there are many lessons that we should be taking from messaging that 
seem to be missing from this proposal. Even lessons or ideas learned from 
social media messaging (Twitter messages???) is missing from the proposal.

The goals that I pointed to all specify all work against the flexibility of 
allowing non-hieracrchal taxonomies. In other words, allow users of the 
framework to define categories that the makers of the framework cannot not 
possibly conjure up and then allow those categories to be collected or acted 
upon independent of other categories. This scheme does not nor should not 
preclude one from using a hierarchal logging system. However it does not limit 
one to a hierarchical scheme when that scheme does not make sense.

If we take GC logging into consideration. The current system of flags allow me 
to understand what it is I’m getting as I turn on each flag. I can turn on 
combinations of flags to get different bits of information as the need arises. 
Unfortunately the newer code is translating these flags to Fine, Finer, and 
Finest which means some developer is not deciding what categories of logging 
goes into each of these limited number of categories which may.. or may not be 
an appropriate mix. My experience with logging in the field is that everything 
sooner or later degenerates into the lowest common denominator… which is why I 
get called in as the logging becomes the primary bottleneck in the application. 
To let you know how much of a problem this it now represents about 30-40% of 
all rescue tuning that I’m involved in. Now I know you guys are all very clever 
and you don’t think that you’d be susceptible to making the same type of 
mistake… however I should caution that a number of the teams that locked 
themselves into the logging/bottleneck problem were also very clever. This is 
just something that happens organically… over time without anyone really taking 
notice.. much like the frog in the pot or stone soup story.

I completely agree with this statement; It is outside the scope of this JEP to 
add the actual logging calls from all JVM components. This JEP will only 
provide the infrastructure to do the logging.” I’d add to this that the 
implementation should not over reach in that defining categories is not within 
the scope of the framework

Finally, the statement It is also outside the scope of the JEP to enforce a 
logging format, apart from the format of the decorations and the use of 
human-readable plain text” is also objectionable as it precludes logging binary 
formats. This IMHO is a huge over-reach based on the dogma that logs should be 
in human readable form. The last time I checked ASCII was not human readable in 
that I need a program to view it in a meaningful format on a screen.

Please, lets not let the problems found in the current set of logging 
frameworks that Java developers have had been forced to use leak into the JVM. 
Lets reconsider this JEP in it’s current form before moving to an 
implementation.

Kind regards,
Kirk Pepperdine


On May 17, 2014, at 9:47 AM, Chris Newland cnewl...@chrisnewland.com wrote:

 Hi Fredrik,
 
 The discussion I had with David Holmes and John Rose on hotspot-dev back
 in February might be relevant to this JEP:
 http://mail.openjdk.java.net/pipermail/hotspot-dev/2014-February/012718.html
 
 I'm the author of a JITWatch[1], a JIT compiler log analysis tool which
 relies on -XX:+TraceClassLoading -XX:+LogCompilation -XX:+PrintAssembly
 all being present in hotspot.log
 
 Occasionally the lack of thread safety in the VM logging presents me with
 jumbled output:
 
 [Entry Point]
 [Constants]
   # {method} apos;ind
 writer thread='3340'/
 [Loaded sun.nio.cs.ThreadLocalCoders from C:\Program
 Files\Java\jre7\lib\rt.jar]
 writer thread='2660'/
 exOfapos; apos;(II)Iapos; in apos;java/lang/Stringapos;
   # this: ecx   = apos;java/lang/Stringapos;
   # parm0:edx   = int
   # parm1:[sp+0x20]   = int  (sp of caller)
   0x009e07e0: nop

Re: Need comments on JEP-158: Unified JVM Logging

[Kirk Pepperdine]

Hi Fredrik,

I do need more information but I think a flat structure would provide less 
overhead rather than passing messages through a network of “sub” loggers.

Peter Lawrey wrote JChronicle which at it’s core is a messaging system but it’s 
also a wickedly fast logger. That said, there isn’t a formalized logging 
interface to it though it’s something being discussed/thought about as we speak.
https://github.com/peter-lawrey/Java-Chronicle.

I’m not suggesting you take this implementation but there might be things to 
learn from it as this spec is being strengthened.

Kind regards,
Kirk Pepperdine

On May 18, 2014, at 11:47 AM, Richard Warburton richard.warbur...@gmail.com 
wrote:

 Hi Fredrik,
 
 In the JEP I tried to explain the concept of 'sub' loggers but I did not
 put in anything about the intent of having them. I think that your
 requirement easily could be solved using sub loggers, or sub components if
 you like that wording better.
 
 Loggers are ordered in an hierarchical tree where 'gc' might be the root
 for all logging in the gc 'area'. The 'details', 'cause' and 'safepoint'
 content types could be translated in to sub loggers to the 'gc' logger. By
 setting log level for the separate sub loggers to enable logging I cant see
 that you would not be able to get the filtering you want. By using log
 levels wisely you will be able to get even more control over what gets
 output in the logs. The above was a much simplified example, in reality you
 would probably have a more elaborate tree of gc loggers. There is nothing
 in the logger three hierarchy approach that implies that they should be big
 blocks of stuff.
 
 Logging topics, tags, or markers were something we considered at the
 beginning of the design phase but we came to the conclusion that we
 probably could solve the requirements by using sub loggers instead.
 
 Please tell me if you don't think this approach would work, and in that
 case why.
 
 
 Thanks for clarifying this point. I think this explanation addresses how
 that particular use case is met. I suspect that it might be worth adding a
 bit more explanation around this topic to the JEP itself, because I don't
 think I'm the only person with this concern.
 
 Also a heads up that there's discussion happening related to this topic
 off-list at:
 https://groups.google.com/a/jclarity.com/forum/#!topic/friends/NA0EyOJk6bs
 
 regards,
 
  Richard Warburton
 
  http://insightfullogic.com
  @RichardWarburto http://twitter.com/richardwarburto

Re: Logging JEP

[Kirk Pepperdine]

Hi Fredrik,

comments below…

Regards,
Kirk

 
 My primary objection with the current specification is that the JEP is both 
 over-reaching and restrictive. It suggest taxonomy a closed hierarchical 
 taxonomy instead of a more open and flexible arrangement. The JEP 
 overreaches in that it defines or suggests extremely limited levels which 
 are all completely devoid of context or context that allow one to pull out 
 certain pieces of information in exclusion to other pieces of information. 
 It also allows for developers to make very arbitrary decisions as to what to 
 log at each “levels”. Where as PrintAdapativeSizingPolicy gives a very clear 
 indication as what will be logged, G1.logging=WARNING is… well... I don’t 
 know what is and is not suppose to be included. In addition, maybe 
 PrintAdaptiveSizingPolicy is not interesting but PrintTenuringDistribution 
 is. And lets say they are both at WARNING log level. Even better, lets say 
 that with CMS you get PrintTenuringDistribution at FINE and with G1 you get 
 it at FINEST. The point of all this is the logging framework is complicating 
 questions that are none of it’s concern. So I would suggest that the logging 
 framework get out of the business of trying to predefine categories of 
 logging, especially hierarchical categories of logging.
 With the hierarchical logger design proposed in the JEP I cant see why you 
 could not get the behaviour you're asking for. By targeting sub/child loggers 
 it is possible to enable/disable arbitrary levels in the tree. And at the 
 same time you could enable logging for lets say all GC related activity by 
 enable the top most logger (root). It is possible to create sparse 
 configurations including/excluding single loggers.

I think that pub/sub is a much simpler model that is easier to understand than 
what you’ve described.

 The logger hierarchy does not need to be 'component oriented' even if I think 
 it makes the most sense.
 I think you put far too much emphasis on log levels.

Funny but I think the spec puts far too much emphasis on log levels.

 Log levels should not be used to categorise information.

In practice they do categorize information because there is no other practical 
way to categorize it. I set verbosity to debug because I expect all debug 
messages to show up.

 It should be used to select the verbosity of the log data output.

I guess to understand this further I’d need to see an example of how it would 
be used. For example, this is a single G1GC event

1.047: [Full GC (System.gc()) 17M-3728K(13M), 0.0212570 secs]
   [Eden: 18.0M(24.0M)-0.0B(5120.0K) Survivors: 0.0B-0.0B Heap: 
17.2M(256.0M)-3728.7K(13.0M)]

How would you suggest that the JVM use the logging framework to log this 
message?


 We have only five levels of verbosity in the JEP (plus disabled). I think 
 that is a well balanced number.

I don’t know how to respond to this without risking sounding inflammatory and 
I’d really like not like to be inflammatory. ;-)

 By pre defining all available loggers(categories) in the source code we 
 actually give the user the possibility to enumerate all those, and see what 
 different type of information that are in there.

Why do you need to know levels prior to iterating? Pub/Sub does not come with 
this requirement.

 On top of that we get compile time checks of that its valid.
 Your proposal would require any module implementer to carefully document all 
 'tags' used in some place in some way. The latter would not make it easier 
 for users, it would make it even worse than it is today.

What I read here, and maybe I’m reading it wrong, is that the log levels are 
used to categorize messages and that simply using your predefined tags 
alleviates developers from carefully documenting the information that is 
included with each tag/level. This is exactly one of the bigger problems that I 
see in the field. Developers drop things into a log level and they don’t 
document what comes with each level. You can’t relieve developers of the 
responsibility of documenting what will happen when I turn the logging level to 
debug (or some other level).

 Instead I believe it is more useful for the framework to support more of a 
 pub/sub model where each of the frameworks or components in the JVM are free 
 to devise a logging scheme or taxonomy that makes sense for that component.
 This would not be something targeted by this JEP. The JEP title is: *Unified* 
 JVM Logging. what you describe is similar to what we have today. All modules 
 have their own way of doing logging. This whole discussion is about 
 addressing that problem, not making it a rule. I have gotten heaps of 
 feedback from developers asking me to bring an end to this arbitrary 'log as 
 you like' situation they live in today.

I’m not advocating what is in the JVM today. Unifying the logging scheme makes 
sense. It allows for better standardization of tooling, just like we have with 
JMS client technology.

 For example, 

Re: Logging JEP

[Kirk Pepperdine]

Hi Fredrik,

More comments inlined.

Regards,
Kirk

 
 My primary objection with the current specification is that the JEP is 
 both over-reaching and restrictive. It suggest taxonomy a closed 
 hierarchical taxonomy instead of a more open and flexible arrangement. The 
 JEP overreaches in that it defines or suggests extremely limited levels 
 which are all completely devoid of context or context that allow one to 
 pull out certain pieces of information in exclusion to other pieces of 
 information. It also allows for developers to make very arbitrary 
 decisions as to what to log at each “levels”. Where as 
 PrintAdapativeSizingPolicy gives a very clear indication as what will be 
 logged, G1.logging=WARNING is… well... I don’t know what is and is not 
 suppose to be included. In addition, maybe PrintAdaptiveSizingPolicy is 
 not interesting but PrintTenuringDistribution is. And lets say they are 
 both at WARNING log level. Even better, lets say that with CMS you get 
 PrintTenuringDistribution at FINE and with G1 you get it at FINEST. The 
 point of all this is the logging framework is complicating questions that 
 are none of it’s concern. So I would suggest that the logging framework 
 get out of the business of trying to predefine categories of logging, 
 especially hierarchical categories of logging.
 With the hierarchical logger design proposed in the JEP I cant see why you 
 could not get the behaviour you're asking for. By targeting sub/child 
 loggers it is possible to enable/disable arbitrary levels in the tree. And 
 at the same time you could enable logging for lets say all GC related 
 activity by enable the top most logger (root). It is possible to create 
 sparse configurations including/excluding single loggers.
 I think that pub/sub is a much simpler model that is easier to understand 
 than what you’ve described.
 I don't think the difference is that big. Enabling logging on a logger is 
 essentially the same as subscribing to a topic.

the only exception is that the JEP eliminates all possibilities besides the 
topics that the JEP decided are “needed”.

 
 The logger hierarchy does not need to be 'component oriented' even if I 
 think it makes the most sense.
 I think you put far too much emphasis on log levels.
 Funny but I think the spec puts far too much emphasis on log levels.
 Loggers and Adapters are the most important things in the JEP. Log levels are 
 just to a way to select the verbosity of the output.

So my argument has been, why not just use user defined topics to select the 
verbosity? The topics will have semantic meaning where as these predefined 
levels have none.

 Log levels should not be used to categorise information.
 In practice they do categorize information because there is no other 
 practical way to categorize it. I set verbosity to debug because I expect 
 all debug messages to show up.
 
 It should be used to select the verbosity of the log data output.
 I guess to understand this further I’d need to see an example of how it 
 would be used. For example, this is a single G1GC event
 
 1.047: [Full GC (System.gc()) 17M-3728K(13M), 0.0212570 secs]
[Eden: 18.0M(24.0M)-0.0B(5120.0K) Survivors: 0.0B-0.0B Heap: 
 17.2M(256.0M)-3728.7K(13.0M)]
 
 How would you suggest that the JVM use the logging framework to log this 
 message?
 This information should probably be logged in the 'info' level since the 
 purpose of the information is to provide information. Maybe 'debug' level 
 could be used to log even more information about how the numbers were 
 computed.

This information is logged at PrintGCDetails level in the 1.8 and at 
PrintGCDetails PrintGCCause in the 1.7.0_45 and better. There is more detail 
available but that comes to you via different flags. For example adaptive 
sizing and reference processing. Are these both debug and if so how could I get 
one without getting the other?


 
 We have only five levels of verbosity in the JEP (plus disabled). I think 
 that is a well balanced number.
 I don’t know how to respond to this without risking sounding inflammatory 
 and I’d really like not like to be inflammatory. ;-)
 Well, you just did ;)
 By pre defining all available loggers(categories) in the source code we 
 actually give the user the possibility to enumerate all those, and see what 
 different type of information that are in there.
 Why do you need to know levels prior to iterating? Pub/Sub does not come 
 with this requirement.
 If you are not to get a lot of noise/unwanted information you would have to 
 know what topics to subscribe to. I can't really see the difference here. its 
 like saying that you want news, and then start subscribing to all newspapers. 
 You will get news, but a lot of it you're not interested in.

What I’m saying is that of course one is going to have to know what to 
subscribe to. As it is one needs to know which flags should be set if you need 
to get a information from the JVM. If I subscribe to the wrong topics 

Re: RFR: 7164841: Improvements to the GC log file rotation

[Kirk Pepperdine]

 Append vs clobber: with or without rotation, -Xloggc always clobbers the last 
 log. That's why people want a time stamp or pid in the gc file name, IMHO.

My client just clobbered his log files yet... I'd still *not* want the time 
stamp because this can potentially accidentally fill production disks. Better 
that my client over wrote the logs than filled his production disk because he 
didn't realize the behaviour. And, many don't realize the behaviour, they just 
follow the admin manual.

 
 Alternatively, give an option to append when -Xloggc is enabled, vs default 
 overwrite.

Allowing an append will break all current analysis tooling.

Regards,
Kirk

Re: RFR: 7164841: Improvements to the GC log file rotation

[Kirk Pepperdine]

On 2013-08-27, at 10:01 AM, Bengt Rutisson bengt.rutis...@oracle.com wrote:

 
 Yumin,
 
 On 8/26/13 7:01 PM, Yumin Qi wrote:
 Bengt,
 
  Thanks for your comments.
  Yes, as you said, the purpose of rotating logs is primarily targeted for 
 saving disk space. This bug is supplying  customers another option to 
 prevent the previous gc logs from removed by restart app without making 
 copy. Now with this pid and time stamp included in file name,  we have more 
 options for users. It is up to user to make decision to or not to keep the 
 logs. We cannot handle all the requests in JVM, but we can offer the choices 
 for users I think. Any way, with either the previous rotating version, or 
 the one I am working, the logs will not grow constantly without limit to 
 blow storage out as long as users give enough attention.
 
  My concern is for no log rotation, should we still use time stamp in file 
 name? I have one version for this, I hope get more comments for that.
 
 Sorry if I wasn't clear before. I am not worried about the case when log 
 rotation is turned on. What I was worried about was the case where a user is 
 not using log rotation but is still piping the GC output to a file. That file 
 will be overwritten every time the VM is restarted. If we add time stamps to 
 the file name for this case then the file will not be overwritten. I think 
 that is a bit of a scary change. That's all.

This falls into the category of unexpected behaviour and I would agree that 
unless log rotation has been specified I would not like to see the name 
changed.. even though I'll admit to be bitten by having a log file 
accidentally over written. That said, it would be excellent if the new %x 
macros (where it makes sense of course) could be applied to the -Xloggc flag 
even when rotation is not turned on.

Regards,
Kirk

Re: [PATCH] EnableTracing: output from multiple threads may be mixed together

[Kirk Pepperdine]

What effect with this have on the other mixed up log files?

Regards,
Kirk

On 2013-05-02, at 2:57 PM, Staffan Larsen staffan.lar...@oracle.com wrote:

 Looks good. (not a reviewer)
 
 /Staffan
 
 On 2 maj 2013, at 05:07, 云达(Yunda) yunda@taobao.com wrote:
 
 Could anyone review this for me, please?
  
 Regards,
 Yunda
  
 From: 云达(Yunda) 
 Sent: Friday, April 19, 2013 4:26 PM
 To: hotspot-runtime-...@openjdk.java.net; serviceability-dev@openjdk.java.net
 Subject: [PATCH] EnableTracing: output from multiple threads may be mixed 
 together
  
 Hi all,
  
 I found that the output from multiple threads may be mixed together when 
 using EnableTracing. It happens many times in my test case like this:
  
 Allocation outside TLAB: [Allocation in new TLAB: [Allocation in new TLAB: 
 [Allocation in new TLAB: [Class = [C, Allocation in new TLAB: [Class = [I, 
 Allocation in new TLAB: [Java Monitor Wait: [Class = java/lang/String, 
 Allocation Size = 24, Allocation Size = 24, Class = java/lang/String, Class 
 = [I, Allocation in new TLAB: [Class = [C, Allocation in new TLAB: 
 [Allocation Size = 192]
 Class = com/sun/org/apache/xerces/internal/dom/DeferredElementNSImpl, 
 Allocation Size = 24, Allocation Size = 24, TLAB Size = 23712280]
 TLAB Size = 24607080]
 Allocation Size = 24, Monitor Class = java/lang/ref/Reference$Lock, TLAB 
 Size = 25054480]
 TLAB Size = 25054480]
 Allocation in new TLAB: [Class = [CTLAB Size = 24607080]
 Allocation Size = 72, Class = [C, TLAB Size = 24159728]
 , Allocation Size = 32, TLAB Size = 23712288]
  
 It's very confusing and it's even not easy to tell how many events there 
 are. I think the reason is that the writeEvent() method of each Event* class 
 output the fields of event one by one without using any lock. So I made a 
 small patch which add ttyLocker to writeEvent() method and after applying 
 this patch there's no output mixed together in my test case(against 
 http://hg.openjdk.java.net/hsx/hsx24/hotspot/):
 diff -r edd1619a3ae4 src/share/vm/trace/traceEventClasses.xsl
 --- a/src/share/vm/trace/traceEventClasses.xslThu Apr 18 13:50:58 2013 
 -0700
 +++ b/src/share/vm/trace/traceEventClasses.xsl Fri Apr 19 16:12:38 2013 +0800
 @@ -132,6 +132,7 @@
void writeEvent(void) {
  ResourceMark rm;
  HandleMark hm;
 +ttyLocker ttyl;
  TraceStream ts(*tty);
  ts.print(xsl:value-of select=@label/: [);
 xsl:apply-templates select=value|structvalue mode=write-data/
  
  
 I searched before sending this mail I didn't find anyone who covering thisJ
  
 Regards,
 Yunda
  
 
 
 This email (including any attachments) is confidential and may be legally 
 privileged. If you received this email in error, please delete it 
 immediately and do not copy it or use it for any purpose or disclose its 
 contents to any other person. Thank you.
 
 本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮件之内容。谢谢。
 

Re: [PATCH] EnableTracing: output from multiple threads may be mixed together

[Kirk Pepperdine]

Hi Yunda,

Sorry for not being clear about this. The CMS collector injects records into 
the middle of young gen (ParNew/DefNew) collections. Sorry for the noise but 
this is due to a different issue in how logging is performed.

Regards,
Kirk

On 2013-05-02, at 3:42 PM, 云达(Yunda) yunda@taobao.com wrote:

 Kirk,
 
 I'm not quite clear what the other mixed up log files means, since when 
 using EnableTracing the output uses stdout by default.
 
 
 Regards,
 Yunda
 
 From: Kirk Pepperdine [k...@kodewerk.com]
 Sent: Thursday, May 02, 2013 21:05
 To: Staffan Larsen
 Cc: 云达(Yunda); serviceability-dev@openjdk.java.net; 
 hotspot-runtime-...@openjdk.java.net
 Subject: Re: [PATCH] EnableTracing: output from multiple threads may be mixed 
 together
 
 What effect with this have on the other mixed up log files?
 
 Regards,
 Kirk
 
 On 2013-05-02, at 2:57 PM, Staffan Larsen 
 staffan.lar...@oracle.commailto:staffan.lar...@oracle.com wrote:
 
 Looks good. (not a reviewer)
 
 /Staffan
 
 On 2 maj 2013, at 05:07, 云达(Yunda) 
 yunda@taobao.commailto:yunda@taobao.com wrote:
 
 Could anyone review this for me, please?
 
 Regards,
 Yunda
 
 From: 云达(Yunda)
 Sent: Friday, April 19, 2013 4:26 PM
 To: 
 hotspot-runtime-...@openjdk.java.netmailto:hotspot-runtime-...@openjdk.java.net;
  
 serviceability-dev@openjdk.java.netmailto:serviceability-dev@openjdk.java.net
 Subject: [PATCH] EnableTracing: output from multiple threads may be mixed 
 together
 
 Hi all,
 
 I found that the output from multiple threads may be mixed together when 
 using EnableTracing. It happens many times in my test case like this:
 
 Allocation outside TLAB: [Allocation in new TLAB: [Allocation in new TLAB: 
 [Allocation in new TLAB: [Class = [C, Allocation in new TLAB: [Class = [I, 
 Allocation in new TLAB: [Java Monitor Wait: [Class = java/lang/String, 
 Allocation Size = 24, Allocation Size = 24, Class = java/lang/String, Class = 
 [I, Allocation in new TLAB: [Class = [C, Allocation in new TLAB: [Allocation 
 Size = 192]
 Class = com/sun/org/apache/xerces/internal/dom/DeferredElementNSImpl, 
 Allocation Size = 24, Allocation Size = 24, TLAB Size = 23712280]
 TLAB Size = 24607080]
 Allocation Size = 24, Monitor Class = java/lang/ref/Reference$Lock, TLAB Size 
 = 25054480]
 TLAB Size = 25054480]
 Allocation in new TLAB: [Class = [CTLAB Size = 24607080]
 Allocation Size = 72, Class = [C, TLAB Size = 24159728]
 , Allocation Size = 32, TLAB Size = 23712288]
 
 It's very confusing and it's even not easy to tell how many events there are. 
 I think the reason is that the writeEvent() method of each Event* class 
 output the fields of event one by one without using any lock. So I made a 
 small patch which add ttyLocker to writeEvent() method and after applying 
 this patch there's no output mixed together in my test case(against 
 http://hg.openjdk.java.net/hsx/hsx24/hotspot/):
 diff -r edd1619a3ae4 src/share/vm/trace/traceEventClasses.xsl
 --- a/src/share/vm/trace/traceEventClasses.xslThu Apr 18 13:50:58 2013 
 -0700
 +++ b/src/share/vm/trace/traceEventClasses.xsl Fri Apr 19 16:12:38 2013 +0800
 @@ -132,6 +132,7 @@
   void writeEvent(void) {
 ResourceMark rm;
 HandleMark hm;
 +ttyLocker ttyl;
 TraceStream ts(*tty);
 ts.print(xsl:value-of select=@label/: [);
 xsl:apply-templates select=value|structvalue mode=write-data/
 
 
 I searched before sending this mail I didn't find anyone who covering this☺
 
 Regards,
 Yunda
 
 
 
 
 This email (including any attachments) is confidential and may be legally 
 privileged. If you received this email in error, please delete it immediately 
 and do not copy it or use it for any purpose or disclose its contents to any 
 other person. Thank you.
 
 本电邮(包括任何附件)可能含有机密资料并受法律保护。如您不是正确的收件人，请您立即删除本邮件。请不要将本电邮进行复制并用作任何其他用途、或透露本邮件之内容。谢谢。
 
 

Re: JEP 158: Unified JVM logging

[Kirk Pepperdine]

Hi Staffan,

Thanks for the response.

You call it levels, I call is at hierarchy. I'm happy to change to the work 
levels.
On 2012-08-17, at 10:20 PM, Staffan Larsen staffan.lar...@oracle.com wrote:

 
 On 15 aug 2012, at 16:19, Dmitry Samersoff dmitry.samers...@oracle.com 
 wrote:
 
 On 2012-08-15 12:44, Kirk Pepperdine wrote:
 
 The current system of command-line (experimental) options follow a
 format that is quite consistent. The options provide semantic meaning as
 to what will be logged. This semantic meaning is lost when we reduce
 everything to the level of error, warning, info, debug and trace. 
 
 I guess Levels are independent to categories. i.e. I think
 
 Log:info,gc,younggen should be possible
 
 The suggestion in the JEP is that each category/component has several levels, 
 so in that sense they are not independent. To enable logging you would 
 specify a list of {component, level} tuples, like:
 
  gc:info, younggen:trace
 
 But to make it easier to type in the most common case the 'info' level is 
 considered 'default' if nothing else is specified. So the above would be 
 equivalent to saying just
 
  gc, younggen:trace

Right, and this is at the heart of one of my objections is that the predefined 
levels used by everyone lack semantic meaning where as being able to define 
tags or groups of tags would allow one to define levels but not force one to 
use levels. There certainly is a case for levels and I would not want them 
precluded but I do not believe it is desirable for all components to 
hierarchically reduce log messages.

If I had a better idea of the schedule for the JEP I'd have a better 
understanding of when I might be able to pitch in rather then just play seagull.

Regards,
Kirk

Re: JEP 158: Unified JVM logging

[Kirk Pepperdine]

Hi Dmitry,

Lets start with this.

Common logging command-line options for all components
Logging is performed at different levels: error, warning, info, debug, trace
Logging messages are in human-readable plain text

IMHO, these current set of goals are based on the assumptions that all 
component is hierarchical in nature and that they will log the same way without 
the possibility of binary dumps. I would argue these assumptions over-reach and 
preclude certain types of logging that take place in the JVM today.

The current system of command-line (experimental) options follow a format that 
is quite consistent. The options provide semantic meaning as to what will be 
logged. This semantic meaning is lost when we reduce everything to the level of 
error, warning, info, debug and trace. It's sets up arbitrary categories in 
which developers will be left to decide what level a particular activity should 
be logged at. To make this real I'll use the -XX:+PrintTenuringDistribution 
flag as an example. When I set that flag it's very clear what's to be logged. 
My question is, should this be logged at the info level? How about debug? Maybe 
trace? And if I set those levels, what else comes along for the ride? IOWs, if 
I attempt to keep the same model of GC logging, we'd need different levels. If 
we need different levels then we're about to violate the first goal.

What this example suggests is that gc logging would be better served by using 
categories (or subjects if we can take something from the pub/sub metaphor) or 
tags or labels or what ever you want to call them. Using tags, I can create a 
system of levels using error, warning, info, debug, and trace if that fits how 
I'm setting up logging. But from levels it's difficult to implement things 
using categories. This is a case where the one of the non-goals should be, we 
shouldn't prevent people from structuring their logs as they see best fits the 
component they are instrumenting.

The non-goals that should be reconsidered is to force human-readability. I 
would argue that this is again over-reaching in that properly structured log 
files should almost *never* be read by a human. In the cases why force an 
unnecessary requirement. I say this because I've run into a number of 
applications where logging was the primary bottleneck. In those cases there 
were two primary problems, one was bulk and the other was threading (lock 
contention which I shan't address here). Now while trying to solve the problem 
of bulkiness and lock contention might be considered a premature optimization 
at this stage and I certainty wouldn't disagree, I would not like to see 
designs or specifications that would naturally preclude some very obvious 
solutions. One of the solutions to bulk is to use a more compact format be some 
form of binary. That option has been taken away for what I would call an 
ideological requirement. There is no technical reason why I shouldn't be able 
to log in binary other than connivence. In addition, you don't know what else 
is happening on the system (disk??) you're logging to so it behoves you to be 
as light as possible.

I'll leave it at this for now.

-- Kirk

On 2012-08-14, at 12:17 PM, Dmitry Samersoff dmitry.samers...@oracle.com 
wrote:

 Kirk,
 
 However I do have very serious concerns about this JEP in that it
 doesn't fix the problems that exist in current logging frameworks,
 it only mimics them.
 
 http://openjdk.java.net/jeps/158
 
 Any comments is much appreciated.
 
 Personally, I think that log rotation is out of scope and responsibility
 of JVM.
 
 -Dmitry
 
 
 On 2012-08-14 13:26, Kirk Pepperdine wrote:
 Hi Yasumasa,
 
 I'm not sure that log file rotation is a part of this JEP. 
 However I do have very serious concerns about this JEP in that it doesn't 
 fix the problems that exist in current logging frameworks, it only mimics 
 them.
 
 Regards,
 Kirk
 
 On 2012-08-14, at 11:20 AM, Yasumasa Suenaga 
 suenaga.yasum...@lab.ntt.co.jp wrote:
 
 Hi Staffan,
 
 May I ask you 2 questions about this JEP?
 
 1. One of goals of this JEP is defined as following:
 File rotation of log files by size (similar to what is available for 
 GC logs today)
   My patch realizes log rotation by external trigger.
   7090324 is included in this JEP?
   (Is 7090324 included in Logging can be configured dynamically at runtime 
 via jcmd or MBeans ? )
 
 2. I've posted a patch for 6950794: Make the GC log file name 
 parameterized .
 
 http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2012-July/004758.html
   Could you include this RFE in this JEP?
   If we use log rotation, I think that name of logs is very important for 
 log management.
 
 
 Thanks,
 
 Yasumasa
 
 
 On 2012/08/14 16:50, Staffan Larsen wrote:
 Hi Yasumasa,
 
 This work should be done in the context of the Unified Logging JEP [1]. 
 Unfortunately, that work has not yet started due to resource constraints. 
 Comments on the proposal are welcome.
 
 [1] http

Re: JEP 158: Unified JVM logging

[Kirk Pepperdine]

Hi Dmitry,

I should have added that Neal Ford has an excellent talk on taxonomy systems 
and tags vs. hierarchies. It includes a bit on the 5 kingdoms that all 
organisms are split into. Yet many organisms show characteristics from more 
than one kingdom while some others show characteristics of none. Problem is, 
the current hierarchical system forces categorization in a way that doesn't 
work where as a tag system would easily be able to cope. I'm trying to find a 
link to the talk.

Regards,
Kirk

On 2012-08-14, at 12:17 PM, Dmitry Samersoff dmitry.samers...@oracle.com 
wrote:

 Kirk,
 
 However I do have very serious concerns about this JEP in that it
 doesn't fix the problems that exist in current logging frameworks,
 it only mimics them.
 
 http://openjdk.java.net/jeps/158
 
 Any comments is much appreciated.
 
 Personally, I think that log rotation is out of scope and responsibility
 of JVM.
 
 -Dmitry
 
 
 On 2012-08-14 13:26, Kirk Pepperdine wrote:
 Hi Yasumasa,
 
 I'm not sure that log file rotation is a part of this JEP. 
 However I do have very serious concerns about this JEP in that it doesn't 
 fix the problems that exist in current logging frameworks, it only mimics 
 them.
 
 Regards,
 Kirk
 
 On 2012-08-14, at 11:20 AM, Yasumasa Suenaga 
 suenaga.yasum...@lab.ntt.co.jp wrote:
 
 Hi Staffan,
 
 May I ask you 2 questions about this JEP?
 
 1. One of goals of this JEP is defined as following:
 File rotation of log files by size (similar to what is available for 
 GC logs today)
   My patch realizes log rotation by external trigger.
   7090324 is included in this JEP?
   (Is 7090324 included in Logging can be configured dynamically at runtime 
 via jcmd or MBeans ? )
 
 2. I've posted a patch for 6950794: Make the GC log file name 
 parameterized .
 
 http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2012-July/004758.html
   Could you include this RFE in this JEP?
   If we use log rotation, I think that name of logs is very important for 
 log management.
 
 
 Thanks,
 
 Yasumasa
 
 
 On 2012/08/14 16:50, Staffan Larsen wrote:
 Hi Yasumasa,
 
 This work should be done in the context of the Unified Logging JEP [1]. 
 Unfortunately, that work has not yet started due to resource constraints. 
 Comments on the proposal are welcome.
 
 [1] http://openjdk.java.net/jeps/158
 
 /Staffan
 
 On 31 jul 2012, at 12:45, Yasumasa Suenaga suenaga.yasum...@lab.ntt.co.jp 
 mailto:suenaga.yasum...@lab.ntt.co.jp wrote:
 
 Hi,
 
 I've posted a patch for gc log rotation via jinfo tool last year.
 Then I received an email that essence of my patch will be implemented as
 subcommands of jcmd.
 
 Now, jcmd is provided by OpenJDK. However, current jcmd does not seem to
 implement function of gclog rotation.
 
 Will this RFE be implemented?
 
 I need to rotate gclog which is synchronized with signal, cron, etc... on 
 Linux.
 So, if this RFE is not implemented for a while, I would like to contribute
 patch for jcmd.
 
 
 Regards,
 
 Yasumasa
 
 
 On 2011/09/29 21:15, James Melvin wrote:
 Hi Yasumasa,
 
 Thanks very much for your OpenJDK contribution! As part of the effort to
 port JRockit features to HotSpot, we plan to introduce a consolidated
 commandline serviceability tool (jcmd) to potentially replace many of
 the existing tools in the bin directory, such as jmap, jstack, jinfo and
 others. Over the next few update releases, we plan to add several jcmd
 *subcommands* instead to accomplish specific tasks or affect the running
 JVM instance. We'd like to use the essence of your contribution in one
 of the jcmd subcommands (instead of extending jinfo) to ask the JVM to
 begin rotating GC logs. We hope you find this attractive and hope you
 will help review and perfect the change.
 
 Thanks,
 
 Jim Melvin
 Sr. Engineering Manager
 Oracle
 
 
 
 On 9/26/11 6:38 AM, Yasumasa Suenaga wrote:
 (I've changed subject of this email to new RFE.)
 
 This RFE is enhancement of current gclog implementation.
 So, I'd like to discuss about rotating gclog.
 
 My customers need gclog rotation which is invoked by external trigger.
 So I've requested this RFE and made a patch.
 
 
 In many case on Linux, logfile is rotated by signal (e.g. SIGHUP) .
 The aim of this RFE is to synchronize gclog and the other logs.
 
 
 Thanks,
 
 Yasumasa
 
 (2011/09/22 20:55), Rainer Jung wrote:
 On 22.09.2011 13:20, Dmitry Samersoff wrote:
 
 Yasumasa,
 
 On 2011-09-22 04:47, Yasumasa Suenaga wrote:
 If we can think Java on Linux and Solaris only, syslog is best
 solution.
 However, Windows usually doesn't have syslog.
 
 So, I think that gclog is needed for logging GC stats with platform
 independent in realtime.
 
 Windows has it's own logging API as reach as syslog is or ever better
 as well as numerous syslog implementations.
 
 Native windows logging API was completely redesigned for Windows 2008
 server and now it allows for developers to send a structured events 
 from
 theirs application.
 
 AFAIK log rotation for loggc is already

Re: JEP 158: Unified JVM logging

[Kirk Pepperdine]

Hi Dmitry,

Neal Ford talk is call Abstraction Distraction.. it's downloadable and you 
can see it here. http://vimeo.com/44235657

-- Kirk

On 2012-08-14, at 12:17 PM, Dmitry Samersoff dmitry.samers...@oracle.com 
wrote:

 Kirk,
 
 However I do have very serious concerns about this JEP in that it
 doesn't fix the problems that exist in current logging frameworks,
 it only mimics them.
 
 http://openjdk.java.net/jeps/158
 
 Any comments is much appreciated.
 
 Personally, I think that log rotation is out of scope and responsibility
 of JVM.
 
 -Dmitry
 
 
 On 2012-08-14 13:26, Kirk Pepperdine wrote:
 Hi Yasumasa,
 
 I'm not sure that log file rotation is a part of this JEP. 
 However I do have very serious concerns about this JEP in that it doesn't 
 fix the problems that exist in current logging frameworks, it only mimics 
 them.
 
 Regards,
 Kirk
 
 On 2012-08-14, at 11:20 AM, Yasumasa Suenaga 
 suenaga.yasum...@lab.ntt.co.jp wrote:
 
 Hi Staffan,
 
 May I ask you 2 questions about this JEP?
 
 1. One of goals of this JEP is defined as following:
 File rotation of log files by size (similar to what is available for 
 GC logs today)
   My patch realizes log rotation by external trigger.
   7090324 is included in this JEP?
   (Is 7090324 included in Logging can be configured dynamically at runtime 
 via jcmd or MBeans ? )
 
 2. I've posted a patch for 6950794: Make the GC log file name 
 parameterized .
 
 http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2012-July/004758.html
   Could you include this RFE in this JEP?
   If we use log rotation, I think that name of logs is very important for 
 log management.
 
 
 Thanks,
 
 Yasumasa
 
 
 On 2012/08/14 16:50, Staffan Larsen wrote:
 Hi Yasumasa,
 
 This work should be done in the context of the Unified Logging JEP [1]. 
 Unfortunately, that work has not yet started due to resource constraints. 
 Comments on the proposal are welcome.
 
 [1] http://openjdk.java.net/jeps/158
 
 /Staffan
 
 On 31 jul 2012, at 12:45, Yasumasa Suenaga suenaga.yasum...@lab.ntt.co.jp 
 mailto:suenaga.yasum...@lab.ntt.co.jp wrote:
 
 Hi,
 
 I've posted a patch for gc log rotation via jinfo tool last year.
 Then I received an email that essence of my patch will be implemented as
 subcommands of jcmd.
 
 Now, jcmd is provided by OpenJDK. However, current jcmd does not seem to
 implement function of gclog rotation.
 
 Will this RFE be implemented?
 
 I need to rotate gclog which is synchronized with signal, cron, etc... on 
 Linux.
 So, if this RFE is not implemented for a while, I would like to contribute
 patch for jcmd.
 
 
 Regards,
 
 Yasumasa
 
 
 On 2011/09/29 21:15, James Melvin wrote:
 Hi Yasumasa,
 
 Thanks very much for your OpenJDK contribution! As part of the effort to
 port JRockit features to HotSpot, we plan to introduce a consolidated
 commandline serviceability tool (jcmd) to potentially replace many of
 the existing tools in the bin directory, such as jmap, jstack, jinfo and
 others. Over the next few update releases, we plan to add several jcmd
 *subcommands* instead to accomplish specific tasks or affect the running
 JVM instance. We'd like to use the essence of your contribution in one
 of the jcmd subcommands (instead of extending jinfo) to ask the JVM to
 begin rotating GC logs. We hope you find this attractive and hope you
 will help review and perfect the change.
 
 Thanks,
 
 Jim Melvin
 Sr. Engineering Manager
 Oracle
 
 
 
 On 9/26/11 6:38 AM, Yasumasa Suenaga wrote:
 (I've changed subject of this email to new RFE.)
 
 This RFE is enhancement of current gclog implementation.
 So, I'd like to discuss about rotating gclog.
 
 My customers need gclog rotation which is invoked by external trigger.
 So I've requested this RFE and made a patch.
 
 
 In many case on Linux, logfile is rotated by signal (e.g. SIGHUP) .
 The aim of this RFE is to synchronize gclog and the other logs.
 
 
 Thanks,
 
 Yasumasa
 
 (2011/09/22 20:55), Rainer Jung wrote:
 On 22.09.2011 13:20, Dmitry Samersoff wrote:
 
 Yasumasa,
 
 On 2011-09-22 04:47, Yasumasa Suenaga wrote:
 If we can think Java on Linux and Solaris only, syslog is best
 solution.
 However, Windows usually doesn't have syslog.
 
 So, I think that gclog is needed for logging GC stats with platform
 independent in realtime.
 
 Windows has it's own logging API as reach as syslog is or ever better
 as well as numerous syslog implementations.
 
 Native windows logging API was completely redesigned for Windows 2008
 server and now it allows for developers to send a structured events 
 from
 theirs application.
 
 AFAIK log rotation for loggc is already implemented though not
 necessarily yet released. The change discussed here is about supporting
 an externally generated rotation trigger, e.g. via a signal, instead of
 only rotating by size or time via a startup parameter.
 
 If you want support for syslog or Windows APIs included, it would be
 best to start a new discussion.
 
 A GC log for an application under

Re: JEP 158: Unified JVM logging

[Kirk Pepperdine]

On 2012-08-15, at 11:29 AM, Dmitry Samersoff dmitry.samers...@oracle.com 
wrote:

 Kirk,
 
 Thank you. I'm downloading it.
 
 PS:
 Tags vs hierarchy discussion is as old as taxonomy it self.

which is why I'm surprised we're here ;-)

Kirk

Re: JEP 158: Unified JVM logging

[Kirk Pepperdine]

On 2012-08-15, at 3:28 PM, Dmitry Samersoff dmitry.samers...@oracle.com wrote:

 Kirk,
 
 On 2012-08-15 12:56, Kirk Pepperdine wrote:
 I should have added that Neal Ford has an excellent talk on taxonomy systems 
 and tags vs. hierarchies. 
 It includes a bit on the 5 kingdoms that all organisms are split into. Yet 
 many organisms show characteristics 
 from more than one kingdom while some others show characteristics of none. 
 Problem is, 
 the current hierarchical system forces categorization in a way that doesn't 
 work where 
 as a tag system would easily be able to cope. I'm trying to find a link to 
 the talk.
 
 Really good talk.
 
 Stepping aside from logging problem -
 
 hierarchy require additional efforts from a people who builds hierarchy,
 but tags put the same efforts to people who uses it.
 
 If I'm constructing new butterfly and check description of Insecta,
 Lepidoptera I should get a list of the most important characteristics of
 all butterflies. It's responsibility of persons building hierarchy to
 create this list. I trust them and believe that if I give all these
 characteristics to my being it become a butterfly.

I think the point of Neal's talk is this type of taxonomy is better suited to 
tags. As you mentioned, a butterfly has wings.. but so do bees.. and birds.. 
and bats. So maybe not such a good idea to use wings a differentiator in a 
hierarchy. But as a tag it works out very well. A taxonomy for butterflies 
would simply include all the relevant tags.

 
 But with tags I should build the list of characteristics by my self,
 e.g. start from insects with tags four wings, and then refine the mix
 of Lepidoptera and Hymenoptera using other tags to distinguish between
 butterfly and bees.
 
 So tags give you more power but require much more efforts in regular case.

I guess what I've also implicitly said is that the current GC logging system is 
essentially tag based. My feeling is that tags could be cleaned up which means 
I could probably get everything that I might want in about a dozen flags or 
less. I don't really want to get into an implementation discussion so lets just 
call the following a bad implementation and go from there.

java -log:gc,pauseTime,tenuring,stoppedTime,occupancyConfiguredSize,tag5,tag6  
my.main

This seems about the same amount of effort as it to log today which isn't 
insignificant but it's also not a big burden that it's worth the loss of 
flexibility that is currently enjoyed. And again, tags don't preclude someone 
defining hierarchies of tags if that makes sense to they way their component 
works.

-- Kirk

Re: 7090324: gclog rotation via external tool

[Kirk Pepperdine]

Hi Yasumasa,

I'm not sure that log file rotation is a part of this JEP. However I do have 
very serious concerns about this JEP in that it doesn't fix the problems that 
exist in current logging frameworks, it only mimics them.

Regards,
Kirk

On 2012-08-14, at 11:20 AM, Yasumasa Suenaga suenaga.yasum...@lab.ntt.co.jp 
wrote:

 Hi Staffan,
 
 May I ask you 2 questions about this JEP?
 
 1. One of goals of this JEP is defined as following:
  File rotation of log files by size (similar to what is available for GC 
 logs today)
My patch realizes log rotation by external trigger.
7090324 is included in this JEP?
(Is 7090324 included in Logging can be configured dynamically at runtime 
 via jcmd or MBeans ? )
 
 2. I've posted a patch for 6950794: Make the GC log file name parameterized 
 .
  
 http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2012-July/004758.html
Could you include this RFE in this JEP?
If we use log rotation, I think that name of logs is very important for 
 log management.
 
 
 Thanks,
 
 Yasumasa
 
 
 On 2012/08/14 16:50, Staffan Larsen wrote:
 Hi Yasumasa,
 
 This work should be done in the context of the Unified Logging JEP [1]. 
 Unfortunately, that work has not yet started due to resource constraints. 
 Comments on the proposal are welcome.
 
 [1] http://openjdk.java.net/jeps/158
 
 /Staffan
 
 On 31 jul 2012, at 12:45, Yasumasa Suenaga suenaga.yasum...@lab.ntt.co.jp 
 mailto:suenaga.yasum...@lab.ntt.co.jp wrote:
 
 Hi,
 
 I've posted a patch for gc log rotation via jinfo tool last year.
 Then I received an email that essence of my patch will be implemented as
 subcommands of jcmd.
 
 Now, jcmd is provided by OpenJDK. However, current jcmd does not seem to
 implement function of gclog rotation.
 
 Will this RFE be implemented?
 
 I need to rotate gclog which is synchronized with signal, cron, etc... on 
 Linux.
 So, if this RFE is not implemented for a while, I would like to contribute
 patch for jcmd.
 
 
 Regards,
 
 Yasumasa
 
 
 On 2011/09/29 21:15, James Melvin wrote:
 Hi Yasumasa,
 
 Thanks very much for your OpenJDK contribution! As part of the effort to
 port JRockit features to HotSpot, we plan to introduce a consolidated
 commandline serviceability tool (jcmd) to potentially replace many of
 the existing tools in the bin directory, such as jmap, jstack, jinfo and
 others. Over the next few update releases, we plan to add several jcmd
 *subcommands* instead to accomplish specific tasks or affect the running
 JVM instance. We'd like to use the essence of your contribution in one
 of the jcmd subcommands (instead of extending jinfo) to ask the JVM to
 begin rotating GC logs. We hope you find this attractive and hope you
 will help review and perfect the change.
 
 Thanks,
 
 Jim Melvin
 Sr. Engineering Manager
 Oracle
 
 
 
 On 9/26/11 6:38 AM, Yasumasa Suenaga wrote:
 (I've changed subject of this email to new RFE.)
 
 This RFE is enhancement of current gclog implementation.
 So, I'd like to discuss about rotating gclog.
 
 My customers need gclog rotation which is invoked by external trigger.
 So I've requested this RFE and made a patch.
 
 
 In many case on Linux, logfile is rotated by signal (e.g. SIGHUP) .
 The aim of this RFE is to synchronize gclog and the other logs.
 
 
 Thanks,
 
 Yasumasa
 
 (2011/09/22 20:55), Rainer Jung wrote:
 On 22.09.2011 13:20, Dmitry Samersoff wrote:
 
 Yasumasa,
 
 On 2011-09-22 04:47, Yasumasa Suenaga wrote:
 If we can think Java on Linux and Solaris only, syslog is best
 solution.
 However, Windows usually doesn't have syslog.
 
 So, I think that gclog is needed for logging GC stats with platform
 independent in realtime.
 
 Windows has it's own logging API as reach as syslog is or ever better
 as well as numerous syslog implementations.
 
 Native windows logging API was completely redesigned for Windows 2008
 server and now it allows for developers to send a structured events from
 theirs application.
 
 AFAIK log rotation for loggc is already implemented though not
 necessarily yet released. The change discussed here is about supporting
 an externally generated rotation trigger, e.g. via a signal, instead of
 only rotating by size or time via a startup parameter.
 
 If you want support for syslog or Windows APIs included, it would be
 best to start a new discussion.
 
 A GC log for an application under load might easily produce a block of
 about 1.5 KB size every few seconds. I seriously doubt, that the need
 for log file rotation can be argued away by referring to syslog or
 Windows log API as the correct solution.
 
 The messages are not really line formatted, the format can vary a lot
 (depending on the excat XX switches), the traffic can be quite high and
 AFAIK the JVM writes it synchronously, so there must be absolutely no
 risk in writing it out with very little latency. In addition, for
 analysis, you wouldn't want to look at each event individually, but
 instead process the whole file through a 

Re: JEP 158

[Kirk Pepperdine]

Hi Staffan,

I've noticed that the specification has been updated but still contains the 
hierarchical level system.  Is it your intention to maintain level in the 
specification?

Regards,
Kirk

On 2012-06-20, at 12:50 PM, Staffan Larsen wrote:

 Hi,
 
 Thanks for the feedback! 
 
 It is going to be hard to find the right balance between a system that 
 provides great power and one that is simple to use and to implement. In 
 general, I will lean more towards making it simple, but I want to make sure 
 we cover the major use cases as well.
 
 It looks like there are three possible ways to select log messages being 
 discussed here. I'd like to discuss how to solve your problem for each one.
 
 1) Selection based on component/level. (This is what is described in the JEP.)
 In this case we can have any number of components. So we can have gc and 
 tenuring as different component. They will each have a level associated with 
 them. However there is no relationship between the gc and the tenuring 
 component, so to enable both you need to say -Xverbose:gc,tenuring. To only 
 enable tenuring you can say -Xverbose:tenuring. To enable all gc messages, 
 except tenuring you say -Xverbose:gc.
 
 2) Selection based on component/level where the component is a hierarchy.
 In this case the different steps in the hierarchy can have different levels 
 associated with them, but if there is no explicit level associated, the 
 parent level will be used. So to enable both tenuring and gc you would say 
 -Xverbose:gc. To enable only gc you would say -Xverbose:gc,gc.tenuring=none 
 (I made up the 'none' level). To enable only tenuring you would say 
 -Xverbose:gc.tenuring.
 
 3) Selection based on tags. 
 In this case log messages are associated with a set of tags instead of a 
 component/level tuple. You select messages by specifying tags you want to 
 see. I imagine that in this case the tenuring messages would be tagged with 
 both the gc and the tenuring tags, but that there would be other messages 
 tagged with gc only. To enable gc and tenuring you would say 
 -Xverbose:gc,tenuring. To enable all gc messages you say -Xverbose:gc. There 
 is no way to see only gc messages without seeing the tenuring messages.
 
 Is this a fair description of the different ways? I'm especially interested 
 in the last one - I'm not sure I captured your idea correctly there.
 
 Thanks,
 /Staffan
 
 
 On 20 jun 2012, at 09:32, Jesper Wilhelmsson wrote:
 
 Hi Kirk,
 
 I'm CC'ing serviceability on this since this is really their JEP and 
 discussions around it should go on the serviceability list, even though it 
 seems you are mainly interested in GC logging at this point.
 
 I understand what you want and I see that the logging level won't help us 
 get there. I don't agree that the logging we have today can't fit nicely 
 into a hierarchical scheme though, it just needs to be more fine grained to 
 achieve what you want.
 
 We can be pretty generous with modules and in principal have one module for 
 each verbose flag that exists today. Personally I don't think that is a good 
 idea, but it certainly is possible. I would rather like to propose a 
 different solution.
 
 How about we have a gc module that can be filtered based on different sub 
 modules. The sub modules could be fairly close to the existing verbose flags 
 that exists today if that turns out to be a good way to divide information. 
 It could look like this
 
 -Xverbose:gc=info,gc.tenuring=debug
 
 to set regular gc verbose to info level (I would say that is close to 
 PrintGC) and turn on more detailed logging for tenuring. Or
 
 -Xverbose:gc.tenuring
 
 that could be equal to what that flag prints today. Let's see what the 
 serviceability team thinks since they are the ones who will actually 
 implement this in the end.
 
 Another solution that I don't really like but guess is easier to implement 
 is to add the current verbose flag to the actual message so that the logs 
 can be filtered based on that. But this will clutter the messages and we 
 would still have the problem to decide on which level things should get 
 logged.
 /Jesper
 
 
 On 2012-06-20 07:28, Kirk Pepperdine wrote:
 
 Hi Jesper,
 
 I did read the spec and I do like the ability to specify the component 
 that you'd like to log information from. So I feel that is a great 
 improvement over the (broken) pattern established in every major logging 
 Java framework. I'm going to stick to GC logging just because I've spent so 
 much time puzzling over them and adjusting my parser to deal with all the 
 changes that have continuously crept into them. While 'm certainly not 
 going to argue for keeping the current GC logging framework what I will say 
 is that it's not all bad in that the flags that have been provided to me 
 are almost always semantically meaningful in that they tell me what I'm 
 going to see. In this spirit I'd like to see a category like 
 TenuringDetails for example. Is this information

Re: JEP 158

[Kirk Pepperdine]

Hi Staffan,

 Thanks for the feedback! 
 
 It is going to be hard to find the right balance between a system that 
 provides great power and one that is simple to use and to implement. In 
 general, I will lean more towards making it simple, but I want to make sure 
 we cover the major use cases as well.

agreed, levels is brain dead simple but at the same time it is brain dead.. 
where as tag based systems might be a bit more difficult to admin... but I 
would argue that they are also simple and certainly more powerful.

 
 It looks like there are three possible ways to select log messages being 
 discussed here. I'd like to discuss how to solve your problem for each one.
 
 1) Selection based on component/level. (This is what is described in the JEP.)
 In this case we can have any number of components. So we can have gc and 
 tenuring as different component. They will each have a level associated with 
 them. However there is no relationship between the gc and the tenuring 
 component, so to enable both you need to say -Xverbose:gc,tenuring. To only 
 enable tenuring you can say -Xverbose:tenuring. To enable all gc messages, 
 except tenuring you say -Xverbose:gc.

This is great start on the path to tag based logging

 
 2) Selection based on component/level where the component is a hierarchy.
 In this case the different steps in the hierarchy can have different levels 
 associated with them, but if there is no explicit level associated, the 
 parent level will be used. So to enable both tenuring and gc you would say 
 -Xverbose:gc. To enable only gc you would say -Xverbose:gc,gc.tenuring=none 
 (I made up the 'none' level). To enable only tenuring you would say 
 -Xverbose:gc.tenuring.

And this is where things jump the rails IMHO.

 
 3) Selection based on tags. 
 In this case log messages are associated with a set of tags instead of a 
 component/level tuple. You select messages by specifying tags you want to 
 see. I imagine that in this case the tenuring messages would be tagged with 
 both the gc and the tenuring tags, but that there would be other messages 
 tagged with gc only. To enable gc and tenuring you would say 
 -Xverbose:gc,tenuring. To enable all gc messages you say -Xverbose:gc. There 
 is no way to see only gc messages without seeing the tenuring messages.

If would be unfortunate if there was no way to see only GC messages without 
having to eat the tenuring records. As described this isn't a tag system, or an 
aspect oriented log system, it's still levels. I guess it would be good to have 
connivence tags that turned on a bundle of other tags. that way people who like 
error, warning, info fine, finner and finest can have it that way. Where as 
those that want to turn on logging for a particular aspect can just get to 
those messages. This makes more sense for GC logging but I think it also makes 
more sense for HotSpot also. In that environment there are things that I might 
only want at a debug level but if I turn on a debug level log I might get a ton 
of other unintended stuff with it. If I can specify a tag (like I can currently 
do with the current set of JVM flags) then I can get the data I need.

I see this level of granularity somewhat important given that I often have to 
dig into production environments where everyone is very sensitive to the 
possible latency that maybe induced. It's often very comforting to say I can 
turn this on, we get this small amount of data that is very relevant to the 
problem at hand without flooding the system with tons and tons of messages that 
simply take up disk bandwidth and disk space.

 
 Is this a fair description of the different ways? I'm especially interested 
 in the last one - I'm not sure I captured your idea correctly there.

Hopefully this last description fixes things  ;-)

Cheers,
Kirk

Re: JEP 158

[Kirk Pepperdine]

Hi Jesper, 

On 2012-06-20, at 9:32 AM, Jesper Wilhelmsson wrote:

 Hi Kirk,
 
 I'm CC'ing serviceability on this since this is really their JEP and 
 discussions around it should go on the serviceability list, even though it 
 seems you are mainly interested in GC logging at this point.

I'm not only interested in GC but I'm using GC as an exemplar

 
 I understand what you want and I see that the logging level won't help us get 
 there. I don't agree that the logging we have today can't fit nicely into a 
 hierarchical scheme though, it just needs to be more fine grained to achieve 
 what you want.

I think to do this you have to assume structure which may or may not apply to 
everyone. In this case I'd rather drop the assumption and work towards a 
solution that doesn't prevent but enables.

 
 We can be pretty generous with modules and in principal have one module for 
 each verbose flag that exists today. Personally I don't think that is a good 
 idea, but it certainly is possible. I would rather like to propose a 
 different solution.
 
 How about we have a gc module that can be filtered based on different sub 
 modules. The sub modules could be fairly close to the existing verbose flags 
 that exists today if that turns out to be a good way to divide information. 
 It could look like this
 
 -Xverbose:gc=info,gc.tenuring=debug
 
 to set regular gc verbose to info level (I would say that is close to 
 PrintGC) and turn on more detailed logging for tenuring. Or
 
 -Xverbose:gc.tenuring

I predicted that you'd come back with the way to shoehorn the problem into 
leveling. I don't really see this as an appropriate solution as in this case 
because tenuring distribution is only one aspect of logging. Maybe that's what 
I we need, a new term for logging.. I'll call this Aspect Oriented Logging 
which I see as being completely different than hierarchical logging which is 
the quagmire we've been stuck in for far too long.

 
 that could be equal to what that flag prints today. Let's see what the 
 serviceability team thinks since they are the ones who will actually 
 implement this in the end.
 
 Another solution that I don't really like but guess is easier to implement is 
 to add the current verbose flag to the actual message so that the logs can be 
 filtered based on that. But this will clutter the messages and we would still 
 have the problem to decide on which level things should get logged.

IMHO, we don't have a good taxonomy for logging categories and instead of 
over-reaching and forcing one on everybody, why not come to a specification 
that would allow groups to define their own. So again, I ask the question, what 
would the specification look like with levels taken out.

Regards,
Kirk


 /Jesper
 
 
 On 2012-06-20 07:28, Kirk Pepperdine wrote:
 
 Hi Jesper,
 
 I did read the spec and I do like the ability to specify the component 
 that you'd like to log information from. So I feel that is a great 
 improvement over the (broken) pattern established in every major logging 
 Java framework. I'm going to stick to GC logging just because I've spent so 
 much time puzzling over them and adjusting my parser to deal with all the 
 changes that have continuously crept into them. While 'm certainly not going 
 to argue for keeping the current GC logging framework what I will say is 
 that it's not all bad in that the flags that have been provided to me are 
 almost always semantically meaningful in that they tell me what I'm going to 
 see. In this spirit I'd like to see a category like TenuringDetails for 
 example. Is this information INFO, DEBUG, or TRACE? hard to say but it's 
 clearly TenuringDetails and so this is a subcategory that I'd like to define 
 and it's clearly not a subcategory that you'd want to define a generalize 
 logging framework. And it is here that this specification over-reaches. It 
 tries to define logging categories that are not only are devoid of meaning, 
 they assume a hierarchical structure to them. Going back to GC logging I 
 would argue that while there is some hierarchy in there, most of the 
 messages don't nicely fit into this imposed hierarchical developer centric 
 list of categories.
 
 I think we could easily both agree that it would be ridiculous for me to ask 
 that you add PrintTunuring to the list of levels yet that is exactly what I 
 want. So I guess what I'm asking is, what would the spec look like if you 
 removed the log levels from it and allowed me to define my own or to 
 not even use levels at all.
 
 Regards,
 Kirk
 
 On 2012-06-20, at 1:03 AM, Jesper Wilhelmsson wrote:
 
 Hi Kirk,
 
 To select what should be logged there should be logging modules. A module 
 could be for example class loading, gc, jit compiler etc. The logging level 
 is just a way to control how much logging you want. Setting gc=info would 
 give you some basic gc logging while gc=debug would give you more detailed 
 info.
 
 A typical command line could look like

Re: JEP 158

[Kirk Pepperdine]

Hi Staffan,

I should have added, the existing frameworks are very developer oriented and 
that's ok except that more often than not if operational people that are 
looking at these logs and their perspective and purpose is often quite 
different. I would just like to bring a more operational POV into the 
specification.

Cheers,
Kirk

On 2012-06-20, at 12:50 PM, Staffan Larsen wrote:

 Hi,
 
 Thanks for the feedback! 
 
 It is going to be hard to find the right balance between a system that 
 provides great power and one that is simple to use and to implement. In 
 general, I will lean more towards making it simple, but I want to make sure 
 we cover the major use cases as well.
 
 It looks like there are three possible ways to select log messages being 
 discussed here. I'd like to discuss how to solve your problem for each one.
 
 1) Selection based on component/level. (This is what is described in the JEP.)
 In this case we can have any number of components. So we can have gc and 
 tenuring as different component. They will each have a level associated with 
 them. However there is no relationship between the gc and the tenuring 
 component, so to enable both you need to say -Xverbose:gc,tenuring. To only 
 enable tenuring you can say -Xverbose:tenuring. To enable all gc messages, 
 except tenuring you say -Xverbose:gc.
 
 2) Selection based on component/level where the component is a hierarchy.
 In this case the different steps in the hierarchy can have different levels 
 associated with them, but if there is no explicit level associated, the 
 parent level will be used. So to enable both tenuring and gc you would say 
 -Xverbose:gc. To enable only gc you would say -Xverbose:gc,gc.tenuring=none 
 (I made up the 'none' level). To enable only tenuring you would say 
 -Xverbose:gc.tenuring.
 
 3) Selection based on tags. 
 In this case log messages are associated with a set of tags instead of a 
 component/level tuple. You select messages by specifying tags you want to 
 see. I imagine that in this case the tenuring messages would be tagged with 
 both the gc and the tenuring tags, but that there would be other messages 
 tagged with gc only. To enable gc and tenuring you would say 
 -Xverbose:gc,tenuring. To enable all gc messages you say -Xverbose:gc. There 
 is no way to see only gc messages without seeing the tenuring messages.
 
 Is this a fair description of the different ways? I'm especially interested 
 in the last one - I'm not sure I captured your idea correctly there.
 
 Thanks,
 /Staffan
 
 
 On 20 jun 2012, at 09:32, Jesper Wilhelmsson wrote:
 
 Hi Kirk,
 
 I'm CC'ing serviceability on this since this is really their JEP and 
 discussions around it should go on the serviceability list, even though it 
 seems you are mainly interested in GC logging at this point.
 
 I understand what you want and I see that the logging level won't help us 
 get there. I don't agree that the logging we have today can't fit nicely 
 into a hierarchical scheme though, it just needs to be more fine grained to 
 achieve what you want.
 
 We can be pretty generous with modules and in principal have one module for 
 each verbose flag that exists today. Personally I don't think that is a good 
 idea, but it certainly is possible. I would rather like to propose a 
 different solution.
 
 How about we have a gc module that can be filtered based on different sub 
 modules. The sub modules could be fairly close to the existing verbose flags 
 that exists today if that turns out to be a good way to divide information. 
 It could look like this
 
 -Xverbose:gc=info,gc.tenuring=debug
 
 to set regular gc verbose to info level (I would say that is close to 
 PrintGC) and turn on more detailed logging for tenuring. Or
 
 -Xverbose:gc.tenuring
 
 that could be equal to what that flag prints today. Let's see what the 
 serviceability team thinks since they are the ones who will actually 
 implement this in the end.
 
 Another solution that I don't really like but guess is easier to implement 
 is to add the current verbose flag to the actual message so that the logs 
 can be filtered based on that. But this will clutter the messages and we 
 would still have the problem to decide on which level things should get 
 logged.
 /Jesper
 
 
 On 2012-06-20 07:28, Kirk Pepperdine wrote:
 
 Hi Jesper,
 
 I did read the spec and I do like the ability to specify the component 
 that you'd like to log information from. So I feel that is a great 
 improvement over the (broken) pattern established in every major logging 
 Java framework. I'm going to stick to GC logging just because I've spent so 
 much time puzzling over them and adjusting my parser to deal with all the 
 changes that have continuously crept into them. While 'm certainly not 
 going to argue for keeping the current GC logging framework what I will say 
 is that it's not all bad in that the flags that have been provided to me 
 are almost always semantically meaningful

Re: JEP 158

[Kirk Pepperdine]

 
 
 I understand what you want and I see that the logging level won't help us 
 get there. I don't agree that the logging we have today can't fit nicely 
 into a hierarchical scheme though, it just needs to be more fine grained to 
 achieve what you want.
 
 I think to do this you have to assume structure which may or may not apply 
 to everyone. In this case I'd rather drop the assumption and work towards a 
 solution that doesn't prevent but enables.
 
 Well, the structure is there in the VM. We can't really take that away. 
 Tenuring is and will most likely always be part of the GC.
 I'm not sure I understand how the structure of the VM will prevent people 
 from using the logs efficiently.

I'm not sure that I understand what you don't understand here ;-)

 
 
 
 We can be pretty generous with modules and in principal have one module for 
 each verbose flag that exists today. Personally I don't think that is a 
 good idea, but it certainly is possible. I would rather like to propose a 
 different solution.
 
 How about we have a gc module that can be filtered based on different sub 
 modules. The sub modules could be fairly close to the existing verbose 
 flags that exists today if that turns out to be a good way to divide 
 information. It could look like this
 
 -Xverbose:gc=info,gc.tenuring=debug
 
 to set regular gc verbose to info level (I would say that is close to 
 PrintGC) and turn on more detailed logging for tenuring. Or
 
 -Xverbose:gc.tenuring
 
 I predicted that you'd come back with the way to shoehorn the problem into 
 leveling. I don't really see this as an appropriate solution as in this case 
 because tenuring distribution is only one aspect of logging. Maybe that's 
 what I we need, a new term for logging.. I'll call this Aspect Oriented 
 Logging which I see as being completely different than hierarchical logging 
 which is the quagmire we've been stuck in for far too long.
 
 What I did here was to extend the module system. That is completely separate 
 from the logging levels.

Sorry but your first response suggests logging levels.

 
 that could be equal to what that flag prints today. Let's see what the 
 serviceability team thinks since they are the ones who will actually 
 implement this in the end.
 
 Another solution that I don't really like but guess is easier to implement 
 is to add the current verbose flag to the actual message so that the logs 
 can be filtered based on that. But this will clutter the messages and we 
 would still have the problem to decide on which level things should get 
 logged.
 
 IMHO, we don't have a good taxonomy for logging categories and instead of 
 over-reaching and forcing one on everybody, why not come to a specification 
 that would allow groups to define their own. So again, I ask the question, 
 what would the specification look like with levels taken out.
 
 Well, if you remove the levels from the current specification you'd be left 
 with just the modules. That would work I guess, but it would be a slightly 
 less powerful logging framework. Let me describe what I think that you want 
 to achieve. Please correct me if I'm wrong.

I think you'd be left with a more powerful logging system. More assumption == 
more constraints == less adaptive to things you may need to do. In this case I 
don't think you need to make that extra assumption. Just my opinion.


 
 You'd like to specify that you are for instance interested in tenuring. The 
 logs that you are interested in in this case should contain information about 
 heap, eden and survivor sizes. They should contain tenuring threshold and 
 information about objects that has been tenured like number of objects and 
 their sizes etc. And maybe some more stuff.
 
 A different aspect could be that you are interested in fragmentation. Again 
 you'd like information about heap sizes, free memory in different areas, 
 sizes of the individual free areas and their count. Maybe some timestamps 
 that could tell you how long ago an area was compacted etc.
 
 Obviously aspects could cut through other parts of the VM as well, not just 
 GC logging. Is this roughly what you are looking for?

Yes, this sounds like were heading on the right track.

 
 To be able to achieve this we would have to divide all verbose data into 
 different categories and have a mechanism to based on the aspect select the 
 correct categories for output.

Well, this is how the current set of flags work.

 
 I would say that this is exactly what we can achieve with the proposed 
 logging framework. The only difference is that the user is able to pick and 
 choose exactly what information to include instead of just having an aspect 
 that will give a predefined set of logging events. We could easily provide a 
 set of predefined config files that selects different logging modules based 
 on the desired aspect.

Indeed!

 
 If we add the levels into this we could also specify that some aspect would 
 like to provide very