Re: JDK 15 hidden classes & stack trace visibility

[Mandy Chung]

Can you use StackWalker with StackWalker.Option::SHOW_HIDDEN_FRAMES to 
programmatically find the appropriate frame as an interim solution?


ClassOption.SHOW_IN_STACKTRACE might be one possibility.   IMO we should 
take the time and consider together with JDK-8212620 what other options 
could be explored before we provide an easy simple fix just for this 
specific case.


Mandy

On 9/7/20 11:39 AM, Uwe Schindler wrote:

Thanks Mandy.

In my opinion for the hidden classes an additional enum constant in 
ClassOption would be fine: ClassOption.SHOW_IN_STACKTRACE (or similar).


That would be simplest to implement and allows also for the caller to 
enable it like strong references or nestmate features. So it's 
consistent with the other API parts.


The annotations are good for fine granular hiding (like complex 
classes recursively calling methods). Or for frameworks like Spring.


Should I add a comment to this issue?

We are now doing some testing in Lucene's expressions module (it's 
about scoring formulas in query execution). Here's the draft PR: 
https://github.com/apache/lucene-solr/pull/1837/files 



We use the new features in Lucene when the static initializer detects 
defineHiddenClass using a publicLookup.


Currently a test fails, because of missing debug information in stack 
traces. The executed scoring formula is part of the filename attribute 
in class file. As this is missing, test fails. 


Uwe

Am September 7, 2020 4:44:18 PM UTC schrieb Mandy Chung 
:


Hi Uwe,

It's a future enhancement to allow a class/method to request
filtering from stack trace:
https://bugs.openjdk.java.net/browse/JDK-8212620

This RFE should also take hidden classes into account.

Mandy

On 9/5/20 4:55 AM, Uwe Schindler wrote:

Hi,

I am doing a mockup for dynamically compiled scripts in Apache Lucene (later
also painless scripting in elastcsearch), where I tried to use
Lookup#defineHiddenClass (with default parameters, so weak and no nestmates
- but: for painless scripts of Elasticsearch, nestmates will be great). It
all looks great, as sometimes for every query a new class is generated.

The current approach (Java 8, Java 11) uses a Classloader per script, which
is mostly OK, but hidden classes would be better, especially under high load
with many new classes. The problem I see is: The hidden class and their
methods are also hidden from stack traces, so people neither get the script
source code reference, nor they get the method, which was called at all.

My question is: is it possible to mark such a hidden class in byte code or
by a flag to defineHiddenClass() so it is "unhidden"? - I know you can do
this by a command line option, but that's for developers and debugging only.
In some cases, this is really wanted (like for scripting languages), but
code still wants to use the better class unloading. If this is not possible,
how about extending the Lookup.ClassOption enum (last parameter of
defineHiddenClass), to also pass a flag to show them in stack traces?

Uwe

-
Uwe Schindler
uschind...@apache.org  
ASF Member, Apache Lucene PMC / Committer

Bremen, Germany
https://lucene.apache.org/

Re: JDK 15 hidden classes & stack trace visibility

[Uwe Schindler]

Thanks Mandy.

In my opinion for the hidden classes an additional enum constant in ClassOption 
would be fine: ClassOption.SHOW_IN_STACKTRACE (or similar).

That would be simplest to implement and allows also for the caller to enable it 
like strong references or nestmate features. So it's consistent with the other 
API parts.

The annotations are good for fine granular   hiding (like complex classes 
recursively calling methods). Or for frameworks like Spring.

Should I add a comment to this issue?

We are now doing some testing in Lucene's expressions module (it's about 
scoring formulas in query execution). Here's the draft PR: 
https://github.com/apache/lucene-solr/pull/1837/files

We use the new features in Lucene when the static initializer detects 
defineHiddenClass using a publicLookup.

Currently a test fails, because of missing debug information in stack traces. 
The executed scoring formula is part of the filename attribute in class file. 
As this is missing, test fails. 

Uwe

Am September 7, 2020 4:44:18 PM UTC schrieb Mandy Chung 
:
>Hi Uwe,
>
>It's a future enhancement to allow a class/method to request filtering 
>from stack trace:
>     https://bugs.openjdk.java.net/browse/JDK-8212620
>
>This RFE should also take hidden classes into account.
>
>Mandy
>
>On 9/5/20 4:55 AM, Uwe Schindler wrote:
>> Hi,
>>
>> I am doing a mockup for dynamically compiled scripts in Apache Lucene
>(later
>> also painless scripting in elastcsearch), where I tried to use
>> Lookup#defineHiddenClass (with default parameters, so weak and no
>nestmates
>> - but: for painless scripts of Elasticsearch, nestmates will be
>great). It
>> all looks great, as sometimes for every query a new class is
>generated.
>>
>> The current approach (Java 8, Java 11) uses a Classloader per script,
>which
>> is mostly OK, but hidden classes would be better, especially under
>high load
>> with many new classes. The problem I see is: The hidden class and
>their
>> methods are also hidden from stack traces, so people neither get the
>script
>> source code reference, nor they get the method, which was called at
>all.
>>
>> My question is: is it possible to mark such a hidden class in byte
>code or
>> by a flag to defineHiddenClass() so it is "unhidden"? - I know you
>can do
>> this by a command line option, but that's for developers and
>debugging only.
>> In some cases, this is really wanted (like for scripting languages),
>but
>> code still wants to use the better class unloading. If this is not
>possible,
>> how about extending the Lookup.ClassOption enum (last parameter of
>> defineHiddenClass), to also pass a flag to show them in stack traces?
>>
>> Uwe
>>
>> -
>> Uwe Schindler
>> uschind...@apache.org
>> ASF Member, Apache Lucene PMC / Committer
>> Bremen, Germany
>> https://lucene.apache.org/
>>
>>

Re: JDK 15 hidden classes & stack trace visibility

[Mandy Chung]

Hi Uwe,

It's a future enhancement to allow a class/method to request filtering 
from stack trace:

    https://bugs.openjdk.java.net/browse/JDK-8212620

This RFE should also take hidden classes into account.

Mandy

On 9/5/20 4:55 AM, Uwe Schindler wrote:

Hi,

I am doing a mockup for dynamically compiled scripts in Apache Lucene (later
also painless scripting in elastcsearch), where I tried to use
Lookup#defineHiddenClass (with default parameters, so weak and no nestmates
- but: for painless scripts of Elasticsearch, nestmates will be great). It
all looks great, as sometimes for every query a new class is generated.

The current approach (Java 8, Java 11) uses a Classloader per script, which
is mostly OK, but hidden classes would be better, especially under high load
with many new classes. The problem I see is: The hidden class and their
methods are also hidden from stack traces, so people neither get the script
source code reference, nor they get the method, which was called at all.

My question is: is it possible to mark such a hidden class in byte code or
by a flag to defineHiddenClass() so it is "unhidden"? - I know you can do
this by a command line option, but that's for developers and debugging only.
In some cases, this is really wanted (like for scripting languages), but
code still wants to use the better class unloading. If this is not possible,
how about extending the Lookup.ClassOption enum (last parameter of
defineHiddenClass), to also pass a flag to show them in stack traces?

Uwe

-
Uwe Schindler
uschind...@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
https://lucene.apache.org/

Re: JDK 15 hidden classes & stack trace visibility

[Uwe Schindler]

Hi Remy,

Those classes would get a strong relationship to class loader. So I would like 
to have the anonymous class features in combination with make them visible in 
stack traces.

The current approach using separate classloader works fine for that as it 
unloads perfectly (exactly what's discussed in the JEP), so I would like to 
only reduce overhead of the added classloader (which is used only to allow 
unloading of those short living objects).

The hidden classes work exactly like expected for this dynamic situation of 
short living classes, but the backside is the "hidden" feature. I'd like to 
make the hidden-ness a feature. My question is if this can be decoupled. The 
hidden feature is wanted for lambdas and such stuff, but in some situations 
stack traces are essential.

Uwe

Am September 5, 2020 11:18:35 PM UTC schrieb Remi Forax :
>Hi Uwe,
>
>- Mail original -
>> De: "Uwe Schindler" 
>> À: "core-libs-dev" 
>> Envoyé: Samedi 5 Septembre 2020 13:55:03
>> Objet: JDK 15 hidden classes & stack trace visibility
>
>> Hi,
>> 
>> I am doing a mockup for dynamically compiled scripts in Apache Lucene
>(later
>> also painless scripting in elastcsearch), where I tried to use
>> Lookup#defineHiddenClass (with default parameters, so weak and no
>nestmates
>> - but: for painless scripts of Elasticsearch, nestmates will be
>great). It
>> all looks great, as sometimes for every query a new class is
>generated.
>> 
>> The current approach (Java 8, Java 11) uses a Classloader per script,
>which
>> is mostly OK, but hidden classes would be better, especially under
>high load
>> with many new classes. The problem I see is: The hidden class and
>their
>> methods are also hidden from stack traces, so people neither get the
>script
>> source code reference, nor they get the method, which was called at
>all.
>> 
>> My question is: is it possible to mark such a hidden class in byte
>code or
>> by a flag to defineHiddenClass() so it is "unhidden"? - I know you
>can do
>> this by a command line option, but that's for developers and
>debugging only.
>> In some cases, this is really wanted (like for scripting languages),
>but
>> code still wants to use the better class unloading. If this is not
>possible,
>> how about extending the Lookup.ClassOption enum (last parameter of
>> defineHiddenClass), to also pass a flag to show them in stack traces?
>
>If you don't want a hidden class, why not using Lookup.defineClass()
>for 11 and calling ClassLoader.defineClass() by reflection in 8 ?
>With defineClass you get a resolvable real name and methods are visible
>in the stacktrace.
>
>Obviously, it means that you are grouping several classes in the same
>classloader which may be not what you want.
>
>> 
>> Uwe
>
>Rémi
>
>> 
>> -
>> Uwe Schindler
>> uschind...@apache.org
>> ASF Member, Apache Lucene PMC / Committer
>> Bremen, Germany
>> https://lucene.apache.org/

Re: JDK 15 hidden classes & stack trace visibility

[Remi Forax]

Hi Uwe,

- Mail original -
> De: "Uwe Schindler" 
> À: "core-libs-dev" 
> Envoyé: Samedi 5 Septembre 2020 13:55:03
> Objet: JDK 15 hidden classes & stack trace visibility

> Hi,
> 
> I am doing a mockup for dynamically compiled scripts in Apache Lucene (later
> also painless scripting in elastcsearch), where I tried to use
> Lookup#defineHiddenClass (with default parameters, so weak and no nestmates
> - but: for painless scripts of Elasticsearch, nestmates will be great). It
> all looks great, as sometimes for every query a new class is generated.
> 
> The current approach (Java 8, Java 11) uses a Classloader per script, which
> is mostly OK, but hidden classes would be better, especially under high load
> with many new classes. The problem I see is: The hidden class and their
> methods are also hidden from stack traces, so people neither get the script
> source code reference, nor they get the method, which was called at all.
> 
> My question is: is it possible to mark such a hidden class in byte code or
> by a flag to defineHiddenClass() so it is "unhidden"? - I know you can do
> this by a command line option, but that's for developers and debugging only.
> In some cases, this is really wanted (like for scripting languages), but
> code still wants to use the better class unloading. If this is not possible,
> how about extending the Lookup.ClassOption enum (last parameter of
> defineHiddenClass), to also pass a flag to show them in stack traces?

If you don't want a hidden class, why not using Lookup.defineClass() for 11 and 
calling ClassLoader.defineClass() by reflection in 8 ?
With defineClass you get a resolvable real name and methods are visible in the 
stacktrace.

Obviously, it means that you are grouping several classes in the same 
classloader which may be not what you want.

> 
> Uwe

Rémi

> 
> -
> Uwe Schindler
> uschind...@apache.org
> ASF Member, Apache Lucene PMC / Committer
> Bremen, Germany
> https://lucene.apache.org/

JDK 15 hidden classes & stack trace visibility

[Uwe Schindler]

Hi,

I am doing a mockup for dynamically compiled scripts in Apache Lucene (later
also painless scripting in elastcsearch), where I tried to use
Lookup#defineHiddenClass (with default parameters, so weak and no nestmates
- but: for painless scripts of Elasticsearch, nestmates will be great). It
all looks great, as sometimes for every query a new class is generated. 

The current approach (Java 8, Java 11) uses a Classloader per script, which
is mostly OK, but hidden classes would be better, especially under high load
with many new classes. The problem I see is: The hidden class and their
methods are also hidden from stack traces, so people neither get the script
source code reference, nor they get the method, which was called at all. 

My question is: is it possible to mark such a hidden class in byte code or
by a flag to defineHiddenClass() so it is "unhidden"? - I know you can do
this by a command line option, but that's for developers and debugging only.
In some cases, this is really wanted (like for scripting languages), but
code still wants to use the better class unloading. If this is not possible,
how about extending the Lookup.ClassOption enum (last parameter of
defineHiddenClass), to also pass a flag to show them in stack traces?

Uwe

-
Uwe Schindler
uschind...@apache.org 
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
https://lucene.apache.org/