@Francesco: can you *javap -v -cp $yourclasspath the.class.which.Fail* when it works vs when it fails please?
Romain Manni-Bucau @rmannibucau <https://twitter.com/rmannibucau> | Blog <https://rmannibucau.metawerx.net/> | Old Blog <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> | LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book <https://www.packtpub.com/application-development/java-ee-8-high-performance> Le lun. 3 juil. 2023 à 17:23, Mark Struberg <strub...@yahoo.de.invalid> a écrit : > Does not yet ring any bells, sorry :( > > Right now trying to fire up the enhancer manually in the debugger. > Difference really is that we now do the ASM part way earlier, but still > class per class. > Before that it was done as part of the ASMAdapter. > > Still digging. > > LieGrue, > strub > > > > Am 03.07.2023 um 17:00 schrieb Francesco Chicchiriccò < > ilgro...@apache.org>: > > > > Adding some info. > > > > First of all, the class to which the error message is referring is not > always the same. > > > > Secondly, at least about AccountPolicy as reported below, the original > error message is stating > > > > Execution enhancer of goal > org.apache.openjpa:openjpa-maven-plugin:4.0.0-SNAPSHOT:enhance failed: An > error occurred while enhancing > org.apache.syncope.core.persistence.jpa.entity.JPARealm. Exception message: > Type org/apache/syncope/core/persistence/api/entity/policy/AccountPolicy > not present: > org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy > > > > If you look at JPARealm: > > > > > https://github.com/apache/syncope/blob/master/core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/entity/JPARealm.java#L82 > > > > it declares > > > > @ManyToOne(fetch = FetchType.EAGER) > > private JPAAccountPolicy accountPolicy; > > > > e.g. a reference to the JPA entity JPAAccountPolicy, not to the > interface that this class implements, e.g. AccountPolicy > > > > It looks to me that somewhere the OpenJPA enhancement process is trying > to resolve more than it should do. > > > > Does this make ringing any bell? > > > > Regards. > > > > On 03/07/23 16:32, Francesco Chicchiriccò wrote: > >> On 03/07/23 16:27, Mark Struberg wrote: > >>> Went forward and compiled the Syncope main branch. > >>> > >>> Getting this error when running with mvn -X > >>> > >>> Caused by: java.lang.ClassNotFoundException: > org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy > >>> at > org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass > (SelfFirstStrategy.java:50) > >>> at > org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass > (ClassRealm.java:271) > >>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass > (ClassRealm.java:247) > >>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass > (ClassRealm.java:239) > >>> at java.lang.Class.forName0 (Native Method) > >>> at java.lang.Class.forName (Class.java:467) > >>> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass > (ClassWriter.java:1043) > >>> at org.apache.xbean.asm9.SymbolTable.addMergedType > (SymbolTable.java:1202) > >>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299) > >>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244) > >>> at org.apache.xbean.asm9.MethodWriter.computeAllFrames > (MethodWriter.java:1611) > >>> at org.apache.xbean.asm9.MethodWriter.visitMaxs > (MethodWriter.java:1547) > >>> at org.apache.xbean.asm9.tree.MethodNode.accept > (MethodNode.java:767) > >>> at org.apache.xbean.asm9.tree.MethodNode.accept > (MethodNode.java:647) > >>> at org.apache.xbean.asm9.tree.ClassNode.accept > (ClassNode.java:468) > >>> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass > (AsmHelper.java:146) > >>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:606) > >>> at org.apache.openjpa.enhance.PCEnhancer.run > (PCEnhancer.java:5889) > >>> at org.apache.openjpa.enhance.PCEnhancer.run > (PCEnhancer.java:5829) > >>> at org.apache.openjpa.enhance.PCEnhancer$1.run > (PCEnhancer.java:5799) > >>> at org.apache.openjpa.lib.conf.Configurations.launchRunnable > (Configurations.java:760) > >>> at > org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors > (Configurations.java:745) > >>> > >>> > >>> Is this the one you experience too? > >> > >> Yes, exactly. > >> > >> As you might have found out, AccountPolicy is an interface implemented > by JPAAccountPolicy, which is the corresponding JPA entity. > >> We take this approach for all JPA entities, actually. > >> > >> I thought the reason why AccountPolicy or JPAAccountPolicy are not > found is that enhancement does not actually finds them. > >> > >>> Will take a look at it now. > >> > >> Thank you very much! > >> > >> Regards. > >>>> Am 03.07.2023 um 15:49 schrieb Mark Struberg > <strub...@yahoo.de.INVALID>: > >>>> > >>>> Hi Francesco! > >>>> > >>>> Can you see which one of the classes fail to enhance? > >>>> > >>>> In which case you might try to run the enhancement of that very class > manually in your IDE. > >>>> > >>>> Main class: org.apache.openjpa.enhance.PCEnhancer > >>>> VMOptions:: -ea > -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true > -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver > >>>> or something similar for your connection > >>>> > >>>> Then you also need the parameter pointing to the class file which you > want to enhance, e.g. > >>>> > target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class > >>>> > >>>> Please keep me updated! > >>>> > >>>> > >>>> LieGrue, > >>>> strub > >>>> > >>>> PS: does it add value if we'd write all this up? You will likely know > most of that stuff, but others who dig into OpenJPA do not. > >>>> > >>>> > >>>> > >>>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò < > ilgro...@apache.org>: > >>>>> > >>>>> Hi Mark, > >>>>> thanks for your reply. > >>>>> > >>>>> So I am proceeding as follows. > >>>>> > >>>>> Took the executions at > >>>>> > >>>>> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/ > >>>>> > >>>>> (the Jenkins job which is publishing SNAPSHOT artifacts from master > to ASF Maven repo) and saw that the artifacts with which the Syncope build > succeeded last Friday are the one from May 16th, so the corresponding > commit id is > >>>>> > >>>>> b238417dabeac935846fe1ce2fef28aafaeef205 > >>>>> > >>>>> I confirm that building locally from that commit id everything is > working. > >>>>> > >>>>> As soon as I move to next commit (dated May 22nd, pushed and > deployed today) with id > >>>>> > >>>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d > >>>>> > >>>>> I see the same behavior as with HEAD, e.g. > >>>>> > >>>>> 2b9b024f273d63e479a02cad751c28b8ef974ace > >>>>> > >>>>> I will try to take a closer look at the code. > >>>>> Regards. > >>>>> > >>>>> On 03/07/23 14:44, Mark Struberg wrote: > >>>>>> Hi Francesco! > >>>>>> > >>>>>> That's very valuable information! > >>>>>> Could you please do a git-bisect to find out which change did break > it? > >>>>>> > >>>>>> Actually the algorithm to select which classes are to be enhanced > should not have changed. Thus I'm really curious what I did break. > >>>>>> > >>>>>> Most times the amount of enhanced classes are the same but slightly > differ in the generated code. That might be due to now we really generate > Java11++ code whereas before we only generated Java 1.1 code (yes, not even > 1.5!) > >>>>>> > >>>>>> What I usually do is to have 2 checkouts. You might want to change > the pom GAV version of the working checkout to something something > different so you might have both in your ~/.m2/repository in parallel. > >>>>>> > >>>>>> The first thing I do is to compare the class files in > target/classes of both versions with 'compare with clipboard'. > >>>>>> I just open the class file in target/classes and let > Intellij/Netbeans do the decompilation for me. > >>>>>> > >>>>>> You might find some differences here in which case this needs > further investigation. Note that the order of the methods might have > changed. > >>>>>> Sometimes you will also see some 'cannot decompile' in the code > block. This happens when something in the bytecode is messed up. You can > also watch out for ClassNotFound or VerifyError, etc. > >>>>>> > >>>>>> If this is the case I then use > >>>>>> $> javap -c -p -constants > target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile > >>>>>> on both versions and check whether something is different. > >>>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs > ALOAD 2) > >>>>>> > >>>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of > Class constants (LoaD Constant). This was only introduced in Java 1.4 or > 1.5 bytecode afair. Before that a dynamic Class.forName was called and > stored in a static variable. Ugly stuff, and happy to finally get rid of > all those hacks... > >>>>>> > >>>>>> Hope that helps to get you started. Feel free to ping me again over > here or at the #apachee ASF slack channel. > >>>>>> > >>>>>> txs and LieGrue, > >>>>>> strub > >>>>>> > >>>>>> > >>>>>> > >>>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò < > ilgro...@apache.org>: > >>>>>>> > >>>>>>> From a very preliminary analysis it seems that just a subset of > the persistence-capable classes are effectively found and enhanced: now > [2], it used to be [3]. > >>>>>>> > >>>>>>> I'll continue investigating. > >>>>>>> > >>>>>>> Regards. > >>>>>>> > >>>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote: > >>>>>>>> Hi Mark, > >>>>>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT > is now failing - it used to work at least until last June 30th. > >>>>>>>> > >>>>>>>> The issues seem to be related to entity enhancement, which we > perform via openjpa-maven-plugin - see [1]. > >>>>>>>> > >>>>>>>> Is there anything obvious we should be changing? > >>>>>>>> Meanwhile I'll try to dig to see if I am able to understand > something more. > >>>>>>>> > >>>>>>>> Regards. > >>>>>>>> > >>>>>>>> [1] > https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174 > >>>>>>> [2] https://paste.apache.org/gxcm6 > >>>>>>> [3] https://paste.apache.org/1g9k5 > >>>>>>> > >>>>>>>> On 03/07/23 12:15, Mark Struberg wrote: > >>>>>>>>> Update: I've now merged the wip to our master and pushed it to > the ASF gitox repo. > >>>>>>>>> > >>>>>>>>> I think we already came pretty far and it looks really > promising. So there is a high chance that we succeed, although there is > still plenty of work in front of us. > >>>>>>>>> > >>>>>>>>> How it works: > >>>>>>>>> > >>>>>>>>> Basically there are 2 sets of information in the PCEnhancer > right now: > >>>>>>>>> > >>>>>>>>> _managed: the BCClass version of the original class > >>>>>>>>> _pc: the BCClass of the generated/modified class. Might be the > same as _managed but in case of subclassing or interface case it is > different. In those cases _managed represents the original class and _pc > the subclass/concrete implementation. > >>>>>>>>> > >>>>>>>>> managed and pc are the ASM variants of _managed and _pc. > >>>>>>>>> > >>>>>>>>> The main entry point for debugging into it is PCEnhancer#run() > >>>>>>>>> > >>>>>>>>> There are a few methods in ASMHelper to update from BCClass -> > ASM and vice versa: > >>>>>>>>> > >>>>>>>>> AsmHelper.readIntoBCClass(pc, _pc); > >>>>>>>>> takes all the information from the ASM pc, creates a class > byte[] and reads that back into the existing _pc instance. > >>>>>>>>> > >>>>>>>>> pc = AsmHelper.toClassNode(_pc); > >>>>>>>>> is the corresponding method in the other direction. from BCClass > to ASM. > >>>>>>>>> > >>>>>>>>> Right now we have the following methods left to migrate in > PCEnhancer > >>>>>>>>> addAttachDetachCode(); > >>>>>>>>> addSerializationCode(); > >>>>>>>>> addCloningCode(); > >>>>>>>>> runAuxiliaryEnhancers(); > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> I'll be off for a week on holidays in Venice. > >>>>>>>>> If someone want's to join in with the effort then please go on! > You might want to compare the current PCEnhancer code with an older version > to get an idea how it can be done and to see some patterns I've used. > >>>>>>>>> > >>>>>>>>> As a quick hint: Java is basically a stack based virtual CPU. > The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that > comes the parameters. The size of the param on the stack depends on the > type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method > parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type). > >>>>>>>>> In case of a static method there is ofc no 'this' on the stack, > so the parameters start with zero offset. > >>>>>>>>> After all the parameters there are the unnamed 'local' > variables. I tried to consistently name them xxxVarPos. > >>>>>>>>> See also the various helper methods in ASMHelper. For example > for the various LOAD, RETURN, STORE operations which depends based on the > type (ALOAD vs ILOAD vs IALOAD etc) > >>>>>>>>> Also read the guide to ASM PDF from objectweb. > >>>>>>>>> > >>>>>>>>> I'll be mostly offline till Sunday, but will try to answer mails > if there are some questions. > >>>>>>>>> > >>>>>>>>> txs and LieGrue, > >>>>>>>>> strub > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich < > mattr...@gmail.com>: > >>>>>>>>>> > >>>>>>>>>> +1 thanks for tackling this, Mark! > >>>>>>>>>> > >>>>>>>>>> ASM definitely more widely used going forward. > >>>>>>>>>> > >>>>>>>>>> -Matt > >>>>>>>>>> > >>>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg > <strub...@yahoo.de.INVALID> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Small update: > >>>>>>>>>>> > >>>>>>>>>>> I've worked on it over the last few weeks, and I'm getting > closer > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance > >>>>>>>>>>> > >>>>>>>>>>> contains the latest work on the PCEnhancer. Right in the > middle of replacing Serp with native ASM code. > >>>>>>>>>>> > >>>>>>>>>>> Note that I also had to modify a few interfaces with a few > more to follow from BCClass to ASM ClassNode. > >>>>>>>>>>> > >>>>>>>>>>> Happy to get some feedback! > >>>>>>>>>>> > >>>>>>>>>>> If anybody wants to help with this effort I'm happy to also > push this feature branch to our ASF repo. It looks reasonably promising > already. > >>>>>>>>>>> > >>>>>>>>>>> LieGrue, > >>>>>>>>>>> strub > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg > <strub...@yahoo.de.INVALID>: > >>>>>>>>>>>> > >>>>>>>>>>>> Hi! > >>>>>>>>>>>> > >>>>>>>>>>>> Right now I'm trying to get rid of serp step by step. > >>>>>>>>>>>> The code is right now at my own github repo in the > fb_asmEnhance branch: > >>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> The approach I took for now is to create a few methods in > AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other > way around. That way we should be able to replace functionality part by > part but still keep all things afloat. > >>>>>>>>>>>> > >>>>>>>>>>>> For now I've started with the PCSubclassValidator. > >>>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM > and then compare the results. > >>>>>>>>>>>> If something is fishy, you'll see the following > >>>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " > " + field); > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> It would be extremely helpful, if you could go through your > projects and let it run and report how it's going. > >>>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed > something up. > >>>>>>>>>>>> > >>>>>>>>>>>> I'll gonna finally remove the BCClass handling from those > parts in a few days. Current commit is > >>>>>>>>>>>> > https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b > >>>>>>>>>>>> > >>>>>>>>>>>> txs and LieGrue, > >>>>>>>>>>>> strub > >> > > > > -- > > Francesco Chicchiriccò > > > > Tirasa - Open Source Excellence > > http://www.tirasa.net/ > > > > Member at The Apache Software Foundation > > Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail > > http://home.apache.org/~ilgrosso/ > > > >
