Francesco, I've now pushed this change. Can you please try it out? Looks good over here now. Now I started the full Syncope build and hope all comes green again.
txs and LieGrue, strub > Am 03.07.2023 um 17:55 schrieb Mark Struberg <strub...@yahoo.de>: > > Think I found the solution, but first want to run all tests in OpenJPA. > > I did use the standard ClassWriter in ASMHelper. This does use Class.forName. > Changed ASMHelper to use the BCClassWriter you wrote some time ago. That > should do the trick I hope. > > Gimme 10 minutes plz. > > LieGrue, > strub > > >> Am 03.07.2023 um 17:34 schrieb Romain Manni-Bucau <rmannibu...@gmail.com>: >> >> @Francesco: can you *javap -v -cp $yourclasspath the.class.which.Fail* when >> it works vs when it fails please? >> >> Romain Manni-Bucau >> @rmannibucau <https://twitter.com/rmannibucau> | Blog >> <https://rmannibucau.metawerx.net/> | Old Blog >> <http://rmannibucau.wordpress.com> | Github <https://github.com/rmannibucau> >> | >> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book >> <https://www.packtpub.com/application-development/java-ee-8-high-performance> >> >> >> Le lun. 3 juil. 2023 à 17:23, Mark Struberg <strub...@yahoo.de.invalid> a >> écrit : >> >>> Does not yet ring any bells, sorry :( >>> >>> Right now trying to fire up the enhancer manually in the debugger. >>> Difference really is that we now do the ASM part way earlier, but still >>> class per class. >>> Before that it was done as part of the ASMAdapter. >>> >>> Still digging. >>> >>> LieGrue, >>> strub >>> >>> >>>> Am 03.07.2023 um 17:00 schrieb Francesco Chicchiriccò < >>> ilgro...@apache.org>: >>>> >>>> Adding some info. >>>> >>>> First of all, the class to which the error message is referring is not >>> always the same. >>>> >>>> Secondly, at least about AccountPolicy as reported below, the original >>> error message is stating >>>> >>>> Execution enhancer of goal >>> org.apache.openjpa:openjpa-maven-plugin:4.0.0-SNAPSHOT:enhance failed: An >>> error occurred while enhancing >>> org.apache.syncope.core.persistence.jpa.entity.JPARealm. Exception message: >>> Type org/apache/syncope/core/persistence/api/entity/policy/AccountPolicy >>> not present: >>> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy >>>> >>>> If you look at JPARealm: >>>> >>>> >>> https://github.com/apache/syncope/blob/master/core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/entity/JPARealm.java#L82 >>>> >>>> it declares >>>> >>>> @ManyToOne(fetch = FetchType.EAGER) >>>> private JPAAccountPolicy accountPolicy; >>>> >>>> e.g. a reference to the JPA entity JPAAccountPolicy, not to the >>> interface that this class implements, e.g. AccountPolicy >>>> >>>> It looks to me that somewhere the OpenJPA enhancement process is trying >>> to resolve more than it should do. >>>> >>>> Does this make ringing any bell? >>>> >>>> Regards. >>>> >>>> On 03/07/23 16:32, Francesco Chicchiriccò wrote: >>>>> On 03/07/23 16:27, Mark Struberg wrote: >>>>>> Went forward and compiled the Syncope main branch. >>>>>> >>>>>> Getting this error when running with mvn -X >>>>>> >>>>>> Caused by: java.lang.ClassNotFoundException: >>> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy >>>>>> at >>> org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass >>> (SelfFirstStrategy.java:50) >>>>>> at >>> org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass >>> (ClassRealm.java:271) >>>>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass >>> (ClassRealm.java:247) >>>>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass >>> (ClassRealm.java:239) >>>>>> at java.lang.Class.forName0 (Native Method) >>>>>> at java.lang.Class.forName (Class.java:467) >>>>>> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass >>> (ClassWriter.java:1043) >>>>>> at org.apache.xbean.asm9.SymbolTable.addMergedType >>> (SymbolTable.java:1202) >>>>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299) >>>>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244) >>>>>> at org.apache.xbean.asm9.MethodWriter.computeAllFrames >>> (MethodWriter.java:1611) >>>>>> at org.apache.xbean.asm9.MethodWriter.visitMaxs >>> (MethodWriter.java:1547) >>>>>> at org.apache.xbean.asm9.tree.MethodNode.accept >>> (MethodNode.java:767) >>>>>> at org.apache.xbean.asm9.tree.MethodNode.accept >>> (MethodNode.java:647) >>>>>> at org.apache.xbean.asm9.tree.ClassNode.accept >>> (ClassNode.java:468) >>>>>> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass >>> (AsmHelper.java:146) >>>>>> at org.apache.openjpa.enhance.PCEnhancer.run (PCEnhancer.java:606) >>>>>> at org.apache.openjpa.enhance.PCEnhancer.run >>> (PCEnhancer.java:5889) >>>>>> at org.apache.openjpa.enhance.PCEnhancer.run >>> (PCEnhancer.java:5829) >>>>>> at org.apache.openjpa.enhance.PCEnhancer$1.run >>> (PCEnhancer.java:5799) >>>>>> at org.apache.openjpa.lib.conf.Configurations.launchRunnable >>> (Configurations.java:760) >>>>>> at >>> org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors >>> (Configurations.java:745) >>>>>> >>>>>> >>>>>> Is this the one you experience too? >>>>> >>>>> Yes, exactly. >>>>> >>>>> As you might have found out, AccountPolicy is an interface implemented >>> by JPAAccountPolicy, which is the corresponding JPA entity. >>>>> We take this approach for all JPA entities, actually. >>>>> >>>>> I thought the reason why AccountPolicy or JPAAccountPolicy are not >>> found is that enhancement does not actually finds them. >>>>> >>>>>> Will take a look at it now. >>>>> >>>>> Thank you very much! >>>>> >>>>> Regards. >>>>>>> Am 03.07.2023 um 15:49 schrieb Mark Struberg >>> <strub...@yahoo.de.INVALID>: >>>>>>> >>>>>>> Hi Francesco! >>>>>>> >>>>>>> Can you see which one of the classes fail to enhance? >>>>>>> >>>>>>> In which case you might try to run the enhancement of that very class >>> manually in your IDE. >>>>>>> >>>>>>> Main class: org.apache.openjpa.enhance.PCEnhancer >>>>>>> VMOptions:: -ea >>> -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true >>> -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver >>>>>>> or something similar for your connection >>>>>>> >>>>>>> Then you also need the parameter pointing to the class file which you >>> want to enhance, e.g. >>>>>>> >>> target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class >>>>>>> >>>>>>> Please keep me updated! >>>>>>> >>>>>>> >>>>>>> LieGrue, >>>>>>> strub >>>>>>> >>>>>>> PS: does it add value if we'd write all this up? You will likely know >>> most of that stuff, but others who dig into OpenJPA do not. >>>>>>> >>>>>>> >>>>>>> >>>>>>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò < >>> ilgro...@apache.org>: >>>>>>>> >>>>>>>> Hi Mark, >>>>>>>> thanks for your reply. >>>>>>>> >>>>>>>> So I am proceeding as follows. >>>>>>>> >>>>>>>> Took the executions at >>>>>>>> >>>>>>>> https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/ >>>>>>>> >>>>>>>> (the Jenkins job which is publishing SNAPSHOT artifacts from master >>> to ASF Maven repo) and saw that the artifacts with which the Syncope build >>> succeeded last Friday are the one from May 16th, so the corresponding >>> commit id is >>>>>>>> >>>>>>>> b238417dabeac935846fe1ce2fef28aafaeef205 >>>>>>>> >>>>>>>> I confirm that building locally from that commit id everything is >>> working. >>>>>>>> >>>>>>>> As soon as I move to next commit (dated May 22nd, pushed and >>> deployed today) with id >>>>>>>> >>>>>>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d >>>>>>>> >>>>>>>> I see the same behavior as with HEAD, e.g. >>>>>>>> >>>>>>>> 2b9b024f273d63e479a02cad751c28b8ef974ace >>>>>>>> >>>>>>>> I will try to take a closer look at the code. >>>>>>>> Regards. >>>>>>>> >>>>>>>> On 03/07/23 14:44, Mark Struberg wrote: >>>>>>>>> Hi Francesco! >>>>>>>>> >>>>>>>>> That's very valuable information! >>>>>>>>> Could you please do a git-bisect to find out which change did break >>> it? >>>>>>>>> >>>>>>>>> Actually the algorithm to select which classes are to be enhanced >>> should not have changed. Thus I'm really curious what I did break. >>>>>>>>> >>>>>>>>> Most times the amount of enhanced classes are the same but slightly >>> differ in the generated code. That might be due to now we really generate >>> Java11++ code whereas before we only generated Java 1.1 code (yes, not even >>> 1.5!) >>>>>>>>> >>>>>>>>> What I usually do is to have 2 checkouts. You might want to change >>> the pom GAV version of the working checkout to something something >>> different so you might have both in your ~/.m2/repository in parallel. >>>>>>>>> >>>>>>>>> The first thing I do is to compare the class files in >>> target/classes of both versions with 'compare with clipboard'. >>>>>>>>> I just open the class file in target/classes and let >>> Intellij/Netbeans do the decompilation for me. >>>>>>>>> >>>>>>>>> You might find some differences here in which case this needs >>> further investigation. Note that the order of the methods might have >>> changed. >>>>>>>>> Sometimes you will also see some 'cannot decompile' in the code >>> block. This happens when something in the bytecode is messed up. You can >>> also watch out for ClassNotFound or VerifyError, etc. >>>>>>>>> >>>>>>>>> If this is the case I then use >>>>>>>>> $> javap -c -p -constants >>> target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile >>>>>>>>> on both versions and check whether something is different. >>>>>>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 vs >>> ALOAD 2) >>>>>>>>> >>>>>>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of >>> Class constants (LoaD Constant). This was only introduced in Java 1.4 or >>> 1.5 bytecode afair. Before that a dynamic Class.forName was called and >>> stored in a static variable. Ugly stuff, and happy to finally get rid of >>> all those hacks... >>>>>>>>> >>>>>>>>> Hope that helps to get you started. Feel free to ping me again over >>> here or at the #apachee ASF slack channel. >>>>>>>>> >>>>>>>>> txs and LieGrue, >>>>>>>>> strub >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò < >>> ilgro...@apache.org>: >>>>>>>>>> >>>>>>>>>> From a very preliminary analysis it seems that just a subset of >>> the persistence-capable classes are effectively found and enhanced: now >>> [2], it used to be [3]. >>>>>>>>>> >>>>>>>>>> I'll continue investigating. >>>>>>>>>> >>>>>>>>>> Regards. >>>>>>>>>> >>>>>>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote: >>>>>>>>>>> Hi Mark, >>>>>>>>>>> the build on Syncope master branch, using OpenJPA 4.0.0-SNAPSHOT >>> is now failing - it used to work at least until last June 30th. >>>>>>>>>>> >>>>>>>>>>> The issues seem to be related to entity enhancement, which we >>> perform via openjpa-maven-plugin - see [1]. >>>>>>>>>>> >>>>>>>>>>> Is there anything obvious we should be changing? >>>>>>>>>>> Meanwhile I'll try to dig to see if I am able to understand >>> something more. >>>>>>>>>>> >>>>>>>>>>> Regards. >>>>>>>>>>> >>>>>>>>>>> [1] >>> https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174 >>>>>>>>>> [2] https://paste.apache.org/gxcm6 >>>>>>>>>> [3] https://paste.apache.org/1g9k5 >>>>>>>>>> >>>>>>>>>>> On 03/07/23 12:15, Mark Struberg wrote: >>>>>>>>>>>> Update: I've now merged the wip to our master and pushed it to >>> the ASF gitox repo. >>>>>>>>>>>> >>>>>>>>>>>> I think we already came pretty far and it looks really >>> promising. So there is a high chance that we succeed, although there is >>> still plenty of work in front of us. >>>>>>>>>>>> >>>>>>>>>>>> How it works: >>>>>>>>>>>> >>>>>>>>>>>> Basically there are 2 sets of information in the PCEnhancer >>> right now: >>>>>>>>>>>> >>>>>>>>>>>> _managed: the BCClass version of the original class >>>>>>>>>>>> _pc: the BCClass of the generated/modified class. Might be the >>> same as _managed but in case of subclassing or interface case it is >>> different. In those cases _managed represents the original class and _pc >>> the subclass/concrete implementation. >>>>>>>>>>>> >>>>>>>>>>>> managed and pc are the ASM variants of _managed and _pc. >>>>>>>>>>>> >>>>>>>>>>>> The main entry point for debugging into it is PCEnhancer#run() >>>>>>>>>>>> >>>>>>>>>>>> There are a few methods in ASMHelper to update from BCClass -> >>> ASM and vice versa: >>>>>>>>>>>> >>>>>>>>>>>> AsmHelper.readIntoBCClass(pc, _pc); >>>>>>>>>>>> takes all the information from the ASM pc, creates a class >>> byte[] and reads that back into the existing _pc instance. >>>>>>>>>>>> >>>>>>>>>>>> pc = AsmHelper.toClassNode(_pc); >>>>>>>>>>>> is the corresponding method in the other direction. from BCClass >>> to ASM. >>>>>>>>>>>> >>>>>>>>>>>> Right now we have the following methods left to migrate in >>> PCEnhancer >>>>>>>>>>>> addAttachDetachCode(); >>>>>>>>>>>> addSerializationCode(); >>>>>>>>>>>> addCloningCode(); >>>>>>>>>>>> runAuxiliaryEnhancers(); >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I'll be off for a week on holidays in Venice. >>>>>>>>>>>> If someone want's to join in with the effort then please go on! >>> You might want to compare the current PCEnhancer code with an older version >>> to get an idea how it can be done and to see some patterns I've used. >>>>>>>>>>>> >>>>>>>>>>>> As a quick hint: Java is basically a stack based virtual CPU. >>> The 'this' pointer is on position 0 of the stack, so ALOAD 0. After that >>> comes the parameters. The size of the param on the stack depends on the >>> type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st method >>> parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type). >>>>>>>>>>>> In case of a static method there is ofc no 'this' on the stack, >>> so the parameters start with zero offset. >>>>>>>>>>>> After all the parameters there are the unnamed 'local' >>> variables. I tried to consistently name them xxxVarPos. >>>>>>>>>>>> See also the various helper methods in ASMHelper. For example >>> for the various LOAD, RETURN, STORE operations which depends based on the >>> type (ALOAD vs ILOAD vs IALOAD etc) >>>>>>>>>>>> Also read the guide to ASM PDF from objectweb. >>>>>>>>>>>> >>>>>>>>>>>> I'll be mostly offline till Sunday, but will try to answer mails >>> if there are some questions. >>>>>>>>>>>> >>>>>>>>>>>> txs and LieGrue, >>>>>>>>>>>> strub >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich < >>> mattr...@gmail.com>: >>>>>>>>>>>>> >>>>>>>>>>>>> +1 thanks for tackling this, Mark! >>>>>>>>>>>>> >>>>>>>>>>>>> ASM definitely more widely used going forward. >>>>>>>>>>>>> >>>>>>>>>>>>> -Matt >>>>>>>>>>>>> >>>>>>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg >>> <strub...@yahoo.de.INVALID> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> Small update: >>>>>>>>>>>>>> >>>>>>>>>>>>>> I've worked on it over the last few weeks, and I'm getting >>> closer >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance >>>>>>>>>>>>>> >>>>>>>>>>>>>> contains the latest work on the PCEnhancer. Right in the >>> middle of replacing Serp with native ASM code. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Note that I also had to modify a few interfaces with a few >>> more to follow from BCClass to ASM ClassNode. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Happy to get some feedback! >>>>>>>>>>>>>> >>>>>>>>>>>>>> If anybody wants to help with this effort I'm happy to also >>> push this feature branch to our ASF repo. It looks reasonably promising >>> already. >>>>>>>>>>>>>> >>>>>>>>>>>>>> LieGrue, >>>>>>>>>>>>>> strub >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg >>> <strub...@yahoo.de.INVALID>: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi! >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Right now I'm trying to get rid of serp step by step. >>>>>>>>>>>>>>> The code is right now at my own github repo in the >>> fb_asmEnhance branch: >>>>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> The approach I took for now is to create a few methods in >>> AsmHelper to be able to move from BCClass -> ASM ClassWriter and the other >>> way around. That way we should be able to replace functionality part by >>> part but still keep all things afloat. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> For now I've started with the PCSubclassValidator. >>>>>>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM >>> and then compare the results. >>>>>>>>>>>>>>> If something is fishy, you'll see the following >>>>>>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + " >>> " + field); >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> It would be extremely helpful, if you could go through your >>> projects and let it run and report how it's going. >>>>>>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I messed >>> something up. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I'll gonna finally remove the BCClass handling from those >>> parts in a few days. Current commit is >>>>>>>>>>>>>>> >>> https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> txs and LieGrue, >>>>>>>>>>>>>>> strub >>>>> >>>> >>>> -- >>>> Francesco Chicchiriccò >>>> >>>> Tirasa - Open Source Excellence >>>> http://www.tirasa.net/ >>>> >>>> Member at The Apache Software Foundation >>>> Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail >>>> http://home.apache.org/~ilgrosso/ >>>> >>> >>> >
