Thanks Mark, your last commit seems to have fixed, our build is green again.
Nice job! Il lun 3 lug 2023, 18:10 Mark Struberg <strub...@yahoo.de.invalid> ha scritto: > Francesco, I've now pushed this change. Can you please try it out? Looks > good over here now. > Now I started the full Syncope build and hope all comes green again. > > txs and LieGrue, > strub > > > Am 03.07.2023 um 17:55 schrieb Mark Struberg <strub...@yahoo.de>: > > > > Think I found the solution, but first want to run all tests in OpenJPA. > > > > I did use the standard ClassWriter in ASMHelper. This does use > Class.forName. > > Changed ASMHelper to use the BCClassWriter you wrote some time ago. That > should do the trick I hope. > > > > Gimme 10 minutes plz. > > > > LieGrue, > > strub > > > > > >> Am 03.07.2023 um 17:34 schrieb Romain Manni-Bucau < > rmannibu...@gmail.com>: > >> > >> @Francesco: can you *javap -v -cp $yourclasspath the.class.which.Fail* > when > >> it works vs when it fails please? > >> > >> Romain Manni-Bucau > >> @rmannibucau <https://twitter.com/rmannibucau> | Blog > >> <https://rmannibucau.metawerx.net/> | Old Blog > >> <http://rmannibucau.wordpress.com> | Github < > https://github.com/rmannibucau> | > >> LinkedIn <https://www.linkedin.com/in/rmannibucau> | Book > >> < > https://www.packtpub.com/application-development/java-ee-8-high-performance > > > >> > >> > >> Le lun. 3 juil. 2023 à 17:23, Mark Struberg <strub...@yahoo.de.invalid> > a > >> écrit : > >> > >>> Does not yet ring any bells, sorry :( > >>> > >>> Right now trying to fire up the enhancer manually in the debugger. > >>> Difference really is that we now do the ASM part way earlier, but still > >>> class per class. > >>> Before that it was done as part of the ASMAdapter. > >>> > >>> Still digging. > >>> > >>> LieGrue, > >>> strub > >>> > >>> > >>>> Am 03.07.2023 um 17:00 schrieb Francesco Chicchiriccò < > >>> ilgro...@apache.org>: > >>>> > >>>> Adding some info. > >>>> > >>>> First of all, the class to which the error message is referring is not > >>> always the same. > >>>> > >>>> Secondly, at least about AccountPolicy as reported below, the original > >>> error message is stating > >>>> > >>>> Execution enhancer of goal > >>> org.apache.openjpa:openjpa-maven-plugin:4.0.0-SNAPSHOT:enhance failed: > An > >>> error occurred while enhancing > >>> org.apache.syncope.core.persistence.jpa.entity.JPARealm. Exception > message: > >>> Type > org/apache/syncope/core/persistence/api/entity/policy/AccountPolicy > >>> not present: > >>> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy > >>>> > >>>> If you look at JPARealm: > >>>> > >>>> > >>> > https://github.com/apache/syncope/blob/master/core/persistence-jpa/src/main/java/org/apache/syncope/core/persistence/jpa/entity/JPARealm.java#L82 > >>>> > >>>> it declares > >>>> > >>>> @ManyToOne(fetch = FetchType.EAGER) > >>>> private JPAAccountPolicy accountPolicy; > >>>> > >>>> e.g. a reference to the JPA entity JPAAccountPolicy, not to the > >>> interface that this class implements, e.g. AccountPolicy > >>>> > >>>> It looks to me that somewhere the OpenJPA enhancement process is > trying > >>> to resolve more than it should do. > >>>> > >>>> Does this make ringing any bell? > >>>> > >>>> Regards. > >>>> > >>>> On 03/07/23 16:32, Francesco Chicchiriccò wrote: > >>>>> On 03/07/23 16:27, Mark Struberg wrote: > >>>>>> Went forward and compiled the Syncope main branch. > >>>>>> > >>>>>> Getting this error when running with mvn -X > >>>>>> > >>>>>> Caused by: java.lang.ClassNotFoundException: > >>> org.apache.syncope.core.persistence.api.entity.policy.AccountPolicy > >>>>>> at > >>> org.codehaus.plexus.classworlds.strategy.SelfFirstStrategy.loadClass > >>> (SelfFirstStrategy.java:50) > >>>>>> at > >>> > org.codehaus.plexus.classworlds.realm.ClassRealm.unsynchronizedLoadClass > >>> (ClassRealm.java:271) > >>>>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass > >>> (ClassRealm.java:247) > >>>>>> at org.codehaus.plexus.classworlds.realm.ClassRealm.loadClass > >>> (ClassRealm.java:239) > >>>>>> at java.lang.Class.forName0 (Native Method) > >>>>>> at java.lang.Class.forName (Class.java:467) > >>>>>> at org.apache.xbean.asm9.ClassWriter.getCommonSuperClass > >>> (ClassWriter.java:1043) > >>>>>> at org.apache.xbean.asm9.SymbolTable.addMergedType > >>> (SymbolTable.java:1202) > >>>>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1299) > >>>>>> at org.apache.xbean.asm9.Frame.merge (Frame.java:1244) > >>>>>> at org.apache.xbean.asm9.MethodWriter.computeAllFrames > >>> (MethodWriter.java:1611) > >>>>>> at org.apache.xbean.asm9.MethodWriter.visitMaxs > >>> (MethodWriter.java:1547) > >>>>>> at org.apache.xbean.asm9.tree.MethodNode.accept > >>> (MethodNode.java:767) > >>>>>> at org.apache.xbean.asm9.tree.MethodNode.accept > >>> (MethodNode.java:647) > >>>>>> at org.apache.xbean.asm9.tree.ClassNode.accept > >>> (ClassNode.java:468) > >>>>>> at org.apache.openjpa.util.asm.AsmHelper.readIntoBCClass > >>> (AsmHelper.java:146) > >>>>>> at org.apache.openjpa.enhance.PCEnhancer.run > (PCEnhancer.java:606) > >>>>>> at org.apache.openjpa.enhance.PCEnhancer.run > >>> (PCEnhancer.java:5889) > >>>>>> at org.apache.openjpa.enhance.PCEnhancer.run > >>> (PCEnhancer.java:5829) > >>>>>> at org.apache.openjpa.enhance.PCEnhancer$1.run > >>> (PCEnhancer.java:5799) > >>>>>> at org.apache.openjpa.lib.conf.Configurations.launchRunnable > >>> (Configurations.java:760) > >>>>>> at > >>> org.apache.openjpa.lib.conf.Configurations.runAgainstAllAnchors > >>> (Configurations.java:745) > >>>>>> > >>>>>> > >>>>>> Is this the one you experience too? > >>>>> > >>>>> Yes, exactly. > >>>>> > >>>>> As you might have found out, AccountPolicy is an interface > implemented > >>> by JPAAccountPolicy, which is the corresponding JPA entity. > >>>>> We take this approach for all JPA entities, actually. > >>>>> > >>>>> I thought the reason why AccountPolicy or JPAAccountPolicy are not > >>> found is that enhancement does not actually finds them. > >>>>> > >>>>>> Will take a look at it now. > >>>>> > >>>>> Thank you very much! > >>>>> > >>>>> Regards. > >>>>>>> Am 03.07.2023 um 15:49 schrieb Mark Struberg > >>> <strub...@yahoo.de.INVALID>: > >>>>>>> > >>>>>>> Hi Francesco! > >>>>>>> > >>>>>>> Can you see which one of the classes fail to enhance? > >>>>>>> > >>>>>>> In which case you might try to run the enhancement of that very > class > >>> manually in your IDE. > >>>>>>> > >>>>>>> Main class: org.apache.openjpa.enhance.PCEnhancer > >>>>>>> VMOptions:: -ea > >>> > -Dopenjpa.ConnectionURL=jdbc:derby:target/database/openjpa-derby-database;create=true > >>> -Dopenjpa.ConnectionDriverName=org.apache.derby.jdbc.EmbeddedDriver > >>>>>>> or something similar for your connection > >>>>>>> > >>>>>>> Then you also need the parameter pointing to the class file which > you > >>> want to enhance, e.g. > >>>>>>> > >>> > target/test-classes/org/apache/openjpa/persistence/proxy/entities/Annuity.class > >>>>>>> > >>>>>>> Please keep me updated! > >>>>>>> > >>>>>>> > >>>>>>> LieGrue, > >>>>>>> strub > >>>>>>> > >>>>>>> PS: does it add value if we'd write all this up? You will likely > know > >>> most of that stuff, but others who dig into OpenJPA do not. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> Am 03.07.2023 um 15:28 schrieb Francesco Chicchiriccò < > >>> ilgro...@apache.org>: > >>>>>>>> > >>>>>>>> Hi Mark, > >>>>>>>> thanks for your reply. > >>>>>>>> > >>>>>>>> So I am proceeding as follows. > >>>>>>>> > >>>>>>>> Took the executions at > >>>>>>>> > >>>>>>>> > https://ci-builds.apache.org/job/OpenJPA/job/OpenJPA-master-deploy/ > >>>>>>>> > >>>>>>>> (the Jenkins job which is publishing SNAPSHOT artifacts from > master > >>> to ASF Maven repo) and saw that the artifacts with which the Syncope > build > >>> succeeded last Friday are the one from May 16th, so the corresponding > >>> commit id is > >>>>>>>> > >>>>>>>> b238417dabeac935846fe1ce2fef28aafaeef205 > >>>>>>>> > >>>>>>>> I confirm that building locally from that commit id everything is > >>> working. > >>>>>>>> > >>>>>>>> As soon as I move to next commit (dated May 22nd, pushed and > >>> deployed today) with id > >>>>>>>> > >>>>>>>> b51d003ed98aa7663465aad7c9a1b554b006cc7d > >>>>>>>> > >>>>>>>> I see the same behavior as with HEAD, e.g. > >>>>>>>> > >>>>>>>> 2b9b024f273d63e479a02cad751c28b8ef974ace > >>>>>>>> > >>>>>>>> I will try to take a closer look at the code. > >>>>>>>> Regards. > >>>>>>>> > >>>>>>>> On 03/07/23 14:44, Mark Struberg wrote: > >>>>>>>>> Hi Francesco! > >>>>>>>>> > >>>>>>>>> That's very valuable information! > >>>>>>>>> Could you please do a git-bisect to find out which change did > break > >>> it? > >>>>>>>>> > >>>>>>>>> Actually the algorithm to select which classes are to be enhanced > >>> should not have changed. Thus I'm really curious what I did break. > >>>>>>>>> > >>>>>>>>> Most times the amount of enhanced classes are the same but > slightly > >>> differ in the generated code. That might be due to now we really > generate > >>> Java11++ code whereas before we only generated Java 1.1 code (yes, not > even > >>> 1.5!) > >>>>>>>>> > >>>>>>>>> What I usually do is to have 2 checkouts. You might want to > change > >>> the pom GAV version of the working checkout to something something > >>> different so you might have both in your ~/.m2/repository in parallel. > >>>>>>>>> > >>>>>>>>> The first thing I do is to compare the class files in > >>> target/classes of both versions with 'compare with clipboard'. > >>>>>>>>> I just open the class file in target/classes and let > >>> Intellij/Netbeans do the decompilation for me. > >>>>>>>>> > >>>>>>>>> You might find some differences here in which case this needs > >>> further investigation. Note that the order of the methods might have > >>> changed. > >>>>>>>>> Sometimes you will also see some 'cannot decompile' in the code > >>> block. This happens when something in the bytecode is messed up. You > can > >>> also watch out for ClassNotFound or VerifyError, etc. > >>>>>>>>> > >>>>>>>>> If this is the case I then use > >>>>>>>>> $> javap -c -p -constants > >>> target/classes/com/foo/mycorp/MyBlaEntity.class > MyBlaEntity.decompile > >>>>>>>>> on both versions and check whether something is different. > >>>>>>>>> Often it's a wrong offset in the stack calculation (e.g. ALOAD 1 > vs > >>> ALOAD 2) > >>>>>>>>> > >>>>>>>>> Note that the 'old' Java 1.1 bytecode doesn't understand LDC of > >>> Class constants (LoaD Constant). This was only introduced in Java 1.4 > or > >>> 1.5 bytecode afair. Before that a dynamic Class.forName was called and > >>> stored in a static variable. Ugly stuff, and happy to finally get rid > of > >>> all those hacks... > >>>>>>>>> > >>>>>>>>> Hope that helps to get you started. Feel free to ping me again > over > >>> here or at the #apachee ASF slack channel. > >>>>>>>>> > >>>>>>>>> txs and LieGrue, > >>>>>>>>> strub > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>>> Am 03.07.2023 um 14:13 schrieb Francesco Chicchiriccò < > >>> ilgro...@apache.org>: > >>>>>>>>>> > >>>>>>>>>> From a very preliminary analysis it seems that just a subset of > >>> the persistence-capable classes are effectively found and enhanced: now > >>> [2], it used to be [3]. > >>>>>>>>>> > >>>>>>>>>> I'll continue investigating. > >>>>>>>>>> > >>>>>>>>>> Regards. > >>>>>>>>>> > >>>>>>>>>> On 03/07/23 13:56, Francesco Chicchiriccò wrote: > >>>>>>>>>>> Hi Mark, > >>>>>>>>>>> the build on Syncope master branch, using OpenJPA > 4.0.0-SNAPSHOT > >>> is now failing - it used to work at least until last June 30th. > >>>>>>>>>>> > >>>>>>>>>>> The issues seem to be related to entity enhancement, which we > >>> perform via openjpa-maven-plugin - see [1]. > >>>>>>>>>>> > >>>>>>>>>>> Is there anything obvious we should be changing? > >>>>>>>>>>> Meanwhile I'll try to dig to see if I am able to understand > >>> something more. > >>>>>>>>>>> > >>>>>>>>>>> Regards. > >>>>>>>>>>> > >>>>>>>>>>> [1] > >>> > https://github.com/apache/syncope/blob/master/core/persistence-jpa/pom.xml#L143-L174 > >>>>>>>>>> [2] https://paste.apache.org/gxcm6 > >>>>>>>>>> [3] https://paste.apache.org/1g9k5 > >>>>>>>>>> > >>>>>>>>>>> On 03/07/23 12:15, Mark Struberg wrote: > >>>>>>>>>>>> Update: I've now merged the wip to our master and pushed it to > >>> the ASF gitox repo. > >>>>>>>>>>>> > >>>>>>>>>>>> I think we already came pretty far and it looks really > >>> promising. So there is a high chance that we succeed, although there is > >>> still plenty of work in front of us. > >>>>>>>>>>>> > >>>>>>>>>>>> How it works: > >>>>>>>>>>>> > >>>>>>>>>>>> Basically there are 2 sets of information in the PCEnhancer > >>> right now: > >>>>>>>>>>>> > >>>>>>>>>>>> _managed: the BCClass version of the original class > >>>>>>>>>>>> _pc: the BCClass of the generated/modified class. Might be the > >>> same as _managed but in case of subclassing or interface case it is > >>> different. In those cases _managed represents the original class and > _pc > >>> the subclass/concrete implementation. > >>>>>>>>>>>> > >>>>>>>>>>>> managed and pc are the ASM variants of _managed and _pc. > >>>>>>>>>>>> > >>>>>>>>>>>> The main entry point for debugging into it is PCEnhancer#run() > >>>>>>>>>>>> > >>>>>>>>>>>> There are a few methods in ASMHelper to update from BCClass -> > >>> ASM and vice versa: > >>>>>>>>>>>> > >>>>>>>>>>>> AsmHelper.readIntoBCClass(pc, _pc); > >>>>>>>>>>>> takes all the information from the ASM pc, creates a class > >>> byte[] and reads that back into the existing _pc instance. > >>>>>>>>>>>> > >>>>>>>>>>>> pc = AsmHelper.toClassNode(_pc); > >>>>>>>>>>>> is the corresponding method in the other direction. from > BCClass > >>> to ASM. > >>>>>>>>>>>> > >>>>>>>>>>>> Right now we have the following methods left to migrate in > >>> PCEnhancer > >>>>>>>>>>>> addAttachDetachCode(); > >>>>>>>>>>>> addSerializationCode(); > >>>>>>>>>>>> addCloningCode(); > >>>>>>>>>>>> runAuxiliaryEnhancers(); > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> I'll be off for a week on holidays in Venice. > >>>>>>>>>>>> If someone want's to join in with the effort then please go > on! > >>> You might want to compare the current PCEnhancer code with an older > version > >>> to get an idea how it can be done and to see some patterns I've used. > >>>>>>>>>>>> > >>>>>>>>>>>> As a quick hint: Java is basically a stack based virtual CPU. > >>> The 'this' pointer is on position 0 of the stack, so ALOAD 0. After > that > >>> comes the parameters. The size of the param on the stack depends on the > >>> type. Mostly 1, but for long eg 2 positions. So ALOAD 1 is the 1st > method > >>> parameter, ALOAD 2 the 2nd (except 1st param was a 2-pos type). > >>>>>>>>>>>> In case of a static method there is ofc no 'this' on the > stack, > >>> so the parameters start with zero offset. > >>>>>>>>>>>> After all the parameters there are the unnamed 'local' > >>> variables. I tried to consistently name them xxxVarPos. > >>>>>>>>>>>> See also the various helper methods in ASMHelper. For example > >>> for the various LOAD, RETURN, STORE operations which depends based on > the > >>> type (ALOAD vs ILOAD vs IALOAD etc) > >>>>>>>>>>>> Also read the guide to ASM PDF from objectweb. > >>>>>>>>>>>> > >>>>>>>>>>>> I'll be mostly offline till Sunday, but will try to answer > mails > >>> if there are some questions. > >>>>>>>>>>>> > >>>>>>>>>>>> txs and LieGrue, > >>>>>>>>>>>> strub > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> Am 19.06.2023 um 17:11 schrieb Matt Pavlovich < > >>> mattr...@gmail.com>: > >>>>>>>>>>>>> > >>>>>>>>>>>>> +1 thanks for tackling this, Mark! > >>>>>>>>>>>>> > >>>>>>>>>>>>> ASM definitely more widely used going forward. > >>>>>>>>>>>>> > >>>>>>>>>>>>> -Matt > >>>>>>>>>>>>> > >>>>>>>>>>>>>> On Jun 17, 2023, at 3:01 PM, Mark Struberg > >>> <strub...@yahoo.de.INVALID> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Small update: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I've worked on it over the last few weeks, and I'm getting > >>> closer > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> contains the latest work on the PCEnhancer. Right in the > >>> middle of replacing Serp with native ASM code. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Note that I also had to modify a few interfaces with a few > >>> more to follow from BCClass to ASM ClassNode. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Happy to get some feedback! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> If anybody wants to help with this effort I'm happy to also > >>> push this feature branch to our ASF repo. It looks reasonably promising > >>> already. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> LieGrue, > >>>>>>>>>>>>>> strub > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Am 25.05.2023 um 18:36 schrieb Mark Struberg > >>> <strub...@yahoo.de.INVALID>: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi! > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Right now I'm trying to get rid of serp step by step. > >>>>>>>>>>>>>>> The code is right now at my own github repo in the > >>> fb_asmEnhance branch: > >>>>>>>>>>>>>>> https://github.com/struberg/openjpa/tree/fb_asmEnhance > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> The approach I took for now is to create a few methods in > >>> AsmHelper to be able to move from BCClass -> ASM ClassWriter and the > other > >>> way around. That way we should be able to replace functionality part by > >>> part but still keep all things afloat. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> For now I've started with the PCSubclassValidator. > >>>>>>>>>>>>>>> Right now this evaluates the attributs using Serp plus ASM > >>> and then compare the results. > >>>>>>>>>>>>>>> If something is fishy, you'll see the following > >>>>>>>>>>>>>>> throw new IllegalStateException("MSX ASMTODO " + bcField + > " > >>> " + field); > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> It would be extremely helpful, if you could go through your > >>> projects and let it run and report how it's going. > >>>>>>>>>>>>>>> If you see that "MSX ASMTODO" somewhere then we know I > messed > >>> something up. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'll gonna finally remove the BCClass handling from those > >>> parts in a few days. Current commit is > >>>>>>>>>>>>>>> > >>> > https://github.com/struberg/openjpa/commit/3ea2412003028d37f2a69971a47bb20abf589f8b > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> txs and LieGrue, > >>>>>>>>>>>>>>> strub > >>>>> > >>>> > >>>> -- > >>>> Francesco Chicchiriccò > >>>> > >>>> Tirasa - Open Source Excellence > >>>> http://www.tirasa.net/ > >>>> > >>>> Member at The Apache Software Foundation > >>>> Syncope, Cocoon, Olingo, CXF, OpenJPA, PonyMail > >>>> http://home.apache.org/~ilgrosso/ > >>>> > >>> > >>> > > > >
