I have the impression that this might be caused by a bug in the JDK, see similar issues: https://gitlab.ow2.org/asm/asm/-/issues/317789 https://bugs.openjdk.org/browse/JDK-8144185 https://bugs.openjdk.org/browse/JDK-8187805
On Thu, Dec 21, 2023 at 8:58 PM Julian Hyde <jh...@apache.org> wrote: > I think I was mistaken about Guava. I agree with Benchao's analysis > and #2 seems to be caused by something other than Guava. > > On Thu, Dec 21, 2023 at 3:53 AM Benchao Li <libenc...@apache.org> wrote: > > > > There are two exceptions here: > > > > # 1 > > java.lang.ArrayIndexOutOfBoundsException: Index 65536 out of bounds > > for length 297 > > at org.objectweb.asm.ClassReader.readLabel(ClassReader.java:2695) > > at org.objectweb.asm.ClassReader.createLabel(ClassReader.java:2711) > > at > org.objectweb.asm.ClassReader.readTypeAnnotations(ClassReader.java:2777) > > at org.objectweb.asm.ClassReader.readCode(ClassReader.java:1929) > > at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515) > > at org.objectweb.asm.ClassReader.accept(ClassReader.java:745) > > at org.objectweb.asm.ClassReader.accept(ClassReader.java:425) > > at remapper.bug.RemapperTest.runTest(RemapperTest.java:60) > > at remapper.bug.RemapperTest.calcite36(RemapperTest.java:36) > > > > # 2 > > java.lang.IllegalArgumentException: Invalid end label (must be visited > first) > > at > org.objectweb.asm.util.CheckMethodAdapter.checkLabel(CheckMethodAdapter.java:1453) > > at > org.objectweb.asm.util.CheckMethodAdapter.visitLocalVariableAnnotation(CheckMethodAdapter.java:996) > > at > org.objectweb.asm.MethodVisitor.visitLocalVariableAnnotation(MethodVisitor.java:757) > > at > org.objectweb.asm.commons.MethodRemapper.visitLocalVariableAnnotation(MethodRemapper.java:257) > > at org.objectweb.asm.ClassReader.readCode(ClassReader.java:2614) > > at org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515) > > at org.objectweb.asm.ClassReader.accept(ClassReader.java:745) > > at org.objectweb.asm.ClassReader.accept(ClassReader.java:425) > > at remapper.bug.RemapperTest.runTest(RemapperTest.java:53) > > at remapper.bug.RemapperTest.calcite36WithCheck(RemapperTest.java:39) > > > > > > Sorry that I'm not clear in my previous email. My testing results are > > all about #1. > > > > I tested 1.34.0/1.35.0/1.36.0 with JDK8 on both Linux and Mac for #2, > > they all reproduced the problem. Hence this is not a problem of > > platform, and also not a problem of 1.35.0->1.36.0, I guess this is an > > asm usage problem related to JDK8's output. > > > > In summary, > > for #1 problem, it seems to be a problem of the Guava version we're > > compiling against, > > for #2 problem, it seems not to be a problem since both > > 1.34.0/1.35.0/1.36.0 can reproduce it (I assume that this does not > > affect your usage since you only spotted this after 1.36.0). > > > > Guillaume Masse <masse.guilla...@narrative.io.invalid> 于2023年12月21日周四 > 03:29写道： > > > > > > calcite-1.35.0 with JDK8(1.8.0_391) on MacOS with M1 Pro chip: can > reproduce > > > > > > RemapperTest > calcite35WithCheck() FAILED > > > java.lang.IllegalArgumentException: Invalid start label (must be > > > visited first) > > > at > org.objectweb.asm.util.CheckMethodAdapter.checkLabel(CheckMethodAdapter.java:1453) > > > at > org.objectweb.asm.util.CheckMethodAdapter.visitLocalVariableAnnotation(CheckMethodAdapter.java:995) > > > at > org.objectweb.asm.MethodVisitor.visitLocalVariableAnnotation(MethodVisitor.java:757) > > > at > org.objectweb.asm.commons.MethodRemapper.visitLocalVariableAnnotation(MethodRemapper.java:257) > > > at > org.objectweb.asm.ClassReader.readCode(ClassReader.java:2614) > > > at > org.objectweb.asm.ClassReader.readMethod(ClassReader.java:1515) > > > at org.objectweb.asm.ClassReader.accept(ClassReader.java:745) > > > at org.objectweb.asm.ClassReader.accept(ClassReader.java:425) > > > at remapper.bug.RemapperTest.runTest(RemapperTest.java:53) > > > at > remapper.bug.RemapperTest.calcite35WithCheck(RemapperTest.java:32) > > > > > > > https://github.com/MasseGuillaume/asm-remapper-bug/blob/main/SqlFunctions-1.35.0.class > > > > > > If you kept the .class for your tests, can you try to run javap -p -c > -v > > > > > > On Wed, Dec 20, 2023 at 1:11 PM Julian Hyde <jhyde.apa...@gmail.com> > wrote: > > > > > > > The version of Guava that you use at compile time is important. If > it’s > > > > earlier than 20 (-ish) the byte codes will be different because of > the > > > > presence/absence of certain varargs methods. > > > > > > > > > > > > > On Dec 19, 2023, at 7:28 PM, Benchao Li <libenc...@apache.org> > wrote: > > > > > > > > > > I tried to reproduce this using Guillaume's > > > > > project(https://github.com/MasseGuillaume/asm-remapper-bug), and > the > > > > > findings are below: > > > > > > > > > > - compiled latest main branch > > > > > (50a20824c4536450dcae963b5e757cf4bbc7e406) with JDK8(1.8.0_391) on > > > > > MacOS with M2 chip: reproduced > > > > > - compiled latest main branch > > > > > (50a20824c4536450dcae963b5e757cf4bbc7e406) with JDK8(1.8.0_391) on > > > > > Linux with x64 architecture: reproduced > > > > > - compiled latest main branch > > > > > (50a20824c4536450dcae963b5e757cf4bbc7e406) with JDK11(1.11.0_21) on > > > > > MacOS with M2 chip: cannot reproduce > > > > > - compiled latest main branch > > > > > (50a20824c4536450dcae963b5e757cf4bbc7e406) with JDK17(1.17.0_7) on > > > > > MacOS with M2 chip: cannot reproduce > > > > > > > > > > It seems unrelated to platforms, and I assume it should not either, > > > > > since Java's bytecode should be platform independent. > > > > > > > > > > @Guillaume You said above that with 1.8.0_371-b11 on macos x64, you > > > > > cannot reproduce the problem, can you confirm that? Per my testing, > > > > > JDK8's output could all reproduce the problem > > > > > > > > > > Guillaume Masse <masse.guilla...@narrative.io.invalid> > 于2023年12月18日周一 > > > > 23:13写道： > > > > >> > > > > >> If I build 1.36.0 and 1.35.0 with the same java version/build > javap can > > > > >> read the classfiles. Since the release process is manual it's > really > > > > hard > > > > >> to know what causes the problem. My solution is to build and > release > > > > each > > > > >> Calcite version myself and host it on our own S3 repository. > > > > >> > > > > >> On Mon, Dec 18, 2023, 8:56 AM Ruben Q L <rube...@gmail.com> > wrote: > > > > >> > > > > >>> Hello, > > > > >>> > > > > >>> My project also recently upgraded to Calcite 1.36, and we are > facing > > > > the > > > > >>> exact same issue when trying to shade with ASM. > > > > >>> > > > > >>> @Guillame , I can see that > > > > https://gitlab.ow2.org/asm/asm/-/issues/318008 > > > > >>> has > > > > >>> been closed, but I can't really understand why, since the > explanation > > > > >>> mentions SqlFunctions-1.35.0, but the issue is with 1.36, do you > have > > > > more > > > > >>> info on that? > > > > >>> > > > > >>> Best, > > > > >>> Ruben > > > > >>> > > > > >>> > > > > >>> On Fri, Nov 24, 2023 at 10:09 PM Guillaume Masse > > > > >>> <masse.guilla...@narrative.io.invalid> wrote: > > > > >>> > > > > >>>> We run spark in AWS EMR and it overrides the classpath, so we > don't > > > > have > > > > >>>> control over it: > > > > >>>> > > > > >>>> Sparks will pull guava 14.0.1: > > > > >>>> > > > > >>>> > > > > >>> > > > > > https://github.com/apache/spark/blob/master/dev/deps/spark-deps-hadoop-3-hive-2.3#L68 > > > > >>>> > > > > >>>> That's why we need to shade guava. > > > > >>>> > > > > >>>> > > > > >>>> On Fri, Nov 24, 2023 at 4:31 PM Julian Hyde <jh...@apache.org> > wrote: > > > > >>>> > > > > >>>>> The version of Guava we compile against is relevant. In 1.35 we > > > > >>>>> compiled against Guava 19.0; in 1.36 we compiled against Guava > > > > >>>>> 32.1.3-jre. The effect would be the same if we compiled > against any > > > > >>>>> version of Guava 20.0 or higher, due to the addition of > > > > >>>>> Preconditions.checkArgument methods that in earlier Guava > versions > > > > >>>>> would be handled by varargs method. > > > > >>>>> > > > > >>>>> See https://issues.apache.org/jira/browse/CALCITE-5477 (which > was > > > > >>>>> fixed in 1.35) and > > > > https://issues.apache.org/jira/browse/CALCITE-5763 > > > > >>>>> (which was fixed in 1.36 and essentially reverted 5477). > > > > >>>>> > > > > >>>>> Julian > > > > >>>>> > > > > >>>>> On Fri, Nov 24, 2023 at 10:20 AM Guillaume Masse > > > > >>>>> <masse.guilla...@narrative.io.invalid> wrote: > > > > >>>>>> > > > > >>>>>> Hi Benchao Li, > > > > >>>>>> > > > > >>>>>> thanks for giving me more details like the java version, > > > > >>>>>> > > > > >>>>>> I compiled with 1.8.0_371-b11 on macos x64 (emulated) and I > was > > > > still > > > > >>>>> able > > > > >>>>>> to transform the classfile. The bug must come from something > else. > > > > >>> Was > > > > >>>>> this > > > > >>>>>> compiled on your personal machine? Was it on > windows/linux/mac? > > > > >>> What's > > > > >>>>> the > > > > >>>>>> processor architecture? > > > > >>>>>> > > > > >>>>>> 1.35.0 also pases the transformation and the same piece of > code is > > > > >>>> there. > > > > >>>>>> How was 1.35.0 released? > > > > >>>>>> > > > > >>>>>> Thanks! > > > > >>>>>> > > > > >>>>>> > > > > >>>>>> On Thu, Nov 23, 2023 at 9:59 PM Benchao Li < > libenc...@apache.org> > > > > >>>> wrote: > > > > >>>>>> > > > > >>>>>>> Guillaume, > > > > >>>>>>> > > > > >>>>>>> The binary release are always compiled with JDK8, this is a > > > > >>> required > > > > >>>>>>> procedure[1]. > > > > >>>>>>> > > > > >>>>>>> For 1.36.0, the JDK version I'm using is: > > > > >>>>>>> $ java -version > > > > >>>>>>> java version "1.8.0_371" > > > > >>>>>>> Java(TM) SE Runtime Environment (build 1.8.0_371-b11) > > > > >>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.371-b11, mixed > mode) > > > > >>>>>>> > > > > >>>>>>> [1] > > > > >>>>> > > > > > https://calcite.apache.org/docs/howto.html#making-a-release-candidate > > > > >>>>>>> > > > > >>>>>>> Guillaume Masse <masse.guilla...@narrative.io.invalid> > > > > >>>> 于2023年11月24日周五 > > > > >>>>>>> 07:22写道： > > > > >>>>>>>> > > > > >>>>>>>> Hi all, > > > > >>>>>>>> > > > > >>>>>>>> We recently upgraded to calcite 1.36.0 and it broke our > > > > >>> deployment > > > > >>>>>>> process. > > > > >>>>>>>> We are using java asm <https://asm.ow2.io/> to shade > (rename > > > > >>>>> packages > > > > >>>>>>> from > > > > >>>>>>>> the bytecode) since there are multiple dependencies clash > when > > > > >>>> using > > > > >>>>> with > > > > >>>>>>>> Apache Spark. For example, Apache Spark is using a much > older > > > > >>>>> version of > > > > >>>>>>>> guava. If I build calcite locally with Java Correto > 17.0.7-amzn I > > > > >>>> can > > > > >>>>>>>> transform calcite's bytecode without any problem. Where can > I > > > > >>> find > > > > >>>>> the > > > > >>>>>>> java > > > > >>>>>>>> version used to compiled calcite? Is there a process in > place to > > > > >>>>> have a > > > > >>>>>>>> consistent java version? > > > > >>>>>>>> > > > > >>>>>>>> https://gitlab.ow2.org/asm/asm/-/issues/318008 > > > > >>>>>>>> > > > > >>>>>>>> > > > > >>>>>>>> -- > > > > >>>>>>>> Guillaume Massé > > > > >>>>>>>> [Gee-OHM] > > > > >>>>>>>> (马赛卫) > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> > > > > >>>>>>> -- > > > > >>>>>>> > > > > >>>>>>> Best, > > > > >>>>>>> Benchao Li > > > > >>>>>>> > > > > >>>>> > > > > >>>> > > > > >>> > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > Best, > > > > > Benchao Li > > > > > > > > > > > > > > > > -- > > > > Best, > > Benchao Li >
