The version of Guava that you use at compile time is important. If it’s earlier than 20 (-ish) the byte codes will be different because of the presence/absence of certain varargs methods.
> On Dec 19, 2023, at 7:28 PM, Benchao Li <[email protected]> wrote: > > I tried to reproduce this using Guillaume's > project(https://github.com/MasseGuillaume/asm-remapper-bug), and the > findings are below: > > - compiled latest main branch > (50a20824c4536450dcae963b5e757cf4bbc7e406) with JDK8(1.8.0_391) on > MacOS with M2 chip: reproduced > - compiled latest main branch > (50a20824c4536450dcae963b5e757cf4bbc7e406) with JDK8(1.8.0_391) on > Linux with x64 architecture: reproduced > - compiled latest main branch > (50a20824c4536450dcae963b5e757cf4bbc7e406) with JDK11(1.11.0_21) on > MacOS with M2 chip: cannot reproduce > - compiled latest main branch > (50a20824c4536450dcae963b5e757cf4bbc7e406) with JDK17(1.17.0_7) on > MacOS with M2 chip: cannot reproduce > > It seems unrelated to platforms, and I assume it should not either, > since Java's bytecode should be platform independent. > > @Guillaume You said above that with 1.8.0_371-b11 on macos x64, you > cannot reproduce the problem, can you confirm that? Per my testing, > JDK8's output could all reproduce the problem > > Guillaume Masse <[email protected]> 于2023年12月18日周一 23:13写道： >> >> If I build 1.36.0 and 1.35.0 with the same java version/build javap can >> read the classfiles. Since the release process is manual it's really hard >> to know what causes the problem. My solution is to build and release each >> Calcite version myself and host it on our own S3 repository. >> >> On Mon, Dec 18, 2023, 8:56 AM Ruben Q L <[email protected]> wrote: >> >>> Hello, >>> >>> My project also recently upgraded to Calcite 1.36, and we are facing the >>> exact same issue when trying to shade with ASM. >>> >>> @Guillame , I can see that https://gitlab.ow2.org/asm/asm/-/issues/318008 >>> has >>> been closed, but I can't really understand why, since the explanation >>> mentions SqlFunctions-1.35.0, but the issue is with 1.36, do you have more >>> info on that? >>> >>> Best, >>> Ruben >>> >>> >>> On Fri, Nov 24, 2023 at 10:09 PM Guillaume Masse >>> <[email protected]> wrote: >>> >>>> We run spark in AWS EMR and it overrides the classpath, so we don't have >>>> control over it: >>>> >>>> Sparks will pull guava 14.0.1: >>>> >>>> >>> https://github.com/apache/spark/blob/master/dev/deps/spark-deps-hadoop-3-hive-2.3#L68 >>>> >>>> That's why we need to shade guava. >>>> >>>> >>>> On Fri, Nov 24, 2023 at 4:31 PM Julian Hyde <[email protected]> wrote: >>>> >>>>> The version of Guava we compile against is relevant. In 1.35 we >>>>> compiled against Guava 19.0; in 1.36 we compiled against Guava >>>>> 32.1.3-jre. The effect would be the same if we compiled against any >>>>> version of Guava 20.0 or higher, due to the addition of >>>>> Preconditions.checkArgument methods that in earlier Guava versions >>>>> would be handled by varargs method. >>>>> >>>>> See https://issues.apache.org/jira/browse/CALCITE-5477 (which was >>>>> fixed in 1.35) and https://issues.apache.org/jira/browse/CALCITE-5763 >>>>> (which was fixed in 1.36 and essentially reverted 5477). >>>>> >>>>> Julian >>>>> >>>>> On Fri, Nov 24, 2023 at 10:20 AM Guillaume Masse >>>>> <[email protected]> wrote: >>>>>> >>>>>> Hi Benchao Li, >>>>>> >>>>>> thanks for giving me more details like the java version, >>>>>> >>>>>> I compiled with 1.8.0_371-b11 on macos x64 (emulated) and I was still >>>>> able >>>>>> to transform the classfile. The bug must come from something else. >>> Was >>>>> this >>>>>> compiled on your personal machine? Was it on windows/linux/mac? >>> What's >>>>> the >>>>>> processor architecture? >>>>>> >>>>>> 1.35.0 also pases the transformation and the same piece of code is >>>> there. >>>>>> How was 1.35.0 released? >>>>>> >>>>>> Thanks! >>>>>> >>>>>> >>>>>> On Thu, Nov 23, 2023 at 9:59 PM Benchao Li <[email protected]> >>>> wrote: >>>>>> >>>>>>> Guillaume, >>>>>>> >>>>>>> The binary release are always compiled with JDK8, this is a >>> required >>>>>>> procedure[1]. >>>>>>> >>>>>>> For 1.36.0, the JDK version I'm using is: >>>>>>> $ java -version >>>>>>> java version "1.8.0_371" >>>>>>> Java(TM) SE Runtime Environment (build 1.8.0_371-b11) >>>>>>> Java HotSpot(TM) 64-Bit Server VM (build 25.371-b11, mixed mode) >>>>>>> >>>>>>> [1] >>>>> https://calcite.apache.org/docs/howto.html#making-a-release-candidate >>>>>>> >>>>>>> Guillaume Masse <[email protected]> >>>> 于2023年11月24日周五 >>>>>>> 07:22写道： >>>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> We recently upgraded to calcite 1.36.0 and it broke our >>> deployment >>>>>>> process. >>>>>>>> We are using java asm <https://asm.ow2.io/> to shade (rename >>>>> packages >>>>>>> from >>>>>>>> the bytecode) since there are multiple dependencies clash when >>>> using >>>>> with >>>>>>>> Apache Spark. For example, Apache Spark is using a much older >>>>> version of >>>>>>>> guava. If I build calcite locally with Java Correto 17.0.7-amzn I >>>> can >>>>>>>> transform calcite's bytecode without any problem. Where can I >>> find >>>>> the >>>>>>> java >>>>>>>> version used to compiled calcite? Is there a process in place to >>>>> have a >>>>>>>> consistent java version? >>>>>>>> >>>>>>>> https://gitlab.ow2.org/asm/asm/-/issues/318008 >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Guillaume Massé >>>>>>>> [Gee-OHM] >>>>>>>> (马赛卫) >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Best, >>>>>>> Benchao Li >>>>>>> >>>>> >>>> >>> > > > > -- > > Best, > Benchao Li
