Still working on this and gradually making progress. So far, I ran into these three things along the way:
https://issues.apache.org/jira/browse/CALCITE-5477 https://issues.apache.org/jira/browse/CALCITE-5478 https://issues.apache.org/jira/browse/CALCITE-5479 I've applied the linked PRs locally and continue to make progress integrating this new version into Druid. On 2023/01/04 06:15:00 Gian Merlino wrote: > Thank you! I've built a copy of main and have started working on integrating > it into Druid. I'll let the group know if anything interesting comes up. > There is also a good chunk of work to do here, as it's been a while since we > updated our version of Calcite. > > On 2022/12/11 18:43:08 Julian Hyde wrote: > > The bug is now fixed. (I started from your PR, but I had to do quite a lot > > of extra work. Despite what I said earlier, there were several places where > > Calcite was using APIs added between Guava 16 and 19.) > > > > > On Dec 9, 2022, at 1:30 PM, Gian Merlino <[email protected]> wrote: > > > > > > Hmm. In that case I'm not sure why we came to the belief that newer > > > Calcites could not be used with Guava 16.0.1. Maybe it was from reading > > > the Calcite release notes rather than actually testing it. It would be > > > great if this just works. > > > > > > I raised this PR: https://github.com/apache/calcite/pull/2999 and this > > > JIRA: https://issues.apache.org/jira/browse/CALCITE-5428. > > > > > >> On 2022/12/09 20:54:46 Julian Hyde wrote: > > >> I ran with '-Dguava.version=16.0.1' and it seems to just work. > > >> > > >> If you want this to be the new minimum, please create a Jira case and > > >> PR. Changes will be needed to main.yml (GitHub CI), comments in > > >> gradle.properties, and history.md (release notes for the upcoming > > >> release). > > >> > > >> Julian > > >> > > >>> On Fri, Dec 9, 2022 at 12:30 PM Gian Merlino <[email protected]> wrote: > > >>> > > >>> If Calcite ends up dropping the minimum to 16, that would certainly be > > >>> helpful, and appreciated, since it would mean Druid can upgrade Calcite > > >>> without needing to also tackle the Hadoop 2 problem. Please let us know > > >>> if it seems like that would be in the cards! > > >>> > > >>> In case Calcite sticks on Guava 19+, I asked on the Druid issue > > >>> (https://github.com/apache/druid/issues/13532) to see if anyone's tried > > >>> Hadoop 2 with Guava 19. I'm not sure if we've tried that specific > > >>> combination; if not, maybe that would work for us. If that doesn't work > > >>> out, I think our best remaining idea would be to build our own version > > >>> of Calcite that uses a relocated Guava. > > >>> > > >>> On 2022/12/08 20:49:17 Julian Hyde wrote: > > >>>> If it helps, it wouldn’t be impossible for Calcite to move its minimum > > >>>> Guava version down to 16. (If we rely on APIs introduced after 16 we > > >>>> can create a facade for those APIs.) > > >>>> > > >>>> I am frustrated that the Hadoop folks get locked into a particular > > >>>> version of Guava (and JDK… don’t get me started on that one) and in > > >>>> Calcite I would like to do the opposite. > > >>>> > > >>>> Julian > > >>>> > > >>>> > > >>>>> On Dec 8, 2022, at 9:24 AM, Gian Merlino <[email protected]> wrote: > > >>>>> > > >>>>> It pains me to admit that the reason Druid still uses Guava 16 is > > >>>>> that we still support Hadoop 2, which relies on APIs that have been > > >>>>> removed in newer versions of Guava. From time to time it is proposed > > >>>>> that the project drops support for Hadoop 2 in order to be free to > > >>>>> move up Guava versions, and also achieve various other benefits. > > >>>>> There are generally concerns from people that are stuck with Hadoop 2 > > >>>>> for various reasons. The most recent thread is here: > > >>>>> https://lists.apache.org/thread/1j5w9dmt1gp8hx31tvrmyomcko4mlp03. I > > >>>>> hope at some point we decide to bite the bullet and do it, but the > > >>>>> community has not yet come to that decision. > > >>>>> > > >>>>> In the meantime, we're stuck on an older Calcite version, and really > > >>>>> want to update it, so we're looking into options. There's a GitHub > > >>>>> issue here with our thoughts: > > >>>>> https://github.com/apache/druid/issues/13532 > > >>>>> > > >>>>> On 2022/03/30 17:55:44 Julian Hyde wrote: > > >>>>>> Wow, I had no idea that anyone was on a version of Guava that old. > > >>>>>> > > >>>>>> I checked our history of Guava versions: > > >>>>>> * In https://issues.apache.org/jira/browse/CALCITE-1590 > > >>>>>> <https://issues.apache.org/jira/browse/CALCITE-1590> (Calcite 1.12) > > >>>>>> we changed the default from 19 to 20, but still supported Guava > > >>>>>> 14.0.1 and higher. > > >>>>>> * In https://issues.apache.org/jira/browse/CALCITE-1715 > > >>>>>> <https://issues.apache.org/jira/browse/CALCITE-1715> (also Calcite > > >>>>>> 1.12) we changed the default version from 20 to 19 > > >>>>>> * In https://issues.apache.org/jira/browse/CALCITE-2191 > > >>>>>> <https://issues.apache.org/jira/browse/CALCITE-2191> (Calcite 1.16) > > >>>>>> we moved the minimum from 14.0.1 to 19 and cited a Druid email > > >>>>>> thread [1] about upgrading Druid’s version of Guava > > >>>>>> > > >>>>>> Foolish of me to assume that 4 years later Druid would have done > > >>>>>> something. > > >>>>>> > > >>>>>> Julian > > >>>>>> > > >>>>>> [1] https://groups.google.com/g/druid-development/c/Dw2Qu1CWbuQ > > >>>>>> > > >>>>>>> On Mar 30, 2022, at 10:22 AM, Abhishek Agarwal > > >>>>>>> <[email protected]> wrote: > > >>>>>>> > > >>>>>>> Thank you, Alessandro and Stamatis, for the info. Currently, the > > >>>>>>> guava > > >>>>>>> version being used in druid is 16. So simple exclusion is unlikely > > >>>>>>> to work. > > >>>>>>> I will still give it a try. If it doesn't, we will shade the jars > > >>>>>>> on our > > >>>>>>> end. > > >>>>>>> > > >>>>>>> On Tue, Mar 29, 2022 at 3:38 PM Alessandro Solimando < > > >>>>>>> [email protected]> wrote: > > >>>>>>> > > >>>>>>>> Hello Abhishek, > > >>>>>>>> maybe I am re-stating what Stamatis suggested without realising, > > >>>>>>>> but since > > >>>>>>>> Calcite works well with a broad range of guava versions, can't you > > >>>>>>>> simply > > >>>>>>>> exclude guava when you include it in druid? > > >>>>>>>> > > >>>>>>>> At that point if the guava version of druid is in the list of those > > >>>>>>>> supported by Calcite (>= 19 IIRC) all should work fine. > > >>>>>>>> > > >>>>>>>> I feel like Stamatis concerning shading. > > >>>>>>>> > > >>>>>>>> Best regards, > > >>>>>>>> Alessandro > > >>>>>>>> > > >>>>>>>> On Tue, 29 Mar 2022 at 11:18, Stamatis Zampetakis > > >>>>>>>> <[email protected]> > > >>>>>>>> wrote: > > >>>>>>>> > > >>>>>>>>> Hi Abhishek, > > >>>>>>>>> > > >>>>>>>>> Calcite supports multiple versions of Guava so in principle it > > >>>>>>>>> should > > >>>>>>>> work > > >>>>>>>>> without problems with older versions. > > >>>>>>>>> Instead of shading you could possibly select explicitly the > > >>>>>>>>> version > > >>>>>>>>> that you want to use via maven/gradle. > > >>>>>>>>> > > >>>>>>>>> I share your pain but shading is quite complex to get right so I > > >>>>>>>>> am not > > >>>>>>>> in > > >>>>>>>>> favor of putting this burden on Calcite. > > >>>>>>>>> > > >>>>>>>>> I've seen it used in many projects and it rarely works as > > >>>>>>>>> expected. Most > > >>>>>>>> of > > >>>>>>>>> the time it creates additional problems that need to be resolved. > > >>>>>>>>> > > >>>>>>>>> Best, > > >>>>>>>>> Stamatis > > >>>>>>>>> > > >>>>>>>> > > >>>>>> > > >>>>>> > > >>>> > > >>>> > > >> > > >
