Hi, Thanks for making it. It's a massive step for us!
I found the compile dependency on "nashorn-core" was newly added, which is licensed under "gpl-v2-with-the-classpath-exception-license". I'm not confident that we can distribute the jar file on a release. I'm still testing the master branch with JDK 17. So far, so good. Best, Okumin On Sat, May 31, 2025 at 4:02 AM Simhadri G <simhad...@apache.org> wrote: > > Congratulations guys! That's really awesome! > > Will test it out and let you know if I run into any bugs! > > Thanks everyone ! > Simhadri G > > > On Sat, May 31, 2025, 12:28 AM Ayush Saxena <ayush...@gmail.com> wrote: >> >> Hi Folks, >> >> The Hive master branch has now been upgraded to JDK 17. This means >> that building or running the latest code from the master branch will >> require JDK 17 going forward. >> You're encouraged to try it out and share your feedback. If you >> encounter any issues, please feel free to report them here or create a >> ticket with the details. >> >> A big thank you to all the contributors who helped make this >> transition possible! >> >> Looking ahead, Hive 4.1.0 is expected to be released by the end of >> June or early July, and will officially support JDK 17. >> >> Best regards, >> Ayush >> >> ---------- Forwarded message --------- >> From: Vlad Rozov <vro...@apache.org> >> Date: Tue, 6 May 2025 at 20:29 >> Subject: Re: Move to JDK-11 >> To: <dev@hive.apache.org> >> >> >> I am looking to upgrade Apache Spark 4.x to use Apache Hive 4.x as >> both compile and runtime dependencies. Spark 4.x requires JDK 17 and >> while working on upgrade I have not encountered issues caused by Hive >> using JDK 8. I will keep an eye and report any, though I don't expect >> to see a problem till Spark upgrades to JDK 21. >> >> Thank you, >> >> Vlad >> >> On 2025/04/08 14:27:33 Butao Zhang wrote: >> > The upgrade to JDK 17 is a significant change that introduces >> > compatibility and adaptation requirements for both Apache Tez and Apache >> > Hive. Relevant work is currently in progress—thanks to all community >> > contributors who have already been involved in adaptation and fixes. We >> > encourage everyone to actively report any issues encountered during usage >> > or integration, so that we can collaborate on discussions and resolutions, >> > and accelerate the overall transition to JDK 17. >> > >> > Thanks, >> > Butao Zhang >> > >> > On 2025/04/07 11:25:33 Shohei Okumiya wrote: >> > > Hi Ayush and all people working on the JDK upgrade, >> > > >> > > Thanks for sharing the timeline and detailing the current progress. I'm >> > > more than happy to hear the good news and look forward to seeing Hive >> > > running on Java 17! >> > > >> > > Mid-May is not so far away. I don't have a firm preference about how to >> > > package 4.1. If anyone is interested, let's discuss it in the thread for >> > > the Hive 4.1.x release. >> > > >> > > Finally, let me express my gratitude to everyone working on the >> > > challenging >> > > upgrade project. It will definitely be one of the most outstanding >> > > achievements! >> > > >> > > Sincerely, >> > > Okumin >> > > >> > > On Mon, Apr 7, 2025 at 7:03 PM Ayush Saxena <ayush...@gmail.com> wrote: >> > > >> > > > Hi Okumin, >> > > > >> > > > I know I'm one of the people responsible for misjudging the timeline >> > > > around the JDK-17 upgrade, which has ultimately blocked the 4.1.0 >> > > > release. >> > > > I want to take a moment to share my thoughts and where things currently >> > > > stand. >> > > > >> > > > Q: Can we confirm the importance and ETA of the JDK upgrade and how we >> > > > should proceed? >> > > > >> > > > Ans: I think we all agree that moving to JDK-17 is a major milestone, >> > > > and >> > > > we're admittedly late in adopting it. Many projects have already moved >> > > > or >> > > > are actively working towards it. Not being on JDK-17 restricts us from >> > > > adopting modern third-party libraries like Iceberg and from leveraging >> > > > several improvements that come with it. >> > > > >> > > > As for the ETA: I realize I previously gave an inaccurate timeline, >> > > > and I >> > > > sincerely regret that. However, I now have a much clearer picture. I'm >> > > > in >> > > > active discussions with folks internally who are working on it. I >> > > > believe >> > > > we should have the code ready by mid-May, at which point we can start >> > > > gathering feedback from the community. >> > > > >> > > > Q: We've been working on this upgrade for over half a year. When will >> > > > it >> > > > merge? PR: https://github.com/apache/hive/pull/5404 >> > > > In my opinion, #5404 should be divided into multiple pull requests. I >> > > > really understand we want to test simultaneously rather than make >> > > > incremental progress. However, the consolidated PR makes it impossible >> > > > for >> > > > other developers to help the initiative, reviewers have to check huge >> > > > changes, and authors must occasionally resolve many conflicts. I don't >> > > > prefer any HOW as long as the delivery is predictable, meaning it is >> > > > okay >> > > > for me if we are confident with the agreed ETA. If it is still >> > > > uncertain, >> > > > we may try another project management, which is >> > > > incremental/predictable and >> > > > where we can invest other developers' resources in it. >> > > > >> > > > Ans: I completely agree with your observations. The original PR started >> > > > more as an experiment to see what changes were needed. At the time, I >> > > > don’t >> > > > think anyone anticipated how large it would become, and unfortunately, >> > > > it >> > > > kept growing incrementally over time. >> > > > That said, I’ve already asked the team to start breaking down the PR >> > > > into >> > > > smaller, more manageable parts. Dependency upgrades will be separated >> > > > out >> > > > soon, and changes like those related to SpotBugs (which account for a >> > > > large >> > > > number of line changes) will also be split. The goal is to make the PR >> > > > easier to review and more accessible for other contributors to jump >> > > > in. I >> > > > understand this is a genuine blocker, and folks are doing their best >> > > > to >> > > > address it. >> > > > >> > > > Q: [1] Should we ship 4.1 without JDK-17, given it's been a long time >> > > > since 4.0.1? >> > > > >> > > > Ans: Honestly, I’m not in a position to oppose this. The community >> > > > kindly >> > > > respected my request last time when I asked for more time for the >> > > > JDK-17 >> > > > upgrade, and unfortunately, I couldn’t follow through as expected. So, >> > > > with >> > > > sincere apologies for being one of the reasons for the delay, I’m okay >> > > > with >> > > > whatever direction the community chooses to take from here. >> > > > >> > > > -Ayush >> > > > >> > > > >> > > > On Sun, 6 Apr 2025 at 11:55, Shohei Okumiya <oku...@apache.org> wrote: >> > > > >> > > >> Hi, >> > > >> >> > > >> I appreciate everyone working on this big and challenging project! >> > > >> >> > > >> Can we confirm the importance and ETA of the JDK upgrade and how we >> > > >> should work on it? >> > > >> >> > > >> I recognize it is one of the most essential initiatives blocking the >> > > >> release of Hive 4.1.0[1] and the upgrade of Iceberg libraries[2]. >> > > >> Users and >> > > >> developers would expect it to be shipped soon. >> > > >> >> > > >> We have to complete this effort to ship JDK 17. We've been working on >> > > >> it >> > > >> for over half a year. Some of us might be interested in when we will >> > > >> merge >> > > >> it. >> > > >> https://github.com/apache/hive/pull/5404 >> > > >> >> > > >> In my opinion, #5404 should be divided into multiple pull requests. I >> > > >> really understand we want to test simultaneously rather than make >> > > >> incremental progress. However, the consolidated PR makes it >> > > >> impossible for >> > > >> other developers to help the initiative, reviewers have to check huge >> > > >> changes, and authors must occasionally resolve many conflicts. I don't >> > > >> prefer any HOW as long as the delivery is predictable, meaning it is >> > > >> okay >> > > >> for me if we are confident with the agreed ETA. If it is still >> > > >> uncertain, >> > > >> we may try another project management, which is >> > > >> incremental/predictable and >> > > >> where we can invest other developers' resources in it. >> > > >> >> > > >> - [1] Honestly, I feel we should ship 4.1 without JDK 17, as it's >> > > >> been a >> > > >> long time since the last release of 4.0.1 >> > > >> - [2] Iceberg 1.7 or later doesn't support Java 8 >> > > >> >> > > >> Best, >> > > >> Okumin >> > > >> >> > > >> On Wed, Dec 11, 2024 at 2:51 AM Ayush Saxena <ayush...@gmail.com> >> > > >> wrote: >> > > >> >> > > >>> Regarding JDK-11, we didn't actively chased that, so its status >> > > >>> remains >> > > >>> uncertain. With lower compile-time support like JDK-8, we can >> > > >>> certainly >> > > >>> ensure runtime support for both JDK-11 and JDK-17, which I believe >> > > >>> should >> > > >>> work for Hive 4.0.1. As far as I know, this approach works >> > > >>> downstream for >> > > >>> us atleast. >> > > >>> >> > > >>> However, maintaining compile-time compatibility across JDK-17, >> > > >>> JDK-11, >> > > >>> and JDK-8 simultaneously may not be feasible. JDK-17 introduces >> > > >>> significant >> > > >>> changes, and I don’t believe it’s naive—or even possible—to compile >> > > >>> Hive >> > > >>> using both JDK-8 and JDK-17 & expect things to work properly. >> > > >>> >> > > >>> The initial plan appeared to be supporting JDK-8 for Hive 4.0.x >> > > >>> release >> > > >>> line, with potential tweaks if required to enable runtime support for >> > > >>> JDK-17. Starting with Hive 4.1.x, we could shift to JDK-17 for >> > > >>> compile-time >> > > >>> support, and later evaluate the feasibility of supporting JDK-21 or >> > > >>> newer >> > > >>> versions. >> > > >>> >> > > >>> Once we see a state when nobody want JDK-8 compile time support & >> > > >>> everyone has moved to JDK-17 or say using 4.1.x release line, We can >> > > >>> then >> > > >>> drop having bug fix releases for 4.0.x >> > > >>> >> > > >>> -Ayush >> > > >>> >> > > >>> On Mon, 9 Dec 2024 at 15:11, Butao Zhang <butaozha...@163.com> wrote: >> > > >>> >> > > >>>> IMO, I think there two challenges if we want to maintain JDK8 & >> > > >>>> JDK11 & >> > > >>>> JDK17 at the same time. >> > > >>>> >> > > >>>> - Different jdk syntactic sugar between multiple jdks, and >> > > >>>> advanced >> > > >>>> feats in jdk17&jdk21, such as jdk vector api. If we want the >> > > >>>> multiple jdks, >> > > >>>> we can not use the advanced feats in high jdk in the future. >> > > >>>> Such as the >> > > >>>> discussion in https://issues.apache.org/jira/browse/HIVE-28614 >> > > >>>> - We need to run the jenkins CI process multiple times to run the >> > > >>>> multiple jdk env. I'm not sure we have enough jenkins resources >> > > >>>> to do this. >> > > >>>> However, i want to say if we can use github actions to run the >> > > >>>> whole CI? I >> > > >>>> have seen Apache Spark have done the good job which using the >> > > >>>> forked >> > > >>>> repository github actions to run the CI. FRY >> > > >>>> https://lists.apache.org/thread/dmv1h2snqc7qtdjlzj4k8qxookclp720 >> > > >>>> >> > > >>>> >> > > >>>> Thanks, >> > > >>>> Butao Zhang >> > > >>>> ---- Replied Message ---- >> > > >>>> From Shohei Okumiya<oku...@apache.org> <undefined> >> > > >>>> Date 12/9/2024 17:06 >> > > >>>> To <dev@hive.apache.org> <dev@hive.apache.org> >> > > >>>> Subject Re: Move to JDK-11 >> > > >>>> Hi, >> > > >>>> >> > > >>>> I appreciate everyone's work on the JDK upgrade. It would be ideal >> > > >>>> if >> > > >>>> we could migrate JDK like this. That will give us an >> > > >>>> incremental migration path. >> > > >>>> >> > > >>>> 1. (now) Hive 4.0 compiled by JDK 8, and run by JDK 8 >> > > >>>> 2. Hive 4.x compiled by JDK 8, and run by JDK 17 >> > > >>>> 3. Hive 4.x compiled by JDK 17, and run by JDK 17 >> > > >>>> >> > > >>>> I guess it is technically possible that Hive supports JDK 8 and 17 >> > > >>>> at >> > > >>>> runtime. Apache Spark supports JDK 8, 11, and 17. What is the most >> > > >>>> challenging point? >> > > >>>> >> > > >>>> Regards, >> > > >>>> Okumin >> > > >>>> >> > > >>>> On Thu, Aug 29, 2024 at 10:48 PM Butao Zhang <butaozha...@163.com> >> > > >>>> wrote: >> > > >>>> >> > > >>>>> Hi Akshat, >> > > >>>>> >> > > >>>>> Thanks for your reply! I agree that supporting both JDK 8 and JDK >> > > >>>>> 17 >> > > >>>>> is not easy. I think we can safely to move to JDK 17 if we can >> > > >>>>> ensure *hive >> > > >>>>> application* can access to HiveServer2/Metastore without forcing >> > > >>>>> to >> > > >>>>> use the JDK 17. >> > > >>>>> For example, if a hive application is developed by JDK 8, it can >> > > >>>>> import the *hive-exec.jar & hive-jdbc.jar* which is compiled by JDK >> > > >>>>> 17 and can access to hs2/hms successfully. If this case can be >> > > >>>>> supported, >> > > >>>>> I think we can move to JDK 17 as soon as possible. >> > > >>>>> >> > > >>>>> BTW, If Apache Hive , Apache Tez and Apache Hadoop are all compiled >> > > >>>>> with JDK17 or higer version, i think the hive job will run more >> > > >>>>> faster. >> > > >>>>> Users will enjoy significant performance improvements due to JDK >> > > >>>>> version >> > > >>>>> upgrades. >> > > >>>>> I'm looking forward to the performance improvement. :) >> > > >>>>> >> > > >>>>> Thanks, >> > > >>>>> Butao Zhang >> > > >>>>> ---- Replied Message ---- >> > > >>>>> From Akshat m<akshatats...@gmail.com> <undefined> >> > > >>>>> Date 8/29/2024 18:44 >> > > >>>>> To <dev@hive.apache.org> <dev@hive.apache.org> >> > > >>>>> Subject Re: Move to JDK-11 >> > > >>>>> Hi Butao, >> > > >>>>> >> > > >>>>> It won’t be feasible to support both JDK 8 and JDK 17 in the same >> > > >>>>> Hive >> > > >>>>> version due to compatibility issues and behavior changes, as >> > > >>>>> detailed in >> > > >>>>> the PR. The internal changes in JDK implementations have made this >> > > >>>>> challenging. Additionally, many other libraries maintain separate >> > > >>>>> versions >> > > >>>>> for JDK 17+ and JDK 8 support. >> > > >>>>> >> > > >>>>> To align with this approach, I suggest we continue with Hive 4 >> > > >>>>> supporting JDK 8. Once the JDK 17 PR is merged, we could plan a >> > > >>>>> separate >> > > >>>>> release that specifically targets JDK 17+ support. This would >> > > >>>>> ensure that >> > > >>>>> Hive stays up-to-date while maintaining stability across different >> > > >>>>> environments. >> > > >>>>> >> > > >>>>> Would love to hear your thoughts or any additional suggestions the >> > > >>>>> community might have! >> > > >>>>> >> > > >>>>> Regards >> > > >>>>> >> > > >>>>> Akshat >> > > >>>>> >> > > >>>>> >> > > >>>>> >> > > >>>>> On Wed, Aug 28, 2024 at 8:00 AM Butao Zhang <butaozha...@163.com> >> > > >>>>> wrote: >> > > >>>>> >> > > >>>>>> Good job! Thanks everyone who put effort into the JDK upgrading. >> > > >>>>>> That's a big good news for Apache Hive community! >> > > >>>>>> Just a question: Given that many people are still using the lower >> > > >>>>>> jdk8 version. Do we consider to keeping compatibility with lower >> > > >>>>>> jdk8 >> > > >>>>>> version? >> > > >>>>>> >> > > >>>>>> >> > > >>>>>> Thanks, >> > > >>>>>> Butao Zhang >> > > >>>>>> ---- Replied Message ---- >> > > >>>>>> From Akshat m<akshatats...@gmail.com> <undefined> >> > > >>>>>> Date 8/27/2024 23:12 >> > > >>>>>> To dev<dev@hive.apache.org> <dev@hive.apache.org> >> > > >>>>>> Cc kokila narayanan<kokilanarayana...@gmail.com>, >> > > >>>>>> <kokilanarayana...@gmail.com>Raghav >> > > >>>>>> Aggarwal<raghavaggarwal03...@gmail.com>, >> > > >>>>>> <raghavaggarwal03...@gmail.com>Ayush Saxena<ayush...@gmail.com>, >> > > >>>>>> <ayush...@gmail.com>tanishq chugh<tanishqchugh4...@gmail.com>, >> > > >>>>>> <tanishqchugh4...@gmail.com><aturo...@cloudera.com> >> > > >>>>>> <aturo...@cloudera.com> >> > > >>>>>> Subject Re: Move to JDK-11 >> > > >>>>>> >> > > >>>>>> Hi everyone, >> > > >>>>>> >> > > >>>>>> I’m excited to share that after months of dedicated work, we’ve >> > > >>>>>> finally achieved a passing PR[1] for running *Apache Hive on JDK >> > > >>>>>> 17*! >> > > >>>>>> >> > > >>>>>> This was a challenging task, requiring to address various >> > > >>>>>> compatibility issues and ensure our comprehensive test suite >> > > >>>>>> passed without >> > > >>>>>> a hitch. >> > > >>>>>> >> > > >>>>>> Thanks Kokila and Tanishq for joining me in the initiative and >> > > >>>>>> Raghav >> > > >>>>>> for contributing fix for the errorProne issue, and thanks to >> > > >>>>>> Attila, Ayush >> > > >>>>>> and other Hive PMC members for their guidance in the initiative >> > > >>>>>> >> > > >>>>>> It’s been a real team effort, and I’m grateful for the >> > > >>>>>> collaboration >> > > >>>>>> and support from everyone involved. >> > > >>>>>> >> > > >>>>>> I’d appreciate it if you could take some time to review the PR. >> > > >>>>>> Community feedback is really important to make this happen. >> > > >>>>>> >> > > >>>>>> The PR #5404 addresses the integration of Apache Hive with JDK 17, >> > > >>>>>> introducing a range of changes aimed at ensuring compatibility and >> > > >>>>>> improving the codebase. Here’s a detailed breakdown: >> > > >>>>>> >> > > >>>>>> 1. Core Code Adjustments: >> > > >>>>>> >> > > >>>>>> - Compatibility Fixes: Various parts of the Hive codebase have >> > > >>>>>> been modified to ensure they work seamlessly with JDK 17. This >> > > >>>>>> includes >> > > >>>>>> addressing language or API changes that JDK 17 introduces. >> > > >>>>>> >> > > >>>>>> - Reflection and Modules: Adjustments related to how Hive >> > > >>>>>> interacts with Java's reflection API, which has stricter module >> > > >>>>>> system >> > > >>>>>> rules in JDK 17. >> > > >>>>>> >> > > >>>>>> 2. Dependency Management: >> > > >>>>>> >> > > >>>>>> - Version Upgrades: Several third-party dependencies have been >> > > >>>>>> updated to versions that are compatible with JDK 17. This is >> > > >>>>>> critical >> > > >>>>>> because older versions of some libraries may not work properly or >> > > >>>>>> might use >> > > >>>>>> deprecated features removed in JDK 17. >> > > >>>>>> >> > > >>>>>> - Data nucleus upgrade: To ensure compatibility with Java 17, >> > > >>>>>> we >> > > >>>>>> upgraded DataNucleus to version 6.x. This upgrade, however, >> > > >>>>>> introduced an >> > > >>>>>> issue with Hive schema creation. The issue stems from a commit in >> > > >>>>>> DataNucleus[2] that alters the handling of primary key column >> > > >>>>>> names by >> > > >>>>>> enclosing them in double quotes. This change caused conflicts >> > > >>>>>> during Hive >> > > >>>>>> schema generation. As an interim solution, we implemented a >> > > >>>>>> patched >> > > >>>>>> DataNucleus-rdbms module to address the issue in Hive. The >> > > >>>>>> necessary fix is >> > > >>>>>> now included in the DataNucleus repository[3]. Once DataNucleus >> > > >>>>>> 6.0.8 is >> > > >>>>>> officially released, we will upgrade to this version and remove >> > > >>>>>> the >> > > >>>>>> temporary patched module. >> > > >>>>>> >> > > >>>>>> - Dependency Cleanup: Unused or outdated dependencies were >> > > >>>>>> removed, >> > > >>>>>> reducing potential conflicts and simplifying the build process. >> > > >>>>>> >> > > >>>>>> 3. Testing Enhancements: >> > > >>>>>> >> > > >>>>>> - Test Suite Updates: The PR includes updates to Hive’s test >> > > >>>>>> suite >> > > >>>>>> to ensure all tests pass under JDK 17. This likely involved >> > > >>>>>> modifying tests >> > > >>>>>> that needed adjustments to align with JDK 17’s behaviour. >> > > >>>>>> >> > > >>>>>> - New Tests: New tests may have been added to cover areas where >> > > >>>>>> JDK >> > > >>>>>> 17 introduced new features or changes that could impact Hive. >> > > >>>>>> >> > > >>>>>> 4. Compatibility Checks: >> > > >>>>>> >> > > >>>>>> - JDK Version Verification: A new compatibility check has been >> > > >>>>>> introduced to verify the JDK version at runtime. This ensures >> > > >>>>>> that Hive is >> > > >>>>>> running on an appropriate JDK version, preventing unexpected >> > > >>>>>> errors or >> > > >>>>>> behaviour due to version mismatches. >> > > >>>>>> >> > > >>>>>> 5. Build Updates: >> > > >>>>>> >> > > >>>>>> - Build System Tweaks: The PR likely includes changes to the >> > > >>>>>> Maven >> > > >>>>>> build configuration to accommodate JDK 17, ensuring that the >> > > >>>>>> build process >> > > >>>>>> is smooth and compatible. >> > > >>>>>> >> > > >>>>>> Overall, this PR is a comprehensive effort to bring Apache Hive >> > > >>>>>> up to >> > > >>>>>> date with the latest Java platform, ensuring long-term >> > > >>>>>> maintainability and >> > > >>>>>> performance improvements. >> > > >>>>>> >> > > >>>>>> As the next steps, we’ll be focusing on further sanity checks and >> > > >>>>>> extensive testing to make sure everything functions smoothly. >> > > >>>>>> Community >> > > >>>>>> involvement in this process will be crucial, so please do share >> > > >>>>>> any issues >> > > >>>>>> or feedback as you test the changes. >> > > >>>>>> >> > > >>>>>> I’m looking forward to seeing the benefits this will bring to our >> > > >>>>>> community and users. >> > > >>>>>> >> > > >>>>>> >> > > >>>>>> [1] https://github.com/apache/hive/pull/5404 >> > > >>>>>> >> > > >>>>>> [2] >> > > >>>>>> https://github.com/datanucleus/datanucleus-rdbms/commit/b99df3c803862f9e9feacc7891461bcb21160b38 >> > > >>>>>> >> > > >>>>>> [3] >> > > >>>>>> https://github.com/datanucleus/datanucleus-rdbms/commit/b99df3c803862f9e9feacc7891461bcb21160b38 >> > > >>>>>> >> > > >>>>>> >> > > >>>>>> Best regards, >> > > >>>>>> >> > > >>>>>> Akshat Mathur >> > > >>>>>> >> > > >>>>>> >> > > >>>>>> On Tue, 11 Jul 2023 at 2:25 PM, Attila Turoczy < >> > > >>>>>> aturo...@cloudera.com.invalid> wrote: >> > > >>>>>> >> > > >>>>>>> Returning to this topic, I kindly request those who would like to >> > > >>>>>>> advocate >> > > >>>>>>> for the continued support of JDK8 to please share their >> > > >>>>>>> reasoning and >> > > >>>>>>> insights with us. Your input and perspective are greatly >> > > >>>>>>> appreciated! >> > > >>>>>>> Thank you. >> > > >>>>>>> >> > > >>>>>>> -Attila >> > > >>>>>>> >> > > >>>>>>> On Fri, Jun 2, 2023 at 12:43 PM Attila Turoczy < >> > > >>>>>>> aturo...@cloudera.com> >> > > >>>>>>> wrote: >> > > >>>>>>> >> > > >>>>>>> > Hi All, >> > > >>>>>>> > >> > > >>>>>>> > I know my opinion might not be the most popular, but I advocate >> > > >>>>>>> for using* >> > > >>>>>>> > JDK 17*. Here's why: >> > > >>>>>>> > >> > > >>>>>>> > Let's consider a scenario where a customer wants to use the >> > > >>>>>>> > latest >> > > >>>>>>> version >> > > >>>>>>> > of Apache Hive. They would typically install it locally or on a >> > > >>>>>>> small >> > > >>>>>>> > cluster. In 2023, is it realistic to assume that this customer >> > > >>>>>>> won't be >> > > >>>>>>> > able to install JDK 17 on their cluster? Even in large >> > > >>>>>>> enterprises, it >> > > >>>>>>> > should be feasible to install an LTS JDK, especially >> > > >>>>>>> > considering >> > > >>>>>>> the >> > > >>>>>>> > widespread adoption of cloud computing. Sungwoo Park's >> > > >>>>>>> measurements also >> > > >>>>>>> > support this recommendation to go with JDK 17. It outperforms >> > > >>>>>>> > JDK >> > > >>>>>>> 11 by 8% >> > > >>>>>>> > in terms of runtime speed, and JDK 11 itself is 10+% faster >> > > >>>>>>> > than >> > > >>>>>>> JDK 8. >> > > >>>>>>> > This is a significant value proposition. Who would be the >> > > >>>>>>> > customer >> > > >>>>>>> that >> > > >>>>>>> > says, "I don't want faster query execution! I'd rather use JDK >> > > >>>>>>> > 8 >> > > >>>>>>> and pay >> > > >>>>>>> > more for cloud or data center resources instead of using JDK >> > > >>>>>>> > 17!" >> > > >>>>>>> It >> > > >>>>>>> > doesn't make sense to me. >> > > >>>>>>> > >> > > >>>>>>> > The tech industry has been evolving at an incredible pace, with >> > > >>>>>>> > improvements in serialization, IPC mechanisms, and parallelized >> > > >>>>>>> frameworks >> > > >>>>>>> > since the release of JDK 8 ten years ago. We should leverage >> > > >>>>>>> > these >> > > >>>>>>> > advancements! Couple years ago, we invested a lot to improve >> > > >>>>>>> > 1-2% >> > > >>>>>>> of the >> > > >>>>>>> > execution. We prayed for 3 gods, sacrificed 2 ships and >> > > >>>>>>> > traveled >> > > >>>>>>> around the >> > > >>>>>>> > world to make it happen. :-) Now, the JDK itself provides a >> > > >>>>>>> substantial >> > > >>>>>>> > amount of improvement. So, why would we resist progress just >> > > >>>>>>> because there >> > > >>>>>>> > are a few lazy or conservative admins who don't want to spend >> > > >>>>>>> > two >> > > >>>>>>> minutes >> > > >>>>>>> > installing a JDK? >> > > >>>>>>> > >> > > >>>>>>> > A platform needs to be modern and incorporate the latest >> > > >>>>>>> technologies to >> > > >>>>>>> > attract developers and users. I understand that some may >> > > >>>>>>> > prefer to >> > > >>>>>>> stay >> > > >>>>>>> > with JDK 8 as it seems like the safest position, but I believe >> > > >>>>>>> > in >> > > >>>>>>> taking >> > > >>>>>>> > bold bets to achieve big wins. Even if we decide to stick with >> > > >>>>>>> > JDK >> > > >>>>>>> 8, I >> > > >>>>>>> > would still be happy since we are moving forward and not >> > > >>>>>>> > dwelling >> > > >>>>>>> on a JDK >> > > >>>>>>> > that is a decade old. Personally, I think focusing on one thing >> > > >>>>>>> that brings >> > > >>>>>>> > more value to us and our users is the idealistic path forward. >> > > >>>>>>> > >> > > >>>>>>> > -Attila >> > > >>>>>>> > >> > > >>>>>>> > >> > > >>>>>>> > >> > > >>>>>>> > >> > > >>>>>>> > >> > > >>>>>>> > >> > > >>>>>>> > >> > > >>>>>>> > >> > > >>>>>>> > >> > > >>>>>>> > On Thu, Jun 1, 2023 at 11:23 AM Stamatis Zampetakis < >> > > >>>>>>> zabe...@gmail.com> >> > > >>>>>>> > wrote: >> > > >>>>>>> > >> > > >>>>>>> >> Hey everyone, >> > > >>>>>>> >> >> > > >>>>>>> >> If we claim that Hive supports a certain JDK then we should >> > > >>>>>>> compile and >> > > >>>>>>> >> run >> > > >>>>>>> >> tests with it. >> > > >>>>>>> >> >> > > >>>>>>> >> The more JDKs we can support the better for everyone but this >> > > >>>>>>> comes at a >> > > >>>>>>> >> cost (resources mostly). We should have a precommit run for >> > > >>>>>>> >> every >> > > >>>>>>> >> supported >> > > >>>>>>> >> JDK (frequency to be determined once per day/week) that >> > > >>>>>>> >> compiles >> > > >>>>>>> and run >> > > >>>>>>> >> all tests. >> > > >>>>>>> >> >> > > >>>>>>> >> From my perspective, I would be pretty happy if we could cover >> > > >>>>>>> the two >> > > >>>>>>> >> edge >> > > >>>>>>> >> LTS releases at every point in time. >> > > >>>>>>> >> >> > > >>>>>>> >> Then we have to decide also which JDK shall we use for the >> > > >>>>>>> >> pull >> > > >>>>>>> requests >> > > >>>>>>> >> and local dev environment. I think it makes sense to use the >> > > >>>>>>> latest. >> > > >>>>>>> >> People >> > > >>>>>>> >> like working on modern stuff and also it makes sense that >> > > >>>>>>> >> newer >> > > >>>>>>> releases >> > > >>>>>>> >> will also use newer versions. It would be pretty awkward if >> > > >>>>>>> someone wants >> > > >>>>>>> >> to use the latest Hive version and it turns out that it can >> > > >>>>>>> >> only >> > > >>>>>>> run on >> > > >>>>>>> >> JDK8. >> > > >>>>>>> >> >> > > >>>>>>> >> Best, >> > > >>>>>>> >> Stamatis >> > > >>>>>>> >> >> > > >>>>>>> >> On Thu, Jun 1, 2023, 3:42 AM Sungwoo Park <glap...@gmail.com> >> > > >>>>>>> wrote: >> > > >>>>>>> >> >> > > >>>>>>> >> > Hi, everyone. >> > > >>>>>>> >> > >> > > >>>>>>> >> > I have not tested the master branch with Java 11/17 yet, >> > > >>>>>>> >> > but I >> > > >>>>>>> would >> > > >>>>>>> >> like >> > > >>>>>>> >> > to share my experience with testing a fork of branch-3.1 >> > > >>>>>>> >> > with >> > > >>>>>>> Java 11/17 >> > > >>>>>>> >> > (as part of developing Hive-MR3), in case that it can be >> > > >>>>>>> >> > useful >> > > >>>>>>> for the >> > > >>>>>>> >> > discussion. I merged the patches listed in [1] HIVE-22415 >> > > >>>>>>> >> > and >> > > >>>>>>> updated >> > > >>>>>>> >> the >> > > >>>>>>> >> > Maven configuration for Java 11. >> > > >>>>>>> >> > >> > > >>>>>>> >> > 1. Building Hive was fine and I was able to run it with >> > > >>>>>>> >> > Java 11 >> > > >>>>>>> as well >> > > >>>>>>> >> as >> > > >>>>>>> >> > Java 17. So, it seems that the work reported in [1] is >> > > >>>>>>> >> > indeed >> > > >>>>>>> complete >> > > >>>>>>> >> for >> > > >>>>>>> >> > upgrading to Java 11 (and Java 17) and getting Hive to work. >> > > >>>>>>> >> > >> > > >>>>>>> >> > 2. However, there was a problem with running tests, so this >> > > >>>>>>> >> > can >> > > >>>>>>> be >> > > >>>>>>> >> > additional work for upgrading to Java 11. >> > > >>>>>>> >> > >> > > >>>>>>> >> > 3. For performance, Java 17 gives about 8 percent of (free) >> > > >>>>>>> performance >> > > >>>>>>> >> > improvement. When tested with 10TB TPC-DS, Java 8 takes 8074 >> > > >>>>>>> seconds, >> > > >>>>>>> >> > whereas Java 17 takes 7415 seconds. Considering the >> > > >>>>>>> >> > maturity of >> > > >>>>>>> Hive, I >> > > >>>>>>> >> > think this is not a small improvement because almost every >> > > >>>>>>> query gets >> > > >>>>>>> >> some >> > > >>>>>>> >> > speedup. >> > > >>>>>>> >> > >> > > >>>>>>> >> > Thanks, >> > > >>>>>>> >> > >> > > >>>>>>> >> > --- Sungwoo >> > > >>>>>>> >> > >> > > >>>>>>> >> > [1] https://issues.apache.org/jira/browse/HIVE-22415 >> > > >>>>>>> >> > >> > > >>>>>>> >> > >> > > >>>>>>> >> > On Thu, Jun 1, 2023 at 3:53 AM Sai Hemanth Gantasala >> > > >>>>>>> >> > <saihema...@cloudera.com.invalid> wrote: >> > > >>>>>>> >> > >> > > >>>>>>> >> > > Hi All, >> > > >>>>>>> >> > > >> > > >>>>>>> >> > > I would strongly advocate keeping support for JDK8. >> > > >>>>>>> >> > > Between JDK11 and JDK17, Depending on the amount of >> > > >>>>>>> >> > > effort on >> > > >>>>>>> the >> > > >>>>>>> >> upgrade >> > > >>>>>>> >> > > I'm inclined towards JDK17 (JDK21 LTS will be released in >> > > >>>>>>> >> > > Sep >> > > >>>>>>> 2023). >> > > >>>>>>> >> > > >> > > >>>>>>> >> > > Thanks, >> > > >>>>>>> >> > > Sai. >> > > >>>>>>> >> > > >> > > >>>>>>> >> > > On Wed, May 31, 2023 at 5:39 AM László Bodor < >> > > >>>>>>> >> bodorlaszlo0...@gmail.com> >> > > >>>>>>> >> > > wrote: >> > > >>>>>>> >> > > >> > > >>>>>>> >> > > > *Hi!* >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > > *Should we support both JDK-11 & JDK-8?* >> > > >>>>>>> >> > > > IMO absolutely yes, let's not break up with JDK-8: >> > > >>>>>>> according to its >> > > >>>>>>> >> > > > lifecycle, it's going to stay with us for a long time. >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > > I believe >> > > >>>>>>> >> > > > a) we should be able to compile on JDK8, JDK11, and >> > > >>>>>>> >> > > > JDK17 >> > > >>>>>>> (github >> > > >>>>>>> >> > actions >> > > >>>>>>> >> > > > can cover this conveniently in precommit time, like tez >> > > >>>>>>> >> > > > < >> > > >>>>>>> >> >> > > >>>>>>> https://github.com/apache/tez/blob/master/.github/workflows/build.yml >> > > >>>>>>> >> > >) >> > > >>>>>>> >> > > > b) the release artifacts should be compatible with JDK8 >> > > >>>>>>> >> > > > as >> > > >>>>>>> long as >> > > >>>>>>> >> it >> > > >>>>>>> >> > is >> > > >>>>>>> >> > > > with us. >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > > Regards, >> > > >>>>>>> >> > > > Laszlo Bodor >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > > Butao Zhang <butaozha...@163.com> ezt írta (időpont: >> > > >>>>>>> >> > > > 2023. >> > > >>>>>>> máj. >> > > >>>>>>> >> 31., >> > > >>>>>>> >> > > Sze, >> > > >>>>>>> >> > > > 14:33): >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > > > Thanks Ayush for driving this! Good to know that Hive >> > > >>>>>>> >> > > > > is >> > > >>>>>>> getting >> > > >>>>>>> >> > ready >> > > >>>>>>> >> > > > for >> > > >>>>>>> >> > > > > newer JDK. >> > > >>>>>>> >> > > > > From my opinon, if we have more community energy to >> > > >>>>>>> >> > > > > put >> > > >>>>>>> into it, >> > > >>>>>>> >> we >> > > >>>>>>> >> > can >> > > >>>>>>> >> > > > > support both JDK-11 and JDK-17 like Spark[1]. If we >> > > >>>>>>> >> > > > > have >> > > >>>>>>> to make >> > > >>>>>>> >> a >> > > >>>>>>> >> > > > choice >> > > >>>>>>> >> > > > > between a JDK-11 and JDK-17, i would like to choose >> > > >>>>>>> >> > > > > the >> > > >>>>>>> relatively >> > > >>>>>>> >> > new >> > > >>>>>>> >> > > > > version JDK-17, meanwhile, we should maintain >> > > >>>>>>> compatibility with >> > > >>>>>>> >> > jdk8, >> > > >>>>>>> >> > > as >> > > >>>>>>> >> > > > > JDK-8 is still widely used in most big data platforms. >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > Thanks, >> > > >>>>>>> >> > > > > Butao Zhang >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > [1]https://issues.apache.org/jira/browse/SPARK-33772 >> > > >>>>>>> >> > > > > ---- Replied Message ---- >> > > >>>>>>> >> > > > > | From | Ayush Saxena<ayush...@gmail.com> | >> > > >>>>>>> >> > > > > | Date | 5/31/2023 18:39 | >> > > >>>>>>> >> > > > > | To | dev<dev@hive.apache.org> | >> > > >>>>>>> >> > > > > | Subject | Move to JDK-11 | >> > > >>>>>>> >> > > > > Hi Everyone, >> > > >>>>>>> >> > > > > Want to pull in the attention of folks towards moving >> > > >>>>>>> >> > > > > to >> > > >>>>>>> JDK-11 >> > > >>>>>>> >> > compile >> > > >>>>>>> >> > > > > time support in Hive. There was a ticket in the past >> > > >>>>>>> >> > > > > [1] >> > > >>>>>>> which >> > > >>>>>>> >> talks >> > > >>>>>>> >> > > > about >> > > >>>>>>> >> > > > > it and If I could decode it right, it was blocked >> > > >>>>>>> >> > > > > because >> > > >>>>>>> the >> > > >>>>>>> >> Hadoop >> > > >>>>>>> >> > > > > version used by Hive didn't had JDK-11 runtime >> > > >>>>>>> >> > > > > support, >> > > >>>>>>> But with >> > > >>>>>>> >> [2] >> > > >>>>>>> >> > in >> > > >>>>>>> >> > > > we >> > > >>>>>>> >> > > > > have upgraded the Hadoop version, so that problem is >> > > >>>>>>> sorted out. I >> > > >>>>>>> >> > > > couldn't >> > > >>>>>>> >> > > > > even see any unresolved tickets in the blocked state >> > > >>>>>>> either. >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > I quickly tried* a mvn clean install -DskipTests >> > > >>>>>>> -Piceberg >> > > >>>>>>> >> -Pitests >> > > >>>>>>> >> > > > > -Dmaven.javadoc.skip=true >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > And no surprises it failed with some weird exceptions >> > > >>>>>>> towards the >> > > >>>>>>> >> > end. >> > > >>>>>>> >> > > > But >> > > >>>>>>> >> > > > > I think that should be solvable. >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > So, Questions? >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > - What do folks think about this? Should we put in >> > > >>>>>>> >> > > > > some >> > > >>>>>>> effort >> > > >>>>>>> >> > towards >> > > >>>>>>> >> > > > > JDK-11 >> > > >>>>>>> >> > > > > - Should we support both JDK-11 & JDK-8? >> > > >>>>>>> >> > > > > - Ditch JDK-11 and directly shoot for JDK-17? >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > Let me know your thoughts, In case anyone has some >> > > >>>>>>> experience in >> > > >>>>>>> >> this >> > > >>>>>>> >> > > > area >> > > >>>>>>> >> > > > > and have tried something in the context, feel free to >> > > >>>>>>> share or >> > > >>>>>>> >> may be >> > > >>>>>>> >> > > if >> > > >>>>>>> >> > > > > someone has any potential action plan or so >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > -Ayush >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > [1] https://issues.apache.org/jira/browse/HIVE-22415 >> > > >>>>>>> >> > > > > [2] https://issues.apache.org/jira/browse/HIVE-24484 >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > > * changed the maven.compiler.source & >> > > >>>>>>> maven.compiler.target to 11 >> > > >>>>>>> >> > > > > >> > > >>>>>>> >> > > > >> > > >>>>>>> >> > > >> > > >>>>>>> >> > >> > > >>>>>>> >> >> > > >>>>>>> > >> > > >>>>>>> >> > > >>>>>> >> > > >> >
