I agree, 4.0 is already in the RC stage and I think it's too late to do such a big version bump for the Hive dependency.
We definitely need to do this upgrade and thanks for working on it! On Mon, Mar 24, 2025 at 1:31 PM Ángel Álvarez Pascua < angel.alvarez.pas...@gmail.com> wrote: > That's great news! If you need anything from me, just ask... > > We should also check and update other non-Hive third-party libraries with > high/critical vulnerabilities, as someone mentioned in another email thread. > > Since this is a major change, I think we should leave it for Spark 4.1. > What do you think? > > El lun, 24 mar 2025 a las 0:59, Mich Talebzadeh (< > mich.talebza...@gmail.com>) escribió: > >> For now, I am testing apache-hive-4.0.1-bin which is the latest release >> version from >> >> https://dlcdn.apache.org/hive/hive-4.0.1/ >> >> apache-hive-4.0.1-bin.tar >> >> My metastore is Oracle and upgrade scripts are provided.. >> >> My previous version is Hive 3.1.1 and the metastore upgrade went OK >> without any major headache. >> >> Now I just need to customise various files under $HIVE_HOME/conf and then >> I will have some testing underway. >> >> HTH >> >> Dr Mich Talebzadeh, >> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR >> >> view my Linkedin profile >> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >> >> >> >> >> >> On Sun, 23 Mar 2025 at 17:13, Ángel Álvarez Pascua < >> angel.alvarez.pas...@gmail.com> wrote: >> >>> Well ... and then? When are we going to tackle this? I could help. >>> >>> El mié, 12 mar 2025, 15:50, Mich Talebzadeh <mich.talebza...@gmail.com> >>> escribió: >>> >>>> Agreed. Hive upgrade is more time consuming as it involves backing up >>>> Hive schema on your metastore and then running Hive provided upgrade schema >>>> scripts against Hive schema that could be problematic,but needs to be done >>>> one way or another. >>>> >>>> HTH >>>> >>>> Dr Mich Talebzadeh, >>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR >>>> >>>> view my Linkedin profile >>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>> >>>> >>>> >>>> >>>> >>>> On Wed, 12 Mar 2025 at 12:21, Ángel <angel.alvarez.pas...@gmail.com> >>>> wrote: >>>> >>>>> Not an easy task, I guess, but I'm totally for it too. >>>>> >>>>> The issue SPARK-49910 >>>>> <https://issues.apache.org/jira/browse/SPARK-49910> is related to >>>>> this. >>>>> >>>>> El mar, 11 mar 2025 a las 23:06, Mich Talebzadeh (< >>>>> mich.talebza...@gmail.com>) escribió: >>>>> >>>>>> Yes I am all for it, as I use Hive with Oracle as its metastore >>>>>> extensively. >>>>>> >>>>>> Case in point, on 6th March A Hive user >>>>>> <https://lists.apache.org/thread/vhgxt1cj2ppc862j0lwxl63j6nfc7khh> >>>>>> alluded to it and I quote >>>>>> >>>>>> "I just wanted to highlight that Hive 3.x line is EOL. It has various >>>>>> known security vulnerabilities, many serious bugs (including wrong >>>>>> results >>>>>> and data corruption), and lacks lots of improvements and major features >>>>>> that are available in Hive 4. Upgrading is the right path forward." >>>>>> >>>>>> In summary, Hive 4.x likely includes performance improvements, new >>>>>> features, and bug fixes. Compiling against it would allow Spark to take >>>>>> advantage of these. Plus using the latest versions of both Spark and Hive >>>>>> is important for maintaining a secure data platform. >>>>>> >>>>>> HTH >>>>>> >>>>>> Dr Mich Talebzadeh, >>>>>> Architect | Data Science | Financial Crime | Forensic Analysis | GDPR >>>>>> >>>>>> view my Linkedin profile >>>>>> <https://www.linkedin.com/in/mich-talebzadeh-ph-d-5205b2/> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Tue, 11 Mar 2025 at 19:08, Rozov, Vlad <vro...@amazon.com.invalid> >>>>>> wrote: >>>>>> >>>>>>> Hi All, >>>>>>> >>>>>>> As Apache Hive announced EOL for Hive 2.x [1] and 3.x [2], should >>>>>>> Spark be compiled against Hive 4.x and use it as default? >>>>>>> >>>>>>> Thank you, >>>>>>> >>>>>>> Vlad >>>>>>> >>>>>>> [1] https://lists.apache.org/thread/4ctrzfw60jkhc0hq2xoh1jpqxgt2zd93 >>>>>>> [2] https://lists.apache.org/thread/99h6wr7nk4684r6tkcbm8ydfytgqy6f3 >>>>>>> [3] https://github.com/apache/spark/pull/50213 >>>>>>> >>>>>>
