It's kinda like Scala version upgrade. Historically, we only remove the support of an older Scala version when the newer version is proven to be stable after one or more Spark minor versions.
On Tue, Nov 19, 2019 at 2:07 PM Cheng Lian <lian.cs....@gmail.com> wrote: > Hmm, what exactly did you mean by "remove the usage of forked `hive` in > Apache Spark 3.0 completely officially"? I thought you wanted to remove the > forked Hive 1.2 dependencies completely, no? As long as we still keep the > Hive 1.2 in Spark 3.0, I'm fine with that. I personally don't have a > particular preference between using Hive 1.2 or 2.3 as the default Hive > version. After all, for end-users and providers who need a particular > version combination, they can always build Spark with proper profiles > themselves. > > And thanks for clarifying the Hive 2.3.5 issue. I didn't notice that it's > due to the folder name. > > On Tue, Nov 19, 2019 at 11:15 AM Dongjoon Hyun <dongjoon.h...@gmail.com> > wrote: > >> BTW, `hive.version.short` is a directory name. We are using 2.3.6 only. >> >> For directory name, we use '1.2.1' and '2.3.5' because we just delayed >> the renaming the directories until 3.0.0 deadline to minimize the diff. >> >> We can replace it immediately if we want right now. >> >> >> >> On Tue, Nov 19, 2019 at 11:11 AM Dongjoon Hyun <dongjoon.h...@gmail.com> >> wrote: >> >>> Hi, Cheng. >>> >>> This is irrelevant to JDK11 and Hadoop 3. I'm talking about JDK8 world. >>> If we consider them, it could be the followings. >>> >>> +----------+-----------------+--------------------+ >>> | | Hive 1.2.1 fork | Apache Hive 2.3.6 | >>> +-------------------------------------------------+ >>> |Legitimate| X | O | >>> |JDK11 | X | O | >>> |Hadoop3 | X | O | >>> |Hadoop2 | O | O | >>> |Functions | Baseline | More | >>> |Bug fixes | Baseline | More | >>> +-------------------------------------------------+ >>> >>> To stabilize Spark's Hive 2.3 usage, we should use it by ourselves >>> (including Jenkins/GitHubAction/AppVeyor). >>> >>> For me, AS-IS 3.0 is not enough for that. According to your advices, >>> to give more visibility to the whole community, >>> >>> 1. We need to give additional `hadoop-2.7 with Hive 2.3` pre-built >>> distribution >>> 2. We need to switch our default Hive usage to 2.3 in `master` for 3.1 >>> after `branch-3.0` branch cut. >>> >>> I know that we have been reluctant to (1) and (2) due to its burden. >>> But, it's time to prepare. Without them, we are going to be insufficient >>> again and again. >>> >>> Bests, >>> Dongjoon. >>> >>> >>> >>> >>> On Tue, Nov 19, 2019 at 9:26 AM Cheng Lian <lian.cs....@gmail.com> >>> wrote: >>> >>>> Dongjoon, I'm with Hyukjin. There should be at least one Spark 3.x >>>> minor release to stabilize Hive 2.3 code paths before retiring the Hive 1.2 >>>> fork. Even today, the Hive 2.3.6 version bundled in Spark 3.0 is still >>>> buggy in terms of JDK 11 support. (BTW, I just found that our root POM is >>>> referring both Hive 2.3.6 and 2.3.5 at the moment, see here >>>> <https://github.com/apache/spark/blob/6fb8b8606544f26dc2d9719a2d009eb5aea65ba2/pom.xml#L135> >>>> and here >>>> <https://github.com/apache/spark/blob/6fb8b8606544f26dc2d9719a2d009eb5aea65ba2/pom.xml#L2927> >>>> .) >>>> >>>> Again, I'm happy to get rid of ancient legacy dependencies like Hadoop >>>> 2.7 and the Hive 1.2 fork, but I do believe that we need a safety net for >>>> Spark 3.0. For preview releases, I'm afraid that their visibility is not >>>> good enough for covering such major upgrades. >>>> >>>> On Tue, Nov 19, 2019 at 8:39 AM Dongjoon Hyun <dongjoon.h...@gmail.com> >>>> wrote: >>>> >>>>> Thank you for feedback, Hyujkjin and Sean. >>>>> >>>>> I proposed `preview-2` for that purpose but I'm also +1 for do that at >>>>> 3.1 >>>>> if we can make a decision to eliminate the illegitimate Hive fork >>>>> reference >>>>> immediately after `branch-3.0` cut. >>>>> >>>>> Sean, I'm referencing Cheng Lian's email for the status of >>>>> `hadoop-2.7`. >>>>> >>>>> - >>>>> https://lists.apache.org/thread.html/623dd9a6d4e951daeec985feffede12c7b419e03c2965018de7a72f1@%3Cdev.spark.apache.org%3E >>>>> >>>>> The way I see this is that it's not a user problem. Apache Spark >>>>> community didn't try to drop the illegitimate Hive fork yet. >>>>> We need to drop it by ourselves because we created it and it's our bad. >>>>> >>>>> Bests, >>>>> Dongjoon. >>>>> >>>>> >>>>> >>>>> On Tue, Nov 19, 2019 at 5:06 AM Sean Owen <sro...@gmail.com> wrote: >>>>> >>>>>> Just to clarify, as even I have lost the details over time: hadoop-2.7 >>>>>> works with hive-2.3? it isn't tied to hadoop-3.2? >>>>>> Roughly how much risk is there in using the Hive 1.x fork over Hive >>>>>> 2.x, for end users using Hive via Spark? >>>>>> I don't have a strong opinion, other than sharing the view that we >>>>>> have to dump the Hive 1.x fork at the first opportunity. >>>>>> Question is simply how much risk that entails. Keeping in mind that >>>>>> Spark 3.0 is already something that people understand works >>>>>> differently. We can accept some behavior changes. >>>>>> >>>>>> On Mon, Nov 18, 2019 at 11:11 PM Dongjoon Hyun < >>>>>> dongjoon.h...@gmail.com> wrote: >>>>>> > >>>>>> > Hi, All. >>>>>> > >>>>>> > First of all, I want to put this as a policy issue instead of a >>>>>> technical issue. >>>>>> > Also, this is orthogonal from `hadoop` version discussion. >>>>>> > >>>>>> > Apache Spark community kept (not maintained) the forked Apache Hive >>>>>> > 1.2.1 because there has been no other options before. As we see at >>>>>> > SPARK-20202, it's not a desirable situation among the Apache >>>>>> projects. >>>>>> > >>>>>> > https://issues.apache.org/jira/browse/SPARK-20202 >>>>>> > >>>>>> > Also, please note that we `kept`, not `maintained`, because we know >>>>>> it's not good. >>>>>> > There are several attempt to update that forked repository >>>>>> > for several reasons (Hadoop 3 support is one of the example), >>>>>> > but those attempts are also turned down. >>>>>> > >>>>>> > From Apache Spark 3.0, it seems that we have a new feasible option >>>>>> > `hive-2.3` profile. What about moving forward in this direction >>>>>> further? >>>>>> > >>>>>> > For example, can we remove the usage of forked `hive` in Apache >>>>>> Spark 3.0 >>>>>> > completely officially? If someone still needs to use the forked >>>>>> `hive`, we can >>>>>> > have a profile `hive-1.2`. Of course, it should not be a default >>>>>> profile in the community. >>>>>> > >>>>>> > I want to say this is a goal we should achieve someday. >>>>>> > If we don't do anything, nothing happen. At least we need to >>>>>> prepare this. >>>>>> > Without any preparation, Spark 3.1+ will be the same. >>>>>> > >>>>>> > Shall we focus on what are our problems with Hive 2.3.6? >>>>>> > If the only reason is that we didn't use it before, we can release >>>>>> another >>>>>> > `3.0.0-preview` for that. >>>>>> > >>>>>> > Bests, >>>>>> > Dongjoon. >>>>>> >>>>>
