Sounds good to me - that approach has been successful before - i.e. switching the default in the major version, deprecating the old one, and then removing it some number of releases later. Although in some cases we have been slow to remove the code, e.g. the old decimal behaviour is still accessible.
On Wed, Feb 12, 2020 at 4:37 PM Vihang Karajgaonkar <vih...@cloudera.com> wrote: > Yes, I think changing to local catalog can be done in 4.0 branch. Based on > the feedback received here I think we can start removing the code for 3.x > catalog implementation after we fill in some of the gaps in local catalog. > We can selectively decide which ones are (eg. external data sources, HDFS > caching) important and need to be supported in local catalog mode. Having > just one mode of catalog will simplify implementation of newer features > like catalog HA, fine-grained partition level metadata a lot. > > On Wed, Feb 12, 2020 at 3:46 PM Tim Armstrong <tarmstr...@cloudera.com> > wrote: > > > Do we plan to switch the default catalog implementation to the local > > catalog as well? > > > > On Wed, Jan 29, 2020 at 3:41 PM Tim Armstrong <tarmstr...@cloudera.com> > > wrote: > > > > > We haven't generally had clear guarantees about what OS versions we're > > > going to support. I ran into issues with > > > https://issues.apache.org/jira/browse/IMPALA-8508 where we couldn't > run > > > python 3 on centos 6.4, for example. > > > > > > Would it be possible to explicitly state what OS versions we're not > going > > > to support in Impala 4.0? > > > > > > On Mon, Jan 27, 2020 at 12:05 PM Joe McDonnell < > > joemcdonn...@cloudera.com> > > > wrote: > > > > > >> Given the positive feedback, I will begin moving forward on the > proposed > > >> plan. I will start a new thread to discuss the Impala 3.4 release, and > > we > > >> can start collecting/discussing the breaking changes that we want for > > >> Impala 4.0. > > >> > > >> Feedback about the plan is still welcome on this thread. > Alternatively, > > >> any > > >> concerns can be raised in a new thread here on dev@. > > >> > > >> Thanks, > > >> Joe > > >> > > >> On Wed, Jan 22, 2020 at 9:06 AM Laszlo Gaal <laszlo.g...@cloudera.com > > > > >> wrote: > > >> > > >> > +1 > > >> > > > >> > I'd also add that bumping the major version of Impala opens the > window > > >> > for introducing breaking changes. > > >> > > > >> > I don't intend to hijack this mail thread for that purpose, > > >> > but I'd like to suggest compiling such a list in the context of the > > >> version > > >> > bump. > > >> > > > >> > Thanks, > > >> > > > >> > - LaszloG > > >> > > > >> > On Tue, Jan 21, 2020 at 6:48 PM Anurag Mantripragada < > > >> anu...@cloudera.com> > > >> > wrote: > > >> > > > >> > > This makes sense. > > >> > > +1 > > >> > > > > >> > > On Tue, Jan 21, 2020 at 9:03 AM Andrew Sherman < > > asher...@cloudera.com > > >> > > > >> > > wrote: > > >> > > > > >> > > > +1 > > >> > > > > > >> > > > > > >> > > > On Tue, Jan 21, 2020 at 8:28 AM Sahil Takiar < > > >> takiar.sa...@gmail.com> > > >> > > > wrote: > > >> > > > > > >> > > > > +1 makes sense to me. > > >> > > > > > > >> > > > > On Mon, Jan 20, 2020 at 4:55 PM Tim Armstrong < > > >> > tarmstr...@cloudera.com > > >> > > > > > >> > > > > wrote: > > >> > > > > > > >> > > > > > I think this proposal make sense - we've done well in > enabling > > >> > > parallel > > >> > > > > > development for different Hive versions so far, but it is a > > >> burden. > > >> > > > E.g. > > >> > > > > we > > >> > > > > > still don't have precommit tests for Hive 3+ (I like that > > name) > > >> > and I > > >> > > > > don't > > >> > > > > > know that we want to go about making the suite of precommit > > >> tests > > >> > > even > > >> > > > > > larger. > > >> > > > > > > > >> > > > > > On Fri, Jan 17, 2020 at 4:29 PM Joe McDonnell < > > >> > > > joemcdonn...@cloudera.com > > >> > > > > > > > >> > > > > > wrote: > > >> > > > > > > > >> > > > > > > I wanted to start a conversation around moving to develop > > >> against > > >> > > > Hive > > >> > > > > 3+ > > >> > > > > > > by default. (I describe this as Hive 3+ because it is > close > > to > > >> > Hive > > >> > > > > > master, > > >> > > > > > > which is well beyond any released Hive 3.) There has been > > >> > > > considerable > > >> > > > > > > development effort towards implementing features > integrating > > >> > Impala > > >> > > > > with > > >> > > > > > > Hive 3+ and Hive ACID. This is currently developed under > the > > >> > > > > > > USE_CDP_HIVE=true configuration while regular development > > has > > >> > > > continued > > >> > > > > > > with Hive 2. The Hive 3+ development is now stable enough > to > > >> be > > >> > > used > > >> > > > > for > > >> > > > > > > regular development. It would be nice to reduce our test > and > > >> > > > > > compatibility > > >> > > > > > > matrix and have a unified development environment. > > >> > > > > > > > > >> > > > > > > Changing the major version of Hive is a breaking change, > so > > it > > >> > > would > > >> > > > > > > require an Impala 4.x code line. I have a specific > proposal, > > >> but > > >> > > this > > >> > > > > is > > >> > > > > > > mainly a frame for getting the discussion going. > > >> > > > > > > > > >> > > > > > > I propose that we release Impala 3.4.0 and then update > > master > > >> to > > >> > > 4.0 > > >> > > > > and > > >> > > > > > > allow breaking changes until the Impala 4.0 release. The > > main > > >> > > > breaking > > >> > > > > > > change would be to set USE_CDP_HIVE=true, enabling Hive 3+ > > >> > > > development > > >> > > > > by > > >> > > > > > > default. The Hive 2 configuration would be removed over > > time. > > >> > Other > > >> > > > > > > breaking changes can be proposed and voted on. > > >> > > > > > > > > >> > > > > > > If there are developers interested in maintaining a 3.x > > >> branch, > > >> > we > > >> > > > can > > >> > > > > > > create this branch and add appropriate support to any > > >> > > infrastructure > > >> > > > > > (e.g. > > >> > > > > > > bin/push_to_asf.py) to allow that. > > >> > > > > > > > > >> > > > > > > Thoughts? > > >> > > > > > > > > >> > > > > > > Thanks, > > >> > > > > > > > > >> > > > > > > Joe McDonnell > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > > >> > > > > -- > > >> > > > > Sahil Takiar > > >> > > > > Software Engineer > > >> > > > > takiar.sa...@gmail.com | (510) 673-0309 > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > >
