I agree that it would be good to get this done even if it's not perfect, there's so much good stuff.
On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <joemcdonn...@cloudera.com> wrote: > I agree that we should wrap up Impala 4. A large amount of good work has > gone in and it belongs in a release. > > We need to decide which breaking changes are truly blockers for an Impala 4 > release. My feeling is that we can't hold the release for compatibility > breaking changes unless someone signs up to do those changes. Breaking > compatibility is useful sometimes, but it can't be an indefinite hold on > releases. I think a release discussion thread is a good way to stimulate > this discussion. > > Apart from the two JIRAs you listed, one other change mentioned in the > original email is switching to use the new on-demand metadata by default. > > Thanks, > Joe > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <huangquanl...@gmail.com> > wrote: > > > Sure. If there are no objections, I'll raise a discussion thread for the > > 4.0 release. There are still some unresolved breaking changes, e.g. > > > > - IMPALA-2210: Make Parquet the default file format > > - IMPALA-9690: Bump minimum x86-64 CPU requirements > > > > Thanks > > Quanlong > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <jbap...@apache.org> wrote: > > > > > Thanks for bringing this up again, Quanlong! I would love to see a 4.0 > > > release soon so that 4.1 releases and what not can be prepared. Already > > the > > > changlist from the 3.x line is going to be enormous and could be hard > for > > > users to digest. > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang < > huangquanl...@gmail.com> > > > wrote: > > > > > > > Reviving this thread. I think it's time to create the 4.0.0 branch > and > > > > prepare for the release now. Any breaking change is landing and we > > should > > > > wait? > > > > > > > > Thanks, > > > > Quanlong > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong < > tarmstr...@cloudera.com > > > > > > > wrote: > > > > > > > > > Another thing that we're looking into is changing the encoded > runtime > > > > > profile representation to be more efficient - see > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. So that might > be > > a > > > > > default we will try to change in Impala 4.0. > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong < > > > tarmstr...@cloudera.com> > > > > > wrote: > > > > > > > > > > > An addendum to this - we're also considering whether to increase > > the > > > > > > minimum CPU version so that we can simplify development and focus > > > more > > > > on > > > > > > optimising for the most common CPUs - see > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690 > > > > > > > > > > > > Running on commodity hardware is important and we want people to > be > > > > able > > > > > > to develop on whatever machine they have access to. At most we're > > > > likely > > > > > > going to require AVX2 support, which has been supported by the > vast > > > > > > majority of CPUs for the better part of a decade. > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell < > > > > joemcdonn...@cloudera.com > > > > > > > > > > > > wrote: > > > > > > > > > > > >> Impala is starting development on Impala 4.0. There are many > > > exciting > > > > > >> projects underway including full support for Hive ACID tables, > > > > improved > > > > > >> multithreading support, and fault tolerance improvements, but > > Impala > > > > 4.0 > > > > > >> also serves as a time to alter behavior and platform support. > Here > > > is > > > > > the > > > > > >> current list of notable breaking changes that are planned in > > Impala > > > > 4.0: > > > > > >> > > > > > >> Remove support for older operating systems: > > > > > >> > > > > > >> Maintaining support for older operating systems requires a > > > > considerable > > > > > >> amount of effort, especially with changing security > requirements. > > > Each > > > > > >> extra operating system consumes valuable effort and can conflict > > > with > > > > > >> support for newer operating systems, so we have decided to > remove > > > > > support > > > > > >> for some older operating systems. Specifically, we plan to drop > > > > support > > > > > for: > > > > > >> > > > > > >> - > > > > > >> > > > > > >> Centos 6 > > > > > >> - > > > > > >> > > > > > >> Ubuntu 14 > > > > > >> - > > > > > >> > > > > > >> Debian 8 > > > > > >> > > > > > >> Each of these is approaching or past its end of life. Since > > Centos6 > > > > was > > > > > >> the only operating system still needing Python 2.6 and Python > 2.6 > > is > > > > > long > > > > > >> past its end of life, Python 2.6 will no longer be supported. In > > > > > addition, > > > > > >> Centos 7 support will be focused on newer versions such as 7.5 > and > > > > > above. > > > > > >> > > > > > >> Remove support for Sentry: > > > > > >> > > > > > >> Over the past year and a half, Impala's Ranger authorization > > > > > >> functionality has achieved parity and surpassed the existing > > Sentry > > > > > >> authorization functionality. Impala's Sentry support requires > > > ongoing > > > > > >> effort to test and maintain, including maintenance on Sentry > > itself > > > to > > > > > >> address security updates. Given the diminished activity in the > > > Sentry > > > > > >> community, there is no timeline for addressing these security > > > updates. > > > > > >> Unless there is a maintainer for Sentry support, Impala plans to > > > focus > > > > > its > > > > > >> efforts on Ranger for its authorization system and drop Sentry. > > > > > >> > > > > > >> Remove support for Impala-lzo: > > > > > >> > > > > > >> Impala-lzo provides code to allow Impala to read the LZO > > compressed > > > > > >> tables. LZO is GPL licensed, which is why this support is not > > > included > > > > > >> directly. The Impala-lzo code interacts with internal Impala > code > > > at a > > > > > >> level that is error prone and intricate. Given the low adoption > of > > > LZO > > > > > and > > > > > >> the other compression options available, Impala plans to remove > > > > > Impala-lzo > > > > > >> support along with the low level interface it used. > > > > > >> > > > > > >> Deprecations: > > > > > >> > > > > > >> In addition, we also plan to deprecate several existing features > > so > > > > that > > > > > >> they can be removed in a future release. Here is a summary of > the > > > most > > > > > >> notable ones: > > > > > >> > > > > > >> - > > > > > >> > > > > > >> Impala will default to using on demand metadata management > > (i.e. > > > > the > > > > > >> local catalog implementation) as described in this > > documentation: > > > > > >> > > > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html > > > > > >> The old metadata system is now deprecated and may be removed > > in a > > > > > >> future release. > > > > > >> - > > > > > >> > > > > > >> Impala is deprecating the Beeswax client protocol (i.e. > clients > > > > that > > > > > >> connect via beeswax_port) in favor of the HiveServer2 client > > > > > protocol. > > > > > >> - > > > > > >> > > > > > >> Impala is deprecating the old decimal_v2=false behavior from > > > Impala > > > > > >> 2.x. The decimal_v2 query option has defaulted to true since > > > Impala > > > > > 3. > > > > > >> > > > > > >> > > > > > >> Impala is always open to new developers, and we welcome feedback > > on > > > > > these > > > > > >> plans. Further discussion of other changes and deprecations is > > > ongoing > > > > > at > > > > > >> dev@impala.apache.org. > > > > > >> > > > > > >> > > > > > >> Thanks, > > > > > >> > > > > > >> Joe McDonnell > > > > > >> > > > > > >> > > > > > > > > > > > > > > >
