Hey Jim,
My fix for IMPALA-10482 and IMPALA-10493 (the two issues are fixed in a
single commit) just got merged after the 4.0 branching:
https://gitbox.apache.org/repos/asf?p=impala.git;a=commit;h=f0f083e45e2c77b1499fa6fa08ff8d9dc4a2785f
Can we add this patch to the 4.0 release?
Thanks,
Zoltan
On Fri, May 7, 2021 at 5:36 AM Joe McDonnell <joemcdonn...@cloudera.com>
wrote:
> I agree that it is a good idea for us to get going on the 4.0 release.
>
> I'm working on a change to fix IMPALA-9690. I will post an updated review
> once a test run completes. If everything is looking good, I would like to
> get this in the 4.0 release.
>
> There is also one documentation fix that I think belongs in the Impala 4.0
> release:
> https://issues.apache.org/jira/browse/IMPALA-9770 - Remove Sentry
> references
>
> Thanks,
> Joe
>
> On Thu, May 6, 2021 at 7:55 PM Jim Apple <apa...@jbapple.com> wrote:
>
> > Sure thing! Let us know if you find owners for anything that someone
> wants
> > to champion as a 4.0 blocker.
> >
> > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang <huangquanl...@gmail.com>
> > wrote:
> >
> > > Thank Jim! I planned to go through all the JIRAs target at 4.0:
> > >
> > >
> >
> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22
> > >
> > > But due to some other urgent works and the recent holiday, my progress
> is
> > > slow. Anyway, I think it's time to branch out. We've been waiting too
> > long.
> > > Thanks for creating the branch.
> > >
> > > Regards,
> > > Quanlong
> > >
> > > On Fri, May 7, 2021 at 3:29 AM Jim Apple <apa...@jbapple.com> wrote:
> > >
> > > > I've created a 4.0 branch (with all changes through the end of April)
> > and
> > > > started testing a release candidate. Everyone who wants to vote,
> > > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> is a
> > > > guide to both releasing and voting (at the bottom). I'll let you all
> > know
> > > > when I have something to vote on.
> > > >
> > > > Thanks!
> > > > Jim
> > > >
> > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer <
> > > csringho...@cloudera.com
> > > > >
> > > > wrote:
> > > >
> > > > > About IMPALA-9690 (AVX support):
> > > > > My preferred solution would be to deprecate support for x64 without
> > > AVX2
> > > > in
> > > > > 4.0, but not start removing the related logic yet.
> > > > > - We could even add a DCHECK + flag to crash by default if no AVX2
> is
> > > > > detected, and a message that points them to Impala mailing
> > > lists/related
> > > > > Jira.
> > > > > - After 4.0 is released people would be free to implement things
> that
> > > > > assume AVX2 support on x64, for example compiling Impala/the whole
> > > > > toolchain with -mavx2. I think that currently people are held back
> > from
> > > > > doing this - why put effort in doing+benchmarking a change if you
> > can't
> > > > > push it to master.
> > > > > - If a user appears with a valid use case that needs x64 without
> AVX2
> > > > then
> > > > > we can revise our plans, but we shouldn't ignore low hanging fruits
> > for
> > > > the
> > > > > sake of imaginary users :)
> > > > >
> > > > > -Csaba
> > > > >
> > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian <
> > > > sh...@superdupershant.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > Regarding IMPALA-9690 if someone has the time to clean up the old
> > > > > > conditional cpu logic it would be great, but if the ARM folks
> still
> > > > > > need some of that logic it's not critical enough to block, just a
> > > very
> > > > > > nice to have?
> > > > > >
> > > > > > -Shant
> > > > > >
> > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong <
> > > > tim.g.armstr...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > I agree that it would be good to get this done even if it's not
> > > > > perfect,
> > > > > > > there's so much good stuff.
> > > > > > >
> > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell <
> > > > joemcdonn...@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I agree that we should wrap up Impala 4. A large amount of
> good
> > > > work
> > > > > > has
> > > > > > > > gone in and it belongs in a release.
> > > > > > > >
> > > > > > > > We need to decide which breaking changes are truly blockers
> for
> > > an
> > > > > > Impala 4
> > > > > > > > release. My feeling is that we can't hold the release for
> > > > > compatibility
> > > > > > > > breaking changes unless someone signs up to do those changes.
> > > > > Breaking
> > > > > > > > compatibility is useful sometimes, but it can't be an
> > indefinite
> > > > hold
> > > > > > on
> > > > > > > > releases. I think a release discussion thread is a good way
> to
> > > > > > stimulate
> > > > > > > > this discussion.
> > > > > > > >
> > > > > > > > Apart from the two JIRAs you listed, one other change
> mentioned
> > > in
> > > > > the
> > > > > > > > original email is switching to use the new on-demand metadata
> > by
> > > > > > default.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Joe
> > > > > > > >
> > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong Huang <
> > > > > huangquanl...@gmail.com
> > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Sure. If there are no objections, I'll raise a discussion
> > > thread
> > > > > for
> > > > > > the
> > > > > > > > > 4.0 release. There are still some unresolved breaking
> > changes,
> > > > e.g.
> > > > > > > > >
> > > > > > > > > - IMPALA-2210: Make Parquet the default file format
> > > > > > > > > - IMPALA-9690: Bump minimum x86-64 CPU requirements
> > > > > > > > >
> > > > > > > > > Thanks
> > > > > > > > > Quanlong
> > > > > > > > >
> > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim Apple <
> > jbap...@apache.org
> > > >
> > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Thanks for bringing this up again, Quanlong! I would love
> > to
> > > > see
> > > > > a
> > > > > > 4.0
> > > > > > > > > > release soon so that 4.1 releases and what not can be
> > > prepared.
> > > > > > Already
> > > > > > > > > the
> > > > > > > > > > changlist from the 3.x line is going to be enormous and
> > could
> > > > be
> > > > > > hard
> > > > > > > > for
> > > > > > > > > > users to digest.
> > > > > > > > > >
> > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM Quanlong Huang <
> > > > > > > > huangquanl...@gmail.com>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Reviving this thread. I think it's time to create the
> > 4.0.0
> > > > > > branch
> > > > > > > > and
> > > > > > > > > > > prepare for the release now. Any breaking change is
> > landing
> > > > and
> > > > > > we
> > > > > > > > > should
> > > > > > > > > > > wait?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Quanlong
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim Armstrong <
> > > > > > > > tarmstr...@cloudera.com
> > > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Another thing that we're looking into is changing the
> > > > encoded
> > > > > > > > runtime
> > > > > > > > > > > > profile representation to be more efficient - see
> > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378.
> So
> > > that
> > > > > > might
> > > > > > > > be
> > > > > > > > > a
> > > > > > > > > > > > default we will try to change in Impala 4.0.
> > > > > > > > > > > >
> > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM Tim Armstrong <
> > > > > > > > > > tarmstr...@cloudera.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > An addendum to this - we're also considering
> whether
> > to
> > > > > > increase
> > > > > > > > > the
> > > > > > > > > > > > > minimum CPU version so that we can simplify
> > development
> > > > and
> > > > > > focus
> > > > > > > > > > more
> > > > > > > > > > > on
> > > > > > > > > > > > > optimising for the most common CPUs - see
> > > > > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690
> > > > > > > > > > > > >
> > > > > > > > > > > > > Running on commodity hardware is important and we
> > want
> > > > > > people to
> > > > > > > > be
> > > > > > > > > > > able
> > > > > > > > > > > > > to develop on whatever machine they have access to.
> > At
> > > > most
> > > > > > we're
> > > > > > > > > > > likely
> > > > > > > > > > > > > going to require AVX2 support, which has been
> > supported
> > > > by
> > > > > > the
> > > > > > > > vast
> > > > > > > > > > > > > majority of CPUs for the better part of a decade.
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM Joe McDonnell <
> > > > > > > > > > > joemcdonn...@cloudera.com
> > > > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Impala is starting development on Impala 4.0.
> There
> > > are
> > > > > many
> > > > > > > > > > exciting
> > > > > > > > > > > > >> projects underway including full support for Hive
> > ACID
> > > > > > tables,
> > > > > > > > > > > improved
> > > > > > > > > > > > >> multithreading support, and fault tolerance
> > > > improvements,
> > > > > > but
> > > > > > > > > Impala
> > > > > > > > > > > 4.0
> > > > > > > > > > > > >> also serves as a time to alter behavior and
> platform
> > > > > > support.
> > > > > > > > Here
> > > > > > > > > > is
> > > > > > > > > > > > the
> > > > > > > > > > > > >> current list of notable breaking changes that are
> > > > planned
> > > > > in
> > > > > > > > > Impala
> > > > > > > > > > > 4.0:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Remove support for older operating systems:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Maintaining support for older operating systems
> > > > requires a
> > > > > > > > > > > considerable
> > > > > > > > > > > > >> amount of effort, especially with changing
> security
> > > > > > > > requirements.
> > > > > > > > > > Each
> > > > > > > > > > > > >> extra operating system consumes valuable effort
> and
> > > can
> > > > > > conflict
> > > > > > > > > > with
> > > > > > > > > > > > >> support for newer operating systems, so we have
> > > decided
> > > > to
> > > > > > > > remove
> > > > > > > > > > > > support
> > > > > > > > > > > > >> for some older operating systems. Specifically, we
> > > plan
> > > > to
> > > > > > drop
> > > > > > > > > > > support
> > > > > > > > > > > > for:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Centos 6
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Ubuntu 14
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Debian 8
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Each of these is approaching or past its end of
> > life.
> > > > > Since
> > > > > > > > > Centos6
> > > > > > > > > > > was
> > > > > > > > > > > > >> the only operating system still needing Python 2.6
> > and
> > > > > > Python
> > > > > > > > 2.6
> > > > > > > > > is
> > > > > > > > > > > > long
> > > > > > > > > > > > >> past its end of life, Python 2.6 will no longer be
> > > > > > supported. In
> > > > > > > > > > > > addition,
> > > > > > > > > > > > >> Centos 7 support will be focused on newer versions
> > > such
> > > > as
> > > > > > 7.5
> > > > > > > > and
> > > > > > > > > > > > above.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Remove support for Sentry:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Over the past year and a half, Impala's Ranger
> > > > > authorization
> > > > > > > > > > > > >> functionality has achieved parity and surpassed
> the
> > > > > existing
> > > > > > > > > Sentry
> > > > > > > > > > > > >> authorization functionality. Impala's Sentry
> support
> > > > > > requires
> > > > > > > > > > ongoing
> > > > > > > > > > > > >> effort to test and maintain, including maintenance
> > on
> > > > > Sentry
> > > > > > > > > itself
> > > > > > > > > > to
> > > > > > > > > > > > >> address security updates. Given the diminished
> > > activity
> > > > in
> > > > > > the
> > > > > > > > > > Sentry
> > > > > > > > > > > > >> community, there is no timeline for addressing
> these
> > > > > > security
> > > > > > > > > > updates.
> > > > > > > > > > > > >> Unless there is a maintainer for Sentry support,
> > > Impala
> > > > > > plans to
> > > > > > > > > > focus
> > > > > > > > > > > > its
> > > > > > > > > > > > >> efforts on Ranger for its authorization system and
> > > drop
> > > > > > Sentry.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Remove support for Impala-lzo:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala-lzo provides code to allow Impala to read
> the
> > > LZO
> > > > > > > > > compressed
> > > > > > > > > > > > >> tables. LZO is GPL licensed, which is why this
> > support
> > > > is
> > > > > > not
> > > > > > > > > > included
> > > > > > > > > > > > >> directly. The Impala-lzo code interacts with
> > internal
> > > > > Impala
> > > > > > > > code
> > > > > > > > > > at a
> > > > > > > > > > > > >> level that is error prone and intricate. Given the
> > low
> > > > > > adoption
> > > > > > > > of
> > > > > > > > > > LZO
> > > > > > > > > > > > and
> > > > > > > > > > > > >> the other compression options available, Impala
> > plans
> > > to
> > > > > > remove
> > > > > > > > > > > > Impala-lzo
> > > > > > > > > > > > >> support along with the low level interface it
> used.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Deprecations:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> In addition, we also plan to deprecate several
> > > existing
> > > > > > features
> > > > > > > > > so
> > > > > > > > > > > that
> > > > > > > > > > > > >> they can be removed in a future release. Here is a
> > > > summary
> > > > > > of
> > > > > > > > the
> > > > > > > > > > most
> > > > > > > > > > > > >> notable ones:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala will default to using on demand metadata
> > > > > > management
> > > > > > > > > (i.e.
> > > > > > > > > > > the
> > > > > > > > > > > > >> local catalog implementation) as described in
> > this
> > > > > > > > > documentation:
> > > > > > > > > > > > >>
> > > > > > > > > > >
> > > > > >
> > http://impala.apache.org/docs/build/html/topics/impala_metadata.html
> > > > > > > > > > > > >> The old metadata system is now deprecated and
> may
> > > be
> > > > > > removed
> > > > > > > > > in a
> > > > > > > > > > > > >> future release.
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala is deprecating the Beeswax client
> protocol
> > > > (i.e.
> > > > > > > > clients
> > > > > > > > > > > that
> > > > > > > > > > > > >> connect via beeswax_port) in favor of the
> > > HiveServer2
> > > > > > client
> > > > > > > > > > > > protocol.
> > > > > > > > > > > > >> -
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala is deprecating the old decimal_v2=false
> > > > behavior
> > > > > > from
> > > > > > > > > > Impala
> > > > > > > > > > > > >> 2.x. The decimal_v2 query option has defaulted
> to
> > > > true
> > > > > > since
> > > > > > > > > > Impala
> > > > > > > > > > > > 3.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Impala is always open to new developers, and we
> > > welcome
> > > > > > feedback
> > > > > > > > > on
> > > > > > > > > > > > these
> > > > > > > > > > > > >> plans. Further discussion of other changes and
> > > > > deprecations
> > > > > > is
> > > > > > > > > > ongoing
> > > > > > > > > > > > at
> > > > > > > > > > > > >> dev@impala.apache.org.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Thanks,
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Joe McDonnell
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
