There are some minor conflicts in cherry-picking IMPALA-7501 due to missing IMPALA-10613 (Standup HMS thrift server in Catalog). Other commits can be picked cleanly. Now I'm testing my private 4.0.0 branch with these commits:
- IMPALA-10683: Skip test 'test_double_precision' for non-HDFS test env - IMPALA-9690: Set minimum CPU requirement to AVX2 for x86_64 - IMPALA-10413: fix impalad crashes when canceling the retrying query - IMPALA-5121: Fix AVG() on timestamp col with use_local_tz_for_unix_timestamp_conversions - IMPALA-10704: Fix retried query id not being unregistered when retry fails - IMPALA-9770: [DOCS] Remove Sentry references in documentation - IMPALA-7501: Slim down partition metadata in LocalCatalog mode Jobs are: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/14039/ https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/4296/ Once they pass, I'll push them to branch-4.0.0 and continue the release process. Let me know if you have any concerns. Thanks, Quanlong On Sat, Jun 5, 2021 at 9:41 AM Quanlong Huang <huangquanl...@gmail.com> wrote: > Thank Zoltan for listing these commits! > > IMPALA-10413 can be resolved. I marked the fix version as 4.0 since we are > going to pick it. Also did the same for IMPALA-10704. > > Besides that, I think we should also pick the doc fix about removing > Sentry related stuffs: > > - 1a84a14 IMPALA-9770: [DOCS] Remove Sentry references in documentation > > IMPALA-7501 was recently resolved. I'd be good if we pick it as well, > because it fixes the scalability issue of local-catalog mode on large > tables (avoiding OOM and reducing memory footprints). > > - bb30621 IMPALA-7501: Slim down partition metadata in LocalCatalog > mode > > Thanks, > Quanlong > > On Fri, Jun 4, 2021 at 5:47 PM Zoltán Borók-Nagy <borokna...@apache.org> > wrote: > >> I didn't have any specifics in mind when I wrote there are good commits on >> master since May 1. >> My principle was only that it would be nice to release what's latest, >> since >> some time has passed since 4.0 was branched. >> >> Now that I looked at the commit list, maybe we could consider adding the >> followings at least: >> >> - IMPALA-10683: Skip test 'test_double_precision' for non-HDFS test env >> - IMPALA-9690: Set minimum CPU requirement to AVX2 for x86_64 >> - IMPALA-10413: fix impalad crashes when canceling the retrying query >> - IMPALA-5121: Fix AVG() on timestamp col with >> use_local_tz_for_unix_timestamp_conversions >> - IMPALA-10704: Fix retried query id not being unregistered when retry >> fails >> >> IMPALA-9690 <https://issues.apache.org/jira/browse/IMPALA-9690> and >> IMPALA-5121 <https://issues.apache.org/jira/browse/IMPALA-5121> are even >> targeted for 4.0 in Jira. >> IMPALA-10413 <https://issues.apache.org/jira/browse/IMPALA-10413> is >> targeted for 4.0, but the Jira is not resolved yet. A patch has been >> already merged, so I asked at the Jira ticket whether we can resolve it. >> IMPALA-10704 <https://issues.apache.org/jira/browse/IMPALA-10704> is >> targeted for 4.1, but fixes an issue introduced by IMPALA-10413. >> >> Thanks, >> Zoltan >> >> >> On Fri, Jun 4, 2021 at 5:06 AM Jim Apple <jbap...@apache.org> wrote: >> >> > Once we know the answer to that last question, I'd be interested in >> reading >> > more about your vision for what should go in 4.0, Quanlong -- >> > i.e."everything committed on June 2 except patches X and Y". >> > >> > On Thu, Jun 3, 2021 at 2:10 AM Quanlong Huang <huangquanl...@gmail.com> >> > wrote: >> > >> > > Sorry to be late here! I'm -1 for merging HEAD into 4.0. Because it >> will >> > > bump the CDP_BUILD_NUMBER again which increases the difficulty for 3.x >> > > users (who may still be using Hive 2.x or 3.x) to adopt Impala-4.0. >> There >> > > are two new commits in HEAD that bump the CDP_BUILD_NUMBER. >> > > >> > > We may have already depended on some Hive 4.0 APIs. But those features >> > may >> > > not be needed by 3.x users. They can revert some changes to make >> > Impala-4.0 >> > > compatible with older versions of Hive. >> > > The lower GBN we have, the easier it is for them to upgrade. >> > > >> > > Does those good commits of HEAD depend on bumping the GBN? If not, we >> can >> > > just pick them to the 4.0 branch. >> > > >> > > Thanks, >> > > Quanlong >> > > >> > > On Wed, May 26, 2021 at 8:52 PM Zoltán Borók-Nagy < >> borokna...@apache.org >> > > >> > > wrote: >> > > >> > > > There's been a handful of good commits since May 1, so +1 for >> merging >> > > HEAD >> > > > into 4.0. >> > > > >> > > > Cheers, >> > > > Zoltan >> > > > >> > > > On Wed, May 26, 2021 at 2:11 PM Jim Apple <jbap...@apache.org> >> wrote: >> > > > >> > > > > We're only waiting on one patch for the docs, but since I cut the >> 4.0 >> > > > > branch on May 1, a number of things have landed. Maybe it would >> be >> > > best >> > > > to >> > > > > merge HEAD into 4.0.0 once the Sentry doc change lands, thereby >> > > including >> > > > > everything we've talked about cherrypicking -- including 9690. >> > > > > >> > > > > Thoughts? >> > > > > >> > > > > On Wed, May 26, 2021 at 12:40 AM Quanlong Huang < >> > > huangquanl...@gmail.com >> > > > > >> > > > > wrote: >> > > > > >> > > > > > Hi all, >> > > > > > >> > > > > > I see IMPALA-9690 is done but still missing in branch-4.0.0 and >> I >> > > think >> > > > > we >> > > > > > should pick it >> > > > > > >> > > > > > - c10e7c9 IMPALA-9690: Set minimum CPU requirement to AVX2 >> for >> > > > x86_64 >> > > > > > >> > > > > > I'd be good to also carry on these fixes for query retry: >> > > > > > >> > > > > > - 42684b4 IMPALA-10413: fix impalad crashes when canceling >> the >> > > > > retrying >> > > > > > query >> > > > > > - d111443 IMPALA-10704: Fix retried query id not being >> > > unregistered >> > > > > when >> > > > > > retry fails >> > > > > > >> > > > > > If you are ok, I can push them to branch-4.0.0. >> > > > > > >> > > > > > Thanks, >> > > > > > Quanlong >> > > > > > >> > > > > > On Sun, May 16, 2021 at 6:02 AM Jim Apple <jbap...@apache.org> >> > > wrote: >> > > > > > >> > > > > > > Sounds good! >> > > > > > > >> > > > > > > On Sat, May 15, 2021 at 1:29 PM Joe McDonnell < >> > > > > joemcdonn...@cloudera.com >> > > > > > > >> > > > > > > wrote: >> > > > > > > >> > > > > > > > Hi Jim, >> > > > > > > > >> > > > > > > > I talked with the developer for the Sentry docs bug this >> past >> > > week, >> > > > > and >> > > > > > > we >> > > > > > > > went through the list of changes needed for that. The >> developer >> > > is >> > > > > > hoping >> > > > > > > > to work on it next week. >> > > > > > > > >> > > > > > > > If there is any further delay, I'm open to options of >> working >> > > > around >> > > > > > it. >> > > > > > > > >> > > > > > > > Thanks, >> > > > > > > > Joe >> > > > > > > > >> > > > > > > > On Sat, May 15, 2021 at 12:43 PM Jim Apple < >> jbap...@apache.org >> > > >> > > > > wrote: >> > > > > > > > >> > > > > > > > > This Sentry one I'm concerned about. The ticket has been >> open >> > > > for a >> > > > > > > year. >> > > > > > > > > We could do a 4.0 release without the docs, or we could >> maybe >> > > > put a >> > > > > > > note >> > > > > > > > on >> > > > > > > > > the docs that Sentry support is no longer available. >> > > > > > > > > >> > > > > > > > > Any other ideas on how to tackle this while the ticket >> waits >> > > for >> > > > a >> > > > > > > patch? >> > > > > > > > > >> > > > > > > > > On Thu, May 6, 2021 at 8:36 PM Joe McDonnell < >> > > > > > > joemcdonn...@cloudera.com> >> > > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > I agree that it is a good idea for us to get going on >> the >> > 4.0 >> > > > > > > release. >> > > > > > > > > > >> > > > > > > > > > I'm working on a change to fix IMPALA-9690. I will post >> an >> > > > > updated >> > > > > > > > review >> > > > > > > > > > once a test run completes. If everything is looking >> good, I >> > > > would >> > > > > > > like >> > > > > > > > to >> > > > > > > > > > get this in the 4.0 release. >> > > > > > > > > > >> > > > > > > > > > There is also one documentation fix that I think >> belongs in >> > > the >> > > > > > > Impala >> > > > > > > > > 4.0 >> > > > > > > > > > release: >> > > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9770 - >> Remove >> > > > > Sentry >> > > > > > > > > > references >> > > > > > > > > > >> > > > > > > > > > Thanks, >> > > > > > > > > > Joe >> > > > > > > > > > >> > > > > > > > > > On Thu, May 6, 2021 at 7:55 PM Jim Apple < >> > apa...@jbapple.com >> > > > >> > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > Sure thing! Let us know if you find owners for >> anything >> > > that >> > > > > > > someone >> > > > > > > > > > wants >> > > > > > > > > > > to champion as a 4.0 blocker. >> > > > > > > > > > > >> > > > > > > > > > > On Thu, May 6, 2021 at 7:08 PM Quanlong Huang < >> > > > > > > > huangquanl...@gmail.com >> > > > > > > > > > >> > > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > Thank Jim! I planned to go through all the JIRAs >> target >> > > at >> > > > > 4.0: >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://issues.apache.org/jira/browse/IMPALA-10690?jql=project%20%3D%20IMPALA%20AND%20resolution%20%3D%20Unresolved%20AND%20%22Target%20Version%22%20%3D%20%22Impala%204.0%22 >> > > > > > > > > > > > >> > > > > > > > > > > > But due to some other urgent works and the recent >> > > holiday, >> > > > my >> > > > > > > > > progress >> > > > > > > > > > is >> > > > > > > > > > > > slow. Anyway, I think it's time to branch out. We've >> > been >> > > > > > waiting >> > > > > > > > too >> > > > > > > > > > > long. >> > > > > > > > > > > > Thanks for creating the branch. >> > > > > > > > > > > > >> > > > > > > > > > > > Regards, >> > > > > > > > > > > > Quanlong >> > > > > > > > > > > > >> > > > > > > > > > > > On Fri, May 7, 2021 at 3:29 AM Jim Apple < >> > > > apa...@jbapple.com >> > > > > > >> > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > I've created a 4.0 branch (with all changes >> through >> > the >> > > > end >> > > > > > of >> > > > > > > > > April) >> > > > > > > > > > > and >> > > > > > > > > > > > > started testing a release candidate. Everyone who >> > wants >> > > > to >> > > > > > > vote, >> > > > > > > > > > > > > >> > > > > > > > >> > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release >> > > > > > > > > > is a >> > > > > > > > > > > > > guide to both releasing and voting (at the >> bottom). >> > > I'll >> > > > > let >> > > > > > > you >> > > > > > > > > all >> > > > > > > > > > > know >> > > > > > > > > > > > > when I have something to vote on. >> > > > > > > > > > > > > >> > > > > > > > > > > > > Thanks! >> > > > > > > > > > > > > Jim >> > > > > > > > > > > > > >> > > > > > > > > > > > > On Fri, Apr 23, 2021 at 12:49 AM Csaba Ringhofer < >> > > > > > > > > > > > csringho...@cloudera.com >> > > > > > > > > > > > > > >> > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > > > About IMPALA-9690 (AVX support): >> > > > > > > > > > > > > > My preferred solution would be to deprecate >> support >> > > for >> > > > > x64 >> > > > > > > > > without >> > > > > > > > > > > > AVX2 >> > > > > > > > > > > > > in >> > > > > > > > > > > > > > 4.0, but not start removing the related logic >> yet. >> > > > > > > > > > > > > > - We could even add a DCHECK + flag to crash by >> > > default >> > > > > if >> > > > > > no >> > > > > > > > > AVX2 >> > > > > > > > > > is >> > > > > > > > > > > > > > detected, and a message that points them to >> Impala >> > > > > mailing >> > > > > > > > > > > > lists/related >> > > > > > > > > > > > > > Jira. >> > > > > > > > > > > > > > - After 4.0 is released people would be free to >> > > > implement >> > > > > > > > things >> > > > > > > > > > that >> > > > > > > > > > > > > > assume AVX2 support on x64, for example >> compiling >> > > > > > Impala/the >> > > > > > > > > whole >> > > > > > > > > > > > > > toolchain with -mavx2. I think that currently >> > people >> > > > are >> > > > > > held >> > > > > > > > > back >> > > > > > > > > > > from >> > > > > > > > > > > > > > doing this - why put effort in >> doing+benchmarking a >> > > > > change >> > > > > > if >> > > > > > > > you >> > > > > > > > > > > can't >> > > > > > > > > > > > > > push it to master. >> > > > > > > > > > > > > > - If a user appears with a valid use case that >> > needs >> > > > x64 >> > > > > > > > without >> > > > > > > > > > AVX2 >> > > > > > > > > > > > > then >> > > > > > > > > > > > > > we can revise our plans, but we shouldn't ignore >> > low >> > > > > > hanging >> > > > > > > > > fruits >> > > > > > > > > > > for >> > > > > > > > > > > > > the >> > > > > > > > > > > > > > sake of imaginary users :) >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > -Csaba >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Sun, Apr 11, 2021 at 5:52 PM Shant Hovsepian >> < >> > > > > > > > > > > > > sh...@superdupershant.com >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > > +1 >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Regarding IMPALA-9690 if someone has the time >> to >> > > > clean >> > > > > up >> > > > > > > the >> > > > > > > > > old >> > > > > > > > > > > > > > > conditional cpu logic it would be great, but >> if >> > the >> > > > ARM >> > > > > > > folks >> > > > > > > > > > still >> > > > > > > > > > > > > > > need some of that logic it's not critical >> enough >> > to >> > > > > > block, >> > > > > > > > > just a >> > > > > > > > > > > > very >> > > > > > > > > > > > > > > nice to have? >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > -Shant >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > On Sun, Apr 4, 2021 at 1:09 AM Tim Armstrong < >> > > > > > > > > > > > > tim.g.armstr...@gmail.com> >> > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > I agree that it would be good to get this >> done >> > > even >> > > > > if >> > > > > > > it's >> > > > > > > > > not >> > > > > > > > > > > > > > perfect, >> > > > > > > > > > > > > > > > there's so much good stuff. >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > On Fri, 2 Apr 2021 at 17:40, Joe McDonnell < >> > > > > > > > > > > > > joemcdonn...@cloudera.com> >> > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > I agree that we should wrap up Impala 4. A >> > > large >> > > > > > amount >> > > > > > > > of >> > > > > > > > > > good >> > > > > > > > > > > > > work >> > > > > > > > > > > > > > > has >> > > > > > > > > > > > > > > > > gone in and it belongs in a release. >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > We need to decide which breaking changes >> are >> > > > truly >> > > > > > > > blockers >> > > > > > > > > > for >> > > > > > > > > > > > an >> > > > > > > > > > > > > > > Impala 4 >> > > > > > > > > > > > > > > > > release. My feeling is that we can't hold >> the >> > > > > release >> > > > > > > for >> > > > > > > > > > > > > > compatibility >> > > > > > > > > > > > > > > > > breaking changes unless someone signs up >> to >> > do >> > > > > those >> > > > > > > > > changes. >> > > > > > > > > > > > > > Breaking >> > > > > > > > > > > > > > > > > compatibility is useful sometimes, but it >> > can't >> > > > be >> > > > > an >> > > > > > > > > > > indefinite >> > > > > > > > > > > > > hold >> > > > > > > > > > > > > > > on >> > > > > > > > > > > > > > > > > releases. I think a release discussion >> thread >> > > is >> > > > a >> > > > > > good >> > > > > > > > way >> > > > > > > > > > to >> > > > > > > > > > > > > > > stimulate >> > > > > > > > > > > > > > > > > this discussion. >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Apart from the two JIRAs you listed, one >> > other >> > > > > change >> > > > > > > > > > mentioned >> > > > > > > > > > > > in >> > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > original email is switching to use the new >> > > > > on-demand >> > > > > > > > > metadata >> > > > > > > > > > > by >> > > > > > > > > > > > > > > default. >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > Thanks, >> > > > > > > > > > > > > > > > > Joe >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > On Fri, Apr 2, 2021 at 5:22 PM Quanlong >> > Huang < >> > > > > > > > > > > > > > huangquanl...@gmail.com >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Sure. If there are no objections, I'll >> > raise >> > > a >> > > > > > > > discussion >> > > > > > > > > > > > thread >> > > > > > > > > > > > > > for >> > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > 4.0 release. There are still some >> > unresolved >> > > > > > breaking >> > > > > > > > > > > changes, >> > > > > > > > > > > > > e.g. >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > - IMPALA-2210: Make Parquet the >> default >> > > file >> > > > > > > format >> > > > > > > > > > > > > > > > > > - IMPALA-9690: Bump minimum x86-64 >> CPU >> > > > > > > requirements >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > Thanks >> > > > > > > > > > > > > > > > > > Quanlong >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > On Tue, Mar 30, 2021 at 11:13 PM Jim >> Apple >> > < >> > > > > > > > > > > jbap...@apache.org >> > > > > > > > > > > > > >> > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > Thanks for bringing this up again, >> > > Quanlong! >> > > > I >> > > > > > > would >> > > > > > > > > love >> > > > > > > > > > > to >> > > > > > > > > > > > > see >> > > > > > > > > > > > > > a >> > > > > > > > > > > > > > > 4.0 >> > > > > > > > > > > > > > > > > > > release soon so that 4.1 releases and >> > what >> > > > not >> > > > > > can >> > > > > > > be >> > > > > > > > > > > > prepared. >> > > > > > > > > > > > > > > Already >> > > > > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > changlist from the 3.x line is going >> to >> > be >> > > > > > enormous >> > > > > > > > and >> > > > > > > > > > > could >> > > > > > > > > > > > > be >> > > > > > > > > > > > > > > hard >> > > > > > > > > > > > > > > > > for >> > > > > > > > > > > > > > > > > > > users to digest. >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > On Mon, Mar 29, 2021 at 7:22 AM >> Quanlong >> > > > Huang >> > > > > < >> > > > > > > > > > > > > > > > > huangquanl...@gmail.com> >> > > > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Reviving this thread. I think it's >> time >> > > to >> > > > > > create >> > > > > > > > the >> > > > > > > > > > > 4.0.0 >> > > > > > > > > > > > > > > branch >> > > > > > > > > > > > > > > > > and >> > > > > > > > > > > > > > > > > > > > prepare for the release now. Any >> > breaking >> > > > > > change >> > > > > > > is >> > > > > > > > > > > landing >> > > > > > > > > > > > > and >> > > > > > > > > > > > > > > we >> > > > > > > > > > > > > > > > > > should >> > > > > > > > > > > > > > > > > > > > wait? >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > Thanks, >> > > > > > > > > > > > > > > > > > > > Quanlong >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > On Thu, Jun 18, 2020 at 5:47 AM Tim >> > > > > Armstrong < >> > > > > > > > > > > > > > > > > tarmstr...@cloudera.com >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > Another thing that we're looking >> into >> > > is >> > > > > > > changing >> > > > > > > > > the >> > > > > > > > > > > > > encoded >> > > > > > > > > > > > > > > > > runtime >> > > > > > > > > > > > > > > > > > > > > profile representation to be more >> > > > > efficient - >> > > > > > > see >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9378. >> > > > > > > > > > So >> > > > > > > > > > > > that >> > > > > > > > > > > > > > > might >> > > > > > > > > > > > > > > > > be >> > > > > > > > > > > > > > > > > > a >> > > > > > > > > > > > > > > > > > > > > default we will try to change in >> > Impala >> > > > > 4.0. >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > On Fri, Apr 24, 2020 at 12:37 PM >> Tim >> > > > > > Armstrong >> > > > > > > < >> > > > > > > > > > > > > > > > > > > tarmstr...@cloudera.com> >> > > > > > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > An addendum to this - we're also >> > > > > > considering >> > > > > > > > > > whether >> > > > > > > > > > > to >> > > > > > > > > > > > > > > increase >> > > > > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > > > > minimum CPU version so that we >> can >> > > > > simplify >> > > > > > > > > > > development >> > > > > > > > > > > > > and >> > > > > > > > > > > > > > > focus >> > > > > > > > > > > > > > > > > > > more >> > > > > > > > > > > > > > > > > > > > on >> > > > > > > > > > > > > > > > > > > > > > optimising for the most common >> > CPUs - >> > > > see >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > https://issues.apache.org/jira/browse/IMPALA-9690 >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > Running on commodity hardware is >> > > > > important >> > > > > > > and >> > > > > > > > we >> > > > > > > > > > > want >> > > > > > > > > > > > > > > people to >> > > > > > > > > > > > > > > > > be >> > > > > > > > > > > > > > > > > > > > able >> > > > > > > > > > > > > > > > > > > > > > to develop on whatever machine >> they >> > > > have >> > > > > > > access >> > > > > > > > > to. >> > > > > > > > > > > At >> > > > > > > > > > > > > most >> > > > > > > > > > > > > > > we're >> > > > > > > > > > > > > > > > > > > > likely >> > > > > > > > > > > > > > > > > > > > > > going to require AVX2 support, >> > which >> > > > has >> > > > > > been >> > > > > > > > > > > supported >> > > > > > > > > > > > > by >> > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > vast >> > > > > > > > > > > > > > > > > > > > > > majority of CPUs for the better >> > part >> > > > of a >> > > > > > > > decade. >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 7, 2020 at 11:50 AM >> Joe >> > > > > > > McDonnell < >> > > > > > > > > > > > > > > > > > > > joemcdonn...@cloudera.com >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > > >> Impala is starting development >> on >> > > > Impala >> > > > > > > 4.0. >> > > > > > > > > > There >> > > > > > > > > > > > are >> > > > > > > > > > > > > > many >> > > > > > > > > > > > > > > > > > > exciting >> > > > > > > > > > > > > > > > > > > > > >> projects underway including >> full >> > > > support >> > > > > > for >> > > > > > > > > Hive >> > > > > > > > > > > ACID >> > > > > > > > > > > > > > > tables, >> > > > > > > > > > > > > > > > > > > > improved >> > > > > > > > > > > > > > > > > > > > > >> multithreading support, and >> fault >> > > > > > tolerance >> > > > > > > > > > > > > improvements, >> > > > > > > > > > > > > > > but >> > > > > > > > > > > > > > > > > > Impala >> > > > > > > > > > > > > > > > > > > > 4.0 >> > > > > > > > > > > > > > > > > > > > > >> also serves as a time to alter >> > > > behavior >> > > > > > and >> > > > > > > > > > platform >> > > > > > > > > > > > > > > support. >> > > > > > > > > > > > > > > > > Here >> > > > > > > > > > > > > > > > > > > is >> > > > > > > > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > > > >> current list of notable >> breaking >> > > > changes >> > > > > > > that >> > > > > > > > > are >> > > > > > > > > > > > > planned >> > > > > > > > > > > > > > in >> > > > > > > > > > > > > > > > > > Impala >> > > > > > > > > > > > > > > > > > > > 4.0: >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Remove support for older >> operating >> > > > > > systems: >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Maintaining support for older >> > > > operating >> > > > > > > > systems >> > > > > > > > > > > > > requires a >> > > > > > > > > > > > > > > > > > > > considerable >> > > > > > > > > > > > > > > > > > > > > >> amount of effort, especially >> with >> > > > > changing >> > > > > > > > > > security >> > > > > > > > > > > > > > > > > requirements. >> > > > > > > > > > > > > > > > > > > Each >> > > > > > > > > > > > > > > > > > > > > >> extra operating system consumes >> > > > valuable >> > > > > > > > effort >> > > > > > > > > > and >> > > > > > > > > > > > can >> > > > > > > > > > > > > > > conflict >> > > > > > > > > > > > > > > > > > > with >> > > > > > > > > > > > > > > > > > > > > >> support for newer operating >> > systems, >> > > > so >> > > > > we >> > > > > > > > have >> > > > > > > > > > > > decided >> > > > > > > > > > > > > to >> > > > > > > > > > > > > > > > > remove >> > > > > > > > > > > > > > > > > > > > > support >> > > > > > > > > > > > > > > > > > > > > >> for some older operating >> systems. >> > > > > > > > Specifically, >> > > > > > > > > we >> > > > > > > > > > > > plan >> > > > > > > > > > > > > to >> > > > > > > > > > > > > > > drop >> > > > > > > > > > > > > > > > > > > > support >> > > > > > > > > > > > > > > > > > > > > for: >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> - >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Centos 6 >> > > > > > > > > > > > > > > > > > > > > >> - >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Ubuntu 14 >> > > > > > > > > > > > > > > > > > > > > >> - >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Debian 8 >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Each of these is approaching or >> > past >> > > > its >> > > > > > end >> > > > > > > > of >> > > > > > > > > > > life. >> > > > > > > > > > > > > > Since >> > > > > > > > > > > > > > > > > > Centos6 >> > > > > > > > > > > > > > > > > > > > was >> > > > > > > > > > > > > > > > > > > > > >> the only operating system still >> > > > needing >> > > > > > > Python >> > > > > > > > > 2.6 >> > > > > > > > > > > and >> > > > > > > > > > > > > > > Python >> > > > > > > > > > > > > > > > > 2.6 >> > > > > > > > > > > > > > > > > > is >> > > > > > > > > > > > > > > > > > > > > long >> > > > > > > > > > > > > > > > > > > > > >> past its end of life, Python >> 2.6 >> > > will >> > > > no >> > > > > > > > longer >> > > > > > > > > be >> > > > > > > > > > > > > > > supported. In >> > > > > > > > > > > > > > > > > > > > > addition, >> > > > > > > > > > > > > > > > > > > > > >> Centos 7 support will be >> focused >> > on >> > > > > newer >> > > > > > > > > versions >> > > > > > > > > > > > such >> > > > > > > > > > > > > as >> > > > > > > > > > > > > > > 7.5 >> > > > > > > > > > > > > > > > > and >> > > > > > > > > > > > > > > > > > > > > above. >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Remove support for Sentry: >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Over the past year and a half, >> > > > Impala's >> > > > > > > Ranger >> > > > > > > > > > > > > > authorization >> > > > > > > > > > > > > > > > > > > > > >> functionality has achieved >> parity >> > > and >> > > > > > > > surpassed >> > > > > > > > > > the >> > > > > > > > > > > > > > existing >> > > > > > > > > > > > > > > > > > Sentry >> > > > > > > > > > > > > > > > > > > > > >> authorization functionality. >> > > Impala's >> > > > > > Sentry >> > > > > > > > > > support >> > > > > > > > > > > > > > > requires >> > > > > > > > > > > > > > > > > > > ongoing >> > > > > > > > > > > > > > > > > > > > > >> effort to test and maintain, >> > > including >> > > > > > > > > maintenance >> > > > > > > > > > > on >> > > > > > > > > > > > > > Sentry >> > > > > > > > > > > > > > > > > > itself >> > > > > > > > > > > > > > > > > > > to >> > > > > > > > > > > > > > > > > > > > > >> address security updates. Given >> > the >> > > > > > > diminished >> > > > > > > > > > > > activity >> > > > > > > > > > > > > in >> > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > Sentry >> > > > > > > > > > > > > > > > > > > > > >> community, there is no timeline >> > for >> > > > > > > addressing >> > > > > > > > > > these >> > > > > > > > > > > > > > > security >> > > > > > > > > > > > > > > > > > > updates. >> > > > > > > > > > > > > > > > > > > > > >> Unless there is a maintainer >> for >> > > > Sentry >> > > > > > > > support, >> > > > > > > > > > > > Impala >> > > > > > > > > > > > > > > plans to >> > > > > > > > > > > > > > > > > > > focus >> > > > > > > > > > > > > > > > > > > > > its >> > > > > > > > > > > > > > > > > > > > > >> efforts on Ranger for its >> > > > authorization >> > > > > > > system >> > > > > > > > > and >> > > > > > > > > > > > drop >> > > > > > > > > > > > > > > Sentry. >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Remove support for Impala-lzo: >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Impala-lzo provides code to >> allow >> > > > Impala >> > > > > > to >> > > > > > > > read >> > > > > > > > > > the >> > > > > > > > > > > > LZO >> > > > > > > > > > > > > > > > > > compressed >> > > > > > > > > > > > > > > > > > > > > >> tables. LZO is GPL licensed, >> which >> > > is >> > > > > why >> > > > > > > this >> > > > > > > > > > > support >> > > > > > > > > > > > > is >> > > > > > > > > > > > > > > not >> > > > > > > > > > > > > > > > > > > included >> > > > > > > > > > > > > > > > > > > > > >> directly. The Impala-lzo code >> > > > interacts >> > > > > > with >> > > > > > > > > > > internal >> > > > > > > > > > > > > > Impala >> > > > > > > > > > > > > > > > > code >> > > > > > > > > > > > > > > > > > > at a >> > > > > > > > > > > > > > > > > > > > > >> level that is error prone and >> > > > intricate. >> > > > > > > Given >> > > > > > > > > the >> > > > > > > > > > > low >> > > > > > > > > > > > > > > adoption >> > > > > > > > > > > > > > > > > of >> > > > > > > > > > > > > > > > > > > LZO >> > > > > > > > > > > > > > > > > > > > > and >> > > > > > > > > > > > > > > > > > > > > >> the other compression options >> > > > available, >> > > > > > > > Impala >> > > > > > > > > > > plans >> > > > > > > > > > > > to >> > > > > > > > > > > > > > > remove >> > > > > > > > > > > > > > > > > > > > > Impala-lzo >> > > > > > > > > > > > > > > > > > > > > >> support along with the low >> level >> > > > > interface >> > > > > > > it >> > > > > > > > > > used. >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Deprecations: >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> In addition, we also plan to >> > > deprecate >> > > > > > > several >> > > > > > > > > > > > existing >> > > > > > > > > > > > > > > features >> > > > > > > > > > > > > > > > > > so >> > > > > > > > > > > > > > > > > > > > that >> > > > > > > > > > > > > > > > > > > > > >> they can be removed in a future >> > > > release. >> > > > > > > Here >> > > > > > > > > is a >> > > > > > > > > > > > > summary >> > > > > > > > > > > > > > > of >> > > > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > most >> > > > > > > > > > > > > > > > > > > > > >> notable ones: >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> - >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Impala will default to >> using on >> > > > > demand >> > > > > > > > > metadata >> > > > > > > > > > > > > > > management >> > > > > > > > > > > > > > > > > > (i.e. >> > > > > > > > > > > > > > > > > > > > the >> > > > > > > > > > > > > > > > > > > > > >> local catalog >> implementation) >> > as >> > > > > > > described >> > > > > > > > in >> > > > > > > > > > > this >> > > > > > > > > > > > > > > > > > documentation: >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > >> > > http://impala.apache.org/docs/build/html/topics/impala_metadata.html >> > > > > > > > > > > > > > > > > > > > > >> The old metadata system is >> now >> > > > > > deprecated >> > > > > > > > and >> > > > > > > > > > may >> > > > > > > > > > > > be >> > > > > > > > > > > > > > > removed >> > > > > > > > > > > > > > > > > > in a >> > > > > > > > > > > > > > > > > > > > > >> future release. >> > > > > > > > > > > > > > > > > > > > > >> - >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Impala is deprecating the >> > Beeswax >> > > > > > client >> > > > > > > > > > protocol >> > > > > > > > > > > > > (i.e. >> > > > > > > > > > > > > > > > > clients >> > > > > > > > > > > > > > > > > > > > that >> > > > > > > > > > > > > > > > > > > > > >> connect via beeswax_port) in >> > > favor >> > > > of >> > > > > > the >> > > > > > > > > > > > HiveServer2 >> > > > > > > > > > > > > > > client >> > > > > > > > > > > > > > > > > > > > > protocol. >> > > > > > > > > > > > > > > > > > > > > >> - >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Impala is deprecating the >> old >> > > > > > > > > decimal_v2=false >> > > > > > > > > > > > > behavior >> > > > > > > > > > > > > > > from >> > > > > > > > > > > > > > > > > > > Impala >> > > > > > > > > > > > > > > > > > > > > >> 2.x. The decimal_v2 query >> > option >> > > > has >> > > > > > > > > defaulted >> > > > > > > > > > to >> > > > > > > > > > > > > true >> > > > > > > > > > > > > > > since >> > > > > > > > > > > > > > > > > > > Impala >> > > > > > > > > > > > > > > > > > > > > 3. >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Impala is always open to new >> > > > developers, >> > > > > > and >> > > > > > > > we >> > > > > > > > > > > > welcome >> > > > > > > > > > > > > > > feedback >> > > > > > > > > > > > > > > > > > on >> > > > > > > > > > > > > > > > > > > > > these >> > > > > > > > > > > > > > > > > > > > > >> plans. Further discussion of >> other >> > > > > changes >> > > > > > > and >> > > > > > > > > > > > > > deprecations >> > > > > > > > > > > > > > > is >> > > > > > > > > > > > > > > > > > > ongoing >> > > > > > > > > > > > > > > > > > > > > at >> > > > > > > > > > > > > > > > > > > > > >> dev@impala.apache.org. >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Thanks, >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> Joe McDonnell >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> >
