Re: Action required - re-bootstrap your toolchain when you have a chance

[Philip Zeyliger]

Nice!

On Fri, Sep 20, 2019 at 2:29 PM Tim Armstrong 
wrote:

> https://gerrit.cloudera.org/#/c/14274/ just got merged. This switches to a
> native-toolchain version which has the RUNPATH set correctly in the
> binaries. This will allow solve some issues with LD_LIBRARY_PATH - most of
> the hacks around that in impala-config.sh/bootstrap_system.sh can be
> removed.
>
> I am deferring the LD_LIBRARY_PATH cleanup until everyone has a chance to
> bootstrap with the new binaries, but once I do that, you would likely start
> to see runtime errors about GLIBCXX versions if you are running a newer
> Impala version with the old binaries.
>
> The newer binaries should work fine with older versions of Impala, so once
> you re-bootstrap things should be fine even if you go back to an older
> branch.
>
> To do this you can remove your toolchain components and bootstrap again:
>
>   (set -u; rm -r $IMPALA_TOOLCHAIN/*) && ./bin/bootstrap_toolchain.py
>
> I will hold off on making follow-on changes for at least a few days so that
> everyone has time to do this.
>
> - Tim
>

Re: New Committer: Sahil Takiar

[Philip Zeyliger]

Congrats!

On Wed, May 22, 2019 at 8:21 AM Fredy Wijaya  wrote:

> Congrats!
>
> On Wed, May 22, 2019 at 10:20 AM Joe McDonnell 
> wrote:
>
> > Congrats!
> >
> > On Wed, May 22, 2019 at 8:11 AM Laszlo Gaal 
> > wrote:
> >
> > > Congrats, Sahil!
> > >
> > > On Wed, May 22, 2019 at 4:56 PM David Rorke 
> wrote:
> > >
> > > > Congratulations Sahil!
> > > >
> > > > On Wed, May 22, 2019 at 6:24 AM Andrew Sherman <
> asher...@cloudera.com>
> > > > wrote:
> > > >
> > > > > Congratulations Sahil!
> > > > >
> > > > >
> > > > > On Tue, May 21, 2019 at 11:37 PM Lars Volker 
> > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > The Project Management Committee (PMC) for Apache Impala
> > > > > > has invited Sahil Takiar to become a committer and we are pleased
> > > > > > to announce that he has accepted.
> > > > > >
> > > > > > Welcome and congratulations, Sahil!
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: You'll need to fetch your Impala-lzo...

[Philip Zeyliger]

I think this one is even trickier!

https://jenkins.impala.io/job/ubuntu-16.04-build-only/5395/ doesn't seem to
clone Impala-lzo at all! And, though we're trying to do everything in a
tempdir, the imapla_lzo dir is

IMPALA_LZO  = /home/ubuntu/tmp.CCVBRWuYra/../Impala-lzo

My best guess is that a previous build has checked out Impala-lzo, and, if
the build is re-using a Jenkins worker, it gets Impala-lzo. Otherwise, it
doesn't.

I have some evidence that we only sometimes build IMPALA_LZO in this job.
This looks through the last 30 builds, and notices that some built
impala-lzo and some didn't. Most of them passed...

$for i in $(seq 5400 5430); do echo $i; curl --silent
https://jenkins.impala.io/job/ubuntu-16.04-build-only/$i/consoleText | grep
-i "Built target impala-lzo"; done
5400
5401
[100%] Built target impala-lzo
5402
5403
5404
5405
[100%] Built target impala-lzo
5406
5407
5408
[100%] Built target impala-lzo
5409
5410
[100%] Built target impala-lzo
5411
[100%] Built target impala-lzo
5412
5413
5414
5415
5416
[100%] Built target impala-lzo
5417
[100%] Built target impala-lzo
5418
5419
5420
5421
[100%] Built target impala-lzo
5422
[100%] Built target impala-lzo
5423
[100%] Built target impala-lzo
5424
5425
5426
[100%] Built target impala-lzo
5427
5428
5429
5430

I've added the following to the job:

# Point IMPALA_LZO to a non-existant directory to avoid it being
# built. (Otherwise, sharing of workers will sometimes find an old dir.)
export IMPALA_LZO="$IMPALA_HOME/impala-lzo-not-checked-out"

and am trying it out...

-- Philip

Re: You'll need to fetch your Impala-lzo...

[Philip Zeyliger]

Sorry about that. Some Cloudera-internal automation force-pushed that
branch. It's been fixed now, I hope.

On Wed, Feb 13, 2019 at 2:00 PM Andrew Sherman 
wrote:

> I am debugging some weird build failures which seem to be confused about
> branches in  https://github.com/cloudera/impala-lzo.git
> 'master' has the latest changes, but 'origin/master', the default branch
> when you clone, points to some really old commit.
>
> Does anyone understand this?
>
> -Andrew
>
> asherman@asherman-desktop:~/git/asf/Impala-lzo$ git log -1 --pretty=fuller
> master
> commit dccb1be88a5e237b06ae69cd99b048a38d9f024b (HEAD -> master)
> Author: Philip Zeyliger 
> AuthorDate: Tue Jan 22 14:21:42 2019 -0800
> Commit: Philip Zeyliger 
> CommitDate: Fri Feb 1 11:04:06 2019 -0800
>
> Adapt to new interface for AddDiskIoRange (IMPALA-7980)
>
> The change for IMPALA-7980 moves out the "are we done
> adding disk io ranges" accounting from AddDiskIoRange()
> to a new UpdateRemainingScanRangeSubmissions call.
>
> asherman@asherman-desktop:~/git/asf/Impala-lzo$ git log -1 --pretty=fuller
> origin/master
> commit 9543908ec82245878c4060dec64a1533d7adbee6
> (origin/master-backup-2019-01-23, origin/master)
> Author: Alex Behm 
> AuthorDate: Thu Dec 4 16:53:06 2014 -0800
> Commit: Alex Behm 
> CommitDate: Fri Dec 5 14:28:25 2014 -0800
>
> Remove data compaction flag.
>
> Change-Id: Ife72c7ee630b6ecd9024a3d33b7673b56fcf1fba
>
> I am tracking this at https://issues.apache.org/jira/browse/IMPALA-8200
>
>
> On Wed, Feb 13, 2019 at 9:27 AM Philip Zeyliger 
> wrote:
>
> > I added "IMPALA_LZO_BRANCH=2.x" to the properties file generated in
> > cherrypick-2.x-and-test
> >
> > [image: image.png]
> >
> > On Tue, Feb 12, 2019 at 11:36 PM Quanlong Huang  >
> > wrote:
> >
> >> Thank Tim and Fredy's help!
> >>
> >> However, I still find some downstream Jenkins jobs not using the same
> >> IMPALA_LZO_BRANCH value. There're two failures due to this:
> >>
> >> 1) I ran gerrit-verify-dryrun for patch set 12429
> >> with IMPALA_LZO_BRANCH=2.x. The job is
> >> https://jenkins.impala.io/job/gerrit-verify-dryrun/3760
> >> This downstream job ubuntu-16.04-build-only still used master branch of
> >> Impala-lzo thus failed:
> >>  -  ubuntu-16.04-build-only #5336
> >> https://jenkins.impala.io/job/ubuntu-16.04-build-only/5336/
> >>
> >> Looks like this job does not use IMPALA_LZO_BRANCH so not switch branch
> of
> >> Impala-lzo. I think we should set IMPALA_LZO_BRANCH in impala-config.sh
> >> and
> >> use it in buildall.sh.
> >>
> >> 2) I ran cherrypick-2.x-and-test. The job is
> >> https://jenkins.impala.io/job/cherrypick-2.x-and-test/638/
> >> The problematic downstream jobs:
> >>  - ubuntu-16.04-from-scratch #4470
> >> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/4470/
> >>  - ubuntu-16.04-from-scratch #4471
> >> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/4471/
> >>
> >> Looks like the IMPALA_LZO_BRANCH is not set so they still use the
> default
> >> value (master).
> >> I don't have the permission to modify the job configs. Could anyone help
> >> to
> >> have a look?
> >>
> >> Thanks,
> >> Quanlong
> >>
> >> On Mon, Feb 11, 2019 at 1:32 PM Quanlong Huang  >
> >> wrote:
> >>
> >> > Then branch-2.x needs this again:
> https://gerrit.cloudera.org/c/12339/.
> >> > Could anyone have a look at this? I encountered a build error at
> >> > https://gerrit.cloudera.org/c/12429/ due to this.
> >> >
> >> > Thanks,
> >> > Quanlong
> >> >
> >> > On Wed, Feb 6, 2019 at 1:15 AM Philip Zeyliger 
> >> > wrote:
> >> >
> >> >> Thanks! (And Jim suggested the same thing.)
> >> >>
> >> >> My apologies for the churn. What I didn't realize is that
> >> >> https://jenkins.impala.io/job/parallel-all-tests/build takes a
> >> >> "IMPALA_LZO_BRANCH" parameter. I had foolishly assumed that the
> branch
> >> >> logic was in
> >> >>
> >> >>
> >>
> https://gitbox.apache.org/repos/asf?p=impala.git;a=blob;f=bin/bootstrap_system.sh;h=cb869ffe79f3c35cf45b2f962a472274319d746c;hb=HEAD#l378
> >> >> ,
> >> >> but it's actually passed through in the Jenkins job. Yesterday Lars
> >> and I

Re: You'll need to fetch your Impala-lzo...

[Philip Zeyliger]

I added "IMPALA_LZO_BRANCH=2.x" to the properties file generated in
cherrypick-2.x-and-test

[image: image.png]

On Tue, Feb 12, 2019 at 11:36 PM Quanlong Huang 
wrote:

> Thank Tim and Fredy's help!
>
> However, I still find some downstream Jenkins jobs not using the same
> IMPALA_LZO_BRANCH value. There're two failures due to this:
>
> 1) I ran gerrit-verify-dryrun for patch set 12429
> with IMPALA_LZO_BRANCH=2.x. The job is
> https://jenkins.impala.io/job/gerrit-verify-dryrun/3760
> This downstream job ubuntu-16.04-build-only still used master branch of
> Impala-lzo thus failed:
>  -  ubuntu-16.04-build-only #5336
> https://jenkins.impala.io/job/ubuntu-16.04-build-only/5336/
>
> Looks like this job does not use IMPALA_LZO_BRANCH so not switch branch of
> Impala-lzo. I think we should set IMPALA_LZO_BRANCH in impala-config.sh and
> use it in buildall.sh.
>
> 2) I ran cherrypick-2.x-and-test. The job is
> https://jenkins.impala.io/job/cherrypick-2.x-and-test/638/
> The problematic downstream jobs:
>  - ubuntu-16.04-from-scratch #4470
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/4470/
>  - ubuntu-16.04-from-scratch #4471
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/4471/
>
> Looks like the IMPALA_LZO_BRANCH is not set so they still use the default
> value (master).
> I don't have the permission to modify the job configs. Could anyone help to
> have a look?
>
> Thanks,
> Quanlong
>
> On Mon, Feb 11, 2019 at 1:32 PM Quanlong Huang 
> wrote:
>
> > Then branch-2.x needs this again: https://gerrit.cloudera.org/c/12339/.
> > Could anyone have a look at this? I encountered a build error at
> > https://gerrit.cloudera.org/c/12429/ due to this.
> >
> > Thanks,
> > Quanlong
> >
> > On Wed, Feb 6, 2019 at 1:15 AM Philip Zeyliger 
> > wrote:
> >
> >> Thanks! (And Jim suggested the same thing.)
> >>
> >> My apologies for the churn. What I didn't realize is that
> >> https://jenkins.impala.io/job/parallel-all-tests/build takes a
> >> "IMPALA_LZO_BRANCH" parameter. I had foolishly assumed that the branch
> >> logic was in
> >>
> >>
> https://gitbox.apache.org/repos/asf?p=impala.git;a=blob;f=bin/bootstrap_system.sh;h=cb869ffe79f3c35cf45b2f962a472274319d746c;hb=HEAD#l378
> >> ,
> >> but it's actually passed through in the Jenkins job. Yesterday Lars and
> I
> >> piped some of the stuff through to
> >> https://jenkins.impala.io/job/gerrit-verify-dryrun/build?delay=0sec as
> >> well.
> >>
> >> Again, apologies for causing the churn. I should have worked it out in
> >> more
> >> detail. Please continue to let me know if things are still broken.
> >>
> >> -- Philip
> >>
> >> On Tue, Feb 5, 2019 at 1:00 AM Tim Armstrong 
> >> wrote:
> >>
> >> > I just changed the default to master, because that would make sense.
> >> >
> >> > On Tue, Feb 5, 2019 at 6:56 AM Quanlong Huang <
> huangquanl...@gmail.com>
> >> > wrote:
> >> >
> >> > > Oh, I thought the default is master. Just abandoned my patch since
> >> it's
> >> > > redundant. There's a relative patch for impala-3.x to use the master
> >> > > branch:
> >> > >
> >> > >
> >> >
> >>
> https://github.com/apache/impala/commit/365e35a36f142532044d8a34cbb87646bf9cb240
> >> > >
> >> > > On Sun, Feb 3, 2019 at 4:40 AM Jim Apple 
> >> wrote:
> >> > >
> >> > > > Do you want to change default branch to master, rather than
> >> cdh5-trunk?
> >> > > >
> >> > > > https://help.github.com/articles/setting-the-default-branch/
> >> > > >
> >> > > > On Fri, Feb 1, 2019 at 3:51 PM Philip Zeyliger <
> phi...@cloudera.com
> >> >
> >> > > > wrote:
> >> > > >
> >> > > > > "cdh5-trunk" was that branch, but I also pushed it to 2.x, for a
> >> bit
> >> > > more
> >> > > > > consistency.
> >> > > > >
> >> > > > > -- Philip
> >> > > > >
> >> > > > > On Fri, Feb 1, 2019 at 3:44 PM Quanlong Huang <
> >> > huangquanl...@gmail.com
> >> > > >
> >> > > > > wrote:
> >> > > > >

Welcome Paul Rogers as a Committer

[Philip Zeyliger]

The Project Management Committee (PMC) for Apache Impala has invited Paul
Rogers to become a committer and we are pleased to announce that they have
accepted.
Congratulations and welcome, Paul!

On behalf of the PMC,

-- Philip

You'll need to fetch your Impala-lzo...

[Philip Zeyliger]

Hi folks,

I just pushed
https://gitbox.apache.org/repos/asf?p=impala.git;a=commit;h=a8e30506aafef14646d95a56fb87cf7c28d259d6
and
https://github.com/cloudera/impala-lzo/commit/dccb1be88a5e237b06ae69cd99b048a38d9f024b
to tackle IMPALA-7980. If you're working on master, and you have
../Impala-lzo checked out, you'll need to pull that repo. There's also a
divergence now for Impala-lzo between branches, so if you're still working
on 2.x (say), you'll need to figure out how to manage that.

Please let me know if you run into issues; I'll be happy to help.

-- Philip

Re: Jenkins restart

[Philip Zeyliger]

When I last looked, the issue is that those tests use a Jenkins "pipeline."
These have multiple steps (let's call them A, B, C) that run in serial. If
A is running when you quiesce jenkins, B will get queued but never start,
causing the overall job to hang forever.
https://jenkins.impala.io/job/parallel-all-tests/configure does seem to
have a timeout of 10 hours. It looks (
https://jenkins.impala.io/job/parallel-all-tests/4893/console) like that's
been hit, so now we're looking at Jenkins bugs.
https://issues.jenkins-ci.org/browse/JENKINS-40839 is one. Ideally we
wouldn't even be relying on the timeout.

My experiences with "programming via Jenkins" (i.e., using Jenkins jobs as
functions and tying them together) lead me to try to avoid it.

-- Philip

On Thu, Jan 31, 2019 at 10:05 AM Paul Rogers  wrote:

> Hi All,
>
> Jenkins is restarted.
>
> Looks like we have two jobs that seemingly run forever: parallel-all-tests
> and parallel-all-tests-nightly. Twice now they kept running for more than
> 12 hours and had to be killed. Strangely, the elapsed runtime never exceeds
> about four hours. Perhaps someone familiar with these jobs can figure out
> what’s what.
>
> Thanks,
>
> > On Jan 30, 2019, at 10:01 AM, Paul Rogers  wrote:
> >
> > Hi All,
> >
> > We had a bit of trouble restarting Jenkins: some jobs refused to die
> after waiting 12 hours. So, killed a number of jobs, restarted Jenkins and
> restarted the manually-submiitted dry-run jobs. Looks like some full
> parallel jobs restarted themselves after the restart.
> >
> > Please review your jobs and restart as needed.
> >
> > As it turns out, new security alerts came in yesterday while we were
> fighting this issue so I’ll restart Jenkins again tonight.
> >
> > Thanks,
> >
> > - Paul
> >
>
>

Re: Runtime filter publishing from a join node to its cousin

[Philip Zeyliger]

(If folks are interested in a refresher,
https://impala.apache.org/docs/build/html/topics/impala_runtime_filtering.html
is useful.)

I stared at that code for a bit, and I agree with you that it's plausible.
I'm also confused by the "bottom-up" comment of generateFilters(): it seems
like we walk the plan depth first, and the assignment happens in that
depth-first walk, before all the runtime filters are generated.

A concern is making sure that there are no outer joins impacting
correctness.

-- Philip

On Wed, Jan 30, 2019 at 9:36 PM Todd Lipcon  wrote:

> Hey folks,
>
> I've been digging into a couple of TPCDS queries that are unexpectedly slow
> and uncovered what seems to be some surprising behavior in the planner
> concerning runtime filter assignment. Consider the following query in the
> TPCDS schema:
>
> select straight_join *
>   from promotion a
>   join item b on a.p_item_sk = b.i_item_sk
>   join [shuffle] store_sales c on b.i_item_sk = c.ss_item_sk
>   where b.i_color = 'xxx'
>
> This generates a plan that looks like this:
> http://people.apache.org/~todd/plan.png
>
> Or in text form:
>
> +-+
> | Explain String
>   |
>
> +-+
> | Max Per-Host Resource Reservation: Memory=85.88MB
>|
> | Per-Host Resource Estimates: Memory=298.14GB
>   |
> |
>|
> | F04:PLAN FRAGMENT [UNPARTITIONED] hosts=1 instances=1
>|
> | |  Per-Host Resources: mem-estimate=0B mem-reservation=0B
>|
> | PLAN-ROOT SINK
>   |
> | |  mem-estimate=0B mem-reservation=0B
>|
> | |
>|
> | 08:EXCHANGE [UNPARTITIONED]
>|
> | |  mem-estimate=0B mem-reservation=0B
>|
> | |  tuple-ids=0,1,2 row-size=911B cardinality=14201171
>|
> | |
>|
> | F03:PLAN FRAGMENT [HASH(b.i_item_sk)] hosts=1 instances=1
>|
> | Per-Host Resources: mem-estimate=295.06GB mem-reservation=50.00MB
> runtime-filters-memory=16.00MB|
> | 04:HASH JOIN [INNER JOIN, PARTITIONED]
>   |
> | |  hash predicates: b.i_item_sk = c.ss_item_sk
>   |
> | |  fk/pk conjuncts: none
>   |
> | |  runtime filters: RF000[bloom] <- c.ss_item_sk
>   |
> | |  mem-estimate=295.04GB mem-reservation=34.00MB spill-buffer=2.00MB
>   |
> | |  tuple-ids=0,1,2 row-size=911B cardinality=14201171
>|
> | |
>|
> | |--07:EXCHANGE [HASH(c.ss_item_sk)]
>|
> | |  |  mem-estimate=0B mem-reservation=0B
>   |
> | |  |  tuple-ids=2 row-size=100B cardinality=2879987999
>   |
> | |  |
>   |
> | |  F02:PLAN FRAGMENT [RANDOM] hosts=9 instances=9
>|
> | |  Per-Host Resources: mem-estimate=1.89GB mem-reservation=0B
>|
> | |  02:SCAN HDFS [tpcds_1000_parquet.store_sales c, RANDOM]
>   |
> | | partitions=1824/1824 files=1824 size=189.24GB
>|
> | | stored statistics:
>   |
> | |   table: rows=2879987999 size=unavailable
>|
> | |   partitions: 1824/1824 rows=2879987999
>|
> | |   columns: all
>   |
> | | extrapolated-rows=disabled
>   |
> | | mem-estimate=1.89GB mem-reservation=0B
>   |
> | | tuple-ids=2 row-size=100B cardinality=2879987999
>   |
> | |
>|
> | 06:EXCHANGE [HASH(b.i_item_sk)]
>|
> | |  mem-estimate=0B mem-reservation=0B
>|
> | |  tuple-ids=0,1 row-size=811B cardinality=1500
>|
> | |
>|
> | F00:PLAN FRAGMENT [RANDOM] hosts=1 instances=1
>   |
> | Per-Host Resources: mem-estimate=323.88MB mem-reservation=19.88MB
> runtime-filters-memory=17.00MB|
> | 03:HASH JOIN [INNER JOIN, BROADCAST]
>   |
> | |  hash predicates: a.p_item_sk = b.i_item_sk
>|
> | |  fk/pk conjuncts: a.p_item_sk = b.i_item_sk
>|
> | |  runtime filters: RF002[bloom] <- b.i_item_sk
>|
> | |  mem-estimate=2.88MB mem-reservation=2.88MB spill-buffer=128.00KB
>|
> | |  tuple-ids=0,1 row-size=811B 

Re: Move forward branch-2.x

[Philip Zeyliger]

As for Quanlong's question, I think the answer is however the folks who
want to do the work prefer to do it. As you noticed in the CDH changelists,
Cloudera's distribution has opted for something more like approach (a),
choosing to backport individual features. For a while, we were doing
automation for cherry-picking things automatically, and it got tedious
enough that we decided to turn it off.

On Sun, Jan 27, 2019 at 7:37 PM Paul Rogers  wrote:

> Hi Quanlong,
>
> Thanks for the suggestion. I wonder if there is a third strategy:
>
> c) Isolate the Hadoop 2.x/3.x differences into clearly-defined driver
> layer so that basically all of 3.x can be applied to the 2.x branch. Said
> another way, a single source base can work against either Hadoop 2.x or
> 3.x, with the build (C++) or runtime (Java) choosing the proper “driver”
> classes.
>

We had such a layer for a while, where Impala master could be built against
either Hadoop3 or Hadoop2. We decided to clean it up in commit
e4ae605b083ab536c68552e37ca3c46f6bff4c76.

commit e4ae605b083ab536c68552e37ca3c46f6bff4c76
Author: Fredy Wijaya 
Date:   Thu Jul 12 17:01:13 2018 -0700

IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2

This patch removes the use of IMPALA_MINICLUSTER_PROFILE. The code that
uses IMPALA_MINICLUSTER_PROFILE=2 is removed and it defaults to code
from
IMPALA_MINICLUSTER_PROFILE=3. In order to reduce having too many code
changes in this patch, there is no code change for the shims. The shims
for IMPALA_MINICLUSTER_PROFILE=3 automatically become the default
implementation.

Testing:
- Ran core and exhaustive tests

Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Reviewed-on: http://gerrit.cloudera.org:8080/10940
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 

Re: Does anyone use make_{asan,debug,impala,release}.sh?

[Philip Zeyliger]

+1 to getting rid of them.

On Sun, Jan 27, 2019 at 6:49 PM Quanlong Huang 
wrote:

> +1 for making the infra(bash/python scripts) simpler! It's hard for
> beginners to make clear of their relationships and use them correctly.
>
> On Sat, Jan 26, 2019 at 9:45 AM Tim Armstrong 
> wrote:
>
> > Currently one of the biggest pain points with the build system is the
> > number of layered shell scripts that interact with each other in
> > non-obvious ways.
> >
> > The easiest way to simplify the situation, to start off with, is to
> delete
> > or deprecate some of them as entry points. I deleted some in
> > https://gerrit.cloudera.org/#/c/12271/ and it felt good.
> >
> > The make_*.sh scripts are one candidate. I think the logic there should
> be
> > either moved to CMake (preferably) so we get proper dependency management
> > and a standard interface or become a function in buildall.sh. That way,
> > buildall.sh is the entry point if you want to do a full build or switch
> > build types and make/ninja is the entry point for incremental builds.
> > That's already my workflow and it works well.
> >
> > Does anyone use those make_*.sh scripts directly as part of their
> workflow?
> > If so, which ones and how attached are you to them?
> >
> > - Tim
> >
>

Re: Upcoming restart of jenkins.impala.io

[Philip Zeyliger]

Thanks for taking care of this!

P

On Wed, Jan 9, 2019 at 3:34 PM Pooja Nilangekar <
pooja.nilange...@cloudera.com> wrote:

> The update completed without any issues. I didn't have to kill any jobs
> this time. Please let me know if you come across any issues while
> starting/running jobs.
>
> Thanks,
> Pooja
>
>
> On Wed, Jan 9, 2019 at 9:33 AM Pooja Nilangekar <
> pooja.nilange...@cloudera.com> wrote:
>
> > HI all,
> >
> > I'm working on updating the one remaining plugin. Any new jobs will be
> > queued up. I'll send out an email once the update is complete. I
> apologize
> > for the inconvenience.
> >
> > Thanks,
> > Pooja
> >
> > On Tue, Jan 8, 2019 at 5:09 PM Pooja Nilangekar <
> > pooja.nilange...@cloudera.com> wrote:
> >
> >> Hi all,
> >>
> >> I have updated 2/3 plugins. I am going to wait for a day and ensure all
> >> the jobs run as expected and then upgrade the third plugin which might
> >> cause behavior changes. I have restarted any jobs I had to kill for the
> >> restart to complete. Please let me know if you face any issues.
> >>
> >> Thanks,
> >> Pooja
> >>
> >>
> >> On Tue, Jan 8, 2019 at 1:34 PM Pooja Nilangekar <
> >> pooja.nilange...@cloudera.com> wrote:
> >>
> >>> Hi all,
> >>>
> >>> Jenkins issued a security advisory announcing some security fixes. I'll
> >>> be restarting jenkins to install these fixes. I'll let current running
> jobs
> >>> complete. New jobs will not start until the upgrade is complete. I will
> >>> send out another email once the restart is complete.
> >>>
> >>> Thanks,
> >>> Pooja
> >>>
> >>
>

Re: New Impala PMC member - Zoltán Borók-Nagy

[Philip Zeyliger]

Congrats!

On Sun, Jan 6, 2019 at 10:30 AM Andrew Sherman 
wrote:

> Congratulations Zoltan!
>
> On Sat, Jan 5, 2019 at 8:49 AM Hubert Sun  wrote:
>
> > Congrats Zoltan!
> >
> > On Fri, Jan 4, 2019 at 11:14 PM Gabor Kaszab 
> > wrote:
> >
> > > wow! congrats, Zoli!
> > >
> > > On Sat, Jan 5, 2019 at 1:00 AM Quanlong Huang  >
> > > wrote:
> > >
> > > > Congratulations, Zoltan!
> > > >
> > > > -- Quanlong
> > > >
> > > > On Sat, Jan 5, 2019 at 4:01 AM Csaba Ringhofer <
> > csringho...@cloudera.com
> > > >
> > > > wrote:
> > > >
> > > > > Congratulations Zoli!
> > > > >
> > > > > On Fri, Jan 4, 2019 at 7:42 PM Zoram Thanga 
> > > wrote:
> > > > >
> > > > > > Congratulations, Zoltan!
> > > > > >
> > > > > > -Zoram
> > > > > >
> > > > > > On Fri, Jan 4, 2019 at 8:04 AM Tim Armstrong <
> > > tarmstr...@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > The Project Management Committee (PMC) for Apache Impala has
> > > invited
> > > > > > Zoltán
> > > > > > > Borók-Nagy to become a PMC member and we are pleased to
> announce
> > > that
> > > > > > they
> > > > > > > have accepted.
> > > > > > >
> > > > > > > Congratulations and welcome, Zoltán!
> > > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Zoram Thanga
> > > > > > Cloudera Inc.
> > > > > > 395 Page Mill Road
> > > > > > Palo Alto, CA 94306
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: jenkins.impala.io DNS expired

[Philip Zeyliger]

Thanks, Jim!

On Fri, Nov 16, 2018 at 1:57 PM Jim Apple  wrote:

> Fixed.
>
> On Thu, Nov 15, 2018 at 6:51 PM Jim Apple  wrote:
>
> > Please use 35.164.73.121 in the meantime while we get this fixed.
> >
>

Re: UBSAN in pre-merge testing

[Philip Zeyliger]

On Mon, Nov 12, 2018 at 1:21 PM Jim Apple  wrote:

> I don't think I understand the "one stone" part - are you suggesting that
> we do UBSAN testing within a centos6 container?
>

Exactly. If we're doing multiple builds, we may as well be mutating other
variables to get coverage from them. (Here, you're mutating "build type",
and I'm proposing we also mutate "base OS" to get additional coverage.)

-- Philip



>
> On Mon, Nov 12, 2018 at 1:01 PM Philip Zeyliger 
> wrote:
>
> > Seems useful to me.
> >
> > If you're interested, we could kill multiple birds with one stone.
> > Specifically, I'm also interested in centos6/rh6 pre-merge testing. There
> > are a variety of ways to do so, including running with test-with-docker
> > stuff. I recognize it's more work, but happy to help if you want to try
> it.
> >
> > -- Philip
> >
> > On Sat, Nov 10, 2018 at 11:10 PM Jim Apple  wrote:
> >
> > > C++ has some constructs that have undefined behavior. Shall we test for
> > > this during pre-merge testing?
> > >
> > > When the behavior of C++ code is formally "undefined" by the standard,
> > > compilers can behave erratically, like not taking either branch of a
> > > if/else statement. This can be reproduced in the wild. The standard
> > itself
> > > notes:
> > >
> > > "Using a bool value in ways described by this International Standard as
> > > 'undefined,' such as by examining the value of an uninitialized
> automatic
> > > object, might cause it to behave as if it is neither true nor false."
> > >
> > > Clang has a checker for this called UBSAN, and, after some effort, the
> > data
> > > loading part of our build is now UBSAN-clean. I'm suggesting we add
> that
> > > test to the pre-merge testing. I'm happy to handle the details.
> > >
> > > When it fails, the output will look something like this:
> > >
> > > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3573/console
> > >
> > > exprs/math-functions-ir.cc:405:13: runtime error: signed integer
> > overflow:
> > > 4738381338321616896 * 36 cannot be represented in type 'long'
> > > runtime/decimal-value.inline.h:254:17: runtime error: signed integer
> > > overflow: 0x4b3b4ca85a86c47a098a223f +
> > > 0x4b3b4ca85a86c47a098a223f cannot be represented in type
> > '__int128'
> > > runtime/row-batch-serialize-test.cc:243:18: runtime error: variable
> > length
> > > array bound evaluates to non-positive value 0
> > >
> >
>

Re: UBSAN in pre-merge testing

[Philip Zeyliger]

Seems useful to me.

If you're interested, we could kill multiple birds with one stone.
Specifically, I'm also interested in centos6/rh6 pre-merge testing. There
are a variety of ways to do so, including running with test-with-docker
stuff. I recognize it's more work, but happy to help if you want to try it.

-- Philip

On Sat, Nov 10, 2018 at 11:10 PM Jim Apple  wrote:

> C++ has some constructs that have undefined behavior. Shall we test for
> this during pre-merge testing?
>
> When the behavior of C++ code is formally "undefined" by the standard,
> compilers can behave erratically, like not taking either branch of a
> if/else statement. This can be reproduced in the wild. The standard itself
> notes:
>
> "Using a bool value in ways described by this International Standard as
> 'undefined,' such as by examining the value of an uninitialized automatic
> object, might cause it to behave as if it is neither true nor false."
>
> Clang has a checker for this called UBSAN, and, after some effort, the data
> loading part of our build is now UBSAN-clean. I'm suggesting we add that
> test to the pre-merge testing. I'm happy to handle the details.
>
> When it fails, the output will look something like this:
>
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/3573/console
>
> exprs/math-functions-ir.cc:405:13: runtime error: signed integer overflow:
> 4738381338321616896 * 36 cannot be represented in type 'long'
> runtime/decimal-value.inline.h:254:17: runtime error: signed integer
> overflow: 0x4b3b4ca85a86c47a098a223f +
> 0x4b3b4ca85a86c47a098a223f cannot be represented in type '__int128'
> runtime/row-batch-serialize-test.cc:243:18: runtime error: variable length
> array bound evaluates to non-positive value 0
>

Re: parallel-all-tests update

[Philip Zeyliger]

-everyone

I took advantage of your work on the packer template and wrote
http://github.mtv.cloudera.com/philip/Impala/commit/4c9786eb6455a13f442b771e9296f204a692c2c2
. I'll keep poking at it until it works. Right now it's getting as far as
most of the C++ build. Annoyingly, it's now at the "slow" part.

https://master-02.jenkins.cloudera.com/job/impala-test-with-docker-parameterized/123
is building it; we'll see whether it worked when I wake up :) I'm using
centos:6 as the base.

-- Philip



On Thu, Oct 11, 2018 at 1:15 PM Laszlo Gaal 
wrote:

> I'd be happy to pitch in; this is aligned with other test infra stuff I'm
> working on.
>
> On Thu, Oct 11, 2018 at 7:34 PM Philip Zeyliger 
> wrote:
>
> > I'm working a little bit (in the background) on getting
> > bin/bootstrap_system.sh to work. I'm somewhat optimistic that I can get
> it
> > to work :)
> >
> > I filed https://issues.apache.org/jira/browse/IMPALA-7698. Let me know
> if
> > you want to help dig in!
> >
> > -- Philip
> >
> > On Thu, Oct 11, 2018 at 10:16 AM Pooja Nilangekar <
> > pooja.nilange...@cloudera.com> wrote:
> >
> > > I agree with the approach of running the tests on supported platforms.
> In
> > > the past couple of days, there have been multiple issues (IMPALA-7678
> > > <https://issues.apache.org/jira/browse/IMPALA-7678> & IMPALA-7690
> > > <https://issues.apache.org/jira/browse/IMPALA-7690>) due to python2.6
> > > compatibility issues. Additionally, python2.6 might be one of the place
> > > where multiple linux distributions differ. So it probably makes sense
> to
> > > add another centos-from-scratch or similar, rather than adding ad-hoc
> > tests
> > > to address issues as they come up.
> > >
> > > Thanks,
> > > Pooja
> > >
> > >
> > > On Thu, Sep 27, 2018 at 2:43 PM Philip Zeyliger 
> > > wrote:
> > >
> > > > I think the approach to catch that is to just run the tests on the
> > > relevant
> > > > platform. (E.g., with test-with-docker or something to make the
> Jenkins
> > > > setup less painful.) Looking through some recent commits, I found:
> > > >
> > > > 304e02cf6238a3c1e9537337f802bcf7a92df5db "{}".format(1)
> > > > 9cfa228c2e itertools.count(start=1)
> > > > cf4f314922f13fbf54c6d7300ceaa2229bf5916a shutil.make_archive();
> > Multiple
> > > > with statements in the same context
> > > > d91bc44021f28a5d113285be9c920bd1a646a9b9 {dict_comprehension}
> > > > ??? a set comprehension; remember fixing it; couldn't find it.
> > > >
> > > > So, it's a mix of both classes, but I don't see recurring things that
> > > would
> > > > be easy to catch.
> > > >
> > > > -- Philip
> > > >
> > > >
> > > > On Thu, Sep 27, 2018 at 2:14 PM Tim Armstrong <
> tarmstr...@cloudera.com
> > >
> > > > wrote:
> > > >
> > > > > Is it worth adding some regexes or similar to the gerrit bot to
> catch
> > > > > itertools.count() usage? Or do we not expect repeated bugs of this
> > > form?
> > > > >
> > > > > On Thu, Sep 27, 2018 at 12:59 PM Tim Armstrong <
> > > tarmstr...@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > I also added it to the pre-review tests. Let me know if you see
> any
> > > > > issues.
> > > > > >
> > > > > > On Thu, Sep 27, 2018 at 12:33 PM Philip Zeyliger <
> > > phi...@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Hi folks,
> > > > > >>
> > > > > >> To address IMPALA-6543, there's a new test in parallel-all-tests
> > > that
> > > > > >> makes
> > > > > >> sure that any Python scripts use Python2.6-compatible syntax.
> Note
> > > > that
> > > > > >> this will catch the "try/catch/finally" style of bug, but not
> the
> > > > > >> "itertools.count(start=1)" kind of bug (python2.7 changed the
> > > > signature
> > > > > of
> > > > > >> itertools.count).
> > > > > >>
> > > > > >> The relevant diff was:
> > > > > >>
> > > > > >> $diff -u /tmp/b /tmp/a
> > > > > >> --- /tmp/b 2

Re: parallel-all-tests update

[Philip Zeyliger]

I'm working a little bit (in the background) on getting
bin/bootstrap_system.sh to work. I'm somewhat optimistic that I can get it
to work :)

I filed https://issues.apache.org/jira/browse/IMPALA-7698. Let me know if
you want to help dig in!

-- Philip

On Thu, Oct 11, 2018 at 10:16 AM Pooja Nilangekar <
pooja.nilange...@cloudera.com> wrote:

> I agree with the approach of running the tests on supported platforms. In
> the past couple of days, there have been multiple issues (IMPALA-7678
> <https://issues.apache.org/jira/browse/IMPALA-7678> & IMPALA-7690
> <https://issues.apache.org/jira/browse/IMPALA-7690>) due to python2.6
> compatibility issues. Additionally, python2.6 might be one of the place
> where multiple linux distributions differ. So it probably makes sense to
> add another centos-from-scratch or similar, rather than adding ad-hoc tests
> to address issues as they come up.
>
> Thanks,
> Pooja
>
>
> On Thu, Sep 27, 2018 at 2:43 PM Philip Zeyliger 
> wrote:
>
> > I think the approach to catch that is to just run the tests on the
> relevant
> > platform. (E.g., with test-with-docker or something to make the Jenkins
> > setup less painful.) Looking through some recent commits, I found:
> >
> > 304e02cf6238a3c1e9537337f802bcf7a92df5db "{}".format(1)
> > 9cfa228c2e itertools.count(start=1)
> > cf4f314922f13fbf54c6d7300ceaa2229bf5916a shutil.make_archive();  Multiple
> > with statements in the same context
> > d91bc44021f28a5d113285be9c920bd1a646a9b9 {dict_comprehension}
> > ??? a set comprehension; remember fixing it; couldn't find it.
> >
> > So, it's a mix of both classes, but I don't see recurring things that
> would
> > be easy to catch.
> >
> > -- Philip
> >
> >
> > On Thu, Sep 27, 2018 at 2:14 PM Tim Armstrong 
> > wrote:
> >
> > > Is it worth adding some regexes or similar to the gerrit bot to catch
> > > itertools.count() usage? Or do we not expect repeated bugs of this
> form?
> > >
> > > On Thu, Sep 27, 2018 at 12:59 PM Tim Armstrong <
> tarmstr...@cloudera.com>
> > > wrote:
> > >
> > > > I also added it to the pre-review tests. Let me know if you see any
> > > issues.
> > > >
> > > > On Thu, Sep 27, 2018 at 12:33 PM Philip Zeyliger <
> phi...@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi folks,
> > > >>
> > > >> To address IMPALA-6543, there's a new test in parallel-all-tests
> that
> > > >> makes
> > > >> sure that any Python scripts use Python2.6-compatible syntax. Note
> > that
> > > >> this will catch the "try/catch/finally" style of bug, but not the
> > > >> "itertools.count(start=1)" kind of bug (python2.7 changed the
> > signature
> > > of
> > > >> itertools.count).
> > > >>
> > > >> The relevant diff was:
> > > >>
> > > >> $diff -u /tmp/b /tmp/a
> > > >> --- /tmp/b 2018-09-27 12:29:15.0 -0700
> > > >> +++ /tmp/a 2018-09-27 12:28:58.0 -0700
> > > >> @@ -23,6 +23,14 @@
> > > >>  if (restring != null && !restring.equals("SUCCESS")) {
> > > >>  failed_job_urls.add(result.getAbsoluteUrl())
> > > >>  }
> > > >> +}, Python26Compatibility: {
> > > >> +result = build job: 'python26-incompatibility-check',
> propagate:
> > > >> false, parameters:
> > > >> +[string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > >> + string(name: 'IMPALA_REPO_BRANCH', value: IMPALA_REPO_BRANCH)]
> > > >> +restring = result.getResult()
> > > >> +if (restring != null && !restring.equals("SUCCESS")) {
> > > >> +failed_job_urls.add(result.getAbsoluteUrl())
> > > >> +}
> > > >>  }, TidyAndBuildOnlyAndRat: {
> > > >>  result = build job: 'clang-tidy-ub1604', propagate: false,
> > > >> parameters:
> > > >>  [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> > > >>
> > > >> This will show up in your GVO builds. I've tested it, but of course
> > let
> > > me
> > > >> know if you run into any trouble.
> > > >>
> > > >> Thanks!
> > > >>
> > > >
> > >
> >
>

Re: parallel-all-tests update

[Philip Zeyliger]

I think the approach to catch that is to just run the tests on the relevant
platform. (E.g., with test-with-docker or something to make the Jenkins
setup less painful.) Looking through some recent commits, I found:

304e02cf6238a3c1e9537337f802bcf7a92df5db "{}".format(1)
9cfa228c2e itertools.count(start=1)
cf4f314922f13fbf54c6d7300ceaa2229bf5916a shutil.make_archive();  Multiple
with statements in the same context
d91bc44021f28a5d113285be9c920bd1a646a9b9 {dict_comprehension}
??? a set comprehension; remember fixing it; couldn't find it.

So, it's a mix of both classes, but I don't see recurring things that would
be easy to catch.

-- Philip


On Thu, Sep 27, 2018 at 2:14 PM Tim Armstrong 
wrote:

> Is it worth adding some regexes or similar to the gerrit bot to catch
> itertools.count() usage? Or do we not expect repeated bugs of this form?
>
> On Thu, Sep 27, 2018 at 12:59 PM Tim Armstrong 
> wrote:
>
> > I also added it to the pre-review tests. Let me know if you see any
> issues.
> >
> > On Thu, Sep 27, 2018 at 12:33 PM Philip Zeyliger 
> > wrote:
> >
> >> Hi folks,
> >>
> >> To address IMPALA-6543, there's a new test in parallel-all-tests that
> >> makes
> >> sure that any Python scripts use Python2.6-compatible syntax. Note that
> >> this will catch the "try/catch/finally" style of bug, but not the
> >> "itertools.count(start=1)" kind of bug (python2.7 changed the signature
> of
> >> itertools.count).
> >>
> >> The relevant diff was:
> >>
> >> $diff -u /tmp/b /tmp/a
> >> --- /tmp/b 2018-09-27 12:29:15.0 -0700
> >> +++ /tmp/a 2018-09-27 12:28:58.0 -0700
> >> @@ -23,6 +23,14 @@
> >>  if (restring != null && !restring.equals("SUCCESS")) {
> >>  failed_job_urls.add(result.getAbsoluteUrl())
> >>  }
> >> +}, Python26Compatibility: {
> >> +result = build job: 'python26-incompatibility-check', propagate:
> >> false, parameters:
> >> +[string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> >> + string(name: 'IMPALA_REPO_BRANCH', value: IMPALA_REPO_BRANCH)]
> >> +restring = result.getResult()
> >> +if (restring != null && !restring.equals("SUCCESS")) {
> >> +failed_job_urls.add(result.getAbsoluteUrl())
> >> +}
> >>  }, TidyAndBuildOnlyAndRat: {
> >>  result = build job: 'clang-tidy-ub1604', propagate: false,
> >> parameters:
> >>  [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
> >>
> >> This will show up in your GVO builds. I've tested it, but of course let
> me
> >> know if you run into any trouble.
> >>
> >> Thanks!
> >>
> >
>

parallel-all-tests update

[Philip Zeyliger]

Hi folks,

To address IMPALA-6543, there's a new test in parallel-all-tests that makes
sure that any Python scripts use Python2.6-compatible syntax. Note that
this will catch the "try/catch/finally" style of bug, but not the
"itertools.count(start=1)" kind of bug (python2.7 changed the signature of
itertools.count).

The relevant diff was:

$diff -u /tmp/b /tmp/a
--- /tmp/b 2018-09-27 12:29:15.0 -0700
+++ /tmp/a 2018-09-27 12:28:58.0 -0700
@@ -23,6 +23,14 @@
 if (restring != null && !restring.equals("SUCCESS")) {
 failed_job_urls.add(result.getAbsoluteUrl())
 }
+}, Python26Compatibility: {
+result = build job: 'python26-incompatibility-check', propagate:
false, parameters:
+[string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),
+ string(name: 'IMPALA_REPO_BRANCH', value: IMPALA_REPO_BRANCH)]
+restring = result.getResult()
+if (restring != null && !restring.equals("SUCCESS")) {
+failed_job_urls.add(result.getAbsoluteUrl())
+}
 }, TidyAndBuildOnlyAndRat: {
 result = build job: 'clang-tidy-ub1604', propagate: false, parameters:
 [string(name: 'IMPALA_REPO_URL', value: IMPALA_REPO_URL),

This will show up in your GVO builds. I've tested it, but of course let me
know if you run into any trouble.

Thanks!

Re: Confluence access

[Philip Zeyliger]

I think I granted you access; try now!

On Tue, Sep 18, 2018 at 3:04 PM Paul Rogers  wrote:

> Oh, Wiki user name is “paul-rogers” in case the e-mail is not sufficient.
>
> Thanks,
>
> - Paul
>
> > On Sep 18, 2018, at 3:03 PM, Paul Rogers  wrote:
> >
> > Hi All,
> >
> > I’m a new Impala developer working my way though a development
> environment setup. Have few corrections for the Wiki. Can someone grant me
> write access please?
> >
> > Thanks!
> >
> > - Paul
>
>

Re: FE build failure

[Philip Zeyliger]

Using 'mvn -U' seems to have resolved the issue for Michael. Probably
blowing away ~/.m2 would have as well.

On Wed, Sep 5, 2018 at 1:33 PM Philip Zeyliger  wrote:

> > org.apache.hbase:hbase-procedure:jar:2.0.0-cdh6.x-SNAPSHOT
>
> What command are you trying to build with?
>
> If I had to guess, you're re-building "fe" but haven't rebuilt
> "impala-parent". But it could also be something else, of course. One thing
> you could do is blow away ~/.m2/repository/org/apache, or run with "mvn
> -U", or run a clean build, and so on.
>
> If you'd like to, I'm happy to screen share with you and we can pinpoint
> the problem.
>
> -- Philip
>
> On Wed, Sep 5, 2018 at 1:30 PM Michael Ho  wrote:
>
>> Thanks Philip. I am seeing this on my own machine so far. Also started a
>> private build on some AWS instances to see if it's specific to my
>> environment. The following is the presumably interesting part of maven
>> output. Not sure if my toolchain bootstrapping screwed up or something and
>> I removed toolchain/cdh-components* before retrying. It's complaining
>> about
>> hbase if I understand the output correctly:
>>
>> [DEBUG] Adding failure due to exception
>> org.apache.maven.enforcer.rule.api.EnforcerRuleException: Found Banned
>> Dependency: org.apache.hbase:hbase-procedure:jar:2.0.0-cdh6.x-SNAPSHOT
>> Found Banned Dependency:
>> org.apache.hbase:hbase-zookeeper:jar:2.0.0-cdh6.x-SNAPSHOT
>> Found Banned Dependency:
>> org.apache.hbase:hbase-replication:jar:2.0.0-cdh6.x-SNAPSHOT
>> Found Banned Dependency:
>> org.apache.hbase:hbase-mapreduce:jar:2.0.0-cdh6.x-SNAPSHOT
>> Found Banned Dependency:
>> org.apache.hbase:hbase-server:jar:2.0.0-cdh6.x-SNAPSHOT
>> Found Banned Dependency:
>> org.apache.hbase:hbase-hadoop2-compat:jar:tests:2.0.0-cdh6.x-SNAPSHOT
>> Found Banned Dependency:
>> org.apache.hbase:hbase-http:jar:2.0.0-cdh6.x-SNAPSHOT
>> Found Banned Dependency:
>> org.apache.hbase:hbase-common:jar:tests:2.0.0-cdh6.x-SNAPSHOT
>> Use 'mvn dependency:tree' to locate the source of the banned dependencies.
>> at
>>
>> org.apache.maven.plugins.enforcer.AbstractBanDependencies.execute(AbstractBanDependencies.java:107)
>> at
>>
>> org.apache.maven.plugins.enforcer.EnforceMojo.execute(EnforceMojo.java:202)
>> at
>>
>> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:101)
>> at
>>
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:209)
>> at
>>
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
>> at
>>
>> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
>> at
>>
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:84)
>> at
>>
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:59)
>> at
>>
>> org.apache.maven.lifecycle.internal.LifecycleStarter.singleThreadedBuild(LifecycleStarter.java:183)
>> at
>>
>> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:161)
>> at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:320)
>> at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:156)
>> at org.apache.maven.cli.MavenCli.execute(MavenCli.java:537)
>> at org.apache.maven.cli.MavenCli.doMain(MavenCli.java:196)
>> at org.apache.maven.cli.MavenCli.main(MavenCli.java:141)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:498)
>> at
>>
>> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
>> at
>>
>> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
>> at
>>
>> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
>> at
>> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
>> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies
>> failed with message:
>> Found Banned Dependency:
>> org.apache.hbase:hbase-procedure:jar:2.0.0-cdh6.x-SNAPSHOT
>> Found Banned Dependency:
>> org.apache.hbase:hbase-zookeeper:jar:2.0.0-cdh6.x-SNAPSHOT
>> Found Banned Dependency:
>

Re: FE build failure

[Philip Zeyliger]

pendencies.
> [INFO]
> 
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 4.335s
> [INFO] Finished at: Wed Sep 05 13:26:27 PDT 2018
> [INFO] Final Memory: 35M/859M
> [INFO]
> 
>
> On Wed, Sep 5, 2018 at 1:21 PM Philip Zeyliger 
> wrote:
>
> > Are you seeing this on your own machine? If so, try to add "-X" to the
> mvn
> > execution and capture the (very large) log. I'd be happy to take a look.
> >
> > -- Philip
> >
> > On Wed, Sep 5, 2018 at 12:34 PM Michael Ho  wrote:
> >
> > > Hi all,
> > >
> > > I tried a clean build this morning and kept running into the following
> > > errors when building the FE code:
> > >
> > > [WARNING] Could not transfer metadata
> > > com.cloudera.cdh:cdh-root:6.x-SNAPSHOT/maven-metadata.xml from/to
> > > ${distMgmtSnapshotsId} (${distMgmtSnapshotsUrl}): No connector
> available
> > to
> > > access repository ${distMgmtSnapshotsId} (${distMgmtSnapshotsUrl}) of
> > type
> > > default using the available factories WagonRepositoryConnectorFactory
> > > [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies
> > > failed with message:
> > > [INFO] BUILD FAILURE
> > > [ERROR] Failed to execute goal
> > > org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce
> > > (enforce-banned-dependencies) on project impala-frontend: Some Enforcer
> > > rules have failed. Look above for specific messages explaining why the
> > rule
> > > failed. -> [Help 1]
> > > [ERROR]
> > > [ERROR] To see the full stack trace of the errors, re-run Maven with
> the
> > -e
> > > switch.
> > > [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> > > [ERROR]
> > > [ERROR] For more information about the errors and possible solutions,
> > > please read the following articles:
> > > [ERROR] [Help 1]
> > >
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> > > mvn  -B install -DskipTests exited with code 0
> > > make[3]: *** [fe/CMakeFiles/fe] Error 1
> > > make[2]: *** [fe/CMakeFiles/fe.dir/all] Error 2
> > > make[1]: *** [fe/CMakeFiles/fe.dir/rule] Error 2
> > > make: *** [fe] Error 2
> > >
> > > I believe Bharath filed
> > https://issues.apache.org/jira/browse/IMPALA-7526
> > > for this. Philip mentioned in past code review that log4j2 is the
> > offender.
> > > Did anyone else run into it ?
> > >
> > > --
> > > Thanks,
> > > Michael
> > >
> >
>
>
> --
> Thanks,
> Michael
>

Re: FE build failure

[Philip Zeyliger]

Are you seeing this on your own machine? If so, try to add "-X" to the mvn
execution and capture the (very large) log. I'd be happy to take a look.

-- Philip

On Wed, Sep 5, 2018 at 12:34 PM Michael Ho  wrote:

> Hi all,
>
> I tried a clean build this morning and kept running into the following
> errors when building the FE code:
>
> [WARNING] Could not transfer metadata
> com.cloudera.cdh:cdh-root:6.x-SNAPSHOT/maven-metadata.xml from/to
> ${distMgmtSnapshotsId} (${distMgmtSnapshotsUrl}): No connector available to
> access repository ${distMgmtSnapshotsId} (${distMgmtSnapshotsUrl}) of type
> default using the available factories WagonRepositoryConnectorFactory
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.BannedDependencies
> failed with message:
> [INFO] BUILD FAILURE
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0-M1:enforce
> (enforce-banned-dependencies) on project impala-frontend: Some Enforcer
> rules have failed. Look above for specific messages explaining why the rule
> failed. -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions,
> please read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> mvn  -B install -DskipTests exited with code 0
> make[3]: *** [fe/CMakeFiles/fe] Error 1
> make[2]: *** [fe/CMakeFiles/fe.dir/all] Error 2
> make[1]: *** [fe/CMakeFiles/fe.dir/rule] Error 2
> make: *** [fe] Error 2
>
> I believe Bharath filed https://issues.apache.org/jira/browse/IMPALA-7526
> for this. Philip mentioned in past code review that log4j2 is the offender.
> Did anyone else run into it ?
>
> --
> Thanks,
> Michael
>

Re: Adding profile info from Frontend

[Philip Zeyliger]

I'm certainly aware of tools that read TEventSequence and expect it to have
certain structures. If we change the profiles significantly, we should
insert a version sigil in there so that a client could at least figure out
that they're looking at something new.

A bigger modeling conversation is the use of string keys everywhere. For
the timeline, for example, it's a list of (description, time) pairs, with
the descriptions being strings. An alternative would have been to put the
strings in Thrift itself. The latter, I think, makes tooling easier to
write (since you're calling "get_query_started_time()" instead of "for
description, time in timelines: if description == 'query started': return
time". I recognize working with Thrift is kind of a pain, but I want to
throw the thought out there if we're adding things to profiles.

-- Philip


On Tue, Sep 4, 2018 at 4:02 PM Todd Lipcon  wrote:

> Hey folks,
>
> I'm working on a patch to add some more diagnostics from the planning
> process into query profiles.
>
> Currently, only the planning "Timeline" is reported back as part of the
> response to createExecRequest. As part of the fetch-on-demand catalog work
> I'd like to start exposing various counters such as cache hit/miss counts,
> time spent on remote calls to the catalog, etc. Even in the existing code
> paths, I can see some similar metrics being useful.
>
> My current thinking is to remove the 'timeline'  (TEventSequence) field
> from TExecRequest and replace it with a full TRuntimeProfileNode. I'd then
> add some capability in the Java side to fill in counters, etc, in this
> structure.
>
> Any concerns with this approach before I go down this path? Are there any
> compatibility guarantees I need to uphold with the profile output of
> queries?
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: Impalad JVM OOM minutes after restart

[Philip Zeyliger]

Hi Brock,

If you want to make Eclipse MAT more usable, set JAVA_TOOL_OPTIONS="-Xmx2g
 -XX:+HeapDumpOnOutOfMemoryError" and you should see the max heap at 2GB,
thereby making Eclipse MAT friendlier. Folks have also been using
http://www.jxray.com/.

The query itself will also be interesting. If there's something like an
loop in analyzing it, you could imagine that showing up as an OOM. The heap
dump should tell us.

-- Philip

On Tue, Aug 21, 2018 at 11:32 AM Brock Noland  wrote:

> Hi Jeezy,
>
> Thanks, good tip.
>
> The MS is quite small. Even mysqldump format is only 12MB. The largest
> catalog-update I could find is only 1.5MB which should be easy to
> process with 32GB of of heap. Lastly, it's possible we can reproduce
> by running the query the impalad was processing during the issue,
> going to wait until after the users head home to try, but it doesn't
> appear reproducible in the method you describe. When we restarted, it
> did not reproduce until users started running queries.
>
> I0820 19:45:25.106437 25474 statestore.cc:568] Preparing initial
> catalog-update topic update for impalad@XXX:22000. Size = 1.45 MB
>
> Brock
>
> On Tue, Aug 21, 2018 at 1:18 PM, Jeszy  wrote:
> > Hey,
> >
> > If it happens shortly after a restart, there is a fair chance you're
> > crashing while processing the initial catalog topic update. Statestore
> > logs will tell you how big that was (it takes more memory to process
> > it than the actual size of the update).
> > If this is the case, it should also be reproducible, ie. the daemon
> > will keep restarting and running OOM on initial update until you clear
> > the metadata cache either by restarting catalog or via a (global)
> > invalidate metadata.
> >
> > HTH
> > On Tue, 21 Aug 2018 at 20:13, Brock Noland  wrote:
> >>
> >> Hi folks,
> >>
> >> I've got an Impala CDH 5.14.2 cluster with a handful of users, 2-3, at
> >> any one time. All of a sudden the JVM inside the Impalad started
> >> running out of memory.
> >>
> >> I got a heap dump, but the heap was 32GB, host is 240GB, so it's very
> >> large. Thus I wasn't able to get Memory Analyzer Tool (MAT) to open
> >> it. I was able to get JHAT to opening it when setting JHAT's heap to
> >> 160GB. It's pretty unwieldy so much of the JHAT functionality doesn't
> >> work.
> >>
> >> I am spelunking around, but really curious if there is some places I
> >> should check
> >>
> >> I am only an occasional reader of Impala source so I am just pointing
> >> out things which felt interesting:
> >>
> >> * Impalad was restarted shortly before the JVM OOM
> >> * Joining Parquet on S3 with Kudu
> >> * Only 13  instances of org.apache.impala.catalog.HdfsTable
> >> * 176836 instances of org.apache.impala.analysis.Analyzer - this feels
> >> odd to me. I remember one bug a while back in Hive when it would clone
> >> the query tree until it ran OOM.
> >> * 176796 of those _user fields point at the same user
> >> * org.apache.impala.thrift.TQueryCt@0x7f90975297f8 has 11048
> >> org.apache.impala.analysis.Analyzer@GlobalState objects pointing at
> >> it.
> >> *  There is only a single instance of
> >> org.apache.impala.thrift.TQueryCtx alive in the JVM which appears to
> >> indicate there is only a single query running. I've tracked that query
> >> down in CM. The users need to compute stats, but I don't feel that is
> >> relevant to this JVM OOM condition.
> >>
> >> Any pointers on what I might look for?
> >>
> >> Cheers,
> >> Brock
>

Re: New Impala committer - Quanlong Huang

[Philip Zeyliger]

Congrats!

On Fri, Aug 17, 2018 at 9:29 AM Tim Armstrong 
wrote:

>  The Project Management Committee (PMC) for Apache Impala has invited
> Quanlong Huang to become a committer and we are pleased to announce that
> they have accepted. Congratulations and welcome, Quanlong Huang!
>

Re: hadoop.fs.Path impacts on Impala

[Philip Zeyliger]

Hi Barnabas,

If I may suggest a way to approach this sort of question, I'd take a
heapdump of an impalad and a catalogd (using "jmap") and then use Eclipse
MAT or http://www.jxray.com/ to see if we're using Path. You'll want to
load some tables and partitions ahead of time. Based on a little quick
sleuthing (I'm not well-versed in this area of the
code), fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java seems
to use flat buffers to store these. It's likely we don't use the HDFS Path
object in steady state.

I did a quick look on a cluster we have lying around and found negligible
use at that moment, but I'm not totally confident about what's on that
cluster at the moment.

[root@... philip]# sudo -u impala /usr/java/jdk1.8.0_111/bin/jmap -histo
47116 > /tmp/histo
[root@ philip]# cat /tmp/histo | grep Path
  85:   247  13832  sun.misc.URLClassPath$JarLoader
 567: 3144  sun.misc.URLClassPath
 568: 6144  sun.misc.URLClassPath$FileLoader
 724: 4 96
sun.security.provider.certpath.X509CertPath
 762: 2 80  sun.misc.URLClassPath$1
* 893: 4 64  org.apache.hadoop.fs.Path*
 927: 2 64  sun.nio.fs.UnixPath
1030: 2 48  java.io.File$PathStatus
1211: 1 40
org.apache.hadoop.hdfs.protocol.proto.EncryptionZonesProtos$GetEZForPathRequestProto
1226: 1 40  sun.misc.URLClassPath$2
1411: 1 24  [Ljava.io.File$PathStatus;
1470: 1 24
com.sun.org.apache.bcel.internal.util.ClassPath
1645: 1 16
[Lcom.sun.org.apache.bcel.internal.util.ClassPath$PathEntry;
2035: 1 16
org.apache.hadoop.hdfs.protocol.proto.EncryptionZonesProtos$GetEZForPathRequestProto$1
[root@... philip]# head /tmp/histo

 num #instances #bytes  class name
--
   1:416952   81879632  [B
   2:   1324260   42376320  com.codahale.metrics.LongAdder
   3:794556   38138688  com.codahale.metrics.EWMA
   4:   1060844   25460256  java.util.concurrent.atomic.AtomicLong
   5:264852   14831712
com.codahale.metrics.ExponentiallyDecayingReservoir
   6:264852   12712896  com.codahale.metrics.Meter
   7:264852   12712896
java.util.concurrent.ConcurrentSkipListMap



-- Philip


On Wed, Aug 1, 2018 at 8:19 AM Barnabás Maidics
 wrote:

> On Wed, Aug 1, 2018 at 11:17 AM Barnabás Maidics <
> barnabas.maid...@cloudera.com> wrote:
>
> > Hi Everyone!
> >
> > I'm an intern at Cloudera and analysing where the memory goes in Hive. I
> > was looking at a heapdump with many partitions, and found a memory waste,
> > that comes from HDFS.
> >
> > We store paths in hadoop.fs.Path objects. This uses java.net.URI that
> > stores almost the same strings in 3 different objects (see image and
> > further explanation at the link given below). I think it's a waste of
> > memory and it could be reduced by replacing the URI objects. This is why
> > I've created an issue on HDFS side (HDFS-13752
> > ).
> >
> > I'm curious if you store these objects (hadoop.fs.Path), and if you do
> how
> > much it effects the overall memory usage of Impala. It may be beneficial
> > for you as well, if it can be replaced.
> >
> > Thanks,
> >
> > Barnabas Maidics
> >
> >
>

Re: Impala Incident Sharing

[Philip Zeyliger]

Hi Quanlong,

Thank you for the incident note! You might be interested in
https://gerrit.cloudera.org/#/c/10998/ which is adding some instrumentation
to make it easier to notice with monitoring tools that we're running out of
memory or having large GC pauses.

-- Philip

On Tue, Jul 31, 2018 at 8:21 AM Todd Lipcon 
wrote:

> Hi Quanlong,
>
> Thanks for the writeup. I wonder if we could two of the issues with a
> single approach of using a separate ClassLoader instance when we load and
> instantiate custom UDFs. For one, using a custom ClassLoader means that we
> could probably ensure that the user's jar is checked first even if there is
> a conflicting class on the Impala classpath. For two, if we close the
> custom classloader and ensure that we don't retain any references to it or
> to classes created by it (i.e the UDF classes), then we should be able to
> collect leaked memory on the next GC. In the particular case you pointed
> out, the leak is in static fields of the UDF, and those static fields would
> get destructed when destructing the classloader.
>
> On the downside, making this change could negatively harm performance if
> any UDFs are relying on their static initializer blocks to do expensive
> setup. Maybe a compromise would be to make this behavior optional or to
> cache the ClassLoader via a WeakReference across instantiations of a UDF so
> that, if there isn't memory pressure, the UDF doesn't need to be
> re-class-loaded repeatedly.
>
> As for tracking memory explicitly, the way that HBase does it is via
> somewhat careful accounting inline with all of the code that allocates
> memory. I don't think that approach is feasible in UDFs because the UDF
> code is provided by end users, and they aren't likely to want to carefully
> track their heap usage. I think the only way we could effectively "sandbox"
> UDFs would be to actually execute them in a separate JVM with a limited
> heap size, but that would be somewhat complex to implement without a big
> performance drop.
>
> -Todd
>
> On Tue, Jul 31, 2018 at 7:53 AM, Quanlong Huang 
> wrote:
>
> > Hi all,
> >
> >
> > Just to draw your attention of a patch, I'd like to share an incident
> with
> > you.
> >
> >
> > Two months ago we encountered an incident in one of our Impala clusters
> in
> > version 2.5-cdh5.7.3. Queries were pilled up and most of them were in
> > CREATED state. It looks like an infamous incident when the catalogd is
> > loading metadata of huge tables and finally OOM so metadata loading
> > requests from impalad are stuck. However, the catalogd looks good this
> > time. The logs showed it's not loading any metadata. Then I looked into
> the
> > statestored and found many timeout logs like "Unable to send topic update
> > message to subscriber impalad@xxx:22000, received error: RPC timed out".
> > Chose one of the problematic impalad and dug into its log, I found most
> of
> > them are OOM. Logs look like:
> >
> >
> > E0524 13:46:46.200798  4405impala-server.cc:1344] There was an error
> > processing the impalad catalog update. Requesting a full topic update to
> > recover: OutOfMemoryError: Java heap space
> > at com.cloudera.impala.hive.executor.UdfExecutor.evaluate(
> > UdfExecutor.java:402)
> > at com.cloudera.impala.hive.executor.UdfExecutor.evaluate(
> > UdfExecutor.java:329)
> > Caused by: java.lang.reflect.InvocationTargetException
> > at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
> > at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> > DelegatingMethodAccessorImpl.java:43)E0524 13:49:06.433274
> > 4405impala-server.cc:1344] There was an error processing the impalad
> > catalog update. Requesting a full topic update to recover:
> > OutOfMemoryError: Java heap space
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at com.cloudera.impala.hive.executor.UdfExecutor.evaluate(
> > UdfExecutor.java:394)
> > ... 1 more
> > Caused by: java.lang.OutOfMemoryError: Java heap space
> > E0524 13:56:44.460489  4405impala-server.cc:1344] There was an error
> > processing the impalad catalog update. Requesting a full topic update to
> > recover: OutOfMemoryError: Java heap space
> > E0524 14:03:45.630472  4405impala-server.cc:1344] There was an error
> > processing the impalad catalog update. Requesting a full topic update to
> > recover: OutOfMemoryError: Java heap space
> >
> >
> > At that time I can only restart the whole cluster since most of the
> > impalads were showing OOM errors. It's quite painful but after that the
> > cluster recovered.
> >
> >
> > After I dug into the query history and replayed the queries in a test
> > cluster, I realized it's caused by a UDF we added one day before. Our
> users
> > want to parse JSON strings, but there're no builtin functions for this
> > (IMPALA-376). So we simply added the Hive UDF get_json_object by
> >
> >
> > create function get_json_object location
> 'hdfs://xxx/hive-exec-1.1.0-cdh5.7.3.jar'
> > 

Re: impalad running as superuser in tests

[Philip Zeyliger]

Hi Todd,

You're right: the current minicluster setup makes it impossible to test
some cases.

I think your approach of telling Hadoop a different user is sensible,
though might be a rabbit hole in terms of getting Hive and HBase and
everything happy.

I know we are able to run some tests against clusters deployed on VM. There
is some support for this already. I'm not sure we have an extant annotation
for it, but you could write tests targeting that setup.

Thanks,

-- Philip

On Wed, Jul 18, 2018 at 10:22 AM Todd Lipcon 
wrote:

> On Tue, Jul 17, 2018 at 5:27 PM, Sailesh Mukil
>  > wrote:
>
> > On Tue, Jul 17, 2018 at 2:47 PM, Todd Lipcon 
> > wrote:
> >
> > > Hey folks,
> > >
> > > I'm working on a regression test for IMPALA-7311 and found something
> > > interesting. It appears that in our normal minicluster setup, impalad
> > runs
> > > as the same username as the namenode (namely, the username of the
> > > developer, in my case 'todd').
> > >
> > > This means that the NN treats impala as a superuser, and therefore
> > doesn't
> > > actually enforce permissions. So, tests about the behavior of Impala on
> > > files that it doesn't have access to are somewhat tricky to write.
> > >
> > >
> > What kind of files do you specifically mean? Something that the daemon
> > tries to access directly (Eg: keytab file, log files, etc.) ? I'm
> guessing
> > it's not this since you mentioned the NN.
> >
> > Or files that belong to a table/partition in HDFS? If it's this case, we
> > would go through Sentry before accessing files that belong to a table,
> and
> > access would be determined by Sentry on the "session user" (not the
> impalad
> > user) before Impala even tries to access HDFS. (Eg:
> > tests/authorization/test_authorization.py)
> >
>
> Right, files on HDFS. I mean that, in cases where Sentry is not enabled or
> set up, and even in some cases where it is set up but not synchronized with
> HDFS, it's possible that the user can point table metadata at files or
> directories that aren't writable to the 'impala' user on HDFS. For example,
> I can do:
>
> CREATE EXTERNAL TABLE foo (...) LOCATION '/user/todd/my-dir';
>
> and it's likely that 'my-dir' is not writable by 'impala' on a real
> cluster. Thus, if I try to insert into it, I get an error because "impala"
> does not have HDFS permissions to access this directory.
>
> Currently, the frontend does some checks here to try to produce a nice
> error. But, those checks are based on cached metadata which could be in
> accurate. In the case that it's inaccurate, the error will be thrown from
> the backend when it tries to create a file in a non-writable location.
>
> In the minicluster environment, it's impossible to test this case (actual
> permissions enforced by the NN causing an error) because the backend is
> running as an HDFS superuser. That is to say, it has full permissions
> everywhere. That's due to the special case behavior that HDFS has: it
> determines the name of the superuser to be the username that is running the
> NN. Since in the minicluster, both impala and the NN run as 'todd' in my
> case, impala acts as superuser. In a real cluster (even with security
> disabled) impala typically runs as 'impala' whereas the NN runs as 'hdfs'
> and thus impala does not have superuser privileges.
>
> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: Removing IMPALA_MINICLUSTER_PROFILE=2 from master

[Philip Zeyliger]

I'm +1 on removing it.

On Wed, Jul 11, 2018 at 9:39 AM Sailesh Mukil 
wrote:

> +1 for removing it.
>
> I'm all for reduced complexity if not a lot of users are benefitting from
> it.
>
> On Wed, Jul 11, 2018 at 9:29 AM, Fredy Wijaya  >
> wrote:
>
> > Hi all,
> >
> > In the master branch, we support both Hadoop 2 and Hadoop 3 via
> > IMPALA_MINICLUSTER_PROFILE environment variable. When Impala transitioned
> > from Hadoop 2 to Hadoop 3, we introduced IMPALA_MINICLUSTER_PROFILE as a
> > way to easily switch between Hadoop 2 and Hadoop 3. It made sense at that
> > time. However, I believe the IMPALA_MINICLUSTER_PROFILE=2 has outlived
> its
> > usefulness and has grown more complex to maintain given that we need to
> > introduce shims, feature flags, additional Jenkins jobs, etc. Given that
> we
> > already have 2.x branch for Impala with Hadoop 2 support. I'm proposing
> > that we remove IMPALA_MINICLUSTER_PROFILE=2 in the master branch so that
> we
> > can focus our effort on supporting Hadoop 3 on the master branch.
> >
> > What do others think?
> >
>

Re: Jenkins scripts in repo?

[Philip Zeyliger]

If it's helpful, I'd be happy to send you, off-list, the XML configuration
file (which is Jenkins' serialized format) and/or screenshots of any of our
jobs. It's a little bit of work because I have to make sure they're devoid
of passwords and such.

Thanks!

-- Philip

On Wed, Jul 11, 2018 at 8:16 AM Michael Brown 
wrote:

> That's just the standard Jenkins plugin to read Junit XML. I think it comes
> with a standard Jenkins install on Ubuntu 16.
>
> https://plugins.jenkins.io/junit
>
> Most of our tests already emit Junit XML, so that's taken care of for you
> already.
>
> To use the plugin, you just tell it a series of paths to find find Junit
> XML. For ubuntu-16.04-from-scratch, it's "Impala/logs_static/**/*.xml".
>
>
>
> On Tue, Jul 10, 2018 at 7:19 PM, Quanlong Huang 
> wrote:
>
> > Thank you, Jim! That's really helpful!
> >
> >
> > Could you share more about the fancy test report page? For example,
> > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/
> > lastCompletedBuild/testReport/
> >
> >
> > I think you may have spent some time integrating the Jenkins Test Results
> > plugin with the Impala test framework. It would save us most efforts if
> you
> > can share something about it.
> >
> >
> > Thanks,
> > Quanlong
> > At 2018-07-10 23:40:26, "Jim Apple" 
> wrote:
> > >side note: please change the subject line for new topics; it will help
> > >organize inboxes and list archives.
> > >
> > >The Jenkins jobs mostly use components in the repo. In particular, the
> > >following scrips are each used in one of our core Jenkins jobs:
> > >
> > >bin/bootstrap_development.sh
> > >
> > >bin/bootstrap_build.sh
> > >
> > >bin/check-rat-report.py
> > >
> > >docs/Makefile
> > >
> > >bin/run_clang_tidy.sh
> > >
> > >bin/jenkins/build-all-flag-combinations.sh
> > >
> > >The only one that's used in a non-obvious way is the clang tidy script.
> It
> > >is used as follows:
> > >
> > >./bin/clean.sh
> > >TIDY_BUILD_OUT="${WORKSPACE}/tidylog.txt"
> > >touch "${TIDY_BUILD_OUT}"
> > >if ! ./buildall.sh -skiptests -tidy -so -noclean &>"${TIDY_BUILD_OUT}"
> > >then
> > >  echo "tidy build failed; See ${TIDY_BUILD_OUT} for full tidy build
> > >output. Guess:"
> > >  grep ': error: ' "${TIDY_BUILD_OUT}"
> > >  exit 1
> > >else
> > >  ./bin/clean.sh
> > >  bin/run_clang_tidy.sh &>"${TIDY_BUILD_OUT}"
> > >  ! grep ']' "${TIDY_BUILD_OUT}"
> > >fi
> > >
> > >
> > >On Mon, Jul 9, 2018 at 8:36 PM, Quanlong Huang 
> > >wrote:
> > >
> > >> Hi Lars, though not directly relate to this topic, we'd like to set-up
> > >> testing like the community Jenkins.
> > >>
> > >>
> > >> Are the jenkins scripts open-source? Could you share the configures of
> > the
> > >> Jenkins jobs? Looks like they're not in the Impala repo. I'm unable to
> > >> browse the configures of jobs in jenkins.impala.io since I can't
> login.
> >
>

Re: Impala OS and Java version support

[Philip Zeyliger]

bin/boostrap_development already only works for Ubuntu 16.04+ because it's
painful to get Java8 via apt-get on Ubuntu 14. The "profile=3"
configuration in master already requires Java 8 at runtime, but will still
compile with "java 7 source". Are you proposing to change
https://github.com/cloudera/Impala/blob/2d2579cb31edda24457d33ff5176d79b7c0432c5/fe/pom.xml#L340
from
1.7 to 1.8?

I'm in favor of removing support for CentOS 5. Do we pay a tax for CentOS
6.8 vs older versions of CentOS 6? I worry there are more of those lying
around than we care to admit.

-- Philip

On Fri, Jul 6, 2018 at 9:59 AM Sailesh Mukil 
wrote:

> I'm in favor of the minimum OS version support proposal. From 3.x, I don't
> see any reason to support much older OSes especially since it's becoming
> much more burdensome to special case code for them and have those paths
> tested.
>
> I'll defer to others on the minimum Java version proposal since they may
> have better insight.
>
> On Fri, Jul 6, 2018 at 9:19 AM, Lars Volker 
> wrote:
>
> > Hi All,
> >
> > With the automatic commit flow to the 2.x branch being disabled, I'd like
> > to suggest that we deprecate support for some older operating systems on
> > the 3.x branch. This would allow us to get rid of almost all the checks
> in
> > common/config.h.in as well as some workarounds in the code. It would
> also
> > simplify the upcoming rebase of the KRPC code (IMPALA-7006).
> >
> > I'd like to propose that we pick Ubuntu 14.04, CentOS 6.8, and SLES 12 as
> > the minimum operating system versions on which Impala has to compile and
> > pass all tests. I'd also like to deprecate support for Java 7 and require
> > Java 8 for all changes going forward. To my knowledge these are the
> minimum
> > versions that people use for development and deployment of the 3.x
> branch.
> >
> > I'm curious what others think.
> >
> > Cheers, Lars
> >
>

Re: Re: Branch 2.x

[Philip Zeyliger]

Hi Quanlong,

If someone wishes to step up and maintain the 2.x branch, they're very
welcome to. To be transparent, some of the Cloudera employees are focusing
considerably more on the 3.x line and aren't currently planning to release
manage additional 2.x releases.

On Wed, Jun 20, 2018 at 1:56 AM Quanlong Huang 
wrote:

> Sorry, Philip. Do you mean we won't apply new patches to the 2.x branch? I
> thought you just mean to disable the automatic cherry-pick job.
>
>
> In my limited understanding, there're two major breaking changes for the
> 3.x branch:
>
>
> * Standard SQL syntax: legacy HQL may fail on syntax errors in Impala-3.x
> * Hadoop 3: Impala-3.x can only be deployed on Hadoop 3.x and Hive 2.x
>

If you run with IMPALA_MINICLUSTER_PROFILE=2, you should be able to
continue to build and use against older versions of Hadoop. Taking support
for that out is something we should do, but, frankly, it's a lot less of a
pain to maintain. If you're planning on using this, it'd be useful to set
up some more automated testing for it.

Some of the SQL changes have flags to disable them (e.g., the reserved
words list).

Thanks,

-- Philip

Re: Future of unsupported formats?

[Philip Zeyliger]

I'm supportive of removing them.

-- Philip

On Tue, Jun 19, 2018 at 10:06 AM Tim Armstrong
 wrote:

> Hi Edward,
>   I was talking about write support, specifically. Reading those formats is
> supported without any configuration changes.
>
> - Tim
>
> On Tue, Jun 19, 2018 at 9:32 AM, Edward Capriolo 
> wrote:
>
> > I was going to say. Not a current user ATM, but there are defiantly
> people
> > with text+gzip SequenceFile(Text). It is nice to be able to work with
> > those, I was also at a shop that went hard for AVRO + Impala but since
> > switched off.
> >
> > I also do not understand what is meant by "behind a query option" since
> the
> > version of Impala I had (CDH 5+6) would process all the above formats.
> >
> > Edward
> >
> > On Tue, Jun 19, 2018 at 11:59 AM, Lars Volker 
> > wrote:
> >
> > > I'm in favor of removing unsupported code, especially when doing so
> makes
> > > development of the rest of the codebase easier and saves us cycles for
> > > maintaining it. On the other hand it would suck if users had come to
> rely
> > > on it and we break it, even though we recommend against it.
> > >
> > > We could make a reasonable effort to discover any users of the feature,
> > > e.g. by asking on user@ and by folks on this list checking other
> > > communication channels they might have access to.
> > >
> > > Cheers, Lars
> > >
> > > On Tue, Jun 19, 2018 at 8:26 AM Tim Armstrong
> > >  wrote:
> > >
> > > > I don't think we need to bump a major version to remove something
> that
> > we
> > > > never claimed to support though. The docs are pretty clear:
> > > >
> > > >
> > > > https://impala.apache.org/docs/build/html/topics/impala_
> > > allow_unsupported_formats.html
> > > >
> > > > "An obsolete query option from early work on support for file
> formats.
> > Do
> > > > not use. Might be removed in the future."
> > > >
> > > > On Mon, Jun 18, 2018 at 10:16 PM, Jim Apple
> >  > > >
> > > > wrote:
> > > >
> > > > > As for the time zone case, I’d like to be careful about versioning.
> > If
> > > we
> > > > > remove Avro, that seems like a breaking changedeserving of a major
> > > > version
> > > > > bump.
> > > > >
> > > > > It might be worth taking a survey wider than dev@. User@ or the
> > > > customers
> > > > > of Impala packagers might be good places to start.
> > > > >
> > > > > On Mon, Jun 18, 2018 at 5:10 PM Tim Armstrong
> > > > >  wrote:
> > > > >
> > > > > > For a few years now we've had write support for Sequence, Avro
> and
> > > > > > compressed text hidden behind a query option. We haven't really
> > made
> > > > any
> > > > > > progress on turning it into a supported feature, so I'm wondering
> > if
> > > we
> > > > > > should remove the code and save some overhead of building,
> testing
> > > and
> > > > > code
> > > > > > maintenance.
> > > > > >
> > > > > > I know I've found it useful once or twice to generate test data
> > but I
> > > > > don't
> > > > > > think this is enough to justify maintaining it.
> > > > > >
> > > > > > It seems like we should get it out of this in-between state -
> > either
> > > > > delete
> > > > > > the code or get it to the point where it's supported and tested.
> If
> > > we
> > > > > > delete it, it's always possible for someone to resurrect it
> later.
> > > > > >
> > > > > > What do people think?
> > > > > >
> > > > > > - Tim
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Branch 2.x

[Philip Zeyliger]

I've not heard anything, so I've removed the "trigger" for job
https://jenkins.impala.io/job/cherrypick-2.x-and-test/. It used to "Poll
SCM" at an "@hourly" rate, looking for changes in asf/master.

Thanks!

-- Philip

On Fri, Jun 15, 2018 at 11:23 AM Philip Zeyliger 
wrote:

> Hi folks,
>
> As you know, many of us have been maintaining the 2.x branch by having a
> cherry-pick job (https://jenkins.impala.io/job/cherrypick-2.x-and-test/)
> and, when it fails, manually pushing through the issues. This has proven to
> be quite the burden, and I'm planning on disabling the automatic job.
>
> Please let this thread know if you're interested in adopting the 2.x
> branch. If there are no takers, we can let it lie dormant.
>
> Thanks!
>
> -- Philip
>

Branch 2.x

[Philip Zeyliger]

Hi folks,

As you know, many of us have been maintaining the 2.x branch by having a
cherry-pick job (https://jenkins.impala.io/job/cherrypick-2.x-and-test/)
and, when it fails, manually pushing through the issues. This has proven to
be quite the burden, and I'm planning on disabling the automatic job.

Please let this thread know if you're interested in adopting the 2.x
branch. If there are no takers, we can let it lie dormant.

Thanks!

-- Philip

Configuring Impala to exit on OutOfMemoryError

[Philip Zeyliger]

Hi!

Seeing bugs like IMPALA-7168, it leads me to think that we should recommend
folks who are running Impala to use -XX:OnOutOfMemoryError="kill -9 %p" or
equivalent on their JVMs, to simply crash the process on frontend OOM.

Any thoughts on whether this is the right approach?

-- Philip

Re: Breaking changes after 3.0, versioning, IMPALA-3307

[Philip Zeyliger]

On Mon, Jun 11, 2018 at 2:49 PM Jim Apple  wrote:

> Phil, if Jeszy's solution is possible (feature flag, off by default in the
> 3.x line), would that align with your preference?
>

I'd be fine with "feature flag, 3.x." I'd also be fine with "no feature
flag, 3.x" and "4.x".

-- Philip


> On Mon, Jun 11, 2018 at 2:29 PM, Taras Bobrovytsky 
> wrote:
>
> > I agree with Jim. It would be better to bump to 4.0 in my opinion. We
> > should follow the simple rule: Breaking changes must bump the major
> version
> > number.
> >
> > On Mon, Jun 11, 2018 at 2:17 PM, Jeszy  wrote:
> >
> > > My assumption was that the workaround Phil mentioned must be simple to
> > > toggle (e.g. flag). If it's not, it probably shouldn't be considered a
> > > viable workaround.
> > >
> > > On 11 June 2018 at 10:42, Jim Apple  wrote:
> > >
> > > > Csaba, is that possible with th change similar to how it is now, or
> > would
> > > > it have to be rewritten?
> > > >
> > > > On Mon, Jun 11, 2018 at 1:30 AM Jeszy  wrote:
> > > >
> > > > > I think we should include it in 3.1, with the feature disabled by
> > > default
> > > > > (to not break on a minor upgrade), but recommend enabling it in
> docs
> > > and
> > > > > make it enabled by default in 4.0.
> > > > >
> > > > > On 11 June 2018 at 10:23, Jim Apple  wrote:
> > > > >
> > > > > > Any more thoughts? This question is for everyone in the Impala
> > > > community.
> > > > > >
> > > > > > Right now the plan is to fold it into 3.1, with two to one in
> favor
> > > of
> > > > > that
> > > > > > over bumping to 4.0.
> > > > > >
> > > > > > On Mon, Jun 4, 2018 at 8:48 PM Jim Apple 
> > > wrote:
> > > > > >
> > > > > > > I am more in favor of bumping to 4.0. It is a rapid escalation,
> > but
> > > > we
> > > > > > > wouldn’t be the first open source project to switch to a model
> > with
> > > > > Short
> > > > > > > major versions, as both Clang and Firefox have done so.
> > > > > > >
> > > > > > > I also feel that, both from a semver perspective and as a user
> of
> > > > other
> > > > > > > software, I expect breaking changes to bump the major version
> > > number.
> > > > > > >
> > > > > > > That said, this is not a hill I’m trying to die on. My focus is
> > on
> > > > the
> > > > > > > user experience, and if our users end up well informed of the
> > > > > breakages,
> > > > > > > then I will feel we have done our job, no matter what version
> > > number
> > > > we
> > > > > > > stamp on it.
> > > > > > >
> > > > > > > On Mon, Jun 4, 2018 at 7:57 PM Philip Zeyliger <
> > > phi...@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Csaba!
> > > > > > >>
> > > > > > >> I would be fine with both proposals, with a slight preference
> to
> > > B.
> > > > My
> > > > > > >> understanding is that you're going to expose a way to define
> > > > overrides
> > > > > > for
> > > > > > >> time zone definitions, so there will be pretty workable
> > > workarounds
> > > > > too.
> > > > > > >>
> > > > > > >> -- Philip
> > > > > > >>
> > > > > > >> On Mon, Jun 4, 2018 at 1:45 PM, Csaba Ringhofer <
> > > > > > csringho...@cloudera.com
> > > > > > >> >
> > > > > > >> wrote:
> > > > > > >>
> > > > > > >> > Hi Folks!
> > > > > > >> >
> > > > > > >> >  We had a discussion with a few people about the versioning
> of
> > > > > Impala
> > > > > > >> after
> > > > > > >> > 3.0. The motivation was that IMPALA-3307 (which replaces the
> > > > > timezone
> > > > > > >> > implementation i

Re: Automatically rebase changes before GVO?

[Philip Zeyliger]

Seems fine, especially since we do the rebase as our submission strategy
anyway, so we're already accepting/testing something that's likely to get
rebased, and we may as well minimize that window.

I'd be in favor of the bot also carrying the votes.



On Thu, Jun 7, 2018 at 1:24 PM, Tim Armstrong 
wrote:

> One annoyance with our precommit job is the requirement to manually rebase
> the change before starting the merge. Failure to do so either leads to
> false positives or false negatives - builds that failed because they were
> missing a flaky/broken test fix and builds that succeeded despite
> interacting badly with a previous fix.
>
> What do people think about modifying gerrit-verify-dryrun to automatically
> rebase the patch (by the programmatic equivalent of hitting the "Rebase"
> button) at the start of the job? The patch author would still have to carry
> the +2 but this might make our lives a bit easier.
>
> - Tim
>

Re: Breaking changes after 3.0, versioning, IMPALA-3307

[Philip Zeyliger]

Hi Csaba!

I would be fine with both proposals, with a slight preference to B. My
understanding is that you're going to expose a way to define overrides for
time zone definitions, so there will be pretty workable workarounds too.

-- Philip

On Mon, Jun 4, 2018 at 1:45 PM, Csaba Ringhofer 
wrote:

> Hi Folks!
>
>  We had a discussion with a few people about the versioning of Impala after
> 3.0. The motivation was that IMPALA-3307 (which replaces the timezone
> implementation in Impala, and contains some breaking changes) missed 3.0
> and we are not sure about the version in which it can be released - is it
> 3.1 or 4.0?
>
> A. jumping to 4.0 would communicate clearly that the release contains
> braking changes - if the plan for Impala is to follow semantic versioning,
> than this is the way to go
>
> B. releasing it in 3.1 would communicate that the change is too small for a
> major version bump, and major versions are kept for BIG changes in Impala
>
> My personal preference is for B - if a breaking change is relatively small
> and workarounds are possible + the community agrees, then it should be
> possible to release it in minor a version, while major versions could be
> kept for changes where switching Impala version needs large effort on the
> user's side (for example 2->3 jump needs new Java and Hadoop major
> version), or when a huge improvement is added to Impala which deserves
> extra attention. This is more of an aesthetic than a rational choice on my
> side, so I am totally ok with semantic versioning too, if the community
> prefers it.
>

Re: jenkins.impala.io's ubuntu-16.04-from-scratch will now have Junit test results

[Philip Zeyliger]

ala.io/job/ubuntu-16.04-from-scratch/
> 2186/testReport/junit/impala_bit_functions/xml/_failed_to_read_/>
>- impala_boolean.xml.[failed-to-read]
><https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/
> 2186/testReport/junit/impala_boolean/xml/_failed_to_read_/>
>- impala_breakpad.xml.[failed-to-read]
><https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/
> 2186/testReport/junit/impala_breakpad/xml/_failed_to_read_/>
>- impala_buffer_pool_limit.xml.[failed-to-read]
><https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/
> 2186/testReport/junit/impala_buffer_pool_limit/xml/_failed_to_read_/>
>- impala_char.xml.[failed-to-read]
><https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/
> 2186/testReport/junit/impala_char/xml/_failed_to_read_/>
>- impala_comments.xml.[failed-to-read]
><https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/
> 2186/testReport/junit/impala_comments/xml/_failed_to_read_/>
>- impala_complex_types.xml.[failed-to-read]
><https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/
> 2186/testReport/junit/impala_complex_types/xml/_failed_to_read_/>
>
>
>
> On Tue, May 15, 2018 at 11:59 AM, Philip Zeyliger <phi...@cloudera.com>
> wrote:
>
> > I added the "JUnit Post Build Publisher" to
> > https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/, after testing
> it
> > on a cloned job a while ago. If you run into any trouble, please let me
> > know!
> >
> > -- Philip
> >
>

New message on your code reviews when cherry-picking your change fails.

[Philip Zeyliger]

Hey folks,

I just updated https://jenkins.impala.io/job/cherrypick-2.x-and-test/ to
comment on a code review the first time a change causes the cherry-picking
job to fail. A sample e-mail is forwarded below, with the message
highlighted by me. I've used Gerrit to send e-mail, which, frankly, was
easier to implement for me, but it does end up possibly sending to more
people than a more targeted approach.

Hopefully this helps get failed cherrypicks noticed and handled expediently.

Please holler if you either notice this being buggy or if you hate it. I
figure we can try it for a week and rip it up if we don't like it.

-- Philip

-- Forwarded message --
From: Impala Public Jenkins (Code Review) <ger...@cloudera.org>
Date: Fri, May 11, 2018 at 11:58 AM
Subject: [Impala-ASF-CR] Update version to 3.1.0-SNAPSHOT
To: Sailesh Mukil <sail...@cloudera.com>, impala...@cloudera.com,
revi...@impala.incubator.apache.org


Impala Public Jenkins *posted comments* on this change.

View Change <http://gerrit.cloudera.org:8080/10360>

Patch set 2:

This change did not cherrypick successfully into branch 2.x. To resolve
this, please do the cherry-pick manually and submit it to Gerrit at
refs/for/2.x or add an exception to the branch 2.x copy of
bin/ignored_commits.json. Thanks, your friendly bot at
https://jenkins.impala.io/job/cherrypick-2.x-and-test/479/ .


To view, visit change 10360 <http://gerrit.cloudera.org:8080/10360>. To
unsubscribe, visit settings <http://gerrit.cloudera.org:8080/settings>.
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia2e2df73d27fa332d17fec4e9aa469ea91b14167
Gerrit-Change-Number: 10360
Gerrit-PatchSet: 2
Gerrit-Owner: Sailesh Mukil <sail...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenk...@cloudera.com>
Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com>
Gerrit-Comment-Date: Fri, 11 May 2018 18:58:13 +
Gerrit-HasComments: No

-- 
You received this message because you are subscribed to the Google Groups
"impala-cr" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to impala-cr+unsubscr...@cloudera.com.
For more options, visit https://groups.google.com/a/cloudera.com/d/optout.

Re: [VOTE] 3.0.0 release candidate 1

[Philip Zeyliger]

+1 (Binding)

I checked the hashes and the GPG signatures. Checked the tree hash. Diffed
the tarball with the git RC tag.

I also ran tests via docker/test-with-docker successfully.

-- Philip



On Tue, May 1, 2018 at 4:33 PM, Sailesh Mukil  wrote:

> This is a vote to release Impala 3.0.0
>
> The artifacts for testing can be downloaded from:
> https://dist.apache.org/repos/dist/dev/impala/3.0.0/RC1/
> Git tag: 3.0.0-rc1
> Tree hash: d5bf1fdeb20ac332ce48c6a2b0588bcaf7ac9cb0
>
> Please vote +1 or -1. -1 votes should be accompanied by an explanation of
> the reason. Only PMC members have binding votes, but other
> community members are encouraged to cast non-binding votes. This vote will
> pass if there are 3 binding +1 votes and more binding +1 votes than -1
> votes.
>
> Please also note that this release is a new major release, and does not
> override the recently released 2.12.0 as the latest release. Rather, 2.12.0
> is the latest release from the 2.x branch and 3.0.0 is the first and latest
> release from the 3.x branch.
>
> This wiki page describes how to check the release before you vote:
> https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
>
> The vote will be open until 04:30 PM PST on Friday, May 4
>

new mailing list: impala-issues-all@

[Philip Zeyliger]

Hey folks,

With the help of the Apache INFRA team (INFRA-16440), there's a new mailing
list called impala-issues-all. It gets all JIRA notifications, not just
JIRA creations and their closures.

If you're interested, subscribe by e-mailing
issues-all-subscr...@impala.apache.org .

Thanks!

-- Philip

Re: Unsupported major.minor version 52.0

[Philip Zeyliger]

Hi Jim,

Are you using JDK7? Hadoop3 supports only JDK8+.

-- Philip

On Thu, Apr 26, 2018 at 9:35 AM, Jim Apple  wrote:

> testdata/bin/run-all.sh fails for me with the following error
>
> Exception in thread "main" java.lang.UnsupportedClassVersionError:
> org/apache/hadoop/hdfs/server/namenode/NameNode : Unsupported major.minor
> version 52.0
>
> This is at gerrit HEAD, at tree hash
>
> 6e88e1b26423badb4012506c10a98f32f600dbdb IMPALA-6892:
> CheckHashAndDecrypt()
> includes file and host
>
> aka commit
>
> 518bcd3e148caa8b42011de11e971c2978fb6f3b
>
> This occurs even after a successful buildall.sh
>

Re: Bulk Cherry-Pick FGP Commits to 2.x Branch

[Philip Zeyliger]

Yep.

Sounds good to me.

On Wed, Apr 25, 2018 at 1:13 PM, Bharath Vissapragada  wrote:

> FGP = "Fine-grained privileges" or something else?
>
> On Wed, Apr 25, 2018 at 1:09 PM, Fredy Wijaya 
> wrote:
>
> > Hi,
> >
> > Just to give a heads-up, I'll do bulk cherry-pick FGP commits to 2.x
> > branch. Build is passing in the private build and Alex has verified it.
> >
> > Let me know if you have any concerns.
> >
> > Thanks,
> > Fredy
> >
>

Re: Minicluster MR "your endpoint configuration is wrong" error

[Philip Zeyliger]

: container_1524519161700_0002_02_01
> Exit code: 255
>
> [2018-04-23 23:16:06.475]Container exited with a non-zero exit code 255.
> Error file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering
> org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a
> provider
> class
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as
> a provider class
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices
> as a root resource class
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
> INFO: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25
> AM'
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.
> JAXBContextResolver
> to GuiceManagedComponentProvider with the scope "Singleton"
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to
> GuiceManagedComponentProvider with the scope "Singleton"
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to
> GuiceManagedComponentProvider with the scope "PerRequest"
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
>
> [2018-04-23 23:16:06.476]Container exited with a non-zero exit code 255.
> Error file: prelaunch.err.
> Last 4096 bytes of prelaunch.err :
> Last 4096 bytes of stderr :
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering
> org.apache.hadoop.mapreduce.v2.app.webapp.JAXBContextResolver as a
> provider
> class
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering org.apache.hadoop.yarn.webapp.GenericExceptionHandler as
> a provider class
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory register
> INFO: Registering org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices
> as a root resource class
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
> INFO: Initiating Jersey application, version 'Jersey: 1.19 02/11/2015 03:25
> AM'
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.
> JAXBContextResolver
> to GuiceManagedComponentProvider with the scope "Singleton"
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.yarn.webapp.GenericExceptionHandler to
> GuiceManagedComponentProvider with the scope "Singleton"
> Apr 23, 2018 11:16:03 PM
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory
> getComponentProvider
> INFO: Binding org.apache.hadoop.mapreduce.v2.app.webapp.AMWebServices to
> GuiceManagedComponentProvider with the scope "PerRequest"
> log4j:WARN No appenders could be found for logger
> (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for
> more info.
>
>
> For more detailed output, check the application tracking page:
> http://localhost:8088/cluster/app/application_1524519161700_0002 Then
> click
> on links to logs of each attempt.
> . Failing the application.
> 18/04/23 23:16:06 INFO mapreduce.Job: Counters: 0
> 18/04/23 23:16:06 ERROR streaming.StreamJob: Job not successful!
> Streaming Command Failed!
>
>
>
>
>
>
> On Thu, Apr 19, 2018 at 1:59 PM Tianyi Wang <tw...@cloudera.com> wrote:
>
> > Thanks Philip. It works! I'll update the README file.
> >
> > On Wed, Apr 18, 2018 at 6:49 PM Philip Zeyliger <phi...@cl

Re: New PMC member: Philip Zeyliger

[Philip Zeyliger]

Thanks!

On Mon, Apr 23, 2018 at 1:03 PM, Jim Apple <jbap...@cloudera.com> wrote:

> The Project Management Committee (PMC) for Apache Impala has invited Philip
> Zeyliger to become a PMC member and we are pleased to announce that they
> have accepted.
>
> Congratulations and welcome, Philip!
>

Re: Minicluster MR "your endpoint configuration is wrong" error

[Philip Zeyliger]

Most likely is the following commit, which turned off Yarn by default. The
commit message will tell you how to turn it on ("testdata/cluster/admin -y
start_cluster"). I'm open to changing the defaults the other way around if
we think this is an important use case.

-- Philip

commit 6dc13d933b5ea9a41e584d83e95db72b9e8e19b3
Author: Philip Zeyliger <phi...@cloudera.com>
Date:   Tue Apr 10 09:21:20 2018 -0700

Remove Yarn from minicluster by default. (2nd try)

On Wed, Apr 18, 2018 at 6:25 PM, Tianyi Wang <tw...@cloudera.com> wrote:

> I was trying to run  tests/comparison/data_generator.py, which used to
> work
> before switching to hadoop 3. Now MR claims that it's wrongly configured to
> connect to 0.0.0.0:8032, but I cannot find text "8032" in our minicluster
> configs. Does anybody happen to know this error?
>
>
> Traceback (most recent call last):
>   File "./data_generator.py", line 339, in 
> populator.populate_db(args.table_count, postgresql_conn=postgresql_
> conn)
>   File "./data_generator.py", line 134, in populate_db
> self._run_data_generator_mr_job([g for _, g in table_and_generators],
> self.db_name)
>   File "./data_generator.py", line 244, in _run_data_generator_mr_job
> % (reducer_count, ','.join(files), mapper_input_file, hdfs_output_dir))
>   File "/home/twang/projects/impala/tests/comparison/cluster.py", line
> 476,
> in run_mr_job
> stderr=subprocess.STDOUT, env=env)
>   File "/home/twang/projects/impala/tests/util/shell_util.py", line 113,
> in
> shell
> "\ncmd: %s\nstdout: %s\nstderr: %s") % (retcode, cmd, output, err))
> Exception: Command returned non-zero exit code: 5
> cmd: set -euo pipefail
> hadoop jar
> /home/twang/projects/impala/toolchain/cdh_components/
> hadoop-3.0.0-cdh6.x-SNAPSHOT/share/hadoop/tools/lib/hadoop-
> streaming-3.0.0-cdh6.x-SNAPSHOT.jar
> -D mapred.reduce.tasks=34 \
> -D stream.num.map.output.key.fields=2 \
> -files
> ./common.py,./db_types.py,./data_generator_mapred_common.
> py,./data_generator_mapper.py,./data_generator_reducer.py,./
> random_val_generator.py
> \
> -input /tmp/data_gen_randomness_mr_input_1524095906 \
> -output /tmp/data_gen_randomness_mr_output_1524095906 \
> -mapper data_generator_mapper.py \
> -reducer data_generator_reducer.py
> stdout: packageJobJar: []
> [/home/twang/projects/impala/toolchain/cdh_components/
> hadoop-3.0.0-cdh6.x-SNAPSHOT/share/hadoop/tools/lib/hadoop-
> streaming-3.0.0-cdh6.x-SNAPSHOT.jar]
> /tmp/streamjob6950277591392799099.jar tmpDir=null
> 18/04/18 16:58:30 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
> 18/04/18 16:58:30 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
> 18/04/18 16:58:32 INFO ipc.Client: Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> 18/04/18 16:58:33 INFO ipc.Client: Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
>
> ..
>
> 18/04/18 16:58:51 INFO ipc.Client: Retrying connect to server:
> 0.0.0.0/0.0.0.0:8032. Already tried 9 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
> 18/04/18 16:58:51 INFO retry.RetryInvocationHandler:
> java.net.ConnectException: Your endpoint configuration is wrong; For more
> details see:  http://wiki.apache.org/hadoop/UnsetHostnameOrPort, while
> invoking ApplicationClientProtocolPBClientImpl.getNewApplication over null
> after 1 failover attempts. Trying to failover after sleeping for 16129ms.
>
> --
> Tianyi Wang
>

Re: Maintenance for jenkins.impala.io today

[Philip Zeyliger]

I think https://jenkins.impala.io/job/clang-tidy-ub1604/1330/ has gotten
farther than previously, so perhaps upgrading that plugin was sufficient.

Please continue to let me know if you see anything else awry!

-- Philip

On Wed, Apr 11, 2018 at 2:00 PM, Philip Zeyliger <phi...@cloudera.com>
wrote:

> I am upgrading build-name-setter and am going to do another restart. (
> https://issues.jenkins-ci.org/browse/JENKINS-48944 is the JIRA.)
>
> As before, I'll try to take note of which GVOs were running and restart
> them. On balance, I think it's better to take the disruption right away.
>
> Thanks,
>
> -- Philip
>
> On Wed, Apr 11, 2018 at 1:57 PM, Philip Zeyliger <phi...@cloudera.com>
> wrote:
>
>> Yes, so it seems. I'll look into it.
>>
>> -- Philip
>>
>> On Wed, Apr 11, 2018 at 1:49 PM, Tim Armstrong <tarmstr...@cloudera.com>
>> wrote:
>>
>>> Is it possible that the upgrade broke some of our builds?
>>>
>>> I'm seeing some weird errors like the below, e.g. on this build
>>> https://jenkins.impala.io/job/clang-tidy-ub1604/1327/consoleFull
>>>
>>> *17:38:46* FATAL: java.lang.RuntimeException: Failed to serialize
>>> hudson.model.Actionable#actions for class
>>> hudson.model.FreeStyleBuild*17:38:46*
>>> java.lang.UnsupportedOperationException: Refusing to marshal
>>> java.io.PrintStream for security reasons; see
>>> https://jenkins.io/redirect/class-filter/*17:38:46* at
>>> hudson.util.XStream2$BlacklistedTypesConverter.marshal(XStre
>>> am2.java:543)*17:38:46*
>>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.co
>>> nvert(AbstractReferenceMarshaller.java:69)*17:38:46*
>>> at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(
>>> TreeMarshaller.java:58)*17:38:46*
>>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.
>>> convertAnother(AbstractReferenceMarshaller.java:84)*17:38:46*
>>> at hudson.util.RobustReflectionConverter.marshallField(RobustRe
>>> flectionConverter.java:265)*17:38:46*
>>> at hudson.util.RobustReflectionConverter$2.writeField(RobustRef
>>> lectionConverter.java:252)*17:38:46*
>>> Caused: java.lang.RuntimeException: Failed to serialize
>>> org.jenkinsci.plugins.EnvironmentVarSetter#log for class
>>> org.jenkinsci.plugins.EnvironmentVarSetter*17:38:46*at
>>> hudson.util.RobustReflectionConverter$2.writeField(RobustRef
>>> lectionConverter.java:256)*17:38:46*
>>> at hudson.util.RobustReflectionConverter$2.visit(RobustReflecti
>>> onConverter.java:224)*17:38:46*
>>> at com.thoughtworks.xstream.converters.reflection.PureJavaRefle
>>> ctionProvider.visitSerializableFields(PureJavaReflectionProv
>>> ider.java:138)*17:38:46*
>>> at hudson.util.RobustReflectionConverter.doMarshal(RobustReflec
>>> tionConverter.java:209)*17:38:46*
>>> at hudson.util.RobustReflectionConverter.marshal(RobustReflecti
>>> onConverter.java:150)*17:38:46*
>>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.co
>>> nvert(AbstractReferenceMarshaller.java:69)*17:38:46*
>>> at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(
>>> TreeMarshaller.java:58)*17:38:46*
>>> at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(
>>> TreeMarshaller.java:43)*17:38:46*
>>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.
>>> convertAnother(AbstractReferenceMarshaller.java:88)*17:38:46*
>>> at com.thoughtworks.xstream.converters.collections.AbstractColl
>>> ectionConverter.writeItem(AbstractCollectionConverter.java:64)*17:38:46*
>>> at com.thoughtworks.xstream.converters.collections.CollectionCo
>>> nverter.marshal(CollectionConverter.java:74)*17:38:46*
>>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.co
>>> nvert(AbstractReferenceMarshaller.java:69)*17:38:46*
>>> at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(
>>> TreeMarshaller.java:58)*17:38:46*
>>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.
>>> convertAnother(AbstractReferenceMarshaller.java:84)*17:38:46*
>>> at hudson.util.RobustReflectionConverter.marshallField(RobustRe
>>> flectionConverter.java:265)*17:38:46*
>>> at hudson.util.RobustReflectionConverter$2.writeField(RobustRef
>>> lectionConverter.java:252)*17:38:46*
>>> Caused: ja

Re: Maintenance for jenkins.impala.io today

[Philip Zeyliger]

I am upgrading build-name-setter and am going to do another restart. (
https://issues.jenkins-ci.org/browse/JENKINS-48944 is the JIRA.)

As before, I'll try to take note of which GVOs were running and restart
them. On balance, I think it's better to take the disruption right away.

Thanks,

-- Philip

On Wed, Apr 11, 2018 at 1:57 PM, Philip Zeyliger <phi...@cloudera.com>
wrote:

> Yes, so it seems. I'll look into it.
>
> -- Philip
>
> On Wed, Apr 11, 2018 at 1:49 PM, Tim Armstrong <tarmstr...@cloudera.com>
> wrote:
>
>> Is it possible that the upgrade broke some of our builds?
>>
>> I'm seeing some weird errors like the below, e.g. on this build
>> https://jenkins.impala.io/job/clang-tidy-ub1604/1327/consoleFull
>>
>> *17:38:46* FATAL: java.lang.RuntimeException: Failed to serialize
>> hudson.model.Actionable#actions for class
>> hudson.model.FreeStyleBuild*17:38:46*
>> java.lang.UnsupportedOperationException: Refusing to marshal
>> java.io.PrintStream for security reasons; see
>> https://jenkins.io/redirect/class-filter/*17:38:46* at
>> hudson.util.XStream2$BlacklistedTypesConverter.marshal(
>> XStream2.java:543)*17:38:46*
>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.co
>> nvert(AbstractReferenceMarshaller.java:69)*17:38:46*
>> at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(
>> TreeMarshaller.java:58)*17:38:46*
>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.
>> convertAnother(AbstractReferenceMarshaller.java:84)*17:38:46*
>> at hudson.util.RobustReflectionConverter.marshallField(RobustRe
>> flectionConverter.java:265)*17:38:46*
>> at hudson.util.RobustReflectionConverter$2.writeField(RobustRef
>> lectionConverter.java:252)*17:38:46*
>> Caused: java.lang.RuntimeException: Failed to serialize
>> org.jenkinsci.plugins.EnvironmentVarSetter#log for class
>> org.jenkinsci.plugins.EnvironmentVarSetter*17:38:46*at
>> hudson.util.RobustReflectionConverter$2.writeField(RobustRef
>> lectionConverter.java:256)*17:38:46*
>> at hudson.util.RobustReflectionConverter$2.visit(RobustReflecti
>> onConverter.java:224)*17:38:46*
>> at com.thoughtworks.xstream.converters.reflection.PureJavaRefle
>> ctionProvider.visitSerializableFields(PureJavaReflectionProv
>> ider.java:138)*17:38:46*
>> at hudson.util.RobustReflectionConverter.doMarshal(RobustReflec
>> tionConverter.java:209)*17:38:46*
>> at hudson.util.RobustReflectionConverter.marshal(RobustReflecti
>> onConverter.java:150)*17:38:46*
>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.co
>> nvert(AbstractReferenceMarshaller.java:69)*17:38:46*
>> at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(
>> TreeMarshaller.java:58)*17:38:46*
>> at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(
>> TreeMarshaller.java:43)*17:38:46*
>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.
>> convertAnother(AbstractReferenceMarshaller.java:88)*17:38:46*
>> at com.thoughtworks.xstream.converters.collections.AbstractColl
>> ectionConverter.writeItem(AbstractCollectionConverter.java:64)*17:38:46*
>> at com.thoughtworks.xstream.converters.collections.CollectionCo
>> nverter.marshal(CollectionConverter.java:74)*17:38:46*
>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.co
>> nvert(AbstractReferenceMarshaller.java:69)*17:38:46*
>> at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(
>> TreeMarshaller.java:58)*17:38:46*
>> at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.
>> convertAnother(AbstractReferenceMarshaller.java:84)*17:38:46*
>> at hudson.util.RobustReflectionConverter.marshallField(RobustRe
>> flectionConverter.java:265)*17:38:46*
>> at hudson.util.RobustReflectionConverter$2.writeField(RobustRef
>> lectionConverter.java:252)*17:38:46*
>> Caused: java.lang.RuntimeException: Failed to serialize
>> hudson.model.Actionable#actions for class
>> hudson.model.FreeStyleBuild*17:38:46*   at
>> hudson.util.RobustReflectionConverter$2.writeField(RobustRef
>> lectionConverter.java:256)*17:38:46*
>> at hudson.util.RobustReflectionConverter$2.visit(RobustReflecti
>> onConverter.java:224)*17:38:46*
>> at com.thoughtworks.xstream.converters.reflection.PureJavaRefle
>> ctionProvider.visitSerializableFields(PureJavaReflectionProv
>> ider.java:138)*17:38:46*
>>

Re: Maintenance for jenkins.impala.io today

[Philip Zeyliger]

reeMarshallingStrateg
> y.marshal(AbstractTreeMarshallingStrategy.java:37)*17:38:46*
> at com.thoughtworks.xstream.XStream.marshal(XStream.java:
> 1026)*17:38:46*
> at com.thoughtworks.xstream.XStream.marshal(XStream.java:
> 1015)*17:38:46*
> at com.thoughtworks.xstream.XStream.toXML(XStream.java:
> 988)*17:38:46*
> at hudson.XmlFile.write(XmlFile.java:193)*17:38:46* Caused:
> java.io.IOException*17:38:46*   at
> hudson.XmlFile.write(XmlFile.java:200)
>
>
>
> On Wed, Apr 11, 2018 at 12:42 PM, Lars Volker <l...@cloudera.com> wrote:
>
> > Thank you for doing the maintenance work, Phil!
> >
> > On Wed, Apr 11, 2018 at 9:52 AM, Philip Zeyliger <phi...@cloudera.com>
> > wrote:
> >
> > > This is completed. There were only two patches being built, so I killed
> > > those builds.
> > >
> > > I restarted GVO for change 9976 ("Move some test_spilling debug actions
> > to
> > > exhaustive").
> > >
> > > I *did not* restart builds for change 9988 ("IMPALA-5717: Support for
> > > reading ORC data files"). It's currently private to me, and I thought
> > > https://gerrit.cloudera.org/c/9134/ already went in. Tim, it looks
> like
> > > that was started by you at
> > > https://jenkins.impala.io/job/gerrit-verify-dryrun/2279/. Could you
> > > restart
> > > it yourself if it makes sense to do so?
> > >
> > > Please let me know if you see any Jenkins issues today!
> > >
> > > Thanks!
> > >
> > > -- Philip
> > >
> > > On Wed, Apr 11, 2018 at 9:14 AM, Philip Zeyliger <phi...@cloudera.com>
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > We need to upgrade Jenkins to address a security advisory. I'll be
> > doing
> > > > so today. My apologies for the disruption it will inevitably cause!
> > > >
> > > > -- Philip
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

[Philip Zeyliger]

Sounds like a good plan to me!

-- Philip

Re: Re: Updating your minicluster to Hadoop 3.0...

[Philip Zeyliger]

2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at org.apache.hadoop.hive.ql.session.SessionState.
> createSessionDirs(SessionState.java:640) ~[hive-exec-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:572)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at 
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:539)
> ~[hive-exec-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at 
> org.apache.hive.service.cli.session.HiveSessionImpl.open(HiveSessionImpl.java:169)
> ~[hive-service-2.1.1-cdh6.x-SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_121]
> at sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121]
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121]
> at org.apache.hive.service.cli.session.HiveSessionProxy.
> invoke(HiveSessionProxy.java:78) ~[hive-service-2.1.1-cdh6.x-
> SNAPSHOT.jar:2.1.1-cdh6.x-SNAPSHOT]
> ... 21 more
>
> I just rebase the codes and run ./buildall.sh -format
> Any thoughts?
>
> Thanks,
> Quanlong
>
>
> At 2018-03-30 13:08:33, "Dimitris Tsirogiannis" <dtsirogian...@cloudera.com> 
> wrote:
> >I enabled full logging in my postgres that hosts the sentry and metastore
> >db and I don't see the table being created. If anyone has gone through the
> >process, can you: a) verify that relation SENTRY_ROLE exists in your
> >sentry_policy db, and b) tell me how many relations are in your policy_db.
> >
> >Thanks
> >Dimitris
> >
> >On Thu, Mar 29, 2018 at 9:32 PM, Dimitris Tsirogiannis <
> >dtsirogian...@cloudera.com> wrote:
> >
> >> Good point. I used -format that in theory handles both the metastore and
> >> the sentry policy dB. The sentry_policy db is created and has some tables
> >> but not the SENTRY_ROLE.
> >>
> >> Dimitris
> >>
> >> On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <jbap...@cloudera.com> wrote:
> >>
> >>> I think I might have once fixed that using
> >>>
> >>> ./buildall.sh -notests -format_metastore -format_sentry_policy_db
> >>>
> >>>
> >>> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
> >>> dtsirogian...@cloudera.com> wrote:
> >>>
> >>> > I tried rebuilding my minicluster but Sentry refuses to start. I get
> >>> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does
> >>> that
> >>> > ring any bells?
> >>> >
> >>> > Thanks
> >>> > Dimitris
> >>> >
> >>> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <phi...@cloudera.com>
> >>> > wrote:
> >>> >
> >>> > > Hi folks,
> >>> > >
> >>> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
> >>> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which
> >>> changes
> >>> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry
> >>> 2.0,
> >>> > > and so on. This change *will not* be back-ported to 2.x.
> >>> > >
> >>> > > When you pull that change in, you'll need to re-build your minicluster
> >>> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull
> >>> in
> >>> > the
> >>> > > new dependencies, format your cluster, and load up all the data. As
> >>> you
> >>> > > know, it takes 1-2 hours.
> >>> > >
> >>> > > If you want to hold off, you can elso set export
> >>> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
> >>> > >
> >>> > > Note that this choice between versions happens at build time, and
> >>> CMake
> >>> > > depends on it. So, switching back and forth requires re-running CMake.
> >>> > >
> >>> > > Please let me know if you run into any trouble. This is a big enough
> >>> that
> >>> > > there may be some bumps on the road.
> >>> > >
> >>> > > -- Philip
> >>> > >
> >>> >
> >>>
> >>
>
>

Re: Updating your minicluster to Hadoop 3.0...

[Philip Zeyliger]

Hi Dimitris,

Here's a listing from my machine after I ran "./buildall.sh
-start_minicluster -start_impala_cluster -format -testdata -notests".

-- Philip

$sudo -u postgres psql
psql (9.5.12)
Type "help" for help.

postgres=# \c sentry_policy
You are now connected to database "sentry_policy" as user "postgres".
sentry_policy=# select * from "SENTRY_ROLE";
 ROLE_ID | CREATE_TIME | ROLE_NAME
-+-+---
(0 rows)

sentry_policy=# select * from "SENTRY_VERSION";
 VER_ID | SCHEMA_VERSION |VERSION_COMMENT
++---
  1 | 2.0.0  | Schema version set implicitly
(1 row)

sentry_policy=# select * from "SENTRY_USER";
 USER_ID | CREATE_TIME | USER_NAME
-+-+---
(0 rows)

sentry_policy=# \dt
List of relations
 Schema | Name | Type  |  Owner
+--+---+--
 public | AUTHZ_PATH   | table | hiveuser
 public | AUTHZ_PATHS_MAPPING  | table | hiveuser
 public | AUTHZ_PATHS_SNAPSHOT_ID  | table | hiveuser
 public | SENTRY_DB_PRIVILEGE  | table | hiveuser
 public | SENTRY_GM_PRIVILEGE  | table | hiveuser
 public | SENTRY_GROUP | table | hiveuser
 public | SENTRY_HMS_NOTIFICATION_ID   | table | hiveuser
 public | SENTRY_PATH_CHANGE   | table | hiveuser
 public | SENTRY_PERM_CHANGE   | table | hiveuser
 public | SENTRY_ROLE  | table | hiveuser
 public | SENTRY_ROLE_DB_PRIVILEGE_MAP | table | hiveuser
 public | SENTRY_ROLE_GM_PRIVILEGE_MAP | table | hiveuser
 public | SENTRY_ROLE_GROUP_MAP| table | hiveuser
 public | SENTRY_ROLE_USER_MAP | table | hiveuser
 public | SENTRY_USER  | table | hiveuser
 public | SENTRY_VERSION   | table | hiveuser
 public | SEQUENCE_TABLE   | table | hiveuser
(17 rows)



On Thu, Mar 29, 2018 at 10:08 PM, Dimitris Tsirogiannis <
dtsirogian...@cloudera.com> wrote:

> I enabled full logging in my postgres that hosts the sentry and metastore
> db and I don't see the table being created. If anyone has gone through the
> process, can you: a) verify that relation SENTRY_ROLE exists in your
> sentry_policy db, and b) tell me how many relations are in your policy_db.
>
> Thanks
> Dimitris
>
> On Thu, Mar 29, 2018 at 9:32 PM, Dimitris Tsirogiannis <
> dtsirogian...@cloudera.com> wrote:
>
> > Good point. I used -format that in theory handles both the metastore and
> > the sentry policy dB. The sentry_policy db is created and has some tables
> > but not the SENTRY_ROLE.
> >
> > Dimitris
> >
> > On Thu, Mar 29, 2018 at 6:29 PM Jim Apple <jbap...@cloudera.com> wrote:
> >
> >> I think I might have once fixed that using
> >>
> >> ./buildall.sh -notests -format_metastore -format_sentry_policy_db
> >>
> >>
> >> On Thu, Mar 29, 2018 at 6:15 PM, Dimitris Tsirogiannis <
> >> dtsirogian...@cloudera.com> wrote:
> >>
> >> > I tried rebuilding my minicluster but Sentry refuses to start. I get
> >> > "ERROR: relation "SENTRY_ROLE" does not exist in the sentry logs. Does
> >> that
> >> > ring any bells?
> >> >
> >> > Thanks
> >> > Dimitris
> >> >
> >> > On Tue, Mar 27, 2018 at 2:50 PM, Philip Zeyliger <phi...@cloudera.com
> >
> >> > wrote:
> >> >
> >> > > Hi folks,
> >> > >
> >> > > I just sent off https://gerrit.cloudera.org/#/c/9743/ and
> >> > > https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which
> >> changes
> >> > > the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry
> >> 2.0,
> >> > > and so on. This change *will not* be back-ported to 2.x.
> >> > >
> >> > > When you pull that change in, you'll need to re-build your
> minicluster
> >> > > with, e.g., ./buildall.sh -testdata -format -notests. This will pull
> >> in
> >> > the
> >> > > new dependencies, format your cluster, and load up all the data. As
> >> you
> >> > > know, it takes 1-2 hours.
> >> > >
> >> > > If you want to hold off, you can elso set export
> >> > > IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.
> >> > >
> >> > > Note that this choice between versions happens at build time, and
> >> CMake
> >> > > depends on it. So, switching back and forth requires re-running
> CMake.
> >> > >
> >> > > Please let me know if you run into any trouble. This is a big enough
> >> that
> >> > > there may be some bumps on the road.
> >> > >
> >> > > -- Philip
> >> > >
> >> >
> >>
> >
>

Re: [jira] [Created] (IMPALA-6767) java.lang.NoSuchMethodError: org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient.listAllRoles(Ljava/lang/String;)Ljava/util/Set;

[Philip Zeyliger]

This is a method that changed names between the Hadoop2 and Hadoop3
profiles, and we have a shim in a compat directory in the fe/src. It’s
likely you need to rebuild or are talking to the wrong version.

P



On Thu, Mar 29, 2018 at 12:13 PM Michael Ho (JIRA)  wrote:

> Michael Ho created IMPALA-6767:
> --
>
>  Summary:  java.lang.NoSuchMethodError:
> org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient.listAllRoles(Ljava/lang/String;)Ljava/util/Set;
>  Key: IMPALA-6767
>  URL: https://issues.apache.org/jira/browse/IMPALA-6767
>  Project: IMPALA
>   Issue Type: Bug
>   Components: Infrastructure
> Affects Versions: Impala 2.12.0
> Reporter: Michael Ho
>  Attachments: archive.zip
>
> Sentry failed with the following errors when starting the mini-cluster.
> It's unclear if it's due to a bad jar file or there was something else
> broken. Attaching the build artifacts for reference.
>
> {noformat}
> Stopping Sentry
> log4j:WARN No such property [conversionPattern] in
> org.apache.solr.util.SolrLogLayout.
> 18/03/29 00:59:58 INFO transport.SentryTransportPool: Creating pool for
> localhost with default port 30911
> 18/03/29 00:59:58 INFO transport.SentryTransportPool: Adding endpoint
> localhost:30911
> 18/03/29 00:59:58 INFO transport.SentryTransportPool: Connection pooling
> is enabled
> Exception in thread "main" java.lang.NoSuchMethodError:
> org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClient.listAllRoles(Ljava/lang/String;)Ljava/util/Set;
> at org.apache.impala.util.SentryUtil.listRoles(SentryUtil.java:52)
> at
> org.apache.impala.util.SentryPolicyService.listAllRoles(SentryPolicyService.java:398)
> at
> org.apache.impala.testutil.SentryServicePinger.main(SentryServicePinger.java:75)
> Error in /home/ubuntu/Impala/testdata/bin/run-sentry-service.sh at line
> 36: "$JAVA" -cp $CLASSPATH org.apache.impala.testutil.SentryServicePinger \
> {noformat}
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)
>

Re: accidental push to master

[Philip Zeyliger]

If I had to guess, https://jenkins.impala.io/job/cherrypick-2.x-and-test/
will skip the revert, because it doesn't have a Change-Id. Since it hasn't
happened, you can add 25218487533f6bf6959c32ff4ee38b77e0ab30b5 to the
ignored commits list of 2.x (in branch 2.x) right now, and both commits
will get ignored. If time passes and one commit gets cherry-picked, you'll
just need to revert manually again on 2.x and push that.

-- Philip

On Wed, Mar 28, 2018 at 3:44 PM, Michael Brown  wrote:

> Hi, I accidentally pushed to refs/heads/master, and then immediately
> reverted, though also to refs/heads. This means there is a commit out there
> without a Change-Id again. What's the proper thing here to do for 2.x?
>
> Thanks, and sorry about that.
>
> commit 2c0926e2de22ecafafc460f2b31ca2423b8f7e98
> Author: Michael Brown 
> Date:   Wed Mar 28 15:28:48 2018 -0700
>
> Revert "IMPALA-6759: align stress test memory estimation parse pattern"
>
> This reverts commit 25218487533f6bf6959c32ff4ee38b77e0ab30b5.
>
> commit 25218487533f6bf6959c32ff4ee38b77e0ab30b5
> Author: Michael Brown 
> Date:   Wed Mar 28 15:14:20 2018 -0700
>
> IMPALA-6759: align stress test memory estimation parse pattern
>
> The stress test never expected to see memory estimates on the order of
> PB. Apparently it can happen with TPC DS 1, so update the pattern.
>
> It's not clear how to quickly write a test to catch this, because it
> involves crossing language boundaries and possibly having a
> massively-scaled dataset. I think leaving a comment in both places is
> good enough for now.
>
> Change-Id: I08976f261582b379696fd0e81bc060577e552309
>

Re: FE build failure

[Philip Zeyliger]

https://gerrit.cloudera.org/#/c/9841/ is the review to remove "-noclean"
from bootstrap_development, which I think will prevent this.

-- Philip

On Wed, Mar 28, 2018 at 9:58 AM, Philip Zeyliger <phi...@cloudera.com>
wrote:

> Hi Michael,
>
> What's going on is that we're re-using build machines. I compared
> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/1748/consoleFull
> and https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/
> 1749/consoleFull. They're buliding the same thing, and 1749 is a lot
> farther along, whereas 1748 failed. In 1748, you can see "07:52:11 0
> upgraded, 0 newly installed, 0 to remove and 88 not upgraded.", which
> suggests to me that this machine has been used for a build before. I had
> thought that we use a new machine for every build, but
> https://issues.jenkins-ci.org/browse/JENKINS-8618 doesn't seem to exist,
> so the way it works is just a "1 minute timeout". A previous build ran on
> this machine. It was https://jenkins.impala.io/
> job/all-build-options-ub1604/1198/.
>
> It turns out we run "buildall" with "-noclean" because that's what's in
> "bootstrap_development". I'm going to test that removing "-noclean" would
> help, and, if it does, push that along. We could also change the Jenkins
> jobs to cleanup their workspace after they're done.
>
> Anyway, I think the mystery is cleared up, and it looks like you just
> re-submitted and are unblocked.
>
> -- Philip
>
>
> On Wed, Mar 28, 2018 at 9:37 AM, Michael Ho <k...@cloudera.com> wrote:
>
>> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/1748/
>>
>> On Wed, Mar 28, 2018 at 9:34 AM, Philip Zeyliger <phi...@cloudera.com>
>> wrote:
>>
>> > I see. That's surprising. Could you send me a URL to take a look?
>> >
>> > Thanks!
>> >
>> > -- Philip
>> >
>> > On Wed, Mar 28, 2018 at 9:30 AM, Michael Ho <k...@cloudera.com> wrote:
>> >
>> > > Thanks Philip. This is GVO not my local dev machine. It's presumably
>> > > building from scratch. Could it be left over from some previous GVO ?
>> > >
>> > > On Wed, Mar 28, 2018 at 9:05 AM, Philip Zeyliger <phi...@cloudera.com
>> >
>> > > wrote:
>> > >
>> > > > Hi Michael,
>> > > >
>> > > > Most likely this is on account of switching to Hadoop 3 by default (
>> > > > https://github.com/apache/impala/commit/0812f8737a4301e5b52c
>> ad9dc91158
>> > > > 9f9e98ecdb).
>> > > > I sent an e-mail out about this yesterday, but I think what you're
>> > seeing
>> > > > is a need to re-run buildall, and to possibly clean
>> > fe/generated-sources.
>> > > > Hive1 and Hive2 have differing package names for their Thrift
>> packages,
>> > > and
>> > > > there's a sed script in one of the CMake files which customize that
>> > > > appropriately after running Thrift.
>> > > >
>> > > > The maven.jenkins.cloudera.com issue is harmless. The Sentry pom
>> that
>> > we
>> > > > depend on has an errant "" tag which shouldn't be there.
>> > I'm
>> > > > working on getting that fixed up, but I don't think it's causing
>> your
>> > > > current issue. (That said, if you get an issue about failing to
>> resolve
>> > > > dependencies, it can be related, and but I think I've fixed the one
>> > we're
>> > > > likely to run into already.)
>> > > >
>> > > > -- Philip
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Mar 28, 2018 at 9:00 AM, Michael Ho <k...@cloudera.com>
>> wrote:
>> > > >
>> > > > > During GVO, I ran into the following warning message
>> > > > > (*maven.jenkins.cloudera.com <http://maven.jenkins.cloudera.com>:
>> > Name
>> > > > > or service not known*) and eventually hit a compilation failure
>> > > > > eventually. Not sure if anyone has run into this failure or if the
>> > > > > warning message is a concern:
>> > > > >
>> > > > > [WARNING] Failure to transfer
>> > > > > net.minidev:json-smart/maven-metadata.xml from
>> > > > > http://maven.jenkins.cloudera.com:8081/artifactory/cauldron-
>> > > > > latest-snapshot-local/
>> > > > > was cached in the local 

Re: FE build failure

[Philip Zeyliger]

Hi Michael,

What's going on is that we're re-using build machines. I compared
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/1748/consoleFull
and https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/1749/consoleFull.
They're buliding the same thing, and 1749 is a lot farther along, whereas
1748 failed. In 1748, you can see "07:52:11 0 upgraded, 0 newly installed,
0 to remove and 88 not upgraded.", which suggests to me that this machine
has been used for a build before. I had thought that we use a new machine
for every build, but https://issues.jenkins-ci.org/browse/JENKINS-8618
doesn't seem to exist, so the way it works is just a "1 minute timeout". A
previous build ran on this machine. It was
https://jenkins.impala.io/job/all-build-options-ub1604/1198/.

It turns out we run "buildall" with "-noclean" because that's what's in
"bootstrap_development". I'm going to test that removing "-noclean" would
help, and, if it does, push that along. We could also change the Jenkins
jobs to cleanup their workspace after they're done.

Anyway, I think the mystery is cleared up, and it looks like you just
re-submitted and are unblocked.

-- Philip


On Wed, Mar 28, 2018 at 9:37 AM, Michael Ho <k...@cloudera.com> wrote:

> https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/1748/
>
> On Wed, Mar 28, 2018 at 9:34 AM, Philip Zeyliger <phi...@cloudera.com>
> wrote:
>
> > I see. That's surprising. Could you send me a URL to take a look?
> >
> > Thanks!
> >
> > -- Philip
> >
> > On Wed, Mar 28, 2018 at 9:30 AM, Michael Ho <k...@cloudera.com> wrote:
> >
> > > Thanks Philip. This is GVO not my local dev machine. It's presumably
> > > building from scratch. Could it be left over from some previous GVO ?
> > >
> > > On Wed, Mar 28, 2018 at 9:05 AM, Philip Zeyliger <phi...@cloudera.com>
> > > wrote:
> > >
> > > > Hi Michael,
> > > >
> > > > Most likely this is on account of switching to Hadoop 3 by default (
> > > > https://github.com/apache/impala/commit/
> 0812f8737a4301e5b52cad9dc91158
> > > > 9f9e98ecdb).
> > > > I sent an e-mail out about this yesterday, but I think what you're
> > seeing
> > > > is a need to re-run buildall, and to possibly clean
> > fe/generated-sources.
> > > > Hive1 and Hive2 have differing package names for their Thrift
> packages,
> > > and
> > > > there's a sed script in one of the CMake files which customize that
> > > > appropriately after running Thrift.
> > > >
> > > > The maven.jenkins.cloudera.com issue is harmless. The Sentry pom
> that
> > we
> > > > depend on has an errant "" tag which shouldn't be there.
> > I'm
> > > > working on getting that fixed up, but I don't think it's causing your
> > > > current issue. (That said, if you get an issue about failing to
> resolve
> > > > dependencies, it can be related, and but I think I've fixed the one
> > we're
> > > > likely to run into already.)
> > > >
> > > > -- Philip
> > > >
> > > >
> > > >
> > > > On Wed, Mar 28, 2018 at 9:00 AM, Michael Ho <k...@cloudera.com>
> wrote:
> > > >
> > > > > During GVO, I ran into the following warning message
> > > > > (*maven.jenkins.cloudera.com <http://maven.jenkins.cloudera.com>:
> > Name
> > > > > or service not known*) and eventually hit a compilation failure
> > > > > eventually. Not sure if anyone has run into this failure or if the
> > > > > warning message is a concern:
> > > > >
> > > > > [WARNING] Failure to transfer
> > > > > net.minidev:json-smart/maven-metadata.xml from
> > > > > http://maven.jenkins.cloudera.com:8081/artifactory/cauldron-
> > > > > latest-snapshot-local/
> > > > > was cached in the local repository, resolution will not be
> > reattempted
> > > > > until the update interval of cdh.cauldron.repo has elapsed or
> updates
> > > > > are forced. Original error: Could not transfer metadata
> > > > > net.minidev:json-smart/maven-metadata.xml from/to
> cdh.cauldron.repo
> > > > > (http://maven.jenkins.cloudera.com:8081/artifactory/
> > > > > cauldron-latest-snapshot-local/):
> > > > > maven.jenkins.cloudera.com: Name or service not known
> > > > > [WARNING] Failure to transfer
> > > > > net.minidev:json-smart/m

Re: FE build failure

[Philip Zeyliger]

I see. That's surprising. Could you send me a URL to take a look?

Thanks!

-- Philip

On Wed, Mar 28, 2018 at 9:30 AM, Michael Ho <k...@cloudera.com> wrote:

> Thanks Philip. This is GVO not my local dev machine. It's presumably
> building from scratch. Could it be left over from some previous GVO ?
>
> On Wed, Mar 28, 2018 at 9:05 AM, Philip Zeyliger <phi...@cloudera.com>
> wrote:
>
> > Hi Michael,
> >
> > Most likely this is on account of switching to Hadoop 3 by default (
> > https://github.com/apache/impala/commit/0812f8737a4301e5b52cad9dc91158
> > 9f9e98ecdb).
> > I sent an e-mail out about this yesterday, but I think what you're seeing
> > is a need to re-run buildall, and to possibly clean fe/generated-sources.
> > Hive1 and Hive2 have differing package names for their Thrift packages,
> and
> > there's a sed script in one of the CMake files which customize that
> > appropriately after running Thrift.
> >
> > The maven.jenkins.cloudera.com issue is harmless. The Sentry pom that we
> > depend on has an errant "" tag which shouldn't be there. I'm
> > working on getting that fixed up, but I don't think it's causing your
> > current issue. (That said, if you get an issue about failing to resolve
> > dependencies, it can be related, and but I think I've fixed the one we're
> > likely to run into already.)
> >
> > -- Philip
> >
> >
> >
> > On Wed, Mar 28, 2018 at 9:00 AM, Michael Ho <k...@cloudera.com> wrote:
> >
> > > During GVO, I ran into the following warning message
> > > (*maven.jenkins.cloudera.com <http://maven.jenkins.cloudera.com>: Name
> > > or service not known*) and eventually hit a compilation failure
> > > eventually. Not sure if anyone has run into this failure or if the
> > > warning message is a concern:
> > >
> > > [WARNING] Failure to transfer
> > > net.minidev:json-smart/maven-metadata.xml from
> > > http://maven.jenkins.cloudera.com:8081/artifactory/cauldron-
> > > latest-snapshot-local/
> > > was cached in the local repository, resolution will not be reattempted
> > > until the update interval of cdh.cauldron.repo has elapsed or updates
> > > are forced. Original error: Could not transfer metadata
> > > net.minidev:json-smart/maven-metadata.xml from/to cdh.cauldron.repo
> > > (http://maven.jenkins.cloudera.com:8081/artifactory/
> > > cauldron-latest-snapshot-local/):
> > > maven.jenkins.cloudera.com: Name or service not known
> > > [WARNING] Failure to transfer
> > > net.minidev:json-smart/maven-metadata.xml from
> > > http://maven.jenkins.cloudera.com:8081/artifactory/cauldron-
> > > latest-snapshot-local/
> > > was cached in the local repository, resolution will not be reattempted
> > > until the update interval of cdh.cauldron.repo has elapsed or updates
> > > are forced. Original error: Could not transfer metadata
> > > net.minidev:json-smart/maven-metadata.xml from/to cdh.cauldron.repo
> > > (http://maven.jenkins.cloudera.com:8081/artifactory/
> > > cauldron-latest-snapshot-local/):
> > > maven.jenkins.cloudera.com: Name or service not known
> > > [WARNING] Failure to transfer
> > > net.minidev:json-smart/maven-metadata.xml from
> > > http://maven.jenkins.cloudera.com:8081/artifactory/cauldron-
> > > latest-snapshot-local/
> > > was cached in the local repository, resolution will not be reattempted
> > > until the update interval of cdh.cauldron.repo has elapsed or updates
> > > are forced. Original error: Could not transfer metadata
> > > net.minidev:json-smart/maven-metadata.xml from/to cdh.cauldron.repo
> > > (http://maven.jenkins.cloudera.com:8081/artifactory/
> > > cauldron-latest-snapshot-local/):
> > > maven.jenkins.cloudera.com: Name or service not known
> > >
> > > [INFO] -
> > > [ERROR] COMPILATION ERROR :
> > > [INFO] -
> > > [ERROR] /home/ubuntu/Impala/fe/generated-sources/gen-java/
> > > org/apache/impala/analysis/ParquetHelper.java:[53,1]
> > > duplicate class: org.apache.impala.analysis.ParquetHelper
> > > [ERROR] /home/ubuntu/Impala/fe/generated-sources/gen-java/
> > > org/apache/impala/thrift/TMetadataOpRequest.java:[59,44]
> > > cannot find symbol
> > >   symbol:   class TGetInfoReq
> > >   location: package org.apache.hive.service.cli.thrift
> > > [ERROR] /home/ubuntu/Impal

Re: FE build failure

[Philip Zeyliger]

Hi Michael,

Most likely this is on account of switching to Hadoop 3 by default (
https://github.com/apache/impala/commit/0812f8737a4301e5b52cad9dc911589f9e98ecdb).
I sent an e-mail out about this yesterday, but I think what you're seeing
is a need to re-run buildall, and to possibly clean fe/generated-sources.
Hive1 and Hive2 have differing package names for their Thrift packages, and
there's a sed script in one of the CMake files which customize that
appropriately after running Thrift.

The maven.jenkins.cloudera.com issue is harmless. The Sentry pom that we
depend on has an errant "" tag which shouldn't be there. I'm
working on getting that fixed up, but I don't think it's causing your
current issue. (That said, if you get an issue about failing to resolve
dependencies, it can be related, and but I think I've fixed the one we're
likely to run into already.)

-- Philip



On Wed, Mar 28, 2018 at 9:00 AM, Michael Ho  wrote:

> During GVO, I ran into the following warning message
> (*maven.jenkins.cloudera.com : Name
> or service not known*) and eventually hit a compilation failure
> eventually. Not sure if anyone has run into this failure or if the
> warning message is a concern:
>
> [WARNING] Failure to transfer
> net.minidev:json-smart/maven-metadata.xml from
> http://maven.jenkins.cloudera.com:8081/artifactory/cauldron-
> latest-snapshot-local/
> was cached in the local repository, resolution will not be reattempted
> until the update interval of cdh.cauldron.repo has elapsed or updates
> are forced. Original error: Could not transfer metadata
> net.minidev:json-smart/maven-metadata.xml from/to cdh.cauldron.repo
> (http://maven.jenkins.cloudera.com:8081/artifactory/
> cauldron-latest-snapshot-local/):
> maven.jenkins.cloudera.com: Name or service not known
> [WARNING] Failure to transfer
> net.minidev:json-smart/maven-metadata.xml from
> http://maven.jenkins.cloudera.com:8081/artifactory/cauldron-
> latest-snapshot-local/
> was cached in the local repository, resolution will not be reattempted
> until the update interval of cdh.cauldron.repo has elapsed or updates
> are forced. Original error: Could not transfer metadata
> net.minidev:json-smart/maven-metadata.xml from/to cdh.cauldron.repo
> (http://maven.jenkins.cloudera.com:8081/artifactory/
> cauldron-latest-snapshot-local/):
> maven.jenkins.cloudera.com: Name or service not known
> [WARNING] Failure to transfer
> net.minidev:json-smart/maven-metadata.xml from
> http://maven.jenkins.cloudera.com:8081/artifactory/cauldron-
> latest-snapshot-local/
> was cached in the local repository, resolution will not be reattempted
> until the update interval of cdh.cauldron.repo has elapsed or updates
> are forced. Original error: Could not transfer metadata
> net.minidev:json-smart/maven-metadata.xml from/to cdh.cauldron.repo
> (http://maven.jenkins.cloudera.com:8081/artifactory/
> cauldron-latest-snapshot-local/):
> maven.jenkins.cloudera.com: Name or service not known
>
> [INFO] -
> [ERROR] COMPILATION ERROR :
> [INFO] -
> [ERROR] /home/ubuntu/Impala/fe/generated-sources/gen-java/
> org/apache/impala/analysis/ParquetHelper.java:[53,1]
> duplicate class: org.apache.impala.analysis.ParquetHelper
> [ERROR] /home/ubuntu/Impala/fe/generated-sources/gen-java/
> org/apache/impala/thrift/TMetadataOpRequest.java:[59,44]
> cannot find symbol
>   symbol:   class TGetInfoReq
>   location: package org.apache.hive.service.cli.thrift
> [ERROR] /home/ubuntu/Impala/fe/generated-sources/gen-java/
> org/apache/impala/thrift/TMetadataOpRequest.java:[60,44]
> cannot find symbol
>   symbol:   class TGetTypeInfoReq
>   location: package org.apache.hive.service.cli.thrift
> [ERROR] /home/ubuntu/Impala/fe/generated-sources/gen-java/
> org/apache/impala/thrift/TMetadataOpRequest.java:[61,44]
> cannot find symbol
>   symbol:   class TGetCatalogsReq
>   location: package org.apache.hive.service.cli.thrift
> [ERROR] /home/ubuntu/Impala/fe/generated-sources/gen-java/
> org/apache/impala/thrift/TMetadataOpRequest.java:[62,44]
> cannot find symbol
>   symbol:   class TGetSchemasReq
>   location: package org.apache.hive.service.cli.thrift
>

Updating your minicluster to Hadoop 3.0...

[Philip Zeyliger]

Hi folks,

I just sent off https://gerrit.cloudera.org/#/c/9743/ and
https://issues.apache.org/jira/browse/IMPALA-4277 for GVD, which changes
the default minicluster to be based on Hadoop 3.0, Hive 2.1, Sentry 2.0,
and so on. This change *will not* be back-ported to 2.x.

When you pull that change in, you'll need to re-build your minicluster
with, e.g., ./buildall.sh -testdata -format -notests. This will pull in the
new dependencies, format your cluster, and load up all the data. As you
know, it takes 1-2 hours.

If you want to hold off, you can elso set export
IMPALA_MINICLUSTER_PROFILE_OVERRIDE=2 in your environment.

Note that this choice between versions happens at build time, and CMake
depends on it. So, switching back and forth requires re-running CMake.

Please let me know if you run into any trouble. This is a big enough that
there may be some bumps on the road.

-- Philip

test_semi_joins_exhaustive error?

[Philip Zeyliger]

Hi folks,

I'm curious if the following error looks familiar to folks. I'm running
exhaustive tests on a 32 GB machine with a slightly lowered impalad
memlimit and am pretty much up to date
(commit 623ace0e4c84297606d13f321bd2ceee70a8b9f2 plus the memlimit change).

Thanks!

-- Philip



2018-02-23 06:40:58 === FAILURES
===
2018-02-23 06:40:58
TestSemiJoinQueries.test_semi_joins_exhaustive[batch_size: 0 | exec_option:
{'batch_size': 0, 'num_nodes': 0, 'disable_codegen_rows_threshold': 0,
'disable_codegen': True, 'abort_on_error': 1,
'exec_single_node_rows_threshold': 0} | table_format: parquet/none]
2018-02-23 06:40:58 [gw2] linux2 -- Python 2.7.12
/home/philip/src/Impala/bin/../infra/python/env/bin/python
2018-02-23 06:40:58 query_test/test_join_queries.py:176: in
test_semi_joins_exhaustive
2018-02-23 06:40:58
 self.run_test_case('QueryTest/semi-joins-exhaustive', new_vector)
2018-02-23 06:40:58 common/impala_test_suite.py:397: in run_test_case
2018-02-23 06:40:58 result =
self.__execute_query(target_impalad_client, query, user=user)
2018-02-23 06:40:58 common/impala_test_suite.py:612: in __execute_query
2018-02-23 06:40:58 return impalad_client.execute(query, user=user)
2018-02-23 06:40:58 common/impala_connection.py:160: in execute
2018-02-23 06:40:58 return self.__beeswax_client.execute(sql_stmt,
user=user)


 2018-02-23
06:40:58 beeswax/impala_beeswax.py:173: in execute



 2018-02-23 06:40:58
 handle = self.__execute_query(query_string.strip(), user=user)
2018-02-23 06:40:58 beeswax/impala_beeswax.py:341: in __execute_query
2018-02-23 06:40:58 self.wait_for_completion(handle)



2018-02-23
06:40:58 beeswax/impala_beeswax.py:361: in wait_for_completion
2018-02-23 06:40:58 raise ImpalaBeeswaxException("Query aborted:" +
error_log, None)
2018-02-23 06:40:58 E   ImpalaBeeswaxException: ImpalaBeeswaxException:



 2018-02-23
06:40:58 EQuery aborted:Cannot perform hash join at node with id 4.
Repartitioning did not reduce the size of a spilled partition.
Repartitioning level 1. Number of rows 12100:


 2018-02-23 06:40:58 E   PartitionedHashJoinNode (id=4 op=2
state=RepartitioningBuild #spilled_partitions=0)
2018-02-23 06:40:58 E   PhjBuilder: Hash partitions: 16:
2018-02-23 06:40:58 EHash partition 0 : ptr=0xbc57ac0 Closed
2018-02-23 06:40:58 EHash partition 1 : ptr=0xad6cb60 Spilled
2018-02-23 06:40:58 E   Build Rows: 12100 (Bytes pinned: 0)
2018-02-23 06:40:58 E
2018-02-23 06:40:58 EHash partition 2 : ptr=0x131fa5c0 Closed
2018-02-23 06:40:58 EHash partition 3 : ptr=0x131fb760 Closed
2018-02-23 06:40:58 EHash partition 4 : ptr=0xad6ddc0
Closed



 2018-02-23 06:40:58 EHash partition 5 : ptr=0xad6c340 Closed
2018-02-23 06:40:58 EHash partition 6 : ptr=0xad6c1e0 Closed
2018-02-23 06:40:58 EHash partition 7 : ptr=0xad6d700 Closed
2018-02-23 06:40:58 EHash partition 8 : ptr=0xad6ca20 Closed
2018-02-23 06:40:58 EHash partition 9 : ptr=0xad6d4c0 Closed
2018-02-23 06:40:58 EHash partition 10 : ptr=0xad6df80 Closed
2018-02-23 06:40:58 EHash partition 11 : ptr=0xad6d300
Closed



2018-02-23 06:40:58 EHash partition 12 : ptr=0xad6c420 Closed
2018-02-23 06:40:58 EHash partition 13 : ptr=0x1187f7a0
Closed
2018-02-23 06:40:58 EHash partition 14 : ptr=0x1187e200
Closed
2018-02-23 06:40:58 EHash partition 15 : ptr=0xad6cf00 Closed
2018-02-23 06:40:58 E   Probe hash partitions: 0:
2018-02-23 06:40:58 E   InputPartition: 0x19dc64860
2018-02-23 06:40:58 E  Build Partition Closed
2018-02-23 06:40:58 E  Spilled Probe Rows: 248
2018-02-23 06:40:58 E



 2018-02-23
06:40:58 E0x124d5548 internal state:
{ 0xb306c40 name: HASH_JOIN_NODE id=4 ptr=0x124d5400
write_status:  buffers allocated 0 num_pages: 2 pinned_bytes: 2097152
dirty_unpinned_bytes: 0 in_flight_write_bytes: 0 reservation:
{: reservation_limit 9223372036854775807 reservation
6272581632 used_reservation 2097152 child_reservations 2097152 parent:




2018-02-23 06:40:58 E   : reservation_limit
9223372036854775807 reservation 6272581632 used_reservation 0
child_reservations 6272581632 parent:


   2018-02-23 06:40:58 E   : reservation_limit
6273807482 reservation 6272581632 used_reservation 0 child_reservations
6272581632 parent:
2018-02-23 06:40:58 E   : reservation_limit 6665912320
reservation 6356467712 used_reservation 0 child_reservations 6356467712
parent:


2018-02-23 06:40:58 E   NULL}
2018-02-23 06:40:58 E 1 pinned pages:  0x9ba94a0 len:
2097152 pin_count: 1 buf:  0x9ba9518 client:
0x124d5548/0xb306c40 data: 0x14a0 len: 2097152
2018-02-23 06:40:58 E
2018-02-23 06:40:58 E 0 dirty 

Re: Setting up ASF Git Bot?

[Philip Zeyliger]

Strange. It definitely looked like it was working when that issue
(INFRA-15960) got resolved. Perhaps file an INFRA jira again?

Thanks!

-- Philip



On Wed, Feb 21, 2018 at 9:52 AM, Tim Armstrong <tarmstr...@cloudera.com>
wrote:

> What happened to git bot? I noticed that it stopped posting on JIRAs. E.g.
> I had to copy and paste the commit message manually on this one:
> https://issues.apache.org/jira/browse/IMPALA-6497
>
> On Thu, Feb 1, 2018 at 10:17 AM, Philip Zeyliger <phi...@cloudera.com>
> wrote:
>
> > I filed https://issues.apache.org/jira/browse/INFRA-15960.
> >
> > Obviously, we'll keep an eye out, and if we find we dislike jira-bot, we
> > can disable it again.
> >
> > -- Philip
> >
> > On Thu, Feb 1, 2018 at 9:38 AM, Daniel Hecht <dhe...@cloudera.com>
> wrote:
> >
> > > I'm strongly in favor. Thanks Phil.
> > >
> > > Dan
> > >
> > > On Thu, Feb 1, 2018 at 9:30 AM, Jim Apple <jbap...@cloudera.com>
> wrote:
> > >
> > > > Nice. Thanks for digging in!
> > > >
> > > > On Thu, Feb 1, 2018 at 8:56 AM, Philip Zeyliger <phi...@cloudera.com
> >
> > > > wrote:
> > > > > I see similar messages against git-wip-us, so I think it's
> plausible
> > we
> > > > > don't need to do anything as major as that.
> > > > >
> > > > > https://issues.apache.org/jira/browse/SOLR-11715?
> > > > focusedCommentId=16348814=com.atlassian.jira.
> > > > plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16348814
> > > > >
> > > > > [image: jira-bot]ASF subversion and git services
> > > > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> > name=jira-bot>
> > > > added
> > > > > a comment - 1 hour ago
> > > > >
> > > > > Commit 8817d31a4d5ccce29693da08cca74ff26f7ef992 in lucene-solr's
> > > branch
> > > > > refs/heads/branch_7x from Cassandra Targett
> > > > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?
> > name=ctargett>
> > > > > [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=
> > 8817d31
> > > ]
> > > > >
> > > > > SOLR-11715 <https://issues.apache.org/jira/browse/SOLR-11715>: Fix
> > > > example
> > > > > queries so they appear correctly; add fl param to geodist example
> > > > >
> > > > > On Thu, Feb 1, 2018 at 1:35 AM, Jim Apple <jbap...@cloudera.com>
> > > wrote:
> > > > >
> > > > >> We might have to sign up for ASF gitbox to get that, which might
> > > entail
> > > > >> other workflow changes.
> > > > >>
> > > > >> On Wed, Jan 31, 2018 at 1:37 PM Philip Zeyliger <
> > phi...@cloudera.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Hi folks,
> > > > >> >
> > > > >> > Would people be interested in ASF's bot that connects JIRAs to
> > > > Commits?
> > > > >> You
> > > > >> > can see the "activity stream" of
> > > > >> > https://issues.apache.org/jira/secure/ViewProfile.jspa?
> > > name=jira-bot
> > > > for
> > > > >> > an
> > > > >> > example, and there's a sample below for convenience.
> > > > >> >
> > > > >> > Hopefully it will make it more obvious that we should resolve
> > JIRAs
> > > > once
> > > > >> > they're done, and make it a little easier for folks to follow
> the
> > > > bread
> > > > >> > crumbs. I know many of us already cut/paste the commit link into
> > > JIRA.
> > > > >> >
> > > > >> > I've not looked into the mechanics, but I imagine it's an ASF
> > INFRA
> > > > >> ticket
> > > > >> > away.
> > > > >> >
> > > > >> > Thanks!
> > > > >> >
> > > > >> > -- Philip
> > > > >> >
> > > > >> > *Sample:*
> > > > >> >
> > > > >> > Commit 17a35f464fc283cfe970536011b1d821c4de031e in couchdb's
> > branch
> > > > >> > refs/heads/COUCHDB-3287
> > > > >> > <https://issues.apache.org/jira/browse/COUCHDB-3287
> > > > >> > >-pluggable-storage-engines
> > > > >> > from Paul Joseph Davis
> > > > >> > <
> > > > >> > https://issues.apache.org/jira/secure/ViewProfile.jspa?
> > > > >> name=paul.joseph.davis
> > > > >> > >
> > > > >> > [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=17a35f4 ]
> > > > >> >
> > > > >> > Add storage engine test suite
> > > > >> >
> > > > >> > This allows other storage engine implementations to reuse the
> same
> > > > exact
> > > > >> > test suite without having to resort to shenanigans like keeping
> > > > vendored
> > > > >> > copies up to date.
> > > > >> >
> > > > >> > COUCHDB-3287 <https://issues.apache.org/
> jira/browse/COUCHDB-3287>
> > > > >> >
> > > > >>
> > > >
> > >
> >
>

Re: IMPALA-3343: impala-shell compatibility with Python 3

[Philip Zeyliger]

I'm not aware of anyone having tried it, so I don't think we know. I'd also
check in on Thrift's support for Python3. That seems like the most likely
library. You can certainly give removing print a shot!

-- Philip

On Wed, Feb 14, 2018 at 11:50 AM, Diggs, Asoka 
wrote:

> I have just encountered this error myself after my project made the
> decision to switch from Python 2 to 3.
>
> I’ve added a comment to the JIRA with some additional details from my own
> encounter with the error.
>
>
> Looking at the impala-shell.py file, it’s clear that the error I
> encountered – print statements formulated using Python 2 syntax that is not
> compatible with Python 3 – exist throughout the code.  If this is the
> extent of the incompatibility, then that’s a problem I am ready and willing
> to tackle.
>
> Is there reason to believe that this problem is as simple as changing
> print statement syntax, or reason to believe it is significantly more than
> that?
>
> Thanks for the insight.
> Asoka Diggs
>
>
>
>

Re: TEST_START_CLUSTER_ARGS are not applied in custom cluster tests

[Philip Zeyliger]

It's definitely handy. I've used it for specifying things like --mem_limit
when running inside a container. I think Lars' question is more about what
the custom cluster tests should do with it.

-- Philip

On Tue, Feb 13, 2018 at 10:48 AM, Matthew Jacobs 
wrote:

> I had introduced this while I was working at Cloudera for jenkins jobs
> to specify RM-related parameters to the test environment. At that
> time, I had one or more jenkins jobs that were setting this
> environment variable. I do not know if Cloudera or any other Impala
> developers are trying to use this. If nobody seems to be using this
> anymore, it would be good to remove.
>
> Best,
> mj
>
> commit adf4b4863d7ea38fda21ec76f5699b4fd31e10c0
> Author: Matthew Jacobs 
> Date:   Wed May 13 17:14:11 2015 -0700
>
> Allow specifying cluster start args in run-all-tests.sh
>
> This will enable jenkins jobs to specify custom arguments when
> starting the mini cluster (via start-impala-cluster.py). This
> will be used to create a jenkins job that runs tests with RM
> enabled.
>
> Change-Id: I96a2e8d90db448581bbf448f3df514381f79fb27
> Reviewed-on: http://gerrit.cloudera.org:8080/380
> Reviewed-by: Matthew Jacobs 
> Tested-by: Internal Jenkins
>
> On Tue, Feb 13, 2018 at 10:18 AM, Lars Volker  wrote:
> > Hi All,
> >
> > Mike noticed in a review that TEST_START_CLUSTER_ARGS are not applied in
> > custom cluster tests and we wondered whether that is on purpose or a bug.
> > Can someone with historic knowledge shed some light on it? The relevant
> > code is here:
> >
> > https://github.com/apache/impala/blob/5f7599687748b1e1ce0a5a34d38cc4
> 9a4c4cd9f5/tests/common/custom_cluster_test_suite.py#L124
> >
> > Thanks, Lars
>

Re: Setting up ASF Git Bot?

[Philip Zeyliger]

I filed https://issues.apache.org/jira/browse/INFRA-15960.

Obviously, we'll keep an eye out, and if we find we dislike jira-bot, we
can disable it again.

-- Philip

On Thu, Feb 1, 2018 at 9:38 AM, Daniel Hecht <dhe...@cloudera.com> wrote:

> I'm strongly in favor. Thanks Phil.
>
> Dan
>
> On Thu, Feb 1, 2018 at 9:30 AM, Jim Apple <jbap...@cloudera.com> wrote:
>
> > Nice. Thanks for digging in!
> >
> > On Thu, Feb 1, 2018 at 8:56 AM, Philip Zeyliger <phi...@cloudera.com>
> > wrote:
> > > I see similar messages against git-wip-us, so I think it's plausible we
> > > don't need to do anything as major as that.
> > >
> > > https://issues.apache.org/jira/browse/SOLR-11715?
> > focusedCommentId=16348814=com.atlassian.jira.
> > plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16348814
> > >
> > > [image: jira-bot]ASF subversion and git services
> > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=jira-bot>
> > added
> > > a comment - 1 hour ago
> > >
> > > Commit 8817d31a4d5ccce29693da08cca74ff26f7ef992 in lucene-solr's
> branch
> > > refs/heads/branch_7x from Cassandra Targett
> > > <https://issues.apache.org/jira/secure/ViewProfile.jspa?name=ctargett>
> > > [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8817d31
> ]
> > >
> > > SOLR-11715 <https://issues.apache.org/jira/browse/SOLR-11715>: Fix
> > example
> > > queries so they appear correctly; add fl param to geodist example
> > >
> > > On Thu, Feb 1, 2018 at 1:35 AM, Jim Apple <jbap...@cloudera.com>
> wrote:
> > >
> > >> We might have to sign up for ASF gitbox to get that, which might
> entail
> > >> other workflow changes.
> > >>
> > >> On Wed, Jan 31, 2018 at 1:37 PM Philip Zeyliger <phi...@cloudera.com>
> > >> wrote:
> > >>
> > >> > Hi folks,
> > >> >
> > >> > Would people be interested in ASF's bot that connects JIRAs to
> > Commits?
> > >> You
> > >> > can see the "activity stream" of
> > >> > https://issues.apache.org/jira/secure/ViewProfile.jspa?
> name=jira-bot
> > for
> > >> > an
> > >> > example, and there's a sample below for convenience.
> > >> >
> > >> > Hopefully it will make it more obvious that we should resolve JIRAs
> > once
> > >> > they're done, and make it a little easier for folks to follow the
> > bread
> > >> > crumbs. I know many of us already cut/paste the commit link into
> JIRA.
> > >> >
> > >> > I've not looked into the mechanics, but I imagine it's an ASF INFRA
> > >> ticket
> > >> > away.
> > >> >
> > >> > Thanks!
> > >> >
> > >> > -- Philip
> > >> >
> > >> > *Sample:*
> > >> >
> > >> > Commit 17a35f464fc283cfe970536011b1d821c4de031e in couchdb's branch
> > >> > refs/heads/COUCHDB-3287
> > >> > <https://issues.apache.org/jira/browse/COUCHDB-3287
> > >> > >-pluggable-storage-engines
> > >> > from Paul Joseph Davis
> > >> > <
> > >> > https://issues.apache.org/jira/secure/ViewProfile.jspa?
> > >> name=paul.joseph.davis
> > >> > >
> > >> > [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=17a35f4 ]
> > >> >
> > >> > Add storage engine test suite
> > >> >
> > >> > This allows other storage engine implementations to reuse the same
> > exact
> > >> > test suite without having to resort to shenanigans like keeping
> > vendored
> > >> > copies up to date.
> > >> >
> > >> > COUCHDB-3287 <https://issues.apache.org/jira/browse/COUCHDB-3287>
> > >> >
> > >>
> >
>

Setting up ASF Git Bot?

[Philip Zeyliger]

Hi folks,

Would people be interested in ASF's bot that connects JIRAs to Commits? You
can see the "activity stream" of
https://issues.apache.org/jira/secure/ViewProfile.jspa?name=jira-bot for an
example, and there's a sample below for convenience.

Hopefully it will make it more obvious that we should resolve JIRAs once
they're done, and make it a little easier for folks to follow the bread
crumbs. I know many of us already cut/paste the commit link into JIRA.

I've not looked into the mechanics, but I imagine it's an ASF INFRA ticket
away.

Thanks!

-- Philip

*Sample:*

Commit 17a35f464fc283cfe970536011b1d821c4de031e in couchdb's branch
refs/heads/COUCHDB-3287
-pluggable-storage-engines
from Paul Joseph Davis

[ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=17a35f4 ]

Add storage engine test suite

This allows other storage engine implementations to reuse the same exact
test suite without having to resort to shenanigans like keeping vendored
copies up to date.

COUCHDB-3287 

Re: Impala 3.x (and 2.x)

[Philip Zeyliger]

Hi folks!

As an update here: I've built
https://jenkins.impala.io/job/cherrypick-2.x-and-test/ as a test bed for
how this could work. (Because the cherrypick tool at review 9045 is still
pending GVO, I just hard-coded a single cherrypick.) Whenever "master"
changes, that Jenkins jobs would use a cherrypick tool to cherrypick things
into refs/sandbox/impala-public-jenkins/2.x-staging. Then, it would run
parallel-all-tests. If parallel-all-tests passes, all the cherrypicks are
promoted to asf-gerrit/2.x.

I thought about doing one test run per commit, but I think that's very
liable to get stuck. I think we occasionally have commits that introduce a
test failure, followed by a commit that fixes it. The commit that
introduced the test failure would never pass the gate, and we'd get stuck.
(There's obviously the question of how it passed the original gate, but
that sometimes happens due to races and environmental factors.)

Please let me know if you've got any concerns!

-- Philip


On Fri, Jan 19, 2018 at 4:17 PM, Jim Apple <jbap...@cloudera.com> wrote:

> We could commit first and then GVO -- that way if a commit broke
> something (in a reproducible way), we would at least know which commit
> did it. In this proposal, we don't have to wait for GVO N to finish
> before committing patch N+1. This is a sort of fast-path-slow-path
> idea, but fixing the slow path while keeping 2.x stable could be
> tricky.
>
> On Fri, Jan 19, 2018 at 3:53 PM, Dimitris Tsirogiannis
> <dtsirogian...@cloudera.com> wrote:
> > My ideal and more conservative policy would be:
> > - If conflicts, review.
> > - Always GVO.
> >
> > Dimitris
> >
> > On Fri, Jan 19, 2018 at 2:13 PM, Taras Bobrovytsky <taras...@apache.org>
> > wrote:
> >
> >> I like your policy suggestion. If there are no conflicts, then we can
> push
> >> from master to 2.x without any tests or reviews. If there are conflicts,
> >> then a review AND a GVO test run are required.
> >>
> >> We can at least start off with this policy and then change it if it is
> not
> >> working well for some reason.
> >>
> >> On Fri, Jan 19, 2018 at 1:59 PM, Philip Zeyliger <phi...@cloudera.com>
> >> wrote:
> >>
> >> > An update here!
> >> >
> >> > I'm pretty close to pushing the first master-only change (the very
> >> exciting
> >> > 1-liner at https://gerrit.cloudera.org/#/c/9044/ that bumps
> versions).
> >> > After that, I'll be cherrypicking things into 2.x.
> >> >
> >> > We need a policy, I think, on reviewing these cherry-picks. The most
> >> > heavy-weight would be to do Gerrit and run-tests-before-merge for
> every
> >> > change. The least heavy would be to review only when the cherry-picks
> >> > aren't clean. Are people comfortable with the less heavy policy?
> >> >
> >> > Specifically, I propose that clean cherrypicks from master to 2.x can
> be
> >> > pushed without additional review.
> >> >
> >> > Thanks!
> >> >
> >> > -- Philip
> >> >
> >> > On Wed, Jan 17, 2018 at 9:57 AM, Philip Zeyliger <phi...@cloudera.com
> >
> >> > wrote:
> >> >
> >> > > It sounds like we've reached consensus, so I'm starting to maneuver
> >> this
> >> > > along. Please don't hesitate if you've got concerns. You can follow
> >> along
> >> > > at https://issues.apache.org/jira/browse/IMPALA-6410.
> >> > >
> >> > > The first two reviews are out at:
> >> > >
> >> > > remote:   http://gerrit.cloudera.org:8080/9044 Bumping version to
> 3.0.
> >> > > remote:   http://gerrit.cloudera.org:8080/9045 IMPALA-6410: Tool to
> >> > > cherrypick changes across branches.
> >> > >
> >> > > I've created the branches on Gerrit and Apache as follows:
> >> > >
> >> > > # This created the branch on gerrit.cloudera.org
> >> > > $ssh -p 29418 ph...@gerrit.cloudera.org gerrit create-branch
> >> Impala-ASF
> >> > > 2.x master
> >> > >
> >> > > # This created the branch on the Apache git server:
> >> > > $git push apache asf-gerrit/2.x:refs/heads/2.x
> >> > > Username for 'https://git-wip-us.apache.org': philz
> >> > > Password for 'https://ph...@git-wip-us.apache.org':
> >> > > Total 0 (delta 0), reused 0 (delta 0)
> >> > > To https://git-wip-us.apache.org/repos/asf/impala.git
> >> > 

Re: Impala 3.x (and 2.x)

[Philip Zeyliger]

If we do the GVO route, would folks be comfortable with Jenkins taking care
of promoting cherrypicks to a staging branch (say, 2.x-staging), and then
Jenkins, again, running the tests and pushing that through to gerrit/2.x?
(As always, the push to the official ASF repo is only done by a human,
using bin/push_to_asf.sh.)

Would folks be comfortable if I phased in GVO over the coming days?

Currently, GVO integration is tied up with Gerrit, and generating Gerrit
e-mails for things that we don't want to review seems spammy.

Thanks!

-- Philip

On Fri, Jan 19, 2018 at 3:53 PM, Dimitris Tsirogiannis <
dtsirogian...@cloudera.com> wrote:

> My ideal and more conservative policy would be:
> - If conflicts, review.
> - Always GVO.
>
> Dimitris
>
> On Fri, Jan 19, 2018 at 2:13 PM, Taras Bobrovytsky <taras...@apache.org>
> wrote:
>
> > I like your policy suggestion. If there are no conflicts, then we can
> push
> > from master to 2.x without any tests or reviews. If there are conflicts,
> > then a review AND a GVO test run are required.
> >
> > We can at least start off with this policy and then change it if it is
> not
> > working well for some reason.
> >
> > On Fri, Jan 19, 2018 at 1:59 PM, Philip Zeyliger <phi...@cloudera.com>
> > wrote:
> >
> > > An update here!
> > >
> > > I'm pretty close to pushing the first master-only change (the very
> > exciting
> > > 1-liner at https://gerrit.cloudera.org/#/c/9044/ that bumps versions).
> > > After that, I'll be cherrypicking things into 2.x.
> > >
> > > We need a policy, I think, on reviewing these cherry-picks. The most
> > > heavy-weight would be to do Gerrit and run-tests-before-merge for every
> > > change. The least heavy would be to review only when the cherry-picks
> > > aren't clean. Are people comfortable with the less heavy policy?
> > >
> > > Specifically, I propose that clean cherrypicks from master to 2.x can
> be
> > > pushed without additional review.
> > >
> > > Thanks!
> > >
> > > -- Philip
> > >
> > > On Wed, Jan 17, 2018 at 9:57 AM, Philip Zeyliger <phi...@cloudera.com>
> > > wrote:
> > >
> > > > It sounds like we've reached consensus, so I'm starting to maneuver
> > this
> > > > along. Please don't hesitate if you've got concerns. You can follow
> > along
> > > > at https://issues.apache.org/jira/browse/IMPALA-6410.
> > > >
> > > > The first two reviews are out at:
> > > >
> > > > remote:   http://gerrit.cloudera.org:8080/9044 Bumping version to
> 3.0.
> > > > remote:   http://gerrit.cloudera.org:8080/9045 IMPALA-6410: Tool to
> > > > cherrypick changes across branches.
> > > >
> > > > I've created the branches on Gerrit and Apache as follows:
> > > >
> > > > # This created the branch on gerrit.cloudera.org
> > > > $ssh -p 29418 ph...@gerrit.cloudera.org gerrit create-branch
> > Impala-ASF
> > > > 2.x master
> > > >
> > > > # This created the branch on the Apache git server:
> > > > $git push apache asf-gerrit/2.x:refs/heads/2.x
> > > > Username for 'https://git-wip-us.apache.org': philz
> > > > Password for 'https://ph...@git-wip-us.apache.org':
> > > > Total 0 (delta 0), reused 0 (delta 0)
> > > > To https://git-wip-us.apache.org/repos/asf/impala.git
> > > >  * [new branch]  asf-gerrit/2.x -> 2.x
> > > >
> > > > # Double-check:
> > > > $git-ls-remote apache | grep 2.x; git-ls-remote asf-gerrit | grep 2.x
> > > > 6cc76d72016b8d5672676e8e8a979b0807803ad9refs/heads/2.x
> > > > 6cc76d72016b8d5672676e8e8a979b0807803ad9refs/heads/2.x
> > > >
> > > > # push_to_asf learned of the branch "magically"
> > > > $bin/push_to_asf.py
> > > > INFO:root:Fetching from remote 'asf-gerrit'...
> > > > INFO:root:done
> > > > Branch '2.x':up to date
> > > > Branch 'asf-site':   up to date
> > > > Branch 'branch-2.10.0':  found on Apache but not in gerrit
> > > > Branch 'branch-2.11.0':  found on Apache but not in gerrit
> > > > Branch 'branch-2.7.0':   found on Apache but not in gerrit
> > > > Branch 'branch-2.8.0':   found on Apache but not in gerrit
> > > > Branch 'branch-2.9.0':   found on Apache but not in gerrit
> > > > Branch 'hadoop-next':up to date
> > > > Branch 'master': up to date

Re: Impala 3.x (and 2.x)

[Philip Zeyliger]

An update here!

I'm pretty close to pushing the first master-only change (the very exciting
1-liner at https://gerrit.cloudera.org/#/c/9044/ that bumps versions).
After that, I'll be cherrypicking things into 2.x.

We need a policy, I think, on reviewing these cherry-picks. The most
heavy-weight would be to do Gerrit and run-tests-before-merge for every
change. The least heavy would be to review only when the cherry-picks
aren't clean. Are people comfortable with the less heavy policy?

Specifically, I propose that clean cherrypicks from master to 2.x can be
pushed without additional review.

Thanks!

-- Philip

On Wed, Jan 17, 2018 at 9:57 AM, Philip Zeyliger <phi...@cloudera.com>
wrote:

> It sounds like we've reached consensus, so I'm starting to maneuver this
> along. Please don't hesitate if you've got concerns. You can follow along
> at https://issues.apache.org/jira/browse/IMPALA-6410.
>
> The first two reviews are out at:
>
> remote:   http://gerrit.cloudera.org:8080/9044 Bumping version to 3.0.
> remote:   http://gerrit.cloudera.org:8080/9045 IMPALA-6410: Tool to
> cherrypick changes across branches.
>
> I've created the branches on Gerrit and Apache as follows:
>
> # This created the branch on gerrit.cloudera.org
> $ssh -p 29418 ph...@gerrit.cloudera.org gerrit create-branch Impala-ASF
> 2.x master
>
> # This created the branch on the Apache git server:
> $git push apache asf-gerrit/2.x:refs/heads/2.x
> Username for 'https://git-wip-us.apache.org': philz
> Password for 'https://ph...@git-wip-us.apache.org':
> Total 0 (delta 0), reused 0 (delta 0)
> To https://git-wip-us.apache.org/repos/asf/impala.git
>  * [new branch]  asf-gerrit/2.x -> 2.x
>
> # Double-check:
> $git-ls-remote apache | grep 2.x; git-ls-remote asf-gerrit | grep 2.x
> 6cc76d72016b8d5672676e8e8a979b0807803ad9refs/heads/2.x
> 6cc76d72016b8d5672676e8e8a979b0807803ad9refs/heads/2.x
>
> # push_to_asf learned of the branch "magically"
> $bin/push_to_asf.py
> INFO:root:Fetching from remote 'asf-gerrit'...
> INFO:root:done
> Branch '2.x':up to date
> Branch 'asf-site':   up to date
> Branch 'branch-2.10.0':  found on Apache but not in gerrit
> Branch 'branch-2.11.0':  found on Apache but not in gerrit
> Branch 'branch-2.7.0':   found on Apache but not in gerrit
> Branch 'branch-2.8.0':   found on Apache but not in gerrit
> Branch 'branch-2.9.0':   found on Apache but not in gerrit
> Branch 'hadoop-next':up to date
> Branch 'master': up to date
>
> -- Philip
>
> On Tue, Jan 16, 2018 at 5:29 PM, Alexander Behm <alex.b...@cloudera.com>
> wrote:
>
>> +1
>>
>> On Tue, Jan 16, 2018 at 5:27 PM, Taras Bobrovytsky <taras...@apache.org>
>> wrote:
>>
>> > +1
>> >
>> > On Tue, Jan 16, 2018 at 1:34 PM, Lars Volker <l...@cloudera.com> wrote:
>> >
>> > > +1
>> > >
>> > > On Tue, Jan 16, 2018 at 1:29 PM, Philip Zeyliger <phi...@cloudera.com
>> >
>> > > wrote:
>> > >
>> > > > Hi folks!
>> > > >
>> > > > It sounds like there haven't been objections to having master be
>> "3.0"
>> > > and
>> > > > introducing a 2.x branch. Would folks be alright if I started making
>> > > > changes in that direction?
>> > > >
>> > > > Thanks!
>> > > >
>> > > > -- Philip
>> > > >
>> > > > On Fri, Jan 12, 2018 at 4:10 PM, Philip Zeyliger <
>> phi...@cloudera.com>
>> > > > wrote:
>> > > >
>> > > > >
>> > > > >
>> > > > > On Fri, Jan 12, 2018 at 4:04 PM, Jim Apple <jbap...@cloudera.com>
>> > > wrote:
>> > > > >
>> > > > >> This makes sense to me.
>> > > > >>
>> > > > >> In this mode, for 2.x-only changes and for changes on 3.0 that
>> don't
>> > > > >> apply cleanly, there will be a manual way to do the step labelled
>> > "1.
>> > > > >> Cherrypick tool", and that way is the same way we send patches
>> for
>> > > > >> review now, but pushing to HEAD:refs/for/2.x rather than
>> > > > >> HEAD:refs/for/master, yes?
>> > > > >>
>> > > > >
>> > > > > Exactly. So, non-clean cherrypicks or 2.x-only changes go through
>> > > review
>> > > > > on Gerrit, but we give an implicit review pass to clean
>> cherrypicks.
>> > > > >
>

Re: Impala 3.x (and 2.x)

[Philip Zeyliger]

Hi folks!

It sounds like there haven't been objections to having master be "3.0" and
introducing a 2.x branch. Would folks be alright if I started making
changes in that direction?

Thanks!

-- Philip

On Fri, Jan 12, 2018 at 4:10 PM, Philip Zeyliger <phi...@cloudera.com>
wrote:

>
>
> On Fri, Jan 12, 2018 at 4:04 PM, Jim Apple <jbap...@cloudera.com> wrote:
>
>> This makes sense to me.
>>
>> In this mode, for 2.x-only changes and for changes on 3.0 that don't
>> apply cleanly, there will be a manual way to do the step labelled "1.
>> Cherrypick tool", and that way is the same way we send patches for
>> review now, but pushing to HEAD:refs/for/2.x rather than
>> HEAD:refs/for/master, yes?
>>
>
> Exactly. So, non-clean cherrypicks or 2.x-only changes go through review
> on Gerrit, but we give an implicit review pass to clean cherrypicks.
>
> We could have the cherrypick tool between gerrit/master and gerrit/2.x do
> the cherrypicks and run the tests on Jenkins. Do you think that's
> preferable?
>
> -- Philip
>
>
>
>>
>> On Fri, Jan 12, 2018 at 3:57 PM, Philip Zeyliger <phi...@cloudera.com>
>> wrote:
>> > Picture:
>> > https://gist.github.com/philz/323c8b4cb411dc12eb7231d922c195
>> 1f#file-impala-branch-image-pdf
>> >
>> >
>> > On Fri, Jan 12, 2018 at 3:47 PM, Jim Apple <jbap...@cloudera.com>
>> wrote:
>> >
>> >> Often, this list seems to filter out images. Could you post it and
>> send a
>> >> link?
>> >>
>> >> Thanks for taking this on, Phil!
>> >>
>> >> On Fri, Jan 12, 2018 at 3:15 PM, Philip Zeyliger <phi...@cloudera.com>
>> >> wrote:
>> >>
>> >> > I think most patches go to Gerrit branch 'master', which happens to
>> >> > identify itself as 3.0. (Or 3.x?).
>> >> >
>> >> > Here's a picture:
>> >> >
>> >> > [image: Inline image 1]
>> >> >
>> >> >
>> >> > With this, every time "cherrypick_and_push_to_asf.py" is run, it
>> would
>> >> > first offer to cherrypick changes between master and 2.x. Then, it
>> would
>> >> > offer push those cherrypicks to gerrit/2.x. After that, it continues
>> on
>> >> as
>> >> > before and offers to push changes to ASF. I think this maintains the
>> >> > invariant that pushing to ASF is only done with a human trigger. (We
>> >> could
>> >> > also have step 1 be done by a Jenkins robot, since it's between
>> Gerrit
>> >> and
>> >> > Gerrit.)
>> >> >
>> >> > I looked at the How to Release page, and the main difference would be
>> >> > that, for a 2.x release, the $COMMIT_HASH_YOU_CHOSE would come from
>> the
>> >> 2.x
>> >> > branch, as would any cherrypicks.
>> >> >
>> >> > Does this match what you're thinking?
>> >> >
>> >> > Thanks!
>> >> >
>> >> > -- Philip
>> >> >
>> >> > On Fri, Jan 12, 2018 at 11:07 AM, Jim Apple <jbap...@cloudera.com>
>> >> wrote:
>> >> >
>> >> >> Which gerrit branch were you thinking most patches would go to?
>> >> >>
>> >> >> If they go to 3.0, then push_to_asf.py would have to be amended to
>> >> >> push to gerrit, bypassing code review. I think that's possible, but
>> >> >> I'm not 100%.
>> >> >>
>> >> >> There is also security to think about, since the push_to_asf.py
>> users
>> >> >> can push a few commits at a time, including ones they didn't author
>> or
>> >> >> review.
>> >> >>
>> >> >> We'll also want to clarify
>> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
>> and
>> >> >> keep it consistent with the git & gerrit statuses quo.
>> >> >>
>> >> >> On Fri, Jan 12, 2018 at 10:52 AM, Philip Zeyliger <
>> phi...@cloudera.com>
>> >> >> wrote:
>> >> >> > Hi!
>> >> >> >
>> >> >> >> Should we start tagging all candidates with a common label, e.g.
>> >> >> > include-in-v3?
>> >> >> >
>> >> >> > I agree with Lar

Re: Impala 3.x (and 2.x)

[Philip Zeyliger]

On Fri, Jan 12, 2018 at 4:04 PM, Jim Apple <jbap...@cloudera.com> wrote:

> This makes sense to me.
>
> In this mode, for 2.x-only changes and for changes on 3.0 that don't
> apply cleanly, there will be a manual way to do the step labelled "1.
> Cherrypick tool", and that way is the same way we send patches for
> review now, but pushing to HEAD:refs/for/2.x rather than
> HEAD:refs/for/master, yes?
>

Exactly. So, non-clean cherrypicks or 2.x-only changes go through review on
Gerrit, but we give an implicit review pass to clean cherrypicks.

We could have the cherrypick tool between gerrit/master and gerrit/2.x do
the cherrypicks and run the tests on Jenkins. Do you think that's
preferable?

-- Philip



>
> On Fri, Jan 12, 2018 at 3:57 PM, Philip Zeyliger <phi...@cloudera.com>
> wrote:
> > Picture:
> > https://gist.github.com/philz/323c8b4cb411dc12eb7231d922c195
> 1f#file-impala-branch-image-pdf
> >
> >
> > On Fri, Jan 12, 2018 at 3:47 PM, Jim Apple <jbap...@cloudera.com> wrote:
> >
> >> Often, this list seems to filter out images. Could you post it and send
> a
> >> link?
> >>
> >> Thanks for taking this on, Phil!
> >>
> >> On Fri, Jan 12, 2018 at 3:15 PM, Philip Zeyliger <phi...@cloudera.com>
> >> wrote:
> >>
> >> > I think most patches go to Gerrit branch 'master', which happens to
> >> > identify itself as 3.0. (Or 3.x?).
> >> >
> >> > Here's a picture:
> >> >
> >> > [image: Inline image 1]
> >> >
> >> >
> >> > With this, every time "cherrypick_and_push_to_asf.py" is run, it
> would
> >> > first offer to cherrypick changes between master and 2.x. Then, it
> would
> >> > offer push those cherrypicks to gerrit/2.x. After that, it continues
> on
> >> as
> >> > before and offers to push changes to ASF. I think this maintains the
> >> > invariant that pushing to ASF is only done with a human trigger. (We
> >> could
> >> > also have step 1 be done by a Jenkins robot, since it's between Gerrit
> >> and
> >> > Gerrit.)
> >> >
> >> > I looked at the How to Release page, and the main difference would be
> >> > that, for a 2.x release, the $COMMIT_HASH_YOU_CHOSE would come from
> the
> >> 2.x
> >> > branch, as would any cherrypicks.
> >> >
> >> > Does this match what you're thinking?
> >> >
> >> > Thanks!
> >> >
> >> > -- Philip
> >> >
> >> > On Fri, Jan 12, 2018 at 11:07 AM, Jim Apple <jbap...@cloudera.com>
> >> wrote:
> >> >
> >> >> Which gerrit branch were you thinking most patches would go to?
> >> >>
> >> >> If they go to 3.0, then push_to_asf.py would have to be amended to
> >> >> push to gerrit, bypassing code review. I think that's possible, but
> >> >> I'm not 100%.
> >> >>
> >> >> There is also security to think about, since the push_to_asf.py users
> >> >> can push a few commits at a time, including ones they didn't author
> or
> >> >> review.
> >> >>
> >> >> We'll also want to clarify
> >> >> https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release
> and
> >> >> keep it consistent with the git & gerrit statuses quo.
> >> >>
> >> >> On Fri, Jan 12, 2018 at 10:52 AM, Philip Zeyliger <
> phi...@cloudera.com>
> >> >> wrote:
> >> >> > Hi!
> >> >> >
> >> >> >> Should we start tagging all candidates with a common label, e.g.
> >> >> > include-in-v3?
> >> >> >
> >> >> > I agree with Lars's suggestion for tagging JIRAs with
> include-in-v3.
> >> >> I've
> >> >> > done so, and the relevant query is
> >> >> > https://issues.apache.org/jira/issues/?jql=labels%20%3D%20in
> >> >> clude-in-v3%20and%20project%3Dimpala
> >> >> > .
> >> >> >
> >> >> >> What sort of process were you thinking of for the automation?
> >> >> >
> >> >> > I think amending push_to_asf.py, as you suggest, is a great idea. I
> >> >> think
> >> >> > we have a string ("not for 2.x") which can be used in commit
> messages
> >> to
> >> >> > discourage th

Re: Impala 3.x (and 2.x)

[Philip Zeyliger]

Hi!

> Should we start tagging all candidates with a common label, e.g.
include-in-v3?

I agree with Lars's suggestion for tagging JIRAs with include-in-v3. I've
done so, and the relevant query is
https://issues.apache.org/jira/issues/?jql=labels%20%3D%20include-in-v3%20and%20project%3Dimpala
.

> What sort of process were you thinking of for the automation?

I think amending push_to_asf.py, as you suggest, is a great idea. I think
we have a string ("not for 2.x") which can be used in commit messages to
discourage the cherrypick for the changes we want to be exclusive until we
want to change the defaults in the other direction. (I.e., right now the
string is "not for 2.x", but at some point the string may be "should be
cherrypicked to 2.x".)

I do think that we want to create a gerrit branch to allow 2.x-only changes
to be reviewed in the straight-forward fashion.

-- Philip



On Thu, Jan 11, 2018 at 9:31 AM, Jim Apple  wrote:

> I'm on-board with all of this. (I also would be OK delaying 3.0, if
> that were the consensus).
>
> There is one issue in here I think we should dive into:
>
> > Both master and 2.x would be active, and, at least for the beginning,
> > changes would automatically be pulled into the 2.x line, unless
> explicitly
> > blacklisted.
>
> What sort of process were you thinking of for the automation?
>
> Some context, starting from what we all likely already know:
>
> The bulk of the code review and pre-merge testing results are recorded
> in gerrit. Once the pre-merge testing passes, a patch is cherry-picked
> to the git repo hosted with gerrit. To get the patch to the Impala git
> repo hosted by the ASF, bin/push_to_asf.py is run by a human who
> supplies his or her ASF credentials. That script copies the commit to
> the ASF git repo.
>
> Often, 2-3 commits will pile up in gerrit before some committer runs
> that script and pushes them to ASF.
>
> We could edit that script (bin/push_to_asf.py) to help with the cherry
> picks, so that each time a commit is made, the committer must say
> whether the commit goes in 2.x, 3.0, or both, but the commits are
> often made by people who didn't author the patches, so they may not be
> sure which branch to go in.
>
> Additionally, gerrit code review is intimately tied to the git repo.
> Gerrit runs a git repo under-the-hood, and I believe that the branch
> on gerrit's git that changes are cherry-picked to after pre-merge
> testing is identical to the Impala git repo hosted by the ASF - down
> to the hashes, even. If we think 2.x and 3.0 will diverge enough that
> we'll want different code reviews for different branches, then we
> might want two different branches on gerrit, too.
>

Impala 3.x (and 2.x)

[Philip Zeyliger]

Hi folks,

I'd like to start the conversation around 3.0, spurred in part by some
compatibility-challenging issues. I'm leading with a somewhat concrete
proposal, but that's meant to start the discussion!

Impala's master branch currently self-identifies as version 2.11. (See
https://github.com/apache/impala/blob/master/bin/save-version.sh). There's
a queue of changes that have enough compatibility concerns that we've been
postponing them. (For example, see "Reserving standard SQL keywords next
Impala release" in this mailing list.)

I propose we create a 2.x branch, and update master to be 3.0, thereby
indicating that we'd accept changes with compatibility concerns in master.
Both master and 2.x would be active, and, at least for the beginning,
changes would automatically be pulled into the 2.x line, unless explicitly
blacklisted. After a while, of course, there would be 3.x releases, and 2.x
releases would slow down.

I've gone through labels IN ("incompatibility", "compatibility") and
resolution = "Unresolved" and project=impala

in
JIRA and here's a short, curated sublist:


JIRA

Summary

Comment

IMPALA-4277


Impala should build against latest Hadoop components

Strive to make running both possible.

IMPALA-3916

Reserve SQL:2016 keywords

See thread "Reserving standard SQL keywords next Impala release
(IMPALA-3916)" on the mailing lists.

IMPALA-6204

Remove DataSourceScanNode at next compatibility breaking point


IMPALA-4924

Remove DECIMAL V1 code at next compatibility breaking version

Probably switch to DECIMAL_V2 by default, but retain option.

IMPALA-4319

Remove unused query options in compat-breaking version


IMPALA-4306

Remove deprecated query options at compatibility-breaking version



Creating a new branch has the drawback of yet another entity to think about
(and cherrypick to), but it seems like we need it to provide a place for
changes like those above to land.

Thoughts?

Thanks!

-- Philip

Re: Philip Zeyliger is now an Impala committer

[Philip Zeyliger]

Thanks!

-- Philip

On Tue, Jan 9, 2018 at 12:51 PM, Alexander Behm 
wrote:

> The Project Management Committee (PMC) for Apache Impala has invited Philip
> to become a committer and we are pleased to announce that they have
> accepted.
>
> Congratulations and welcome, Philip!
>

Re: Tianyi Wang is now an Impala committer

[Philip Zeyliger]

Congrats!

On Fri, Jan 5, 2018 at 1:47 PM, Jim Apple  wrote:

> The Project Management Committee (PMC) for Apache Impala has invited
> Tianyi Wang to become a committer and we are pleased to announce that
> they have accepted.
>
> Congratulations and welcome, Tianyi!
>

Re: thrift-server-test

[Philip Zeyliger]

$krb5-config --version
Kerberos 5 release 1.13.2



On Tue, Dec 12, 2017 at 3:39 PM, Michael Ho <k...@cloudera.com> wrote:

> Not that I know the answer to the problem you are hitting. Just wondering
> what version of Kerberos library (krb5-config --version) are you running ?
>
> On Tue, Dec 12, 2017 at 3:25 PM, Philip Zeyliger <phi...@cloudera.com>
> wrote:
>
> > Hi folks,
> >
> > I've been running into issues with thrift-server-test and Kerberos. Below
> > is an excerpt of "KRB5_TRACE=/dev/stderr
> > be/build/debug/rpc/thrift-server-test"; both SslConnectivity/1 and
> > SslConnectivity/2 fail the same way.
> >
> > I'm running Ubuntu16.04. I've seen this both on my host, as well as
> inside
> > of an Ubuntu 16.04 Docker container.
> >
> > Does this ring any bells?
> >
> > Thanks!
> >
> > -- Philip
> >
> >
> > [ RUN  ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> SslConnectivity/2
> > Loading random data
> > Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for realm
> '
> > KRBTEST.COM',
> > master key name 'K/m...@krbtest.com'
> > [31585] 1513120922.459517: Retrieving K/m...@krbtest.com from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > [31586] 1513120922.472314: Retrieving K/m...@krbtest.com from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> setting
> > up network...
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > listening
> > on fd 11: udp 0.0.0.0.51781 (pktinfo)
> > krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > listening
> > on fd 12: udp ::.51781 (pktinfo)
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): set
> up 2
> > sockets
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > commencing operation
> > krb5kdc: starting...
> > Authenticating as principal philip/ad...@krbtest.com with password.
> > [31589] 1513120922.498913: Retrieving K/m...@krbtest.com from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > WARNING: no policy specified for impala/localh...@krbtest.com;
> defaulting
> > to no policy
> > Principal "impala/localh...@krbtest.com" created.
> > Authenticating as principal philip/ad...@krbtest.com with password.
> > [31590] 1513120922.508777: Retrieving K/m...@krbtest.com from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > Entry for principal impala/localhost with kvno 2, encryption type
> > aes256-cts-hmac-sha1-96 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > aes128-cts-hmac-sha1-96 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > des3-cbc-sha1 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > arcfour-hmac added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): AS_REQ
> > (6
> > etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922,
> etypes
> > {rep=18 tkt=18 ses=18}, impala/localh...@krbtest.com for krbtgt/
> > krbtest@krbtest.com
> > [31476] 1513120922.532304: ccselect can't find appropriate cache for
> server
> > principal impala@localhost
> > [31476] 1513120922.532347: Getting credentials impala/
> > localh...@krbtest.com
> > -> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
> > [31476] 1513120922.532382: Retrieving impala/localh...@krbtest.com ->
> > impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
> > -1765328243/Matching credential not found
> > [31476] 1513120922.532407: Retrieving impala/localh...@krbtest.com ->
> > krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
> > result: -1765328243/Matching credential not found
> > [31476] 1513120922.532433: Retrieving impala/localh...@krbtest.com ->
> > krbtgt/krbtest@krbtest.com from FILE:/tmp/krb5cc_impala_internal
> with
> > resu

thrift-server-test

[Philip Zeyliger]

Hi folks,

I've been running into issues with thrift-server-test and Kerberos. Below
is an excerpt of "KRB5_TRACE=/dev/stderr
be/build/debug/rpc/thrift-server-test"; both SslConnectivity/1 and
SslConnectivity/2 fail the same way.

I'm running Ubuntu16.04. I've seen this both on my host, as well as inside
of an Ubuntu 16.04 Docker container.

Does this ring any bells?

Thanks!

-- Philip


[ RUN  ] KerberosOnAndOff/ThriftKerberizedParamsTest.SslConnectivity/2
Loading random data
Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for realm '
KRBTEST.COM',
master key name 'K/m...@krbtest.com'
[31585] 1513120922.459517: Retrieving K/m...@krbtest.com from
FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
[31586] 1513120922.472314: Retrieving K/m...@krbtest.com from
FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): setting
up network...
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): listening
on fd 11: udp 0.0.0.0.51781 (pktinfo)
krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): listening
on fd 12: udp ::.51781 (pktinfo)
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): set up 2
sockets
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
commencing operation
krb5kdc: starting...
Authenticating as principal philip/ad...@krbtest.com with password.
[31589] 1513120922.498913: Retrieving K/m...@krbtest.com from
FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
WARNING: no policy specified for impala/localh...@krbtest.com; defaulting
to no policy
Principal "impala/localh...@krbtest.com" created.
Authenticating as principal philip/ad...@krbtest.com with password.
[31590] 1513120922.508777: Retrieving K/m...@krbtest.com from
FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
result: 0/Success
Entry for principal impala/localhost with kvno 2, encryption type
aes256-cts-hmac-sha1-96 added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Entry for principal impala/localhost with kvno 2, encryption type
aes128-cts-hmac-sha1-96 added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Entry for principal impala/localhost with kvno 2, encryption type
des3-cbc-sha1 added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Entry for principal impala/localhost with kvno 2, encryption type
arcfour-hmac added to keytab
WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): AS_REQ (6
etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922, etypes
{rep=18 tkt=18 ses=18}, impala/localh...@krbtest.com for krbtgt/
krbtest@krbtest.com
[31476] 1513120922.532304: ccselect can't find appropriate cache for server
principal impala@localhost
[31476] 1513120922.532347: Getting credentials impala/localh...@krbtest.com
-> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
[31476] 1513120922.532382: Retrieving impala/localh...@krbtest.com ->
impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
-1765328243/Matching credential not found
[31476] 1513120922.532407: Retrieving impala/localh...@krbtest.com ->
krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
result: -1765328243/Matching credential not found
[31476] 1513120922.532433: Retrieving impala/localh...@krbtest.com ->
krbtgt/krbtest@krbtest.com from FILE:/tmp/krb5cc_impala_internal with
result: 0/Success
[31476] 1513120922.532441: Starting with TGT for client realm: impala/
localh...@krbtest.com -> krbtgt/krbtest@krbtest.com
[31476] 1513120922.532467: Retrieving impala/localh...@krbtest.com ->
krbtgt/localhost@localhost from FILE:/tmp/krb5cc_impala_internal with
result: -1765328243/Matching credential not found
[31476] 1513120922.532475: Requesting TGT krbtgt/localh...@krbtest.com
using TGT krbtgt/krbtest@krbtest.com
[31476] 1513120922.532491: Generated subkey for TGS request: aes256-cts/005D
[31476] 1513120922.532524: etypes requested in TGS request: aes256-cts,
aes128-cts, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts
[31476] 1513120922.532574: Encoding request body and padata into FAST
request
[31476] 1513120922.532616: Sending request (951 bytes) to KRBTEST.COM
[31476] 1513120922.532630: Resolving hostname 127.0.0.1
[31476] 1513120922.532648: Sending initial UDP request to dgram
127.0.0.1:51781
[31586] 1513120922.532790: AP-REQ ticket: impala/localh...@krbtest.com ->
krbtgt/krbtest@krbtest.com, session key aes256-cts/580F
[31586] 1513120922.532814: Negotiated enctype based on authenticator:
aes256-cts
[31586] 1513120922.532820: Authenticator contains subkey: aes256-cts/005D
Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): TGS_REQ

Re: Switch gerrit merge strategy to "rebase always"?

[Philip Zeyliger]

Seems like it's the right thing to do.

On Mon, Dec 11, 2017 at 11:43 AM, Tim Armstrong 
wrote:

> We recently had a bad merge that was allowed by the cherry-pick merge
> strategy merging a simple without its ancestor (since they didn't change
> any nearby lines):
> https://lists.apache.org/thread.html/ee81ee3e396a9a7b1214d92d713a2d
> 28f2f1f7058184504ebc399170@%3Cdev.impala.apache.org%3E
>
> It looks like the "rebase always" merge strategy avoids this by always
> trying to rebase and merge the whole chain of commits. Does anyone have any
> objections or thoughts about switching to this strategy?
>

Upgrading Jenkins

[Philip Zeyliger]

I'll be upgrading Jenkins today in response to a security advisory. I'll be
putting it in "no new jobs" mode shortly.

-- Philip

Re: A test failure at Jenkins build is not reproducible on my local

[Philip Zeyliger]

>
>
> Regarding guidance of E2E tests in "How to load and run Impala tests
>  to+load+and+run+Impala+tests>"
> wiki page, I wish there are more detailed information. Let me put the
> information on the wiki after you answer my question: What is the
> difference between "./test/run-tests.py" and "impala-py.test"?
>

Thank you for volunteering to update the wiki page.

py.test (https://docs.pytest.org/en/latest/) is a framework for writing
tests in Python. It's got a bunch of features.

impala-py.test is a small wrapper that invokes py.test in the right
environment (i.e., from the "virtualenv") and with LD_LIBRARY_PATH updated.
When you're using impala-py.test, you're using py.test's mechanisms.

Meanwhile, test/run-tests.py is a wrapper that invokes py.test a few times.
It uses environment variables for control. Then it executes any "serial"
tests, and then it executes some stress tests, and then, it instructs
py.test to execute the "parallel" tests (those not marked serial or stress)
with a bunch of parallelism. If you plotted your CPU graph while this was
happening, you'd see something like. You can tell where the parallel
testing happens about 3/4 of the way through.
[image: Inline image 1]

To make things even more complicated, there's bin/run-all-tests.sh, which
invokes run-tests.py (but also "mvn test" and the C++ tests), and
buildall.sh, which invokes run-all-tests.sh.

Cheers,

-- Philip