Re: Is there a plan to implement histogram support for computing column stats?

2017-06-20 Thread Alexander Behm 
Histograms can definitely be useful in getting more accurate cardinality
estimates to improve plan choices.

However, adding support for histograms has several challenges which is why
we have no concrete plans on supporting them yet:
- computing stats is already a huge pain for most users due to its cost;
adding histograms might make this problem worse
- perhaps we should support stats on a subset of columns to make it
cheaper; but most users would have difficulties deciding the subset of
columns to pick, so we'd need to provide an automated solution for
suggesting the subset
- there is no support in the Hive Metastore so we'd need to store them
inside the generic TBLPROPERTIES map or similar

Just trying to explain that the user experience needs to be considered when
adding such a new feature, and that dealing with all caveats could be a
substantial amount of design and implementation work.


On Tue, Jun 20, 2017 at 8:53 PM, 吴朱华  wrote:

> let me check it out^_^
>
> 2017-06-21 0:25 GMT+08:00 Jim Apple :
>
> > THis looks like the closest ticket to the question:
> >
> > https://issues.apache.org/jira/browse/IMPALA-2416
> >
> > Feel free to file another, more ambitious, ticket if you'd like.
> >
> > On Tue, Jun 20, 2017 at 4:10 AM, 吴朱华  wrote:
> > > Hi guys:
> > >
> > > Is there a plan to implement the histogram support for computing column
> > > stats? Base on my assumption, if  the histogram support implements, it
> > will
> > > easily and more accurate to  predict the join involved row numbers, and
> > > which will make a better decision for choosing the shuffle or the
> > broadcast.
> > > Above all is all my amateur thoughts, I love to hear your feedbacks^_^
> >
>

Re: Is there a plan to implement histogram support for computing column stats?

2017-06-20 Thread 吴朱华 
let me check it out^_^

2017-06-21 0:25 GMT+08:00 Jim Apple :

> THis looks like the closest ticket to the question:
>
> https://issues.apache.org/jira/browse/IMPALA-2416
>
> Feel free to file another, more ambitious, ticket if you'd like.
>
> On Tue, Jun 20, 2017 at 4:10 AM, 吴朱华  wrote:
> > Hi guys:
> >
> > Is there a plan to implement the histogram support for computing column
> > stats? Base on my assumption, if  the histogram support implements, it
> will
> > easily and more accurate to  predict the join involved row numbers, and
> > which will make a better decision for choosing the shuffle or the
> broadcast.
> > Above all is all my amateur thoughts, I love to hear your feedbacks^_^
>

Re: Re: Impala make install

2017-06-20 Thread Henry Robinson 
I don't think there's any plan for this work. The CMake documentation would
be where I'd start looking for ideas:

https://cmake.org/cmake/help/v3.2/command/install.html

Best,
Henry

On 20 June 2017 at 18:31, sky  wrote:

> Hi Tim，
>Is there a plan for this work? Could you provide a manual copy of the
> example?Thanks.
>
>
>
>
>
>
>
>
> At 2017-06-21 01:41:33, "Tim Armstrong"  wrote:
> >Hi Sky,
> >  We have not implemented an install target yet - for deployment we rely
> on
> >copying out the artifacts manually. I believe CMake has some support for
> >implementing install targets but nobody has picked up that work yet.
> >
> >- Tim
> >
> >On Mon, Jun 19, 2017 at 8:45 PM, sky  wrote:
> >
> >> Hi all,
> >> I am using cdh5.11.1-release,the compilation command is provided in
> >> the documentation(./buildall.sh -notests -so),but there is no command
> >> similar to 'make install'.In the current document compiled, the
> directory
> >> structure is too much and do not need too many files. Could you provide
> an
> >> "install" command to extract compiled files to other directories for
> easy
> >> management
>

Re:Re: Impala make install

2017-06-20 Thread sky 
Hi Tim，
   Is there a plan for this work? Could you provide a manual copy of the 
example?Thanks.








At 2017-06-21 01:41:33, "Tim Armstrong"  wrote:
>Hi Sky,
>  We have not implemented an install target yet - for deployment we rely on
>copying out the artifacts manually. I believe CMake has some support for
>implementing install targets but nobody has picked up that work yet.
>
>- Tim
>
>On Mon, Jun 19, 2017 at 8:45 PM, sky  wrote:
>
>> Hi all,
>> I am using cdh5.11.1-release,the compilation command is provided in
>> the documentation(./buildall.sh -notests -so),but there is no command
>> similar to 'make install'.In the current document compiled, the directory
>> structure is too much and do not need too many files. Could you provide an
>> "install" command to extract compiled files to other directories for easy
>> management

Thrift version used by Impala

2017-06-20 Thread Alexander Kolbasov 
As part of our investigation of Impala/Sentry integration issues it turned
out that Impala uses a version of Thrift that is older then 0.9.3 that's
used by Sentry (and many other components). Is there a fundamental reason
Impala can't move to Thrift 0.9.3? There were some security vulnerabilities
in earlier versions.

- Alex

Re: Impala make install

2017-06-20 Thread Tim Armstrong 
Hi Sky,
  We have not implemented an install target yet - for deployment we rely on
copying out the artifacts manually. I believe CMake has some support for
implementing install targets but nobody has picked up that work yet.

- Tim

On Mon, Jun 19, 2017 at 8:45 PM, sky  wrote:

> Hi all,
> I am using cdh5.11.1-release,the compilation command is provided in
> the documentation(./buildall.sh -notests -so),but there is no command
> similar to 'make install'.In the current document compiled, the directory
> structure is too much and do not need too many files. Could you provide an
> "install" command to extract compiled files to other directories for easy
> management

Re: Broken build from Sentry

2017-06-20 Thread Tim Armstrong 
Yeah my change should build against either version by design.

On 20 Jun. 2017 9:21 am, "Henry Robinson"  wrote:

> Yes, I did. AFAICT it worked fine.
>
> On 20 June 2017 at 09:19, Alexander Behm  wrote:
>
> > Henry, did you try the revert on top of Tim's already-checked-in change?
> >
> > On Tue, Jun 20, 2017 at 9:18 AM, Alexander Behm 
> > wrote:
> >
> > > Let's revert the version to buy us some time. That solution is a
> ticking
> > > time bomb though since that version will disappear soon.
> > >
> > > On Tue, Jun 20, 2017 at 8:56 AM, Henry Robinson 
> > wrote:
> > >
> > >> I was able to run a build with EE and FE tests with Sentry reverted to
> > >> 5.12
> > >> - unless there are objections I'm going to post a patch to revert the
> > >> version bump.
> > >>
> > >> On 20 June 2017 at 06:53, Thomas Tauber-Marshall <
> > tmarsh...@cloudera.com>
> > >> wrote:
> > >>
> > >> > So we've had a successful run of the nightlies now, and I've
> uploaded
> > >> the
> > >> > new jars to the s3 bucket, but Sentry still fails for some reason.
> > >> >
> > >> > I filed: https://issues.apache.org/jira/browse/IMPALA-5540 to track
> > >> this
> > >> >
> > >> > On Tue, Jun 20, 2017 at 1:25 AM Alexander Kolbasov <
> > ak...@cloudera.com>
> > >> > wrote:
> > >> >
> > >> > > Note that Apache upstream story is more complicated - there was a
> > >> change
> > >> > > done upstream that refactored a bunch of Sentry code that will
> cause
> > >> > > similar issue (I think it is SENTRY-1205). The change is present
> in
> > >> > Sentry
> > >> > > master but not in upstream sentry HA branch.
> > >> > >
> > >> > > On Mon, Jun 19, 2017 at 11:02 PM, Dimitris Tsirogiannis <
> > >> > > dtsirogian...@cloudera.com> wrote:
> > >> > >
> > >> > > > +Sasha, who I believe has more up-to-date information on this.
> > >> > > >
> > >> > > > On Mon, Jun 19, 2017 at 10:56 PM, Henry Robinson <
> > he...@apache.org>
> > >> > > wrote:
> > >> > > >
> > >> > > >> FWIW, I've been able to start Sentry by setting:
> > >> > > >>
> > >> > > >> export IMPALA_SENTRY_VERSION=1.5.1-cdh5.12.0-SNAPSHOT
> > >> > > >>
> > >> > > >> (i.e. rolling back to the previous version of Sentry). I
> haven't
> > >> yet
> > >> > > tried
> > >> > > >> to run tests - does anyone know an ETA for a fix coming out of
> > >> > Cloudera
> > >> > > >> for
> > >> > > >> the 5.13-SNAPSHOT? If it might be a while, we should consider
> > >> > regressing
> > >> > > >> the Sentry version to unblock checkins.
> > >> > > >>
> > >> > > >> On 19 June 2017 at 15:31, Tim Armstrong <
> tarmstr...@cloudera.com
> > >
> > >> > > wrote:
> > >> > > >>
> > >> > > >> > It's unfortunately not that simple. The API change has been
> in
> > >> > Apache
> > >> > > >> > sentry
> > >> > > >> >
> > >> > > >> > So rolling back the API change temporarily solves the problem
> > for
> > >> > > >> Cloudera,
> > >> > > >> > but we're going to have to deal with it at some point and get
> > >> Impala
> > >> > > >> > building against both versions of the API.
> > >> > > >> >
> > >> > > >> > On Mon, Jun 19, 2017 at 2:55 PM, Thomas Tauber-Marshall <
> > >> > > >> > tmarsh...@cloudera.com> wrote:
> > >> > > >> >
> > >> > > >> > > Yes, the Sentry team has been contacted and they're going
> to
> > be
> > >> > > >> rolling
> > >> > > >> > it
> > >> > > >> > > back.
> > >> > > >> > >
> > >> > > >> > > On Mon, Jun 19, 2017 at 4:53 PM Todd Lipcon <
> > t...@cloudera.com
> > >> >
> > >> > > >> wrote:
> > >> > > >> > >
> > >> > > >> > > > Quick question from a bystander: it seems like Sentry
> > >> committed
> > >> > an
> > >> > > >> > > > API-incompatible change. Instead of fixing on the Impala
> > >> side,
> > >> > > >> should
> > >> > > >> > the
> > >> > > >> > > > Sentry project be notified that they may want to roll
> back
> > >> such
> > >> > a
> > >> > > >> > change?
> > >> > > >> > > > It seems like an error on their part to do such a thing
> > >> within a
> > >> > > >> minor
> > >> > > >> > > > version.
> > >> > > >> > > >
> > >> > > >> > > > On Mon, Jun 19, 2017 at 1:56 PM, Thomas Tauber-Marshall <
> > >> > > >> > > > tmarsh...@cloudera.com> wrote:
> > >> > > >> > > >
> > >> > > >> > > > > I'm working on getting the s3 jars updated, which
> > >> presumably
> > >> > > will
> > >> > > >> fix
> > >> > > >> > > > that.
> > >> > > >> > > > >
> > >> > > >> > > > > The problem (to my understanding) is that the nightlies
> > >> > haven't
> > >> > > >> > passed
> > >> > > >> > > > > since the change went into Sentry and so the Jenkins
> job
> > >> that
> > >> > > >> > normally
> > >> > > >> > > > > produces the new jars is still pulling in old bits.
> > >> > > >> > > > >
> > >> > > >> > > > > I've been talking with releng and they expect the new
> > jars
> > >> to
> > >> > be
> > >> > > >> > > > available
> > >> > > >> > > > > later today.
> > >> > > >> > > > >
> > >> > > >> > > > > On Mon, Jun 19, 2017 at 3:48 PM Tim Armstrong <
> > >> > > >> >

Re: Is there a plan to implement histogram support for computing column stats?

2017-06-20 Thread Jim Apple 
THis looks like the closest ticket to the question:

https://issues.apache.org/jira/browse/IMPALA-2416

Feel free to file another, more ambitious, ticket if you'd like.

On Tue, Jun 20, 2017 at 4:10 AM, 吴朱华  wrote:
> Hi guys:
>
> Is there a plan to implement the histogram support for computing column
> stats? Base on my assumption, if  the histogram support implements, it will
> easily and more accurate to  predict the join involved row numbers, and
> which will make a better decision for choosing the shuffle or the broadcast.
> Above all is all my amateur thoughts, I love to hear your feedbacks^_^

Re: Broken build from Sentry

2017-06-20 Thread Henry Robinson 
Yes, I did. AFAICT it worked fine.

On 20 June 2017 at 09:19, Alexander Behm  wrote:

> Henry, did you try the revert on top of Tim's already-checked-in change?
>
> On Tue, Jun 20, 2017 at 9:18 AM, Alexander Behm 
> wrote:
>
> > Let's revert the version to buy us some time. That solution is a ticking
> > time bomb though since that version will disappear soon.
> >
> > On Tue, Jun 20, 2017 at 8:56 AM, Henry Robinson 
> wrote:
> >
> >> I was able to run a build with EE and FE tests with Sentry reverted to
> >> 5.12
> >> - unless there are objections I'm going to post a patch to revert the
> >> version bump.
> >>
> >> On 20 June 2017 at 06:53, Thomas Tauber-Marshall <
> tmarsh...@cloudera.com>
> >> wrote:
> >>
> >> > So we've had a successful run of the nightlies now, and I've uploaded
> >> the
> >> > new jars to the s3 bucket, but Sentry still fails for some reason.
> >> >
> >> > I filed: https://issues.apache.org/jira/browse/IMPALA-5540 to track
> >> this
> >> >
> >> > On Tue, Jun 20, 2017 at 1:25 AM Alexander Kolbasov <
> ak...@cloudera.com>
> >> > wrote:
> >> >
> >> > > Note that Apache upstream story is more complicated - there was a
> >> change
> >> > > done upstream that refactored a bunch of Sentry code that will cause
> >> > > similar issue (I think it is SENTRY-1205). The change is present in
> >> > Sentry
> >> > > master but not in upstream sentry HA branch.
> >> > >
> >> > > On Mon, Jun 19, 2017 at 11:02 PM, Dimitris Tsirogiannis <
> >> > > dtsirogian...@cloudera.com> wrote:
> >> > >
> >> > > > +Sasha, who I believe has more up-to-date information on this.
> >> > > >
> >> > > > On Mon, Jun 19, 2017 at 10:56 PM, Henry Robinson <
> he...@apache.org>
> >> > > wrote:
> >> > > >
> >> > > >> FWIW, I've been able to start Sentry by setting:
> >> > > >>
> >> > > >> export IMPALA_SENTRY_VERSION=1.5.1-cdh5.12.0-SNAPSHOT
> >> > > >>
> >> > > >> (i.e. rolling back to the previous version of Sentry). I haven't
> >> yet
> >> > > tried
> >> > > >> to run tests - does anyone know an ETA for a fix coming out of
> >> > Cloudera
> >> > > >> for
> >> > > >> the 5.13-SNAPSHOT? If it might be a while, we should consider
> >> > regressing
> >> > > >> the Sentry version to unblock checkins.
> >> > > >>
> >> > > >> On 19 June 2017 at 15:31, Tim Armstrong  >
> >> > > wrote:
> >> > > >>
> >> > > >> > It's unfortunately not that simple. The API change has been in
> >> > Apache
> >> > > >> > sentry
> >> > > >> >
> >> > > >> > So rolling back the API change temporarily solves the problem
> for
> >> > > >> Cloudera,
> >> > > >> > but we're going to have to deal with it at some point and get
> >> Impala
> >> > > >> > building against both versions of the API.
> >> > > >> >
> >> > > >> > On Mon, Jun 19, 2017 at 2:55 PM, Thomas Tauber-Marshall <
> >> > > >> > tmarsh...@cloudera.com> wrote:
> >> > > >> >
> >> > > >> > > Yes, the Sentry team has been contacted and they're going to
> be
> >> > > >> rolling
> >> > > >> > it
> >> > > >> > > back.
> >> > > >> > >
> >> > > >> > > On Mon, Jun 19, 2017 at 4:53 PM Todd Lipcon <
> t...@cloudera.com
> >> >
> >> > > >> wrote:
> >> > > >> > >
> >> > > >> > > > Quick question from a bystander: it seems like Sentry
> >> committed
> >> > an
> >> > > >> > > > API-incompatible change. Instead of fixing on the Impala
> >> side,
> >> > > >> should
> >> > > >> > the
> >> > > >> > > > Sentry project be notified that they may want to roll back
> >> such
> >> > a
> >> > > >> > change?
> >> > > >> > > > It seems like an error on their part to do such a thing
> >> within a
> >> > > >> minor
> >> > > >> > > > version.
> >> > > >> > > >
> >> > > >> > > > On Mon, Jun 19, 2017 at 1:56 PM, Thomas Tauber-Marshall <
> >> > > >> > > > tmarsh...@cloudera.com> wrote:
> >> > > >> > > >
> >> > > >> > > > > I'm working on getting the s3 jars updated, which
> >> presumably
> >> > > will
> >> > > >> fix
> >> > > >> > > > that.
> >> > > >> > > > >
> >> > > >> > > > > The problem (to my understanding) is that the nightlies
> >> > haven't
> >> > > >> > passed
> >> > > >> > > > > since the change went into Sentry and so the Jenkins job
> >> that
> >> > > >> > normally
> >> > > >> > > > > produces the new jars is still pulling in old bits.
> >> > > >> > > > >
> >> > > >> > > > > I've been talking with releng and they expect the new
> jars
> >> to
> >> > be
> >> > > >> > > > available
> >> > > >> > > > > later today.
> >> > > >> > > > >
> >> > > >> > > > > On Mon, Jun 19, 2017 at 3:48 PM Tim Armstrong <
> >> > > >> > tarmstr...@cloudera.com
> >> > > >> > > >
> >> > > >> > > > > wrote:
> >> > > >> > > > >
> >> > > >> > > > > > Looks like the build still breaks when starting up
> sentry
> >> > > after
> >> > > >> my
> >> > > >> > > fix:
> >> > > >> > > > > >
> >> > > >> > > > > >
> >> > > >> > > > http://jenkins.impala.io:8080/
> job/ubuntu-14.04-from-scratch/
> >> > > >> > 1547/console
> >> > > >> > > > > >
> >> > > >> > > > > >

Re: Broken build from Sentry

2017-06-20 Thread Alexander Behm 
Henry, did you try the revert on top of Tim's already-checked-in change?

On Tue, Jun 20, 2017 at 9:18 AM, Alexander Behm 
wrote:

> Let's revert the version to buy us some time. That solution is a ticking
> time bomb though since that version will disappear soon.
>
> On Tue, Jun 20, 2017 at 8:56 AM, Henry Robinson  wrote:
>
>> I was able to run a build with EE and FE tests with Sentry reverted to
>> 5.12
>> - unless there are objections I'm going to post a patch to revert the
>> version bump.
>>
>> On 20 June 2017 at 06:53, Thomas Tauber-Marshall 
>> wrote:
>>
>> > So we've had a successful run of the nightlies now, and I've uploaded
>> the
>> > new jars to the s3 bucket, but Sentry still fails for some reason.
>> >
>> > I filed: https://issues.apache.org/jira/browse/IMPALA-5540 to track
>> this
>> >
>> > On Tue, Jun 20, 2017 at 1:25 AM Alexander Kolbasov 
>> > wrote:
>> >
>> > > Note that Apache upstream story is more complicated - there was a
>> change
>> > > done upstream that refactored a bunch of Sentry code that will cause
>> > > similar issue (I think it is SENTRY-1205). The change is present in
>> > Sentry
>> > > master but not in upstream sentry HA branch.
>> > >
>> > > On Mon, Jun 19, 2017 at 11:02 PM, Dimitris Tsirogiannis <
>> > > dtsirogian...@cloudera.com> wrote:
>> > >
>> > > > +Sasha, who I believe has more up-to-date information on this.
>> > > >
>> > > > On Mon, Jun 19, 2017 at 10:56 PM, Henry Robinson 
>> > > wrote:
>> > > >
>> > > >> FWIW, I've been able to start Sentry by setting:
>> > > >>
>> > > >> export IMPALA_SENTRY_VERSION=1.5.1-cdh5.12.0-SNAPSHOT
>> > > >>
>> > > >> (i.e. rolling back to the previous version of Sentry). I haven't
>> yet
>> > > tried
>> > > >> to run tests - does anyone know an ETA for a fix coming out of
>> > Cloudera
>> > > >> for
>> > > >> the 5.13-SNAPSHOT? If it might be a while, we should consider
>> > regressing
>> > > >> the Sentry version to unblock checkins.
>> > > >>
>> > > >> On 19 June 2017 at 15:31, Tim Armstrong 
>> > > wrote:
>> > > >>
>> > > >> > It's unfortunately not that simple. The API change has been in
>> > Apache
>> > > >> > sentry
>> > > >> >
>> > > >> > So rolling back the API change temporarily solves the problem for
>> > > >> Cloudera,
>> > > >> > but we're going to have to deal with it at some point and get
>> Impala
>> > > >> > building against both versions of the API.
>> > > >> >
>> > > >> > On Mon, Jun 19, 2017 at 2:55 PM, Thomas Tauber-Marshall <
>> > > >> > tmarsh...@cloudera.com> wrote:
>> > > >> >
>> > > >> > > Yes, the Sentry team has been contacted and they're going to be
>> > > >> rolling
>> > > >> > it
>> > > >> > > back.
>> > > >> > >
>> > > >> > > On Mon, Jun 19, 2017 at 4:53 PM Todd Lipcon > >
>> > > >> wrote:
>> > > >> > >
>> > > >> > > > Quick question from a bystander: it seems like Sentry
>> committed
>> > an
>> > > >> > > > API-incompatible change. Instead of fixing on the Impala
>> side,
>> > > >> should
>> > > >> > the
>> > > >> > > > Sentry project be notified that they may want to roll back
>> such
>> > a
>> > > >> > change?
>> > > >> > > > It seems like an error on their part to do such a thing
>> within a
>> > > >> minor
>> > > >> > > > version.
>> > > >> > > >
>> > > >> > > > On Mon, Jun 19, 2017 at 1:56 PM, Thomas Tauber-Marshall <
>> > > >> > > > tmarsh...@cloudera.com> wrote:
>> > > >> > > >
>> > > >> > > > > I'm working on getting the s3 jars updated, which
>> presumably
>> > > will
>> > > >> fix
>> > > >> > > > that.
>> > > >> > > > >
>> > > >> > > > > The problem (to my understanding) is that the nightlies
>> > haven't
>> > > >> > passed
>> > > >> > > > > since the change went into Sentry and so the Jenkins job
>> that
>> > > >> > normally
>> > > >> > > > > produces the new jars is still pulling in old bits.
>> > > >> > > > >
>> > > >> > > > > I've been talking with releng and they expect the new jars
>> to
>> > be
>> > > >> > > > available
>> > > >> > > > > later today.
>> > > >> > > > >
>> > > >> > > > > On Mon, Jun 19, 2017 at 3:48 PM Tim Armstrong <
>> > > >> > tarmstr...@cloudera.com
>> > > >> > > >
>> > > >> > > > > wrote:
>> > > >> > > > >
>> > > >> > > > > > Looks like the build still breaks when starting up sentry
>> > > after
>> > > >> my
>> > > >> > > fix:
>> > > >> > > > > >
>> > > >> > > > > >
>> > > >> > > > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
>> > > >> > 1547/console
>> > > >> > > > > >
>> > > >> > > > > > *20:08:54*  --> Starting the Sentry Policy
>> Server*20:08:59*
>> > > >> Error
>> > > >> > in
>> > > >> > > > > > /home/ubuntu/Impala/testdata/bin/run-all.sh at line 58:
>> > > >> > > > > > $IMPALA_HOME/testdata/bin/run-sentry-service.sh >
>> > \*20:08:59*
>> > > +
>> > > >> > > > > > onexit*20:08:59* + df -m*20:08:59* Filesystem
>>  1M-blocks
>> > > >> Used
>> > > >> > > > > > Available Use% Mounted on*20:08:59*

Re: Broken build from Sentry

2017-06-20 Thread Alexander Behm 
Let's revert the version to buy us some time. That solution is a ticking
time bomb though since that version will disappear soon.

On Tue, Jun 20, 2017 at 8:56 AM, Henry Robinson  wrote:

> I was able to run a build with EE and FE tests with Sentry reverted to 5.12
> - unless there are objections I'm going to post a patch to revert the
> version bump.
>
> On 20 June 2017 at 06:53, Thomas Tauber-Marshall 
> wrote:
>
> > So we've had a successful run of the nightlies now, and I've uploaded the
> > new jars to the s3 bucket, but Sentry still fails for some reason.
> >
> > I filed: https://issues.apache.org/jira/browse/IMPALA-5540 to track this
> >
> > On Tue, Jun 20, 2017 at 1:25 AM Alexander Kolbasov 
> > wrote:
> >
> > > Note that Apache upstream story is more complicated - there was a
> change
> > > done upstream that refactored a bunch of Sentry code that will cause
> > > similar issue (I think it is SENTRY-1205). The change is present in
> > Sentry
> > > master but not in upstream sentry HA branch.
> > >
> > > On Mon, Jun 19, 2017 at 11:02 PM, Dimitris Tsirogiannis <
> > > dtsirogian...@cloudera.com> wrote:
> > >
> > > > +Sasha, who I believe has more up-to-date information on this.
> > > >
> > > > On Mon, Jun 19, 2017 at 10:56 PM, Henry Robinson 
> > > wrote:
> > > >
> > > >> FWIW, I've been able to start Sentry by setting:
> > > >>
> > > >> export IMPALA_SENTRY_VERSION=1.5.1-cdh5.12.0-SNAPSHOT
> > > >>
> > > >> (i.e. rolling back to the previous version of Sentry). I haven't yet
> > > tried
> > > >> to run tests - does anyone know an ETA for a fix coming out of
> > Cloudera
> > > >> for
> > > >> the 5.13-SNAPSHOT? If it might be a while, we should consider
> > regressing
> > > >> the Sentry version to unblock checkins.
> > > >>
> > > >> On 19 June 2017 at 15:31, Tim Armstrong 
> > > wrote:
> > > >>
> > > >> > It's unfortunately not that simple. The API change has been in
> > Apache
> > > >> > sentry
> > > >> >
> > > >> > So rolling back the API change temporarily solves the problem for
> > > >> Cloudera,
> > > >> > but we're going to have to deal with it at some point and get
> Impala
> > > >> > building against both versions of the API.
> > > >> >
> > > >> > On Mon, Jun 19, 2017 at 2:55 PM, Thomas Tauber-Marshall <
> > > >> > tmarsh...@cloudera.com> wrote:
> > > >> >
> > > >> > > Yes, the Sentry team has been contacted and they're going to be
> > > >> rolling
> > > >> > it
> > > >> > > back.
> > > >> > >
> > > >> > > On Mon, Jun 19, 2017 at 4:53 PM Todd Lipcon 
> > > >> wrote:
> > > >> > >
> > > >> > > > Quick question from a bystander: it seems like Sentry
> committed
> > an
> > > >> > > > API-incompatible change. Instead of fixing on the Impala side,
> > > >> should
> > > >> > the
> > > >> > > > Sentry project be notified that they may want to roll back
> such
> > a
> > > >> > change?
> > > >> > > > It seems like an error on their part to do such a thing
> within a
> > > >> minor
> > > >> > > > version.
> > > >> > > >
> > > >> > > > On Mon, Jun 19, 2017 at 1:56 PM, Thomas Tauber-Marshall <
> > > >> > > > tmarsh...@cloudera.com> wrote:
> > > >> > > >
> > > >> > > > > I'm working on getting the s3 jars updated, which presumably
> > > will
> > > >> fix
> > > >> > > > that.
> > > >> > > > >
> > > >> > > > > The problem (to my understanding) is that the nightlies
> > haven't
> > > >> > passed
> > > >> > > > > since the change went into Sentry and so the Jenkins job
> that
> > > >> > normally
> > > >> > > > > produces the new jars is still pulling in old bits.
> > > >> > > > >
> > > >> > > > > I've been talking with releng and they expect the new jars
> to
> > be
> > > >> > > > available
> > > >> > > > > later today.
> > > >> > > > >
> > > >> > > > > On Mon, Jun 19, 2017 at 3:48 PM Tim Armstrong <
> > > >> > tarmstr...@cloudera.com
> > > >> > > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Looks like the build still breaks when starting up sentry
> > > after
> > > >> my
> > > >> > > fix:
> > > >> > > > > >
> > > >> > > > > >
> > > >> > > > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
> > > >> > 1547/console
> > > >> > > > > >
> > > >> > > > > > *20:08:54*  --> Starting the Sentry Policy
> Server*20:08:59*
> > > >> Error
> > > >> > in
> > > >> > > > > > /home/ubuntu/Impala/testdata/bin/run-all.sh at line 58:
> > > >> > > > > > $IMPALA_HOME/testdata/bin/run-sentry-service.sh >
> > \*20:08:59*
> > > +
> > > >> > > > > > onexit*20:08:59* + df -m*20:08:59* Filesystem
>  1M-blocks
> > > >> Used
> > > >> > > > > > Available Use% Mounted on*20:08:59* udev
>  15070
> > > >>  1
> > > >> > > > > > 15070   1% /dev*20:08:59* tmpfs   3015 1
> > > >> 3015
> > > >> > > > > > 1% /run*20:08:59* /dev/xvda1161129 22275132204
> > > 15%
> > > >> > > > > > /*20:08:59* none   1 0 1   0%
> > > >> > > > > >

Re: Broken build from Sentry

2017-06-20 Thread Henry Robinson 
I was able to run a build with EE and FE tests with Sentry reverted to 5.12
- unless there are objections I'm going to post a patch to revert the
version bump.

On 20 June 2017 at 06:53, Thomas Tauber-Marshall 
wrote:

> So we've had a successful run of the nightlies now, and I've uploaded the
> new jars to the s3 bucket, but Sentry still fails for some reason.
>
> I filed: https://issues.apache.org/jira/browse/IMPALA-5540 to track this
>
> On Tue, Jun 20, 2017 at 1:25 AM Alexander Kolbasov 
> wrote:
>
> > Note that Apache upstream story is more complicated - there was a change
> > done upstream that refactored a bunch of Sentry code that will cause
> > similar issue (I think it is SENTRY-1205). The change is present in
> Sentry
> > master but not in upstream sentry HA branch.
> >
> > On Mon, Jun 19, 2017 at 11:02 PM, Dimitris Tsirogiannis <
> > dtsirogian...@cloudera.com> wrote:
> >
> > > +Sasha, who I believe has more up-to-date information on this.
> > >
> > > On Mon, Jun 19, 2017 at 10:56 PM, Henry Robinson 
> > wrote:
> > >
> > >> FWIW, I've been able to start Sentry by setting:
> > >>
> > >> export IMPALA_SENTRY_VERSION=1.5.1-cdh5.12.0-SNAPSHOT
> > >>
> > >> (i.e. rolling back to the previous version of Sentry). I haven't yet
> > tried
> > >> to run tests - does anyone know an ETA for a fix coming out of
> Cloudera
> > >> for
> > >> the 5.13-SNAPSHOT? If it might be a while, we should consider
> regressing
> > >> the Sentry version to unblock checkins.
> > >>
> > >> On 19 June 2017 at 15:31, Tim Armstrong 
> > wrote:
> > >>
> > >> > It's unfortunately not that simple. The API change has been in
> Apache
> > >> > sentry
> > >> >
> > >> > So rolling back the API change temporarily solves the problem for
> > >> Cloudera,
> > >> > but we're going to have to deal with it at some point and get Impala
> > >> > building against both versions of the API.
> > >> >
> > >> > On Mon, Jun 19, 2017 at 2:55 PM, Thomas Tauber-Marshall <
> > >> > tmarsh...@cloudera.com> wrote:
> > >> >
> > >> > > Yes, the Sentry team has been contacted and they're going to be
> > >> rolling
> > >> > it
> > >> > > back.
> > >> > >
> > >> > > On Mon, Jun 19, 2017 at 4:53 PM Todd Lipcon 
> > >> wrote:
> > >> > >
> > >> > > > Quick question from a bystander: it seems like Sentry committed
> an
> > >> > > > API-incompatible change. Instead of fixing on the Impala side,
> > >> should
> > >> > the
> > >> > > > Sentry project be notified that they may want to roll back such
> a
> > >> > change?
> > >> > > > It seems like an error on their part to do such a thing within a
> > >> minor
> > >> > > > version.
> > >> > > >
> > >> > > > On Mon, Jun 19, 2017 at 1:56 PM, Thomas Tauber-Marshall <
> > >> > > > tmarsh...@cloudera.com> wrote:
> > >> > > >
> > >> > > > > I'm working on getting the s3 jars updated, which presumably
> > will
> > >> fix
> > >> > > > that.
> > >> > > > >
> > >> > > > > The problem (to my understanding) is that the nightlies
> haven't
> > >> > passed
> > >> > > > > since the change went into Sentry and so the Jenkins job that
> > >> > normally
> > >> > > > > produces the new jars is still pulling in old bits.
> > >> > > > >
> > >> > > > > I've been talking with releng and they expect the new jars to
> be
> > >> > > > available
> > >> > > > > later today.
> > >> > > > >
> > >> > > > > On Mon, Jun 19, 2017 at 3:48 PM Tim Armstrong <
> > >> > tarmstr...@cloudera.com
> > >> > > >
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Looks like the build still breaks when starting up sentry
> > after
> > >> my
> > >> > > fix:
> > >> > > > > >
> > >> > > > > >
> > >> > > > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
> > >> > 1547/console
> > >> > > > > >
> > >> > > > > > *20:08:54*  --> Starting the Sentry Policy Server*20:08:59*
> > >> Error
> > >> > in
> > >> > > > > > /home/ubuntu/Impala/testdata/bin/run-all.sh at line 58:
> > >> > > > > > $IMPALA_HOME/testdata/bin/run-sentry-service.sh >
> \*20:08:59*
> > +
> > >> > > > > > onexit*20:08:59* + df -m*20:08:59* Filesystem 1M-blocks
> > >> Used
> > >> > > > > > Available Use% Mounted on*20:08:59* udev   15070
> > >>  1
> > >> > > > > > 15070   1% /dev*20:08:59* tmpfs   3015 1
> > >> 3015
> > >> > > > > > 1% /run*20:08:59* /dev/xvda1161129 22275132204
> > 15%
> > >> > > > > > /*20:08:59* none   1 0 1   0%
> > >> > > > > > /sys/fs/cgroup*20:08:59* none   5 0
> > >>  5
> > >> > >  0%
> > >> > > > > > /run/lock*20:08:59* none   15075 1 15075
> > >>  1%
> > >> > > > > > /run/shm*20:08:59* none 100 0   100
> >  0%
> > >> > > > > > /run/user*20:08:59* + free -m*20:08:59*  total
> > >> >  used
> > >> > > > > >  free sharedbuffers cached*20:08:59* Mem:
> > >> > > > > > 30148  19597  10550 11

Re: Broken build from Sentry

2017-06-20 Thread Thomas Tauber-Marshall 
So we've had a successful run of the nightlies now, and I've uploaded the
new jars to the s3 bucket, but Sentry still fails for some reason.

I filed: https://issues.apache.org/jira/browse/IMPALA-5540 to track this

On Tue, Jun 20, 2017 at 1:25 AM Alexander Kolbasov 
wrote:

> Note that Apache upstream story is more complicated - there was a change
> done upstream that refactored a bunch of Sentry code that will cause
> similar issue (I think it is SENTRY-1205). The change is present in Sentry
> master but not in upstream sentry HA branch.
>
> On Mon, Jun 19, 2017 at 11:02 PM, Dimitris Tsirogiannis <
> dtsirogian...@cloudera.com> wrote:
>
> > +Sasha, who I believe has more up-to-date information on this.
> >
> > On Mon, Jun 19, 2017 at 10:56 PM, Henry Robinson 
> wrote:
> >
> >> FWIW, I've been able to start Sentry by setting:
> >>
> >> export IMPALA_SENTRY_VERSION=1.5.1-cdh5.12.0-SNAPSHOT
> >>
> >> (i.e. rolling back to the previous version of Sentry). I haven't yet
> tried
> >> to run tests - does anyone know an ETA for a fix coming out of Cloudera
> >> for
> >> the 5.13-SNAPSHOT? If it might be a while, we should consider regressing
> >> the Sentry version to unblock checkins.
> >>
> >> On 19 June 2017 at 15:31, Tim Armstrong 
> wrote:
> >>
> >> > It's unfortunately not that simple. The API change has been in Apache
> >> > sentry
> >> >
> >> > So rolling back the API change temporarily solves the problem for
> >> Cloudera,
> >> > but we're going to have to deal with it at some point and get Impala
> >> > building against both versions of the API.
> >> >
> >> > On Mon, Jun 19, 2017 at 2:55 PM, Thomas Tauber-Marshall <
> >> > tmarsh...@cloudera.com> wrote:
> >> >
> >> > > Yes, the Sentry team has been contacted and they're going to be
> >> rolling
> >> > it
> >> > > back.
> >> > >
> >> > > On Mon, Jun 19, 2017 at 4:53 PM Todd Lipcon 
> >> wrote:
> >> > >
> >> > > > Quick question from a bystander: it seems like Sentry committed an
> >> > > > API-incompatible change. Instead of fixing on the Impala side,
> >> should
> >> > the
> >> > > > Sentry project be notified that they may want to roll back such a
> >> > change?
> >> > > > It seems like an error on their part to do such a thing within a
> >> minor
> >> > > > version.
> >> > > >
> >> > > > On Mon, Jun 19, 2017 at 1:56 PM, Thomas Tauber-Marshall <
> >> > > > tmarsh...@cloudera.com> wrote:
> >> > > >
> >> > > > > I'm working on getting the s3 jars updated, which presumably
> will
> >> fix
> >> > > > that.
> >> > > > >
> >> > > > > The problem (to my understanding) is that the nightlies haven't
> >> > passed
> >> > > > > since the change went into Sentry and so the Jenkins job that
> >> > normally
> >> > > > > produces the new jars is still pulling in old bits.
> >> > > > >
> >> > > > > I've been talking with releng and they expect the new jars to be
> >> > > > available
> >> > > > > later today.
> >> > > > >
> >> > > > > On Mon, Jun 19, 2017 at 3:48 PM Tim Armstrong <
> >> > tarmstr...@cloudera.com
> >> > > >
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Looks like the build still breaks when starting up sentry
> after
> >> my
> >> > > fix:
> >> > > > > >
> >> > > > > >
> >> > > > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
> >> > 1547/console
> >> > > > > >
> >> > > > > > *20:08:54*  --> Starting the Sentry Policy Server*20:08:59*
> >> Error
> >> > in
> >> > > > > > /home/ubuntu/Impala/testdata/bin/run-all.sh at line 58:
> >> > > > > > $IMPALA_HOME/testdata/bin/run-sentry-service.sh > \*20:08:59*
> +
> >> > > > > > onexit*20:08:59* + df -m*20:08:59* Filesystem 1M-blocks
> >> Used
> >> > > > > > Available Use% Mounted on*20:08:59* udev   15070
> >>  1
> >> > > > > > 15070   1% /dev*20:08:59* tmpfs   3015 1
> >> 3015
> >> > > > > > 1% /run*20:08:59* /dev/xvda1161129 22275132204
> 15%
> >> > > > > > /*20:08:59* none   1 0 1   0%
> >> > > > > > /sys/fs/cgroup*20:08:59* none   5 0
> >>  5
> >> > >  0%
> >> > > > > > /run/lock*20:08:59* none   15075 1 15075
> >>  1%
> >> > > > > > /run/shm*20:08:59* none 100 0   100
>  0%
> >> > > > > > /run/user*20:08:59* + free -m*20:08:59*  total
> >> >  used
> >> > > > > >  free sharedbuffers cached*20:08:59* Mem:
> >> > > > > > 30148  19597  10550 11 91
> >> > > 14323*20:08:59*
> >> > > > > > -/+ buffers/cache:   5182  24965*20:08:59* Swap:
> >> >   0
> >> > > > > > 0  0*20:08:59* + uptime -p*20:08:59* up 45
> >> > > > > > minutes*20:08:59* + rm -rf /home/ubuntu/Impala/logs_stati
> >> > c*20:08:59*
> >> > > +
> >> > > > > > mkdir -p /home/ubuntu/Impala/logs_static*20:08:59* + cp -r -L
> >> > > > > > /home/ubuntu/Impala/logs /home/ubuntu/Impala/logs_stati
> >> c*20:08:59*
> >> > > > > > Build step 'Execute shell'

Re: Broken build from Sentry

2017-06-20 Thread Dimitris Tsirogiannis 
+Sasha, who I believe has more up-to-date information on this.

On Mon, Jun 19, 2017 at 10:56 PM, Henry Robinson  wrote:

> FWIW, I've been able to start Sentry by setting:
>
> export IMPALA_SENTRY_VERSION=1.5.1-cdh5.12.0-SNAPSHOT
>
> (i.e. rolling back to the previous version of Sentry). I haven't yet tried
> to run tests - does anyone know an ETA for a fix coming out of Cloudera for
> the 5.13-SNAPSHOT? If it might be a while, we should consider regressing
> the Sentry version to unblock checkins.
>
> On 19 June 2017 at 15:31, Tim Armstrong  wrote:
>
> > It's unfortunately not that simple. The API change has been in Apache
> > sentry
> >
> > So rolling back the API change temporarily solves the problem for
> Cloudera,
> > but we're going to have to deal with it at some point and get Impala
> > building against both versions of the API.
> >
> > On Mon, Jun 19, 2017 at 2:55 PM, Thomas Tauber-Marshall <
> > tmarsh...@cloudera.com> wrote:
> >
> > > Yes, the Sentry team has been contacted and they're going to be rolling
> > it
> > > back.
> > >
> > > On Mon, Jun 19, 2017 at 4:53 PM Todd Lipcon  wrote:
> > >
> > > > Quick question from a bystander: it seems like Sentry committed an
> > > > API-incompatible change. Instead of fixing on the Impala side, should
> > the
> > > > Sentry project be notified that they may want to roll back such a
> > change?
> > > > It seems like an error on their part to do such a thing within a
> minor
> > > > version.
> > > >
> > > > On Mon, Jun 19, 2017 at 1:56 PM, Thomas Tauber-Marshall <
> > > > tmarsh...@cloudera.com> wrote:
> > > >
> > > > > I'm working on getting the s3 jars updated, which presumably will
> fix
> > > > that.
> > > > >
> > > > > The problem (to my understanding) is that the nightlies haven't
> > passed
> > > > > since the change went into Sentry and so the Jenkins job that
> > normally
> > > > > produces the new jars is still pulling in old bits.
> > > > >
> > > > > I've been talking with releng and they expect the new jars to be
> > > > available
> > > > > later today.
> > > > >
> > > > > On Mon, Jun 19, 2017 at 3:48 PM Tim Armstrong <
> > tarmstr...@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Looks like the build still breaks when starting up sentry after
> my
> > > fix:
> > > > > >
> > > > > >
> > > > http://jenkins.impala.io:8080/job/ubuntu-14.04-from-scratch/
> > 1547/console
> > > > > >
> > > > > > *20:08:54*  --> Starting the Sentry Policy Server*20:08:59* Error
> > in
> > > > > > /home/ubuntu/Impala/testdata/bin/run-all.sh at line 58:
> > > > > > $IMPALA_HOME/testdata/bin/run-sentry-service.sh > \*20:08:59* +
> > > > > > onexit*20:08:59* + df -m*20:08:59* Filesystem 1M-blocks  Used
> > > > > > Available Use% Mounted on*20:08:59* udev   15070
>  1
> > > > > > 15070   1% /dev*20:08:59* tmpfs   3015 1
> 3015
> > > > > > 1% /run*20:08:59* /dev/xvda1161129 22275132204  15%
> > > > > > /*20:08:59* none   1 0 1   0%
> > > > > > /sys/fs/cgroup*20:08:59* none   5 0 5
> > >  0%
> > > > > > /run/lock*20:08:59* none   15075 1 15075   1%
> > > > > > /run/shm*20:08:59* none 100 0   100   0%
> > > > > > /run/user*20:08:59* + free -m*20:08:59*  total
> >  used
> > > > > >  free sharedbuffers cached*20:08:59* Mem:
> > > > > > 30148  19597  10550 11 91
> > > 14323*20:08:59*
> > > > > > -/+ buffers/cache:   5182  24965*20:08:59* Swap:
> >   0
> > > > > > 0  0*20:08:59* + uptime -p*20:08:59* up 45
> > > > > > minutes*20:08:59* + rm -rf /home/ubuntu/Impala/logs_stati
> > c*20:08:59*
> > > +
> > > > > > mkdir -p /home/ubuntu/Impala/logs_static*20:08:59* + cp -r -L
> > > > > > /home/ubuntu/Impala/logs /home/ubuntu/Impala/logs_
> static*20:08:59*
> > > > > > Build step 'Execute shell' marked build as failure*20:08:59* Set
> > > build
> > > > > > name.*20:08:59* New build name is '#1547
> > > > > > refs/changes/22/7222/3'*20:08:59* Variable with name
> > > > > > 'BUILD_DISPLAY_NAME' already exists, current value: '#1547
> > > > > > refs/changes/22/7222/3', new value: '#1547
> > > > > > refs/changes/22/7222/3'*20:09:12* Archiving artifacts*20:09:21*
> > > > > > Finished: FAILURE
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 19, 2017 at 12:23 PM, Tim Armstrong <
> > > > tarmstr...@cloudera.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > It's unclear if there will be incompatibility between the
> updated
> > > > > client
> > > > > > > and the version of sentry we use for the minicluster. I kicked
> > off
> > > a
> > > > > test
> > > > > > > run to see if it works.
> > > > > > >
> > > > > > > On Mon, Jun 19, 2017 at 12:06 PM, Henry Robinson <
> > he...@apache.org
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > >> Presumably this will break GVO jobs as well - should we