Re: New committer - Abhishek Rawat

[Lars Volker]

Congratulations, Abhishek!

On Tue, Dec 8, 2020 at 4:06 PM Yida Wu  wrote:

> Congrats Abhishek !
>
> On Tue, Dec 8, 2020 at 1:47 PM Wenzhe Zhou  wrote:
>
> > Congratulations!
> >
> > Wenzhe Zhou
> > wz...@cloudera.com
> > 408-568-0101
> >
> >
> > On Tue, Dec 8, 2020 at 1:43 PM Fang-Yu Rao 
> > wrote:
> >
> > > Congratulations Abhishek!
> > >
> > >
> > > On Tue, Dec 8, 2020 at 1:33 PM Vihang Karajgaonkar <
> vih...@cloudera.com>
> > > wrote:
> > >
> > > > Congratulations Abhishek!
> > > >
> > > > On Tue, Dec 8, 2020 at 1:32 PM Kurt Deschler 
> > > > wrote:
> > > >
> > > > > Congratulations, Abhishek!
> > > > >
> > > > > On Tue, Dec 8, 2020 at 4:12 PM Andrew Sherman <
> asher...@cloudera.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Well done Abhishek!
> > > > > >
> > > > > > On Tue, Dec 8, 2020 at 12:31 PM Tim Armstrong <
> > > tarmstr...@cloudera.com
> > > > >
> > > > > > wrote:
> > > > > >
> > > > > > >  The Project Management Committee (PMC) for Apache Impala has
> > > invited
> > > > > > > Abhishek Rawat to become a committer and we are pleased to
> > announce
> > > > > that
> > > > > > > they have accepted.
> > > > > > > Congratulations and welcome, Abhishek!
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: New Apache Impala committer - Shant Hovsepian

[Lars Volker]

Woohoo! Congratulations, Shant!

On Wed, Oct 14, 2020, 02:10 Jeszy  wrote:

> Congrats!
>
> On Wed, Oct 14, 2020 at 11:09 AM Norbert Luksa
>  wrote:
> >
> > Congratulations, Shant!
> >
> > On Wed, Oct 14, 2020 at 10:55 AM Laszlo Gaal 
> > wrote:
> >
> > > Congratulations, Shant!
> > >
> > > On Wed, Oct 14, 2020 at 9:39 AM Tamas Mate  wrote:
> > >
> > > > Congrats, Shant!
> > > >
> > > > On Wed, Oct 14, 2020 at 9:35 AM Zoltán Borók-Nagy <
> borokna...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Congratulations, Shant!
> > > > >
> > > > >
> > > > > On Wed, Oct 14, 2020 at 4:03 AM Shant Hovsepian <
> > > > sh...@superdupershant.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > Humbled and honored, thanks all!
> > > > > >
> > > > > > On Tue, Oct 13, 2020 at 7:59 PM Quanlong Huang <
> > > > huangquanl...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Congratulations Shant!
> > > > > > >
> > > > > > > On Wed, Oct 14, 2020 at 7:11 AM Sahil Takiar <
> > > takiar.sa...@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Congrats Shant!
> > > > > > > >
> > > > > > > > On Tue, Oct 13, 2020 at 3:35 PM Fang-Yu Rao <
> > > > fangyu@cloudera.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Congratulations Shant!
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Oct 13, 2020 at 2:33 PM Vihang Karajgaonkar <
> > > > > > > vih...@cloudera.com
> > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Congratulations Shant!
> > > > > > > > > >
> > > > > > > > > > On Tue, Oct 13, 2020 at 2:29 PM David Rorke <
> > > > dro...@cloudera.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Congrats Shant!
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Oct 13, 2020 at 2:23 PM Andrew Sherman <
> > > > > > > > asher...@cloudera.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Congratulations Shant!!
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Oct 13, 2020 at 2:16 PM Joe McDonnell <
> > > > > > > > > > joemcdonn...@cloudera.com
> > > > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Congratulations, Shant!
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Oct 13, 2020 at 2:14 PM Kurt Deschler <
> > > > > > > > > kdesc...@cloudera.com
> > > > > > > > > > >
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Congrats, Shant!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Oct 13, 2020 at 5:10 PM Wenzhe Zhou <
> > > > > > > > wz...@cloudera.com>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Congratulations, Shant!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Wenzhe Zhou
> > > > > > > > > > > > > > > wz...@cloudera.com
> > > > > > > > > > > > > > > 408-568-0101
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Tue, Oct 13, 2020 at 2:04 PM Riza Suminto <
> > > > > > > > > > > > > riza.sumi...@cloudera.com>
> > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Congratulations, Shant!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Tue, Oct 13, 2020 at 2:03 PM Zoram Thanga
> <
> > > > > > > > > > zo...@cloudera.com
> > > > > > > > > > > >
> > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Congratulations, Shant!
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Tue, Oct 13, 2020 at 1:51 PM Tim
> Armstrong <
> > > > > > > > > > > > > > tarmstr...@cloudera.com
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >  The Project Management Committee (PMC)
> for
> > > > > Apache
> > > > > > > > Impala
> > > > > > > > > > has
> > > > > > > > > > > > > > invited
> > > > > > > > > > > > > > > > > Shant
> > > > > > > > > > > > > > > > > > Hovsepian to become a committer and we
> are
> > > > > pleased
> > > > > > to
> > > > > > > > > > > announce
> > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > they
> > > > > > > > > > > > > > > > > > have accepted.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Congratulations and welcome, Shant!
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > --
> > > > > > > > > > > > > > > > > Zoram Thanga
> > > > > > > > > > > > > > > > > Cloudera Inc.
> > > > > > > > > > > > > > > > > 395 Page Mill Road
> > > > > > > > > > > > > > > > > Palo Alto, CA 94306
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > 

Re: New committer - Aman Sinha

[Lars Volker]

Congratulations, Aman!

On Tue, Sep 8, 2020 at 10:51 AM Andrew Sherman 
wrote:

> Good news!
>
> On Tue, Sep 8, 2020 at 10:50 AM Bikramjeet Vig <
> bikramjeet@cloudera.com>
> wrote:
>
> > Congrats Aman!
> >
> > On Tue, Sep 8, 2020 at 10:31 AM Tim Armstrong 
> > wrote:
> >
> > >  The Project Management Committee (PMC) for Apache Impala has invited
> > Aman
> > > Sinha to become a committer and we are pleased to announce that they
> have
> > > accepted.
> > > Congratulations and welcome, Aman!
> > >
> >
>

Re: New Committer: Anurag Mantripragada

[Lars Volker]

Congratulations Anurag!

On Wed, May 13, 2020, 21:57 Vihang Karajgaonkar  wrote:

> Congrats Anurag!
>
> On Wed, May 13, 2020 at 9:55 PM David Rorke  wrote:
>
> > Congrats Anurag!
> >
> > On Wed, May 13, 2020 at 9:49 PM Fang-Yu Rao 
> > wrote:
> >
> > > Congratulations!
> > >
> > >
> > > On Wed, May 13, 2020 at 8:37 PM Wenzhe Zhou 
> wrote:
> > >
> > > > Congratulations!
> > > >
> > > > Wenzhe Zhou
> > > > wz...@cloudera.com
> > > > 408-568-0101
> > > >
> > > >
> > > > On Wed, May 13, 2020 at 7:49 PM Dinesh Garg 
> > > wrote:
> > > >
> > > > > Congratulations for the very well deserved committership, Anurag.
> > > > >
> > > > > --
> > > > > Regards
> > > > > Dinesh Garg
> > > > > +1510.931.0944
> > > > >
> > > > >
> > > > >
> > > > > On Wed, May 13, 2020 at 7:40 PM Riza Suminto <
> > > riza.sumi...@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > Congratulations!
> > > > > >
> > > > > > On Wed, May 13, 2020 at 7:09 PM Joe McDonnell <
> > > > joemcdonn...@cloudera.com
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Congrats!
> > > > > > >
> > > > > > > On Wed, May 13, 2020 at 7:04 PM Andrew Sherman <
> > > > asher...@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yay! Well done Anurag!
> > > > > > > >
> > > > > > > > On Wed, May 13, 2020 at 7:00 PM Quanlong Huang <
> > > > > > huangquanl...@gmail.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > The Project Management Committee (PMC) for Apache Impala
> has
> > > > > invited
> > > > > > > > Anurag
> > > > > > > > > to become a committer and we are pleased to announce that
> > they
> > > > have
> > > > > > > > > accepted.
> > > > > > > > > Congratulations and welcome, Anurag!
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Jenkins service outage

[Lars Volker]

Thank you for taking care of the updates, Vihang!

On Wed, Jan 15, 2020, 18:41 Vihang Karajgaonkar  wrote:

> Jenkins is back up and running. Some of the jobs related to reviews below
> had to be aborted to restart the server. I retriggered the jobs which were
> killed.
>
> https://gerrit.cloudera.org/#/c/14917/
> https://gerrit.cloudera.org/#/c/15039/
> https://jenkins.impala.io/job/parallel-all-tests-nightly/1161/
>
> Thanks,
> Vihang
>
> On Wed, Jan 15, 2020 at 5:47 PM Vihang Karajgaonkar 
> wrote:
>
> > The public Jenkins server (http://jenkins.impala.io) will be taken
> > offline for security updates in some time. I will send out an email when
> it
> > comes back up.
> >
> > Thanks,
> > Vihang
> >
>

Re: Impala Jenkins service outage

[Lars Volker]

Thank you for installing the updates, Laszlo!

On Tue, Dec 17, 2019 at 8:19 AM Laszlo Gaal 
wrote:

> Jenkins is now back online.
>
> There was a change in the way it interacts with Gerrit, so if you see any
> issues with the way
> Gerrit reviews are processed (or with Jenkins in general), please let me
> know.
>
> On Tue, Dec 17, 2019 at 4:51 PM Laszlo Gaal 
> wrote:
>
> > The Impala public Jenkins instance (jenkins.impala.io) will be taken
> > offline for security update shortly. Mail will be sent when the service
> > becomes available again.
> >
> > Thank you for your patience,
> >
> > - Laszlo Gaal
> >
>

Re: Fixing #hosts in exec summary

[Lars Volker]

I'm in favor of always displaying the number of instances. However, we're
at 100 columns with the summary and should be mindful of users who display
the summary in the impala-shell. How about using a short title, e.g. #Inst
instead of #instances?

On Wed, Nov 20, 2019 at 9:50 AM Tim Armstrong 
wrote:

> > Do you know of any tools that rely on summary having only one of the
> mentioned columns?
> I don't know of any tools that consume the text representation of the exec
> summary. Even if they did, I don't think they can reasonably expect that we
> never change the format.
>
> > I managed to change the signature of the summary to have a #hosts and a
> #instances column with the correct value. My question is should the
> instances column only appear when mt_dop > 0, since the values would be the
> same?
>
> I'd be inclined to always include the instances column so that the format
> is consistent, even if the information is redundant. I don't feel that
> strongly about it though - I can see the argument for reducing the amount
> of data displayed.
>
>
>
> On Wed, Nov 20, 2019 at 4:27 AM Norbert Luksa 
> wrote:
>
> > Hi,
> >
> > I'm working on https://issues.apache.org/jira/browse/IMPALA-4618.
> >
> > I managed to change the signature of the summary to have a #hosts and a
> > #instances column with the correct value. My question is should the
> > instances column only appear when mt_dop > 0, since the values would be
> the
> > same?
> > Do you know of any tools that rely on summary having only one of the
> > mentioned columns?
> >
> > Thanks,
> > Norbert
> >
>

Re: Maintenance for jenkins.impala.io today

[Lars Volker]

Thank you for doing the maintenance work, Attila!

On Wed, Aug 28, 2019 at 9:36 AM Attila Jeges  wrote:

> Jenkins is back up and running. Let me know if you see any issues.
>
> On Wed, Aug 28, 2019 at 5:39 PM Attila Jeges  wrote:
>
> > Hi,
> > We need to upgrade Jenkins to address a security advisory. I'll be doing
> > so
> > today. My apologies for the disruption it will inevitably cause!
> >
>

Re: [VOTE] 3.3.0 release candidate 1

[Lars Volker]

+1 (binding)

   - Checked the signature
   - Ran buildall.sh
   - Ran a handful of queries, created and dropped a table


On Wed, Aug 21, 2019 at 1:31 PM Bharath Vissapragada 
wrote:

> +1 (binding)
> https://jenkins.impala.io/view/Utility/job/release-test-ub1604/26/
>
> The job failed because of a flaky test for which there is a patch in flight
> [1]. It shouldn't block the release IMO.
>
> [1] https://gerrit.cloudera.org/#/c/14117/
>
> On Tue, Aug 20, 2019 at 7:38 PM Jim Apple  wrote:
>
> > +1 (binding), based on
> > https://jenkins.impala.io/view/Utility/job/release-test-ub1604/25/
> >
> > On Mon, Aug 19, 2019 at 1:01 PM Quanlong Huang 
> > wrote:
> >
> > > Hi folks,
> > >
> > > This is a vote for Impala 3.3.0
> > >
> > > The artifacts for testing can be downloaded from:
> > > https://dist.apache.org/repos/dist/dev/impala/3.3.0/RC1/
> > > Git tag: 3.3.0-rc1
> > > Tree hash: 5ef67bca619d19402f8c7186b2ab6895bd0603ba
> > >
> > > Please vote +1 or -1. -1 votes should be accompanied by an explanation
> of
> > > the reason. Only PMC members have binding votes, but other
> > > community members are encouraged to cast non-binding votes. This vote
> > will
> > > pass if there are 3 binding +1 votes and more binding +1 votes than -1
> > > votes.
> > >
> > > This wiki page describes how to check the release before you vote:
> > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> > >
> > > Cheers,
> > > Quanlong
> > >
> >
>

Re: [DISCUSS] 3.3.0 release

[Lars Volker]

+1. We've added lots of new things over the past months and it'll be nice
to get them out in a release.

On Fri, Aug 16, 2019 at 8:51 AM Bharath Vissapragada 
wrote:

> +1. Thanks for volunteering.
>
> On Thu, Aug 15, 2019 at 8:25 PM Quanlong Huang 
> wrote:
>
> > Hi all,
> >
> > Impala-3.2.0 is released since March. A couple of pretty cool features
> and
> > a good number of improvements are checked in since then. I think it's
> time
> > to release 3.3.0 to make them visible to our users.
> >
> > I propose that we release 3.3.0 soon and I volunteer to be its release
> > manager. Please speak up and let the community know if anyone has any
> > objections to this.
> >
> > Thanks,
> > Quanlong
> >
>

Re: jenkins.impala.io will be restarted for updates

[Lars Volker]

Thank you for doing the maintenance, Vihang!

On Wed, Jul 31, 2019, 23:43 Vihang Karajgaonkar  wrote:

> Jenkins has been restarted and is back up and running. Jobs related to
> following two reviews were aborted and I resubmitted them again. Let me
> know if you notice anything wrong.
>
> https://gerrit.cloudera.org/#/c/13740/
> https://gerrit.cloudera.org/#/c/13883/
>
> Thanks,
> Vihang
>
> On Wed, Jul 31, 2019 at 11:24 PM Vihang Karajgaonkar 
> wrote:
>
> > Just a reminder to all, jenkins.impala.io will be down for maintenance.
> I
> > will update on this thread once its back.
> >
> > Thanks,
> > Vihang
> >
> > On Wed, Jul 31, 2019 at 3:13 PM Vihang Karajgaonkar  >
> > wrote:
> >
> >> jenkins.impala.io will be restarted tonight for updates. I will send
> out
> >> a email before starting the maintenance and after its done.
> >>
> >> Thanks,
> >> Vihang
> >>
> >
>

Re: jenkins.impala.io will be restarted for updates

[Lars Volker]

Thank you for doing the maintenance, Sahil!

On Thu, Jul 18, 2019 at 9:15 PM Sahil Takiar  wrote:

> Maintenance is complete, all the killed jobs have been restarted.
>
> Thanks,
> Sahil
>
> On Thu, Jul 18, 2019 at 8:54 PM Sahil Takiar 
> wrote:
>
> > Maintenance is starting now. Unfortunately, some jobs are still running,
> > I'm going to kill them and restart them after the maintenance. If there
> are
> > any issues, let me know.
> >
> > Thanks,
> > Sahil
> >
> > On Thu, Jul 18, 2019 at 6:38 PM Sahil Takiar 
> > wrote:
> >
> >> The restart will happen later tonight. I will send an update before and
> >> after the required maintenance.
> >>
> >> Thanks,
> >> Sahil
> >>
> >
> >
> > --
> > Sahil Takiar
> > Software Engineer
> > takiar.sa...@gmail.com | (510) 673-0309
> >
>
>
> --
> Sahil Takiar
> Software Engineer
> takiar.sa...@gmail.com | (510) 673-0309
>

Re: Apache JIRA Bug Tracker

[Lars Volker]

Done.

Cheers, Lars

On Wed, Jul 17, 2019 at 7:40 AM Kurt Deschler  wrote:

> Hello,
> Please add me as a contributor on the Apache JIRA bug tracker for the
> Impala project. Thanks,
>   -Kurt Deschler
>

Re: Add a parameter in Jenkins job cherrypick-2.x-and-test

[Lars Volker]

Hi Quanlong,

I added a parameter to the jenkins job. This test run looks like it works
ok: https://jenkins.impala.io/job/cherrypick-2.x-and-test/681/console

Please have a look and let me know if you require any further changes.

Cheers, Lars

On Sat, Jun 15, 2019 at 11:36 PM Quanlong Huang 
wrote:

> It happens again that a clean pick introduces build failure:
>
>- Jenkins run:
> https://jenkins.impala.io/job/cherrypick-2.x-and-test/678/
>- Commit required some fix:
>https://github.com/apache/impala/commit/c333b5526
>
> Now the HEAD of 2.x branch is e6c1eb85eb31264eaf5ed92782bf181225ce9581.
> There are 20 commits that can be cleanly applied to the 2.x branch before
> this commit (c333b5526). It'd be better that the Jenkins job can pick them
> first. Then we just need to submit one patch for review, requiring pretty
> less approvals than the last time. So we can move forward faster.
>
> I submitted a patch for giving a terminal commit hash to
> bin/compare_branches.py: https://gerrit.cloudera.org/c/13660
> If this got approved, hope someone have the permission can do me a favor to
> add an optional parameter for this terminal commit hash in
> the cherrypick-2.x-and-test Jenkins job.
>
> Thanks,
> Quanlong
>
>
> On Wed, Mar 20, 2019 at 10:12 AM Quanlong Huang 
> wrote:
>
> > Sure. I just uploaded and published a batch of patches:
> >
> > remote: New Changes:
> >
> > remote:   http://gerrit.cloudera.org:8080/12798 IMPALA-6642 (Part 2):
> > clean up start-impala-cluster.py [DRAFT]
> >
> > remote:   http://gerrit.cloudera.org:8080/12799 IMPALA-6988: Implement
> > ALTER TABLE/VIEW SET OWNER [DRAFT]
> >
> > remote:   http://gerrit.cloudera.org:8080/12800 IMPALA-6223: Gracefully
> > handle malformed 'with' queries in impala-shell [DRAFT]
> >
> > remote:   http://gerrit.cloudera.org:8080/12801 IMPALA-6031: Fix
> executor
> > node count in distributed plans [DRAFT]
> >
> > remote:   http://gerrit.cloudera.org:8080/12802 IMPALA-3956: [DOCS]
> Escape
> > variables with '\' in impala-shell [DRAFT]
> >
> > remote:   http://gerrit.cloudera.org:8080/12803 IMPALA-6918: [DOCS]
> > COMMENT ON COLUMN privileges [DRAFT]
> >
> > remote:   http://gerrit.cloudera.org:8080/12804 [DOCS] Removed an unused
> > keydef [DRAFT]
> >
> > remote:   http://gerrit.cloudera.org:8080/12805 [DOCS] A typo fixed in
> > impala_shell_running_commands [DRAFT]
> >
> > remote:   http://gerrit.cloudera.org:8080/12806 Fix zsh issue in
> > set-pythonpath.sh [DRAFT]
> >
> > remote:   http://gerrit.cloudera.org:8080/12807 IMPALA-7254:
> Inconsistent
> > decimal behavior for IN/BETWEEN predicate [DRAFT]
> >
> > Note that I've ignored c01efd096 in master branch since it's reverted
> > later (df78eaec0). In all the above patches, just the last one
> > (IMPALA-7254) needs some changes to fix test failures. I've added what I
> > changed in the review comments. Hope anyone has time can help to review
> > them.
> >
> > Thanks,
> > Quanlong
> >
> >
> > On Tue, Mar 19, 2019 at 4:22 AM Lars Volker  wrote:
> >
> >> Yes, you could have a local branch, cherry pick all the changes that you
> >> know are working plus the last one that causes issues. Then you can send
> >> the whole branch for review, which will open a new review for each of
> the
> >> changes and then submit them manually.
> >>
> >> Cheers, Lars
> >>
> >> On Fri, Mar 15, 2019 at 6:09 PM Quanlong Huang  >
> >> wrote:
> >>
> >> > I thought all the submits to 2.x and master branch should be done in
> >> > Gerrit. Do you mean I can create a new branch manually with the next
> 10
> >> > patches, and then merge it *manually* to branch 2.x after
> >> > parallel-all-tests run successfully on it?
> >> >
> >> > --
> >> > Quanlong
> >> >
> >> > On Sat, Mar 16, 2019 at 12:17 AM Lars Volker  wrote:
> >> >
> >> > > The Jenkins job uses compare_branches.py
> >> > > <https://github.com/apache/impala/blob/2.x/bin/compare_branches.py>
> >> to
> >> > do
> >> > > the cherry picking. That script takes source_branch and
> target_branch
> >> > > parameters, but they need to be valid keys into the ignored commits
> >> file.
> >> > > You'd probably want to amend that script to take a commit up until
> >> which
> >> > > you want to pick. Alternatively you could do the picking manually by
> >> &g

New PMC Members: Gabor Kaszab and Bikramjeet Vig

[Lars Volker]

The Project Management Committee (PMC) for Apache Impala has invited
Gabor Kaszab and Bikramjeet Vig to become PMC members and we are pleased to
announce
that they have accepted.

Congratulations and welcome, Gabor and Bikram!

Re: Maintenance downtime for the Impala public Jenkins server

[Lars Volker]

Thank you for taking care of this, Laszlo!

On Wed, May 22, 2019 at 8:31 AM Laszlo Gaal 
wrote:

> The service is up and running again.
>
> Thank you for your patience.
>
> On Wed, May 22, 2019 at 5:20 PM Laszlo Gaal 
> wrote:
>
> > The server instance has been taken offline and will be restarted shortly
> > for plugin security updates.
> > The currently running build will be restarted after the maintenance is
> > completed.
> >
>

New Committer: Sahil Takiar

[Lars Volker]

Hi All,

The Project Management Committee (PMC) for Apache Impala
has invited Sahil Takiar to become a committer and we are pleased
to announce that he has accepted.

Welcome and congratulations, Sahil!

Re: Please set a Fix Version when you close a JIRA

[Lars Volker]

Thanks for the reminder, Tim.

If you are curious whether this affects you, you can use the following Jira
query to find out: "assignee = currentUser() and resolution is not empty
and fixVersion is empty order by updated desc"

https://issues.apache.org/jira/issues/?jql=assignee%20%3D%20currentUser()%20and%20resolution%20is%20not%20empty%20and%20fixVersion%20is%20empty%20order%20by%20updated%20desc

On Tue, Apr 30, 2019 at 11:08 AM Tim Armstrong 
wrote:

> There's been an epidemic lately of contributors/committers forgetting to
> set a fix version when they close a JIRA. We rely on these fix versions to
> track what version a fix went into.
>
> This is a reminder to do this.
>
> - Tim
>

Re: Remote read testing in precommit

[Lars Volker]

+1 for turning it on

On Mon, Apr 22, 2019 at 2:14 PM Tim Armstrong 
wrote:

> It's been stable for a while now, with the exception of hitting a flaky
> test that is also flaky on the non-dockerised minicluster (IMPALA-8124) -
> https://jenkins.impala.io/job/ubuntu-16.04-dockerised-tests/
>
> Are there any objections to me modifying parallel-all-tests and therefore
> precommit to run this job? I'll wait a couple of days for lazy consensus
> then go ahead.
>
> Thanks,
> Tim
>
> On Fri, Apr 5, 2019 at 3:03 PM Lars Volker  wrote:
>
> > +1, thanks for working on this!
> >
> > On Fri, Apr 5, 2019 at 11:18 AM Jim Apple  wrote:
> >
> > > I'm in favor. Given the importance of remote reads, I would even be in
> > > favor of these if it DID extend the critical path.
> > >
> > > On Fri, Apr 5, 2019 at 10:41 AM Tim Armstrong  >
> > > wrote:
> > >
> > > > This is really about testing the dockerised minicluster, but gives us
> > > > coverage of remote read code paths for free, and more people care
> about
> > > > that right now.
> > > >
> > > > I got the core end-to-end tests passing locally as part of
> > > > https://issues.apache.org/jira/browse/IMPALA-7995. That change is up
> > for
> > > > review here https://gerrit.cloudera.org/c/12639/. The next step is
> to
> > > get
> > > > a
> > > > Jenkins job running, which I've been working on.
> > > >
> > > > I'd like to run it regularly so we can catch any regressions.
> Initially
> > > > I'll just have it email me when it fails, but after it's stable for a
> > > week
> > > > or two I'd like to make it part of the regular set of jobs.
> > > >
> > > > My preference is to run it as part of the precommit jobs, in parallel
> > to
> > > > the Ubuntu 16.04 tests. It should not extend the critical path of
> > > precommit
> > > > because it only runs the end-to-end tests. We could alternatively run
> > it
> > > as
> > > > a scheduled post-commit job, but that tends to create additional work
> > > when
> > > > it breaks.
> > > >
> > > > What do people think?
> > > >
> > > > - Tim
> > > >
> > >
> >
>

Re: Jenkins security update and server restart (jenkins.impala.io)

[Lars Volker]

Thanks for taking care of the update, Bharath!

On Sat, Apr 13, 2019 at 12:56 PM Bharath Vissapragada 
wrote:

> Jenkins is back up and running. Please let me know if something looks off!
>
> On Wed, Apr 10, 2019 at 9:37 AM Bharath Vissapragada <
> bhara...@cloudera.com>
> wrote:
>
> > Currently running builds will finish but no new builds will be accepted.
> > Will send out an update once finished.
> >
>

Re: New committer - Pooja Nilangekar

[Lars Volker]

Congratulations Pooja and welcome to our round!

On Sat, Apr 13, 2019, 08:59 Tim Armstrong  wrote:

>  The Project Management Committee (PMC) for Apache Impala has invited
> Pooja Nilangekar to become a committer and we are pleased to announce that
> they have accepted.
> Congratulations and welcome, Pooja!
>

New Impala Community Slack Workspace

[Lars Volker]

Hi All,

We have created a Slack community for the Impala project. It is public and
we welcome everyone to join us. You can do so by clicking on this invite
link

[1].
We look forward to seeing you there!

Cheers, Lars

[1]
https://join.slack.com/t/apache-impala/shared_invite/enQtNTgzMzAyNzIyNTk0LTQwMzJjMDI0YzEwOWRmZDk2MzNlZTk5OWZkNTI4M2Y5MmU1MjQ1ZWIzYzQxMWQyMjUzNjNjNWU0NDQ1MTMyNWM

Re: Next round of the Impala community meeting

[Lars Volker]

Thank you all for joining today's meeting. Here are the notes I took:

*Participants and what they would like to talk about:*
Csaba (Cloudera): Scanners, Timestamps
Quanlong (Hulu): *Want to know more about everyone’s focus area, whether
Impala supports PB scale warehouse, can introduce the ORC scanner*
Zoltan (Cloudera): Working on Parquet
Daniel (Cloudera): Delta encoding
Gabor (Cloudera): Working on upstream release, new patterns for timestamp
conversion,* interested short discussion about release notes proposal*
Attila (Cloudera): Working on date type support
Laszlo (Cloudera): Infrastructure, toolchain
Jeszy (Cloudera): Backline support
Joe (Cloudera): Disk io manager, S3 guard
Lars (Cloudera): Backend stuff

*Log of things we talked about*

   - Metadata issues: At 10s of PB, metadata becomes an issue (on 2.x).
   Discussed tips and tricks to improve refresh performance.
   - JDBC data source support, tracked in IMPALA-5741
   <https://issues.apache.org/jira/browse/IMPALA-5741>
   - Release notes
   - Very long list, seems useful to have a shorter list
  - We shouldn't rely on release notes for compatibility, things just
  shouldn’t break.
  - Point out the highlights, focus more on advancements (Metadata,
  KRPC).
  - Gabor will send out a proposal for people to contribute.


Thanks again for joining today. If you find these sessions *not helpful*,
please reach out to me with a suggestion what we could improve. Otherwise
I'll look forward to having another one in a few weeks.

Cheers, Lars



On Fri, Mar 15, 2019 at 6:19 PM Quanlong Huang 
wrote:

> Thank you, Lars!
>
> It'd be good if we can align with the topics first. So we can control the
> time for each part. I think we can poll in something like this:
>
> https://docs.google.com/document/d/166eE52awakVvmyQ0AIT1Y1giZX72mot3V8_37oIjNWc
>
> For people who will attend the meeting, feel free to write down any topics
> of interest :)
>
> Cheers,
> Quanlong
>
> On Sat, Mar 16, 2019 at 12:34 AM Lars Volker  wrote:
>
> > Thanks all for the feedback. I've sent out an invite and added everyone
> who
> > joined last time plus folks who have expressed interest in this thread.
> If
> > you have not received an invite but would like to join, please reach out
> to
> > me directly.
> >
> > We will meet at this time
> > <
> >
> https://www.timeanddate.com/worldclock/meetingdetails.html?year=2019=3=18=15=0=0=50=33=283
> > >
> > :
> >
> >  - Budapest (Hungary) Monday, March 18, 2019 at 4:00:00 pm CET (UTC+1)
> >  - Beijing (China - Beijing Municipality) Monday, March 18, 2019 at
> > 11:00:00 pm CST (UTC+8)
> >  - San Jose (USA - California) Monday, March 18, 2019 at 8:00:00 am PDT
> > (UTC-7)
> >
> > Please note that the Pacific time zone has already switched to daylight
> > savings time, while Central Europe will not switch until March 31st.
> >
> > Cheers, Lars
> >
> > On Mon, Mar 11, 2019 at 9:01 AM Zoltán Borók-Nagy <
> borokna...@cloudera.com
> > >
> > wrote:
> >
> > > Works for me as well!
> > >
> > > Thanks,
> > > Zoltan
> > >
> > >
> > > On Mon, Mar 11, 2019 at 4:46 PM Gabor Kaszab 
> > > wrote:
> > >
> > > > That seems much better from Budapest's pow! I would also be
> interested
> > > > joining if this schedule works out for everyone.
> > > >
> > > > Cheers,
> > > > Gabor
> > > >
> > > >
> > > > On Mon, Mar 11, 2019 at 4:30 PM Lars Volker  wrote:
> > > >
> > > > > How does it work for the folks in Budapest?
> > > > >
> > > > > On Mon, Mar 11, 2019 at 7:37 AM Quanlong Huang <
> > > huangquanl...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > I'm ok with the time. Thank you, Lars!
> > > > > >
> > > > > > On Sat, Mar 9, 2019 at 6:18 AM Lars Volker 
> > wrote:
> > > > > >
> > > > > > > How would moving Quanlong's proposed schedule to Monday, March
> > 18th
> > > > > work
> > > > > > > for everyone?
> > > > > > >
> > > > > > >  - Budapest (Hungary) Monday, March 18, 2019 at 4:00:00 pm CET
> > > UTC+1
> > > > > hour
> > > > > > >  - Beijing (China - Beijing Municipality) Monday, March 18,
> 2019
> > at
> > > > > > > 11:00:00 pm CST UTC+8 hours
> > > > > > >  - San Jose (USA - California) Monday, March 18, 2019 at
> 8:00:00
> > am
> >

Re: Add a parameter in Jenkins job cherrypick-2.x-and-test

[Lars Volker]

Yes, you could have a local branch, cherry pick all the changes that you
know are working plus the last one that causes issues. Then you can send
the whole branch for review, which will open a new review for each of the
changes and then submit them manually.

Cheers, Lars

On Fri, Mar 15, 2019 at 6:09 PM Quanlong Huang 
wrote:

> I thought all the submits to 2.x and master branch should be done in
> Gerrit. Do you mean I can create a new branch manually with the next 10
> patches, and then merge it *manually* to branch 2.x after
> parallel-all-tests run successfully on it?
>
> --
> Quanlong
>
> On Sat, Mar 16, 2019 at 12:17 AM Lars Volker  wrote:
>
> > The Jenkins job uses compare_branches.py
> > <https://github.com/apache/impala/blob/2.x/bin/compare_branches.py> to
> do
> > the cherry picking. That script takes source_branch and target_branch
> > parameters, but they need to be valid keys into the ignored commits file.
> > You'd probably want to amend that script to take a commit up until which
> > you want to pick. Alternatively you could do the picking manually by
> > passing a list of the commits from your spreadsheet into "git commit" (it
> > can handle multiple commits at once). Then just run parallel-all-tests on
> > it.
> >
> > Let me know if you need help with triggering any of this.
> >
> > Cheers, Lars
> >
> > On Fri, Mar 15, 2019 at 8:37 AM Quanlong Huang 
> > wrote:
> >
> > > Hi all,
> > >
> > > Branch 2.x is now at 193a1b5. See more details:
> > >
> > >
> >
> https://docs.google.com/spreadsheets/d/12h1rTAPS1gm0vhlDGxeOXjnRD7rrOcoqzX4rjRRCyBg
> > >
> > > I got a trouble that the cherrypick-2.x-and-test job is able to cleanly
> > > pick the next 15 patches but build fail due to a patch in the middle
> > > (f16436628). It needs some changes to fix the test. But I can't submit
> > the
> > > new patch until branch 2.x has picked the patch before it.
> > >
> > > Is it possible to add a parameter in job cherrypick-2.x-and-test, to
> tell
> > > it that don't consider patches after a commit? So I can let
> > > cherrypick-2.x-and-test
> > > pick only the next 10 patches and submit a new patch for the fix later.
> > >
> > > Thanks,
> > > Quanlong
> > >
> >
>

Re: Next round of the Impala community meeting

[Lars Volker]

Thanks all for the feedback. I've sent out an invite and added everyone who
joined last time plus folks who have expressed interest in this thread. If
you have not received an invite but would like to join, please reach out to
me directly.

We will meet at this time
<https://www.timeanddate.com/worldclock/meetingdetails.html?year=2019=3=18=15=0=0=50=33=283>
:

 - Budapest (Hungary) Monday, March 18, 2019 at 4:00:00 pm CET (UTC+1)
 - Beijing (China - Beijing Municipality) Monday, March 18, 2019 at
11:00:00 pm CST (UTC+8)
 - San Jose (USA - California) Monday, March 18, 2019 at 8:00:00 am PDT
(UTC-7)

Please note that the Pacific time zone has already switched to daylight
savings time, while Central Europe will not switch until March 31st.

Cheers, Lars

On Mon, Mar 11, 2019 at 9:01 AM Zoltán Borók-Nagy 
wrote:

> Works for me as well!
>
> Thanks,
> Zoltan
>
>
> On Mon, Mar 11, 2019 at 4:46 PM Gabor Kaszab 
> wrote:
>
> > That seems much better from Budapest's pow! I would also be interested
> > joining if this schedule works out for everyone.
> >
> > Cheers,
> > Gabor
> >
> >
> > On Mon, Mar 11, 2019 at 4:30 PM Lars Volker  wrote:
> >
> > > How does it work for the folks in Budapest?
> > >
> > > On Mon, Mar 11, 2019 at 7:37 AM Quanlong Huang <
> huangquanl...@gmail.com>
> > > wrote:
> > >
> > > > I'm ok with the time. Thank you, Lars!
> > > >
> > > > On Sat, Mar 9, 2019 at 6:18 AM Lars Volker  wrote:
> > > >
> > > > > How would moving Quanlong's proposed schedule to Monday, March 18th
> > > work
> > > > > for everyone?
> > > > >
> > > > >  - Budapest (Hungary) Monday, March 18, 2019 at 4:00:00 pm CET
> UTC+1
> > > hour
> > > > >  - Beijing (China - Beijing Municipality) Monday, March 18, 2019 at
> > > > > 11:00:00 pm CST UTC+8 hours
> > > > >  - San Jose (USA - California) Monday, March 18, 2019 at 8:00:00 am
> > PDT
> > > > >
> > > > > Cheers, Lars
> > > > >
> > > > > On Wed, Mar 6, 2019 at 10:57 AM Csaba Ringhofer <
> > > > csringho...@cloudera.com>
> > > > > wrote:
> > > > >
> > > > > > >When would be a better time for folks in Hungary (i.e. no public
> > > > > holiday)?
> > > > > > 15. is the only public holiday in March. Some of us will attend
> > > > Dataworks
> > > > > > on 19-22. of March, so the week after 15. is also not the best
> for
> > > us.
> > > > > >
> > > > > > I am interested in the community meeting, but it is not super
> > > important
> > > > > for
> > > > > > me to join.
> > > > > >
> > > > > > On Wed, Mar 6, 2019 at 7:28 PM Lars Volker 
> > wrote:
> > > > > >
> > > > > > > When would be a better time for folks in Hungary (i.e. no
> public
> > > > > > holiday)?
> > > > > > >
> > > > > > > On Wed, Mar 6, 2019 at 7:55 AM Zoltán Borók-Nagy <
> > > > > > borokna...@cloudera.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I'd be interested in joining the meeting, however March 15 is
> > not
> > > > the
> > > > > > > best
> > > > > > > > for Hungarians, since it's public holiday.
> > > > > > > > I'll try to make it, but there's a chance that I won't be
> > > > available.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Zoltan
> > > > > > > >
> > > > > > > >
> > > > > > > > On Wed, Mar 6, 2019 at 2:10 PM Jeszy 
> wrote:
> > > > > > > >
> > > > > > > > > I'd try to make it if the meeting was in a Europe-friendly
> > time
> > > > out
> > > > > > of
> > > > > > > > > interest, but I don't feel I have much to contribute -
> > probably
> > > > not
> > > > > > > > > worth a reschedule by itself.
> > > > > > > > >
> > > > > > > > > On Wed, 6 Mar 2019 at 00:21, Lars Volker 
> > > > wrote:
> > > > 

Re: Add a parameter in Jenkins job cherrypick-2.x-and-test

[Lars Volker]

The Jenkins job uses compare_branches.py
 to do
the cherry picking. That script takes source_branch and target_branch
parameters, but they need to be valid keys into the ignored commits file.
You'd probably want to amend that script to take a commit up until which
you want to pick. Alternatively you could do the picking manually by
passing a list of the commits from your spreadsheet into "git commit" (it
can handle multiple commits at once). Then just run parallel-all-tests on
it.

Let me know if you need help with triggering any of this.

Cheers, Lars

On Fri, Mar 15, 2019 at 8:37 AM Quanlong Huang 
wrote:

> Hi all,
>
> Branch 2.x is now at 193a1b5. See more details:
>
> https://docs.google.com/spreadsheets/d/12h1rTAPS1gm0vhlDGxeOXjnRD7rrOcoqzX4rjRRCyBg
>
> I got a trouble that the cherrypick-2.x-and-test job is able to cleanly
> pick the next 15 patches but build fail due to a patch in the middle
> (f16436628). It needs some changes to fix the test. But I can't submit the
> new patch until branch 2.x has picked the patch before it.
>
> Is it possible to add a parameter in job cherrypick-2.x-and-test, to tell
> it that don't consider patches after a commit? So I can let
> cherrypick-2.x-and-test
> pick only the next 10 patches and submit a new patch for the fix later.
>
> Thanks,
> Quanlong
>

Re: CWiki write access

[Lars Volker]

I added you, please have a look to see if it worked.

Cheers, Lars

On Tue, Mar 12, 2019 at 3:50 PM Joe McDonnell 
wrote:

> Can someone add write access for username "joemcdonnell"? I have some
> changes I'd like to make to the Impala wiki.
>
> Thanks,
> Joe
>

Re: Next round of the Impala community meeting

[Lars Volker]

How does it work for the folks in Budapest?

On Mon, Mar 11, 2019 at 7:37 AM Quanlong Huang 
wrote:

> I'm ok with the time. Thank you, Lars!
>
> On Sat, Mar 9, 2019 at 6:18 AM Lars Volker  wrote:
>
> > How would moving Quanlong's proposed schedule to Monday, March 18th work
> > for everyone?
> >
> >  - Budapest (Hungary) Monday, March 18, 2019 at 4:00:00 pm CET UTC+1 hour
> >  - Beijing (China - Beijing Municipality) Monday, March 18, 2019 at
> > 11:00:00 pm CST UTC+8 hours
> >  - San Jose (USA - California) Monday, March 18, 2019 at 8:00:00 am PDT
> >
> > Cheers, Lars
> >
> > On Wed, Mar 6, 2019 at 10:57 AM Csaba Ringhofer <
> csringho...@cloudera.com>
> > wrote:
> >
> > > >When would be a better time for folks in Hungary (i.e. no public
> > holiday)?
> > > 15. is the only public holiday in March. Some of us will attend
> Dataworks
> > > on 19-22. of March, so the week after 15. is also not the best for us.
> > >
> > > I am interested in the community meeting, but it is not super important
> > for
> > > me to join.
> > >
> > > On Wed, Mar 6, 2019 at 7:28 PM Lars Volker  wrote:
> > >
> > > > When would be a better time for folks in Hungary (i.e. no public
> > > holiday)?
> > > >
> > > > On Wed, Mar 6, 2019 at 7:55 AM Zoltán Borók-Nagy <
> > > borokna...@cloudera.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > I'd be interested in joining the meeting, however March 15 is not
> the
> > > > best
> > > > > for Hungarians, since it's public holiday.
> > > > > I'll try to make it, but there's a chance that I won't be
> available.
> > > > >
> > > > > Thanks,
> > > > > Zoltan
> > > > >
> > > > >
> > > > > On Wed, Mar 6, 2019 at 2:10 PM Jeszy  wrote:
> > > > >
> > > > > > I'd try to make it if the meeting was in a Europe-friendly time
> out
> > > of
> > > > > > interest, but I don't feel I have much to contribute - probably
> not
> > > > > > worth a reschedule by itself.
> > > > > >
> > > > > > On Wed, 6 Mar 2019 at 00:21, Lars Volker 
> wrote:
> > > > > > >
> > > > > > > I'm good with that time too. However, I have not seen interest
> > from
> > > > > folks
> > > > > > > in Europe, neither in this thread nor in the one where we
> planned
> > > the
> > > > > > > previous meeting.
> > > > > > >
> > > > > > > Is anyone in Europe interested in joining us?
> > > > > > >
> > > > > > > Cheers, Lars
> > > > > > >
> > > > > > > On Mon, Mar 4, 2019 at 3:56 PM Quanlong Huang <
> > > > huangquanl...@gmail.com
> > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1 for Tim's suggestion since our majority are in America and
> > > > Europe.
> > > > > > What
> > > > > > > > about this time:
> > > > > > > >
> > > > > > > >  - Budapest (Hungary) Friday, March 15, 2019 at 4:00:00 pm
> CET
> > > > UTC+1
> > > > > > hour
> > > > > > > >  - Beijing (China - Beijing Municipality) Friday, March 15,
> > 2019
> > > at
> > > > > > > > 11:00:00 pm CST UTC+8 hours
> > > > > > > >  - San Jose (USA - California) Friday, March 15, 2019 at
> > 8:00:00
> > > am
> > > > > PDT
> > > > > > > >
> > > > > > > > On Tue, Mar 5, 2019 at 6:23 AM Tim Armstrong <
> > > > > tarmstr...@cloudera.com>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > WFM. I think we should also consider a future meeting time
> > (not
> > > > > this
> > > > > > one)
> > > > > > > > > that will make it easy for people in Europe to join as
> well.
> > > > > > > > >
> > > > > > > > > On Mon, Mar 4, 2019 at 11:08 AM Lars Volker <
> l...@cloudera.com
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi All,
> > > > > > > > > >
> > > > > > > > > > I propose to have the next Impala community meetings on
> > > > > > > > > >
> > > > > > > > > > Thursday, March 14th, 4pm PST/San Francisco  -  Friday,
> > March
> > > > > > 15th, 8am
> > > > > > > > > > CST/Beijing.
> > > > > > > > > >
> > > > > > > > > > If that time doesn't work for you but you would like to
> > join,
> > > > > > please
> > > > > > > > > reply
> > > > > > > > > > to this email with a new time proposal. If there are no
> > > > > objections
> > > > > > I
> > > > > > > > will
> > > > > > > > > > send out an invite in the next few days.
> > > > > > > > > >
> > > > > > > > > > Cheers, Lars
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Next round of the Impala community meeting

[Lars Volker]

How would moving Quanlong's proposed schedule to Monday, March 18th work
for everyone?

 - Budapest (Hungary) Monday, March 18, 2019 at 4:00:00 pm CET UTC+1 hour
 - Beijing (China - Beijing Municipality) Monday, March 18, 2019 at
11:00:00 pm CST UTC+8 hours
 - San Jose (USA - California) Monday, March 18, 2019 at 8:00:00 am PDT

Cheers, Lars

On Wed, Mar 6, 2019 at 10:57 AM Csaba Ringhofer 
wrote:

> >When would be a better time for folks in Hungary (i.e. no public holiday)?
> 15. is the only public holiday in March. Some of us will attend Dataworks
> on 19-22. of March, so the week after 15. is also not the best for us.
>
> I am interested in the community meeting, but it is not super important for
> me to join.
>
> On Wed, Mar 6, 2019 at 7:28 PM Lars Volker  wrote:
>
> > When would be a better time for folks in Hungary (i.e. no public
> holiday)?
> >
> > On Wed, Mar 6, 2019 at 7:55 AM Zoltán Borók-Nagy <
> borokna...@cloudera.com>
> > wrote:
> >
> > > Hi,
> > >
> > > I'd be interested in joining the meeting, however March 15 is not the
> > best
> > > for Hungarians, since it's public holiday.
> > > I'll try to make it, but there's a chance that I won't be available.
> > >
> > > Thanks,
> > > Zoltan
> > >
> > >
> > > On Wed, Mar 6, 2019 at 2:10 PM Jeszy  wrote:
> > >
> > > > I'd try to make it if the meeting was in a Europe-friendly time out
> of
> > > > interest, but I don't feel I have much to contribute - probably not
> > > > worth a reschedule by itself.
> > > >
> > > > On Wed, 6 Mar 2019 at 00:21, Lars Volker  wrote:
> > > > >
> > > > > I'm good with that time too. However, I have not seen interest from
> > > folks
> > > > > in Europe, neither in this thread nor in the one where we planned
> the
> > > > > previous meeting.
> > > > >
> > > > > Is anyone in Europe interested in joining us?
> > > > >
> > > > > Cheers, Lars
> > > > >
> > > > > On Mon, Mar 4, 2019 at 3:56 PM Quanlong Huang <
> > huangquanl...@gmail.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > +1 for Tim's suggestion since our majority are in America and
> > Europe.
> > > > What
> > > > > > about this time:
> > > > > >
> > > > > >  - Budapest (Hungary) Friday, March 15, 2019 at 4:00:00 pm CET
> > UTC+1
> > > > hour
> > > > > >  - Beijing (China - Beijing Municipality) Friday, March 15, 2019
> at
> > > > > > 11:00:00 pm CST UTC+8 hours
> > > > > >  - San Jose (USA - California) Friday, March 15, 2019 at 8:00:00
> am
> > > PDT
> > > > > >
> > > > > > On Tue, Mar 5, 2019 at 6:23 AM Tim Armstrong <
> > > tarmstr...@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > > > WFM. I think we should also consider a future meeting time (not
> > > this
> > > > one)
> > > > > > > that will make it easy for people in Europe to join as well.
> > > > > > >
> > > > > > > On Mon, Mar 4, 2019 at 11:08 AM Lars Volker 
> > > wrote:
> > > > > > >
> > > > > > > > Hi All,
> > > > > > > >
> > > > > > > > I propose to have the next Impala community meetings on
> > > > > > > >
> > > > > > > > Thursday, March 14th, 4pm PST/San Francisco  -  Friday, March
> > > > 15th, 8am
> > > > > > > > CST/Beijing.
> > > > > > > >
> > > > > > > > If that time doesn't work for you but you would like to join,
> > > > please
> > > > > > > reply
> > > > > > > > to this email with a new time proposal. If there are no
> > > objections
> > > > I
> > > > > > will
> > > > > > > > send out an invite in the next few days.
> > > > > > > >
> > > > > > > > Cheers, Lars
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
>

Re: Next round of the Impala community meeting

[Lars Volker]

When would be a better time for folks in Hungary (i.e. no public holiday)?

On Wed, Mar 6, 2019 at 7:55 AM Zoltán Borók-Nagy 
wrote:

> Hi,
>
> I'd be interested in joining the meeting, however March 15 is not the best
> for Hungarians, since it's public holiday.
> I'll try to make it, but there's a chance that I won't be available.
>
> Thanks,
> Zoltan
>
>
> On Wed, Mar 6, 2019 at 2:10 PM Jeszy  wrote:
>
> > I'd try to make it if the meeting was in a Europe-friendly time out of
> > interest, but I don't feel I have much to contribute - probably not
> > worth a reschedule by itself.
> >
> > On Wed, 6 Mar 2019 at 00:21, Lars Volker  wrote:
> > >
> > > I'm good with that time too. However, I have not seen interest from
> folks
> > > in Europe, neither in this thread nor in the one where we planned the
> > > previous meeting.
> > >
> > > Is anyone in Europe interested in joining us?
> > >
> > > Cheers, Lars
> > >
> > > On Mon, Mar 4, 2019 at 3:56 PM Quanlong Huang  >
> > > wrote:
> > >
> > > > +1 for Tim's suggestion since our majority are in America and Europe.
> > What
> > > > about this time:
> > > >
> > > >  - Budapest (Hungary) Friday, March 15, 2019 at 4:00:00 pm CET UTC+1
> > hour
> > > >  - Beijing (China - Beijing Municipality) Friday, March 15, 2019 at
> > > > 11:00:00 pm CST UTC+8 hours
> > > >  - San Jose (USA - California) Friday, March 15, 2019 at 8:00:00 am
> PDT
> > > >
> > > > On Tue, Mar 5, 2019 at 6:23 AM Tim Armstrong <
> tarmstr...@cloudera.com>
> > > > wrote:
> > > >
> > > > > WFM. I think we should also consider a future meeting time (not
> this
> > one)
> > > > > that will make it easy for people in Europe to join as well.
> > > > >
> > > > > On Mon, Mar 4, 2019 at 11:08 AM Lars Volker 
> wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > I propose to have the next Impala community meetings on
> > > > > >
> > > > > > Thursday, March 14th, 4pm PST/San Francisco  -  Friday, March
> > 15th, 8am
> > > > > > CST/Beijing.
> > > > > >
> > > > > > If that time doesn't work for you but you would like to join,
> > please
> > > > > reply
> > > > > > to this email with a new time proposal. If there are no
> objections
> > I
> > > > will
> > > > > > send out an invite in the next few days.
> > > > > >
> > > > > > Cheers, Lars
> > > > > >
> > > > >
> > > >
> >
>

Re: Next round of the Impala community meeting

[Lars Volker]

I'm good with that time too. However, I have not seen interest from folks
in Europe, neither in this thread nor in the one where we planned the
previous meeting.

Is anyone in Europe interested in joining us?

Cheers, Lars

On Mon, Mar 4, 2019 at 3:56 PM Quanlong Huang 
wrote:

> +1 for Tim's suggestion since our majority are in America and Europe. What
> about this time:
>
>  - Budapest (Hungary) Friday, March 15, 2019 at 4:00:00 pm CET UTC+1 hour
>  - Beijing (China - Beijing Municipality) Friday, March 15, 2019 at
> 11:00:00 pm CST UTC+8 hours
>  - San Jose (USA - California) Friday, March 15, 2019 at 8:00:00 am PDT
>
> On Tue, Mar 5, 2019 at 6:23 AM Tim Armstrong 
> wrote:
>
> > WFM. I think we should also consider a future meeting time (not this one)
> > that will make it easy for people in Europe to join as well.
> >
> > On Mon, Mar 4, 2019 at 11:08 AM Lars Volker  wrote:
> >
> > > Hi All,
> > >
> > > I propose to have the next Impala community meetings on
> > >
> > > Thursday, March 14th, 4pm PST/San Francisco  -  Friday, March 15th, 8am
> > > CST/Beijing.
> > >
> > > If that time doesn't work for you but you would like to join, please
> > reply
> > > to this email with a new time proposal. If there are no objections I
> will
> > > send out an invite in the next few days.
> > >
> > > Cheers, Lars
> > >
> >
>

Next round of the Impala community meeting

[Lars Volker]

Hi All,

I propose to have the next Impala community meetings on

Thursday, March 14th, 4pm PST/San Francisco  -  Friday, March 15th, 8am
CST/Beijing.

If that time doesn't work for you but you would like to join, please reply
to this email with a new time proposal. If there are no objections I will
send out an invite in the next few days.

Cheers, Lars

Re: Impala Video Meeting

[Lars Volker]

Google Meet has a chat on the right hand side, we can use that in case
anyone has issues with the connection. I have joined and organized similar
meetings in the Parquet community and this usually worked well.

Cheers, Lars

On Tue, Feb 19, 2019 at 8:58 AM Quanlong Huang 
wrote:

> Thank you, Lars!
>
> Should we create a Slack channel? So we can type something in case anyone
> has hardware issues, or to paste some resources in need. First time to use
> Google meeting. Not sure about this...
>
> Thanks,
> Quanlong
>
> On Sat, Feb 16, 2019 at 3:03 AM Lars Volker  wrote:
>
> > Hi All,
> >
> > Thanks for chiming in. We'll have a hangout at the proposed day and time:
> >
> > Thursday, February 21st, 4pm PST/San Francisco  -  Friday, February 22nd,
> > 8am CST/Beijing
> >
> > Please reach out to me via email if you want to be added to the invite.
> >
> > Cheers, Lars
> >
> > On Wed, Feb 6, 2019 at 1:33 PM Tim Armstrong 
> > wrote:
> >
> > > That would work for me too
> > >
> > > On Wed., 6 Feb. 2019, 13:11 Quanlong Huang, 
> > > wrote:
> > >
> > > > Thank you, Lars! I'm good for this time. Looking forward to it!
> > > >
> > > > Thanks,
> > > > Quanlong
> > > >
> > > > On Wed, Feb 6, 2019 at 6:32 PM Lars Volker  wrote:
> > > >
> > > > > Thanks for the feedback. I looked at Beijing and San Francisco
> > > timezones
> > > > > and 8am Beijing / 4pm San Francisco look like a good option. How
> > would
> > > > this
> > > > > date work for you (Link
> > > > > <
> > > > >
> > > >
> > >
> >
> https://www.timeanddate.com/worldclock/meetingdetails.html?year=2019=2=22=0=0=0=33=224
> > > > > >
> > > > > )?
> > > > >
> > > > > Thursday, February 21st, 4pm PST/San Francisco  -  Friday, February
> > > 22nd,
> > > > > 8am CST/Beijing
> > > > >
> > > > > If these work, I will send out a calendar invite with meeting
> details
> > > > next.
> > > > > Otherwise I'm also happy to move them around as needed.
> > > > >
> > > > > I'm looking forward to it! Cheers, Lars
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Feb 5, 2019 at 6:33 PM Pooja Nilangekar <
> > > > > pooja.nilange...@cloudera.com> wrote:
> > > > >
> > > > > > I would like to be a part of the meeting. I'm flexible about the
> > > > timing.
> > > > > >
> > > > > > Thanks,
> > > > > > Pooja
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Impala Video Meeting

[Lars Volker]

Hi All,

Thanks for chiming in. We'll have a hangout at the proposed day and time:

Thursday, February 21st, 4pm PST/San Francisco  -  Friday, February 22nd,
8am CST/Beijing

Please reach out to me via email if you want to be added to the invite.

Cheers, Lars

On Wed, Feb 6, 2019 at 1:33 PM Tim Armstrong 
wrote:

> That would work for me too
>
> On Wed., 6 Feb. 2019, 13:11 Quanlong Huang, 
> wrote:
>
> > Thank you, Lars! I'm good for this time. Looking forward to it!
> >
> > Thanks,
> > Quanlong
> >
> > On Wed, Feb 6, 2019 at 6:32 PM Lars Volker  wrote:
> >
> > > Thanks for the feedback. I looked at Beijing and San Francisco
> timezones
> > > and 8am Beijing / 4pm San Francisco look like a good option. How would
> > this
> > > date work for you (Link
> > > <
> > >
> >
> https://www.timeanddate.com/worldclock/meetingdetails.html?year=2019=2=22=0=0=0=33=224
> > > >
> > > )?
> > >
> > > Thursday, February 21st, 4pm PST/San Francisco  -  Friday, February
> 22nd,
> > > 8am CST/Beijing
> > >
> > > If these work, I will send out a calendar invite with meeting details
> > next.
> > > Otherwise I'm also happy to move them around as needed.
> > >
> > > I'm looking forward to it! Cheers, Lars
> > >
> > >
> > >
> > >
> > >
> > > On Tue, Feb 5, 2019 at 6:33 PM Pooja Nilangekar <
> > > pooja.nilange...@cloudera.com> wrote:
> > >
> > > > I would like to be a part of the meeting. I'm flexible about the
> > timing.
> > > >
> > > > Thanks,
> > > > Pooja
> > > >
> > >
> >
>

Re: Impala Video Meeting

[Lars Volker]

Thanks for the feedback. I looked at Beijing and San Francisco timezones
and 8am Beijing / 4pm San Francisco look like a good option. How would this
date work for you (Link

)?

Thursday, February 21st, 4pm PST/San Francisco  -  Friday, February 22nd,
8am CST/Beijing

If these work, I will send out a calendar invite with meeting details next.
Otherwise I'm also happy to move them around as needed.

I'm looking forward to it! Cheers, Lars





On Tue, Feb 5, 2019 at 6:33 PM Pooja Nilangekar <
pooja.nilange...@cloudera.com> wrote:

> I would like to be a part of the meeting. I'm flexible about the timing.
>
> Thanks,
> Pooja
>

Impala Video Meeting

[Lars Volker]

Hi All,

I would like to pick up Quanlong's and Jim's idea from a different thread
and organize a video call for the larger Impala development community. This
meeting will be open to all contributors, regardless of being a committer
or not. The goal of the first meeting will be that everyone gets to
introduce themselves and say "Hi!", and to have a discussion about nested
type support for ORC files. Additionally it will allow us to evaluate the
format of such a session itself and whether we want to do it again.

We have larger groups of contributors in PST and CET and I know of
committers in Beijing and Seoul. Looking at potential meeting times

it seems that there is no time that would be convenient for all 4 time
zones. Of course contributors in other timezones are also invited to join.

Please respond to this email if you're interested in joining such a
meeting, and include your timezone in your response. I'll then try to find
a suitable time for everyone who replies, or put up a poll.

Cheers, Lars

Re: [Patch Ready] Add support for reading ORC complex types

[Lars Volker]

I think it's an excellent idea to have a face-to-face video meeting with
the larger community on this mailing list. It might not be convenient to
have a regular schedule due to time zone constraints, but a one-off to say
"hi", introduce each other, and have a discussion would be helpful. I can
start a thread to see who's in which TZ and to find a good time.

On Mon, Jan 7, 2019 at 5:53 PM Quanlong Huang 
wrote:

> Sorry for my words that may make you uncomfortable... Yes, I mean in-person
> because it's more efficient. I thought you have many discussions and
> meetings about the design, roadmap or planning. But different time zones
> are really an obstacle. Having discussions face to face is not quite
> realistic.
>
> Maybe we can start another thread to discuss the interaction of the Impala
> community, e.g. regular meetings (via Slack or WeChat), roadmap and
> planning discussion, etc.
>
> Thanks for all your efforts to make the community grow!
>
> On Tue, Jan 8, 2019 at 6:07 AM Jim Apple  wrote:
>
> > "As a community developer, I can't discuss with you Cloudera folks in
> > real."
> >
> > Do you mean in-person? Because if you mean "real", we want to strive to
> > have dev@impala.apache.org be very real. In particular, I and others I
> > work
> > with at Cloudera, often in the same building, have discussions (that we
> > could have in person) here on dev@ instead in order to do community work
> > in
> > the open.
> >
> > Some other communities, like Parquet, will have regularly scheduled
> > face-to-face meetings over videoconferencing. Time zones are hard, of
> > course, but this is something we could try if someone wants to take
> charge
> > of organizing it, recruiting people to come, sending out notes, etc.
> >
> > In the meantime, let's all work to make dev@ and gerrit great places to
> > have productive discussions.
> >
> > Thanks for making the google doc, BTW: I think that's really helpful for
> > large and complex patches.
> >
> > On Mon, Jan 7, 2019 at 7:28 AM Quanlong Huang 
> > wrote:
> >
> > > Hi all,
> > >
> > > I'm excited that we can support reading ORC complex types (IMPALA-6503)
> > > now!
> > >
> > > I just finish the patch. As a community developer, I can't discuss with
> > you
> > > Cloudera folks in real. It may be hard to make codes clear via comments
> > for
> > > a big patch, so I wrote a doc about the implementation:
> > >
> > >
> >
> https://docs.google.com/document/d/10Gzcge57VUOQ0ZQWfWiIW9cUix8_FumMewl0Gja3ZA4
> > > (I may need to add more details, please let me know)
> > >
> > > Patch link: https://gerrit.cloudera.org/#/c/12168/
> > > Have passed pre-review-test:
> > > https://jenkins.impala.io/job/pre-review-test/269/
> > >
> > > Hopes you can take some time to review it.
> > >
> > > Thanks!
> > > Quanlong
> > >
> >
>

Re: jenkins.impala.io maintenance coming up

[Lars Volker]

I think this preceeded the other thread about the hung builds. In fact, the
build that we saw stuck had been started by Gabor, although I see no reason
to believe that it was something about that particular build that caused
the hang. After the restart we seem to be good but we're keeping an eye on
it. IMPALA-7938 tracks that issue and our work on it.

Apologies for the trouble this causes.

Cheers, Lars

On Sat, Dec 8, 2018, 12:41 Tim Armstrong  I was able to kick one off - let us know if you're still having issues.
>
> On Thu, Dec 6, 2018 at 11:53 PM Gabor Kaszab 
> wrote:
>
> > Hey Lars,
> >
> > Is everything up and running? I'm asking because I'm unable to launch a
> > GVO. Every time I hit the build/rebuild button it times out with 504
> > (Gateway Time-out). Note, I can log in to jenkins.impala.io and such
> just
> > can't run GVO.
> >
> > Cheers,
> > Gabor
> >
> >
> > On Thu, Dec 6, 2018 at 12:14 AM Lars Volker  wrote:
> >
> > > Following this morning's maintenance we have to restart Jenkins again.
> > > Apologies for the short notice. I'll send an all-clear when it's back.
> > >
> > > On Wed, Dec 5, 2018 at 12:04 PM Lars Volker  wrote:
> > >
> > > > Jenkins maintenance has been completed and the service is back up. I
> > had
> > > > to cancel one build and reached out to the owner directly. Please let
> > me
> > > > know if you experience any issues.
> > > >
> > > > Cheers, Lars
> > > >
> > > > On Wed, Dec 5, 2018 at 9:17 AM Lars Volker  wrote:
> > > >
> > > >> Hi All,
> > > >>
> > > >> Maintenance work will start in about an hour. Jenkins will stop
> > > accepting
> > > >> new builds momentarily and running builds might be aborted. I will
> > send
> > > >> another update once maintenance has been completed.
> > > >>
> > > >> Cheers, Lars
> > > >>
> > > >> On Tue, Dec 4, 2018 at 11:09 AM Lars Volker 
> wrote:
> > > >>
> > > >>> Hi All,
> > > >>>
> > > >>> We plan to perform some maintenance on jenkins.impala.io on
> > Wednesday
> > > >>> morning PST. In anticipation of this, I will start a backup of the
> > > current
> > > >>> state shortly. This should not affect the service availability.
> > Please
> > > let
> > > >>> me know if you experience any issues. I will send another
> > notification
> > > >>> tomorrow morning.
> > > >>>
> > > >>> Cheers, Lars
> > > >>>
> > > >>
> > >
> >
>

Re: Jenkins issues

[Lars Volker]

After a restart Jenkins is accepting jobs again. We'll keep an eye on it
for now.

On Thu, Dec 6, 2018 at 2:39 PM Lars Volker  wrote:

> Hi All,
>
> We are currently experiencing issues with Jenkins not accepting new
> pre-submit jobs. You can follow along at IMPALA-7938
> <https://issues.apache.org/jira/browse/IMPALA-7938>.
>
> Cheers, Lars
>

Re: jenkins.impala.io maintenance coming up

[Lars Volker]

Hi All,

Maintenance work will start in about an hour. Jenkins will stop accepting
new builds momentarily and running builds might be aborted. I will send
another update once maintenance has been completed.

Cheers, Lars

On Tue, Dec 4, 2018 at 11:09 AM Lars Volker  wrote:

> Hi All,
>
> We plan to perform some maintenance on jenkins.impala.io on Wednesday
> morning PST. In anticipation of this, I will start a backup of the current
> state shortly. This should not affect the service availability. Please let
> me know if you experience any issues. I will send another notification
> tomorrow morning.
>
> Cheers, Lars
>

jenkins.impala.io maintenance coming up

[Lars Volker]

Hi All,

We plan to perform some maintenance on jenkins.impala.io on Wednesday
morning PST. In anticipation of this, I will start a backup of the current
state shortly. This should not affect the service availability. Please let
me know if you experience any issues. I will send another notification
tomorrow morning.

Cheers, Lars

Gerrit Maintenance tonight at 8pm Pacific

[Lars Volker]

Hi All,

Cloudera runs the Gerrit instance at gerrit.cloudera.org, which we for our
code reviews. Tonight (10/17) at 8pm Pacific time (10/18 3am UTC), their IT
team will be upgrading Gerrit from version 2.14.6 to 2.14.15.

Details on 2.14.15 can be found here:
https://www.gerritcodereview.com/2.14.html

Please reply to this email if you encounter any issues.

Cheers, Lars

Re: Restart jenkins.impala.io

[Lars Volker]

Thank you for taking care of this, Pooja!

On Wed, Oct 10, 2018 at 12:24 PM Pooja Nilangekar <
pooja.nilange...@cloudera.com> wrote:

> Done.
>
> Thanks,
> Pooja
>
>
> On Wed, Oct 10, 2018 at 9:56 AM Pooja Nilangekar <
> pooja.nilange...@cloudera.com> wrote:
>
> > Hi,
> >
> > I'm going to restart Jenkins to upgrade it with the latest security
> fixes.
> > I'll let the current jobs complete. New jobs will not start until the
> > upgrade is completed.
> >
> > Thanks,
> > Pooja
> >
>

Re: The search capability on the Impala documentation site

[Lars Volker]

We could add a search box that links to an external search engine, e.g.
like so:
https://duckduckgo.com/?q=site%3Aimpala.apache.org+inurl%3Adocs+random+replica=web

Would this help? I don't know much about our dita doc generator, there
might be much easier ways that I'm not aware of.

Cheers, Lars

On Tue, Oct 2, 2018 at 3:55 PM Alexandra Rodoni 
wrote:

> Occasionally, I get a question about the missing search functionality in
> ASF docs, including Impala.
>
> Does anyone have any comment if it is worth our time to add the site search
> functionality? And if so, any recommendation on tools that we can use to
> quickly add the search to the site?
>
> Or is it sufficient to rely on external search engines to guide users to
> the doc site?
>
> Thank you.
> alex
>

Re: JSON support: Hive compatibility or ANSI SQL standard

[Lars Volker]

Thanks for the comprehensive summary, Quanlong.

I'm in favor of (c) but I don't feel strongly about the order in which we
should prioritize them. Being compatible to Hive and following the SQL
standard will help users to switch more easily between systems. I'm not
worried about the confusing users too much. Even if the functions are have
similar names, people are not going to confuse them by accident, and we can
add a warning to the docs to point out that get_json_object should not be
confused with the ANSI SQL function json_object.

Cheers, Lars

On Mon, Sep 10, 2018 at 7:57 AM Quanlong Huang 
wrote:

> Hi all,
>
> Recently, I'm working on a patch to add a built-in function for parsing
> JSON strings and extracting values in them:
> https://gerrit.cloudera.org/#/c/10950. We've not reached an agreement on
> the name of this function. So I reach here for a broader discussion.
>
> *-- Background --*
>
> Hive has a built-in function (get_json_object) for this purpose. But it's
> a Java UDF. Impala can't track its memory usage. The current patch adds
> built-in support for this function.
>
> Greg suggested that we name this function as json_value(). It's a function
> in ANSI SQL standard. Note that json_value() only returns scalar types so
> json_query() in the standard may be more fit.
>
> The discussion is about which of the following options we should choose
> for JSON support:
>
> (a) Support get_json_object for compatibility to Hive;
> (b) Support JSON functions in ANSI SQL standard;
> (c) Support both.
>
> I used to be in favor of (c). But Zoltan points out there's a
> json_object() function in ANSI standard with a quite different meaning
> which will confuse users with the Hive get_json_object UDF. Maybe we can
> only choose (a) or (b).
>
> *-- SQL supports in other systems --*
>
> Oracle supports JSON functions in ANSI standard, while MySQL does not.
> There's a JSON_EXTRACT function in MySQL contains all the ability of Hive's
> get_json_object function.
>
> Presto and Greenplum do not support ANSI JSON standards too:
> https://prestodb.io/docs/current/functions/json.html,
> https://gpdb.docs.pivotal.io/5100/admin_guide/query/topics/json-data.html
>
> Here're the previous discussions:
> https://gerrit.cloudera.org/#/c/10950/13/common/function-registry/impala_functions.py@514
>
> What do you think?
>
> Thanks,
> Quanlong
>

Re: [VOTE] Removing emeritus and activity requirements

[Lars Volker]

+1 (binding)

On Sun, Sep 9, 2018, 18:36 Brock Noland  wrote:

> +1
>
> On Sun, Sep 9, 2018 at 8:15 PM, Jim Apple  wrote:
>
> > Please vote on the following diff to the bylaws:
> >
> > https://gerrit.cloudera.org/#/c/11407/
> >
> > The vote will conclude Thursday morning, California time. Only PMC
> > votes are binding, but everyone is welcomed and encouraged to vote.
> > You can vote here or on the diff itself on gerrit. This vote will pass
> > with 3 binding +1 votes and more binding +1 votes than -1 votes.
> >
> > For posterity, this diff is on our reviews@ mailing list, as are all
> > code reviews and gerrit traffic:
> >
> > https://lists.apache.org/api/source.lua/fe0a4c170824112c0efd69265a8951
> > afe162226e0c998d669a4b7216@%3Creviews.impala.apache.org%3E
> >
>

Re: Wrong Jira id in commit (IMPALA-7147 vs IMPALA-7417)

[Lars Volker]

+1, thanks for fixing it.

On Mon, Sep 3, 2018, 12:16 Brock Noland  wrote:

> I would do this, it's quite courteous for future devs.
>
> "Our current idea is to change IMPALA-7417
> to be a duplicate of IMPALA-7147, and create a new Jira with IMPALA-7417's
> original contents."
>
> On Mon, Sep 3, 2018 at 12:12 PM, Csaba Ringhofer  >
> wrote:
>
> > Hi folks!
> >
> >  We have just discovered (thanks Laszlo), that one of my changes was
> pushed
> > with wrong Jira id in the commit message (it fixed IMPALA-7147, but I
> wrote
> > it as IMPALA-7417, which didn't exist at that time).
> >
> >  Luckily there is no commit pushed for IMPALA-7417 yet (it is on review,
> > and happens to be my commit too). Our current idea is to change
> IMPALA-7417
> > to be a duplicate of IMPALA-7147, and create a new Jira with
> IMPALA-7417's
> > original contents.
> >
> >  Does someone have an idea about the best action in this case? My concern
> > is that the wrong Jira id can trick some tools and steal some time from
> > people.
> >
>

Re: help on permissions on Impala jiras

[Lars Volker]

Hi Yongjun,

I added you to the Contributor role in the Jira project. You should be able
to assign issues to yourself now.

Cheers, Las

On Fri, Aug 24, 2018 at 4:14 PM Yongjun Zhang  wrote:

> Hi,
>
> May I get help on getting permissions on Impala jiras?
>
> My apache jira user id is yzhangal
>
> Thanks.
>
> --Yongjun
>

Re: Breaking change for query hints?

[Lars Volker]

Great, glad you found a fix. Looking through the code I noticed that we
will interpret any unknown select hint as 'straigh_join'. IMPALA-7484
<https://issues.apache.org/jira/browse/IMPALA-7484> has the details, you
might want to keep an eye on this when using this annotation technique with
joins.

Cheers, Lars

On Thu, Aug 23, 2018 at 9:46 PM Quanlong Huang 
wrote:

> Thank you, Todd and Lars! We plan to hotfix this by quotes the vars in
> PlanHint:
>
> *diff --git a/fe/src/main/java/org/apache/impala/analysis/PlanHint.java
> b/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
>
> *index d16919f09..4da0e91c2 100644*
>
> *--- a/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
>
> *+++ b/fe/src/main/java/org/apache/impala/analysis/PlanHint.java*
>
> @@ -61,8 +61,10 @@ public class PlanHint {
>
>@Override
>
>public String toString() {
>
>  StringBuilder sb = new StringBuilder();
>
> -sb.append(name_);
>
> -if (!args_.isEmpty()) {
>
> +if (args_.isEmpty()) {
>
> +  sb.append('`').append(name_).append('`');
>
> +} else {
>
> +  sb.append(name_);
>
>sb.append("(");
>
>sb.append(Joiner.on(",").join(args_));
>
>sb.append(")");
>
>
>
> Quanlong Huang  于2018年8月24日周五 下午12:09写道：
>
>> Unfortunately, the queries come from a system that we don't have control.
>> I think we should change the behavior of the CreateView statement to save
>> everything in the comments.
>>
>> Lars Volker  于2018年8月24日周五 上午11:37写道：
>>
>>> We changed query hints from being interpreted as strings to being parsed
>>> properly in commit ce9b332e
>>> <https://github.com/apache/impala/commit/ce9b332ee9e640d79c8ae35e7abb8c7d787ddf78#diff-27ff375b57f7fc982be98c4f503cb04dR2385>
>>>  for IMPALA-4163 <https://issues.apache.org/jira/browse/IMPALA-4163>.
>>> Before that change we called String.trim() on the hint which I suspect
>>> preserved the ``. After this change we parse it as an IDENT, which likely
>>> strips the ``.
>>>
>>> I feel that hints are not the right solution to embed a query id into
>>> the view definition, but I cannot think of a better one, either. With the
>>> current syntax you can write the hint as queryId_abc or queryId(abc) and
>>> while you'll get a bunch of warning it seems to work on my end.
>>>
>>> I also think there's a (unrelated) bug in SelectList.java:87
>>> <https://github.com/apache/impala/blob/64e6719870db5602a6fa85014bc6c264080b9414/fe/src/main/java/org/apache/impala/analysis/SelectList.java#L87>:
>>> It should have a continue statement inside the check, or wrap the call to
>>> setIsStraightJoin() in an else clause.
>>>
>>> On Thu, Aug 23, 2018 at 8:19 PM Quanlong Huang 
>>> wrote:
>>>
>>>> Yeah, the results of CreateView statement is different. We created the
>>>> same view by impala-2.5. The difference is that impala-2.12 delete the
>>>> quotes in the comment.
>>>>
>>>> Found the difference in beeline:
>>>>
>>>> > show create table view_2_12;
>>>>
>>>> createtab_stmt
>>>>
>>>> CREATE VIEW `view_2_12` AS SELECT
>>>>
>>>> -- +queryId=abc
>>>>
>>>>  * FROM test.video_parq
>>>>
>>>> 3 rows selected (4.518 seconds)
>>>>
>>>>
>>>> > show create table view_2_5;
>>>>
>>>> createtab_stmt
>>>>
>>>> CREATE VIEW `view_2_5` AS SELECT
>>>>
>>>> -- +`queryId=abc`
>>>>
>>>>  * FROM test.video_parq
>>>>
>>>> 3 rows selected (1.202 seconds)
>>>>
>>>> view_2_12 is created by impala-2.12, while view_2_5 is created by
>>>> impala-2.5.
>>>>
>>>> Todd Lipcon  于2018年8月24日周五 上午10:53写道：
>>>>
>>>>> I'm curious: can you describe the view using Hive to see what the
>>>>> stored query consists of?
>>>>>
>>>>> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang 
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> After we upgrade Impala from 2.5 to 2.12, we found some queries on
>>>>>> views
>>>>>> failed. The views contain a query hint which I think is the cause. I
>>>>>> can't
>>>>>> find anything about such kind of incompatibility was men

Re: Breaking change for query hints?

[Lars Volker]

We changed query hints from being interpreted as strings to being parsed
properly in commit ce9b332e

 for IMPALA-4163 .
Before that change we called String.trim() on the hint which I suspect
preserved the ``. After this change we parse it as an IDENT, which likely
strips the ``.

I feel that hints are not the right solution to embed a query id into the
view definition, but I cannot think of a better one, either. With the
current syntax you can write the hint as queryId_abc or queryId(abc) and
while you'll get a bunch of warning it seems to work on my end.

I also think there's a (unrelated) bug in SelectList.java:87
:
It should have a continue statement inside the check, or wrap the call to
setIsStraightJoin() in an else clause.

On Thu, Aug 23, 2018 at 8:19 PM Quanlong Huang 
wrote:

> Yeah, the results of CreateView statement is different. We created the
> same view by impala-2.5. The difference is that impala-2.12 delete the
> quotes in the comment.
>
> Found the difference in beeline:
>
> > show create table view_2_12;
>
> createtab_stmt
>
> CREATE VIEW `view_2_12` AS SELECT
>
> -- +queryId=abc
>
>  * FROM test.video_parq
>
> 3 rows selected (4.518 seconds)
>
>
> > show create table view_2_5;
>
> createtab_stmt
>
> CREATE VIEW `view_2_5` AS SELECT
>
> -- +`queryId=abc`
>
>  * FROM test.video_parq
>
> 3 rows selected (1.202 seconds)
>
> view_2_12 is created by impala-2.12, while view_2_5 is created by
> impala-2.5.
>
> Todd Lipcon  于2018年8月24日周五 上午10:53写道：
>
>> I'm curious: can you describe the view using Hive to see what the stored
>> query consists of?
>>
>> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang 
>> wrote:
>>
>>> Hi all,
>>>
>>> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
>>> failed. The views contain a query hint which I think is the cause. I
>>> can't
>>> find anything about such kind of incompatibility was mentioned in
>>>
>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>>> or
>>>
>>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>>
>>> The error can be reproduced by
>>>
>>> [impala-shell:21000] > use test;
>>> [impala-shell:21000] > create view view_2_12 as select /*
>>> +`queryId=abc` */ * from video_parq;
>>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>>> video_parq
>>> ++
>>> | summary|
>>> ++
>>> | View has been created. |
>>> ++
>>> WARNINGS: PLAN hint not recognized: queryId=abc
>>>
>>> Fetched 1 row(s) in 24.00s
>>> [impala-shell:21000] > desc formatted view_2_12;
>>> Query: describe formatted view_2_12
>>> ERROR: AnalysisException: Failed to parse view-definition statement of
>>> view: test.view_2_12
>>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>>> statement of view: test.view_2_12
>>>
>>>
>>> Since this's an emergency for us, I post an email here for help. I'm also
>>> looking into the history of the cup file and looking for a hotfix for
>>> this.
>>> Thanks if anyone can give some pointers!
>>>
>>> Thanks,
>>> Quanlong
>>>
>>

Re: Breaking change for query hints?

[Lars Volker]

The warning seems to come from SelectList.java:87

which
also was in 2.5 (SelectList.java:85
.
Do you see corresponding errors in the logfiles for the AnalysisException?

On Thu, Aug 23, 2018 at 7:53 PM Todd Lipcon  wrote:

> I'm curious: can you describe the view using Hive to see what the stored
> query consists of?
>
> On Thu, Aug 23, 2018, 7:44 PM Quanlong Huang 
> wrote:
>
>> Hi all,
>>
>> After we upgrade Impala from 2.5 to 2.12, we found some queries on views
>> failed. The views contain a query hint which I think is the cause. I can't
>> find anything about such kind of incompatibility was mentioned in
>>
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_new_features.html
>> or
>>
>> https://www.cloudera.com/documentation/enterprise/release-notes/topics/impala_incompatible_changes.html
>>
>> The error can be reproduced by
>>
>> [impala-shell:21000] > use test;
>> [impala-shell:21000] > create view view_2_12 as select /*
>> +`queryId=abc` */ * from video_parq;
>> Query: create view view_2_12 as select /* +`queryId=abc` */ * from
>> video_parq
>> ++
>> | summary|
>> ++
>> | View has been created. |
>> ++
>> WARNINGS: PLAN hint not recognized: queryId=abc
>>
>> Fetched 1 row(s) in 24.00s
>> [impala-shell:21000] > desc formatted view_2_12;
>> Query: describe formatted view_2_12
>> ERROR: AnalysisException: Failed to parse view-definition statement of
>> view: test.view_2_12
>> CAUSED BY: TableLoadingException: Failed to parse view-definition
>> statement of view: test.view_2_12
>>
>>
>> Since this's an emergency for us, I post an email here for help. I'm also
>> looking into the history of the cup file and looking for a hotfix for
>> this.
>> Thanks if anyone can give some pointers!
>>
>> Thanks,
>> Quanlong
>>
>

Re: Improving Kudu Build Support

[Lars Volker]

I'm in favor of not spending developer time and effort to maintain
compatibility with 14.04. Personally I'm still developing on Ubuntu 14.04
so I'd be happy if we can support it without much pain. On the other hand
it EOLs in April 2019, so I might as well go to 18.04 now, should we decide
to drop support. Maybe not many other folks are on 14.04 after all?



On Mon, Aug 20, 2018 at 10:06 AM Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

> Impala community,
>
> For years now, Impala has utilized tarballs built by Cloudera and uploaded
> to S3 for running most of the Hadoop components in the testing minicluster.
> The one exception to this is Kudu, which is instead provided by the
> toolchain.
>
> This was never ideal - native-toolchain makes more sense for libraries
> where we want to build against a fairly static version, but Kudu is under
> active development and we'd like to always build against a relatively
> up-to-date version. As a result, patches just bumping the version of Kudu
> make up a significant portion of the commit history of native-toolchain.
>
> Thanks to work I'm currently doing at Cloudera, there will soon be snapshot
> tarballs of Kudu getting uploaded to S3 along with the other Hadoop
> components. I would like to propose that Impala switch to using those
> instead of the toolchain Kudu.
>
> One problem here is that the new Kudu tarballs will not be getting build
> for Ubuntu 14.04, only 16.04, but we still officially say we support
> development on 14.04.
>
> One option here would be to maintain the toolchain Kudu for now and hide
> downloading of the new tarballs behind a flag. We could also postpone some
> of this work until 14.04 is less common. Or, given that the
> bootstrap_development script already only supports 16.04, we might want to
> just drop support for building on 14.04.
>
> Thoughts?
>

Re: Impala OS and Java version support

[Lars Volker]

Several developers seem to use Ubuntu for their development which makes me
think that Impala would work on Debian, too. However, I think we should not
claim proper support for Debian since to my knowledge there are no
automated Jenkins jobs verifying this. I am also not aware of anyone
developing or using Impala on Debian.

Of course anyone in the community is free to set-up automated testing for
Debian, either on our project Jenkins or on their own infrastructure.



On Fri, Jul 6, 2018 at 12:04 PM Jim Apple 
wrote:

> Should we throw a Debian version in for good measure?
>
> https://github.com/apache/impala/search?q=debian
>
> On Fri, Jul 6, 2018 at 9:19 AM, Lars Volker 
> wrote:
>
> > Hi All,
> >
> > With the automatic commit flow to the 2.x branch being disabled, I'd like
> > to suggest that we deprecate support for some older operating systems on
> > the 3.x branch. This would allow us to get rid of almost all the checks
> in
> > common/config.h.in as well as some workarounds in the code. It would
> also
> > simplify the upcoming rebase of the KRPC code (IMPALA-7006).
> >
> > I'd like to propose that we pick Ubuntu 14.04, CentOS 6.8, and SLES 12 as
> > the minimum operating system versions on which Impala has to compile and
> > pass all tests. I'd also like to deprecate support for Java 7 and require
> > Java 8 for all changes going forward. To my knowledge these are the
> minimum
> > versions that people use for development and deployment of the 3.x
> branch.
> >
> > I'm curious what others think.
> >
> > Cheers, Lars
> >
>

IMPORTANT: you will need to re-download orc

[Lars Volker]

tldr; you have to run "rm -rf $IMPALA_TOOLCHAIN/orc-*"

This change has just been submitted: https://gerrit.cloudera.org/#/c/10755/

It updates protobuf to version 3 and adds libunwind in preparation for
rebasing our KRPC code and to enable porting of more RPCs to KRPC. The new
toolchain build has orc built against protobuf version 3 and you will need
to re-download orc. Otherwise you will see linker errors like this one:

/opt/Impala-Toolchain/orc-1.4.3-p2/lib/liborc.a(orc-proto-wrapper.cc.o):orc-proto-wrapper.cc:function
orc::proto::PostScript::ByteSizeLong() const: error: undefined reference to
'google::protobuf::internal::WireFormatLite::UInt32Size(google::protobuf::RepeatedField const&)'

To re-download orc you can simply remove it by running "rm -rf
$IMPALA_TOOLCHAIN/orc-*" and rebuild. It will get downloaded automatically.

Apologies for the inconvenience and the required manual step. The
alternative would have been to create an empty patch for orc in our
toolchain, but we've discarded it as too hacky in the past.

Let me know if you have any questions.

Cheers, Lars

Re: Impala OS and Java version support

[Lars Volker]

I've seen people use the 2.x branch on CentOS 6.4 a lot, which was released
in April 2013. CentOS 6.8 is from May 2016, so already over two years old.
Since CentOS supports upgrading between minor releases it seems reasonable
to me to require CentOS 6.8. CentOS does not have maintenance releases, so
by default upgrades always happen to the latest minor version.

Philip, do you feel strongly about supporting older releases? Do you have a
particular one in mind?

On Fri, Jul 6, 2018 at 10:47 AM Tim Armstrong
 wrote:

> Sounds like a good idea to me. Java 8 makes sense to me too since the major
> Java implementations (OpenJDK and Oracle) no longer support the older JDKs.
>
> On Fri, Jul 6, 2018 at 9:58 AM, Sailesh Mukil  >
> wrote:
>
> > I'm in favor of the minimum OS version support proposal. From 3.x, I
> don't
> > see any reason to support much older OSes especially since it's becoming
> > much more burdensome to special case code for them and have those paths
> > tested.
> >
> > I'll defer to others on the minimum Java version proposal since they may
> > have better insight.
> >
> > On Fri, Jul 6, 2018 at 9:19 AM, Lars Volker 
> > wrote:
> >
> > > Hi All,
> > >
> > > With the automatic commit flow to the 2.x branch being disabled, I'd
> like
> > > to suggest that we deprecate support for some older operating systems
> on
> > > the 3.x branch. This would allow us to get rid of almost all the checks
> > in
> > > common/config.h.in as well as some workarounds in the code. It would
> > also
> > > simplify the upcoming rebase of the KRPC code (IMPALA-7006).
> > >
> > > I'd like to propose that we pick Ubuntu 14.04, CentOS 6.8, and SLES 12
> as
> > > the minimum operating system versions on which Impala has to compile
> and
> > > pass all tests. I'd also like to deprecate support for Java 7 and
> require
> > > Java 8 for all changes going forward. To my knowledge these are the
> > minimum
> > > versions that people use for development and deployment of the 3.x
> > branch.
> > >
> > > I'm curious what others think.
> > >
> > > Cheers, Lars
> > >
> >
>

Impala OS and Java version support

[Lars Volker]

Hi All,

With the automatic commit flow to the 2.x branch being disabled, I'd like
to suggest that we deprecate support for some older operating systems on
the 3.x branch. This would allow us to get rid of almost all the checks in
common/config.h.in as well as some workarounds in the code. It would also
simplify the upcoming rebase of the KRPC code (IMPALA-7006).

I'd like to propose that we pick Ubuntu 14.04, CentOS 6.8, and SLES 12 as
the minimum operating system versions on which Impala has to compile and
pass all tests. I'd also like to deprecate support for Java 7 and require
Java 8 for all changes going forward. To my knowledge these are the minimum
versions that people use for development and deployment of the 3.x branch.

I'm curious what others think.

Cheers, Lars

Re: Future of unsupported formats?

[Lars Volker]

>
> Even if people were using it, would that affect our decision if there's
> noone to maintain it? I don't think we were ambiguous about whether writing
> those formats was supported or not.
>

I agree, if we cannot maintain it, we should remove it. I was thinking of
the other option you outlined in the original message:

It seems like we should get it out of this in-between state - either delete
> the code or get it to the point where it's supported and tested. If we
> delete it, it's always possible for someone to resurrect it later.


Cheers, Lars


> On Tue, Jun 19, 2018 at 10:04 AM, Tim Armstrong 
> wrote:
>
> > Hi Edward,
> >   I was talking about write support, specifically. Reading those formats
> > is supported without any configuration changes.
> >
> > - Tim
> >
> > On Tue, Jun 19, 2018 at 9:32 AM, Edward Capriolo 
> > wrote:
> >
> >> I was going to say. Not a current user ATM, but there are defiantly
> people
> >> with text+gzip SequenceFile(Text). It is nice to be able to work with
> >> those, I was also at a shop that went hard for AVRO + Impala but since
> >> switched off.
> >>
> >> I also do not understand what is meant by "behind a query option" since
> >> the
> >> version of Impala I had (CDH 5+6) would process all the above formats.
> >>
> >> Edward
> >>
> >> On Tue, Jun 19, 2018 at 11:59 AM, Lars Volker 
> >> wrote:
> >>
> >> > I'm in favor of removing unsupported code, especially when doing so
> >> makes
> >> > development of the rest of the codebase easier and saves us cycles for
> >> > maintaining it. On the other hand it would suck if users had come to
> >> rely
> >> > on it and we break it, even though we recommend against it.
> >> >
> >> > We could make a reasonable effort to discover any users of the
> feature,
> >> > e.g. by asking on user@ and by folks on this list checking other
> >> > communication channels they might have access to.
> >> >
> >> > Cheers, Lars
> >> >
> >> > On Tue, Jun 19, 2018 at 8:26 AM Tim Armstrong
> >> >  wrote:
> >> >
> >> > > I don't think we need to bump a major version to remove something
> >> that we
> >> > > never claimed to support though. The docs are pretty clear:
> >> > >
> >> > >
> >> > > https://impala.apache.org/docs/build/html/topics/impala_
> >> > allow_unsupported_formats.html
> >> > >
> >> > > "An obsolete query option from early work on support for file
> >> formats. Do
> >> > > not use. Might be removed in the future."
> >> > >
> >> > > On Mon, Jun 18, 2018 at 10:16 PM, Jim Apple
> >>  >> > >
> >> > > wrote:
> >> > >
> >> > > > As for the time zone case, I’d like to be careful about
> versioning.
> >> If
> >> > we
> >> > > > remove Avro, that seems like a breaking changedeserving of a major
> >> > > version
> >> > > > bump.
> >> > > >
> >> > > > It might be worth taking a survey wider than dev@. User@ or the
> >> > > customers
> >> > > > of Impala packagers might be good places to start.
> >> > > >
> >> > > > On Mon, Jun 18, 2018 at 5:10 PM Tim Armstrong
> >> > > >  wrote:
> >> > > >
> >> > > > > For a few years now we've had write support for Sequence, Avro
> and
> >> > > > > compressed text hidden behind a query option. We haven't really
> >> made
> >> > > any
> >> > > > > progress on turning it into a supported feature, so I'm
> wondering
> >> if
> >> > we
> >> > > > > should remove the code and save some overhead of building,
> testing
> >> > and
> >> > > > code
> >> > > > > maintenance.
> >> > > > >
> >> > > > > I know I've found it useful once or twice to generate test data
> >> but I
> >> > > > don't
> >> > > > > think this is enough to justify maintaining it.
> >> > > > >
> >> > > > > It seems like we should get it out of this in-between state -
> >> either
> >> > > > delete
> >> > > > > the code or get it to the point where it's supported and tested.
> >> If
> >> > we
> >> > > > > delete it, it's always possible for someone to resurrect it
> later.
> >> > > > >
> >> > > > > What do people think?
> >> > > > >
> >> > > > > - Tim
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Future of unsupported formats?

[Lars Volker]

Tim's original proposal was to remove *write* support for these formats. We
fully support reading these formats and would continue to do so.

On Tue, Jun 19, 2018 at 9:32 AM Edward Capriolo 
wrote:

> I was going to say. Not a current user ATM, but there are defiantly people
> with text+gzip SequenceFile(Text). It is nice to be able to work with
> those, I was also at a shop that went hard for AVRO + Impala but since
> switched off.
>
> I also do not understand what is meant by "behind a query option" since the
> version of Impala I had (CDH 5+6) would process all the above formats.
>
> Edward
>
> On Tue, Jun 19, 2018 at 11:59 AM, Lars Volker 
> wrote:
>
> > I'm in favor of removing unsupported code, especially when doing so makes
> > development of the rest of the codebase easier and saves us cycles for
> > maintaining it. On the other hand it would suck if users had come to rely
> > on it and we break it, even though we recommend against it.
> >
> > We could make a reasonable effort to discover any users of the feature,
> > e.g. by asking on user@ and by folks on this list checking other
> > communication channels they might have access to.
> >
> > Cheers, Lars
> >
> > On Tue, Jun 19, 2018 at 8:26 AM Tim Armstrong
> >  wrote:
> >
> > > I don't think we need to bump a major version to remove something that
> we
> > > never claimed to support though. The docs are pretty clear:
> > >
> > >
> > > https://impala.apache.org/docs/build/html/topics/impala_
> > allow_unsupported_formats.html
> > >
> > > "An obsolete query option from early work on support for file formats.
> Do
> > > not use. Might be removed in the future."
> > >
> > > On Mon, Jun 18, 2018 at 10:16 PM, Jim Apple
>  > >
> > > wrote:
> > >
> > > > As for the time zone case, I’d like to be careful about versioning.
> If
> > we
> > > > remove Avro, that seems like a breaking changedeserving of a major
> > > version
> > > > bump.
> > > >
> > > > It might be worth taking a survey wider than dev@. User@ or the
> > > customers
> > > > of Impala packagers might be good places to start.
> > > >
> > > > On Mon, Jun 18, 2018 at 5:10 PM Tim Armstrong
> > > >  wrote:
> > > >
> > > > > For a few years now we've had write support for Sequence, Avro and
> > > > > compressed text hidden behind a query option. We haven't really
> made
> > > any
> > > > > progress on turning it into a supported feature, so I'm wondering
> if
> > we
> > > > > should remove the code and save some overhead of building, testing
> > and
> > > > code
> > > > > maintenance.
> > > > >
> > > > > I know I've found it useful once or twice to generate test data
> but I
> > > > don't
> > > > > think this is enough to justify maintaining it.
> > > > >
> > > > > It seems like we should get it out of this in-between state -
> either
> > > > delete
> > > > > the code or get it to the point where it's supported and tested. If
> > we
> > > > > delete it, it's always possible for someone to resurrect it later.
> > > > >
> > > > > What do people think?
> > > > >
> > > > > - Tim
> > > > >
> > > >
> > >
> >
>

Re: Future of unsupported formats?

[Lars Volker]

I'm in favor of removing unsupported code, especially when doing so makes
development of the rest of the codebase easier and saves us cycles for
maintaining it. On the other hand it would suck if users had come to rely
on it and we break it, even though we recommend against it.

We could make a reasonable effort to discover any users of the feature,
e.g. by asking on user@ and by folks on this list checking other
communication channels they might have access to.

Cheers, Lars

On Tue, Jun 19, 2018 at 8:26 AM Tim Armstrong
 wrote:

> I don't think we need to bump a major version to remove something that we
> never claimed to support though. The docs are pretty clear:
>
>
> https://impala.apache.org/docs/build/html/topics/impala_allow_unsupported_formats.html
>
> "An obsolete query option from early work on support for file formats. Do
> not use. Might be removed in the future."
>
> On Mon, Jun 18, 2018 at 10:16 PM, Jim Apple 
> wrote:
>
> > As for the time zone case, I’d like to be careful about versioning. If we
> > remove Avro, that seems like a breaking changedeserving of a major
> version
> > bump.
> >
> > It might be worth taking a survey wider than dev@. User@ or the
> customers
> > of Impala packagers might be good places to start.
> >
> > On Mon, Jun 18, 2018 at 5:10 PM Tim Armstrong
> >  wrote:
> >
> > > For a few years now we've had write support for Sequence, Avro and
> > > compressed text hidden behind a query option. We haven't really made
> any
> > > progress on turning it into a supported feature, so I'm wondering if we
> > > should remove the code and save some overhead of building, testing and
> > code
> > > maintenance.
> > >
> > > I know I've found it useful once or twice to generate test data but I
> > don't
> > > think this is enough to justify maintaining it.
> > >
> > > It seems like we should get it out of this in-between state - either
> > delete
> > > the code or get it to the point where it's supported and tested. If we
> > > delete it, it's always possible for someone to resurrect it later.
> > >
> > > What do people think?
> > >
> > > - Tim
> > >
> >
>

Maximum number of concurrent jobs for gerrit-verify-dryrun-external

[Lars Volker]

Hi All,

It seems that we intend to have two parallel jobs at most:

   - Maximum Total Concurrent Builds: 2
   - Maximum Concurrent Builds Per Node: 1

However, all jobs run on master and then spawn subsequent jobs on worker
nodes. This limits the maximum number of concurrent jobs to 1.

If no-one objects, I will change this number to two concurrent jobs.

Cheers, Lars

Re: Automatically rebase changes before GVO?

[Lars Volker]

It seems that Jenkins now posted the "Build started" message twice. I
removed one of them from the config. Tim, can you verify that I removed the
right one?

https://jenkins.impala.io/job/gerrit-verify-dryrun/jobConfigHistory/showDiffFiles?timestamp1=2018-06-11_22-09-04=2018-06-12_02-47-10

On Mon, Jun 11, 2018 at 3:09 PM, Tim Armstrong 
wrote:

> Ok, I applied the changes. Let me know if you run into any issues.
>
> On Mon, Jun 11, 2018 at 3:05 PM, Sailesh Mukil 
> wrote:
>
> > +1
> >
> > On Mon, Jun 11, 2018 at 3:02 PM, Jim Apple  wrote:
> >
> > > No objection from me.
> > >
> > > On Mon, Jun 11, 2018 at 12:06 PM, Tim Armstrong <
> tarmstr...@cloudera.com
> > >
> > > wrote:
> > >
> > > > > On nit: as GVD gets more complex, it becomes harder for new people
> to
> > > > understand the messages and +Ns applied to their patches. That
> doesn't
> > > mean
> > > > we shouldn't do this, only that it's something to keep an eye on.
> > > >
> > > > I think this helps with that problem in net by removing the manual
> > rebase
> > > > step that people have to remember to do.
> > > >
> > > >
> > > > On Mon, Jun 11, 2018 at 12:04 PM, Tim Armstrong <
> > tarmstr...@cloudera.com
> > > >
> > > > wrote:
> > > >
> > > > > I've tried my job a few times and it's working as expected. Any
> > > > objections
> > > > > to me switching over gerrit-verify-dryrun to my approach?
> > > > >
> > > > > On Thu, Jun 7, 2018 at 2:42 PM, Tim Armstrong <
> > tarmstr...@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > >> Ok, I was able to put together a test job that does the automatic
> > > rebase
> > > > >> and carries a +2 here: https://jenkins.impala.io/job/
> > > > >> gerrit-verify-dryrun-tarmstrong/
> > > > >>
> > > > >> The diff in job config required to get it to work is here:
> > > > >> https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstron
> > > > >> g/jobConfigHistory/showDiffFiles?timestamp1=2018-06-07_20-
> > > > >> 41-18=2018-06-07_21-38-58
> > > > >>
> > > > >> I tested by creating some private drafts, adding "Impala Public
> > > Jenkins"
> > > > >> as a reviewer and testing the various scenarios.
> > > > >>
> > > > >> On Thu, Jun 7, 2018 at 2:26 PM, Jim Apple 
> > > wrote:
> > > > >>
> > > > >>> I agree with both of you.
> > > > >>>
> > > > >>> On nit: as GVD gets more complex, it becomes harder for new
> people
> > to
> > > > >>> understand the messages and +Ns applied to their patches. That
> > > doesn't
> > > > >>> mean
> > > > >>> we shouldn't do this, only that it's something to keep an eye on.
> > > > >>>
> > > > >>> On Thu, Jun 7, 2018 at 1:28 PM, Philip Zeyliger <
> > phi...@cloudera.com
> > > >
> > > > >>> wrote:
> > > > >>>
> > > > >>> > Seems fine, especially since we do the rebase as our submission
> > > > >>> strategy
> > > > >>> > anyway, so we're already accepting/testing something that's
> > likely
> > > to
> > > > >>> get
> > > > >>> > rebased, and we may as well minimize that window.
> > > > >>> >
> > > > >>> > I'd be in favor of the bot also carrying the votes.
> > > > >>> >
> > > > >>> >
> > > > >>> >
> > > > >>> > On Thu, Jun 7, 2018 at 1:24 PM, Tim Armstrong <
> > > > tarmstr...@cloudera.com
> > > > >>> >
> > > > >>> > wrote:
> > > > >>> >
> > > > >>> > > One annoyance with our precommit job is the requirement to
> > > manually
> > > > >>> > rebase
> > > > >>> > > the change before starting the merge. Failure to do so either
> > > leads
> > > > >>> to
> > > > >>> > > false positives or false negatives - builds that failed
> because
> > > > they
> > > > >>> were
> > > > >>> > > missing a flaky/broken test fix and builds that succeeded
> > despite
> > > > >>> > > interacting badly with a previous fix.
> > > > >>> > >
> > > > >>> > > What do people think about modifying gerrit-verify-dryrun to
> > > > >>> > automatically
> > > > >>> > > rebase the patch (by the programmatic equivalent of hitting
> the
> > > > >>> "Rebase"
> > > > >>> > > button) at the start of the job? The patch author would still
> > > have
> > > > to
> > > > >>> > carry
> > > > >>> > > the +2 but this might make our lives a bit easier.
> > > > >>> > >
> > > > >>> > > - Tim
> > > > >>> > >
> > > > >>> >
> > > > >>>
> > > > >>
> > > > >>
> > > > >
> > > >
> > >
> >
>

Re: Automatically rebase changes before GVO?

[Lars Volker]

This is great, I'm in favor of automatic rebasing. It's be nice to have a
flag that turns it off though, since rebases sometimes can clutter a review
with many patch sets.

On Mon, Jun 11, 2018 at 12:06 PM, Tim Armstrong 
wrote:

> > On nit: as GVD gets more complex, it becomes harder for new people to
> understand the messages and +Ns applied to their patches. That doesn't mean
> we shouldn't do this, only that it's something to keep an eye on.
>
> I think this helps with that problem in net by removing the manual rebase
> step that people have to remember to do.
>
>
> On Mon, Jun 11, 2018 at 12:04 PM, Tim Armstrong 
> wrote:
>
> > I've tried my job a few times and it's working as expected. Any
> objections
> > to me switching over gerrit-verify-dryrun to my approach?
> >
> > On Thu, Jun 7, 2018 at 2:42 PM, Tim Armstrong 
> > wrote:
> >
> >> Ok, I was able to put together a test job that does the automatic rebase
> >> and carries a +2 here: https://jenkins.impala.io/job/
> >> gerrit-verify-dryrun-tarmstrong/
> >>
> >> The diff in job config required to get it to work is here:
> >> https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstron
> >> g/jobConfigHistory/showDiffFiles?timestamp1=2018-06-07_20-
> >> 41-18=2018-06-07_21-38-58
> >>
> >> I tested by creating some private drafts, adding "Impala Public Jenkins"
> >> as a reviewer and testing the various scenarios.
> >>
> >> On Thu, Jun 7, 2018 at 2:26 PM, Jim Apple  wrote:
> >>
> >>> I agree with both of you.
> >>>
> >>> On nit: as GVD gets more complex, it becomes harder for new people to
> >>> understand the messages and +Ns applied to their patches. That doesn't
> >>> mean
> >>> we shouldn't do this, only that it's something to keep an eye on.
> >>>
> >>> On Thu, Jun 7, 2018 at 1:28 PM, Philip Zeyliger 
> >>> wrote:
> >>>
> >>> > Seems fine, especially since we do the rebase as our submission
> >>> strategy
> >>> > anyway, so we're already accepting/testing something that's likely to
> >>> get
> >>> > rebased, and we may as well minimize that window.
> >>> >
> >>> > I'd be in favor of the bot also carrying the votes.
> >>> >
> >>> >
> >>> >
> >>> > On Thu, Jun 7, 2018 at 1:24 PM, Tim Armstrong <
> tarmstr...@cloudera.com
> >>> >
> >>> > wrote:
> >>> >
> >>> > > One annoyance with our precommit job is the requirement to manually
> >>> > rebase
> >>> > > the change before starting the merge. Failure to do so either leads
> >>> to
> >>> > > false positives or false negatives - builds that failed because
> they
> >>> were
> >>> > > missing a flaky/broken test fix and builds that succeeded despite
> >>> > > interacting badly with a previous fix.
> >>> > >
> >>> > > What do people think about modifying gerrit-verify-dryrun to
> >>> > automatically
> >>> > > rebase the patch (by the programmatic equivalent of hitting the
> >>> "Rebase"
> >>> > > button) at the start of the job? The patch author would still have
> to
> >>> > carry
> >>> > > the +2 but this might make our lives a bit easier.
> >>> > >
> >>> > > - Tim
> >>> > >
> >>> >
> >>>
> >>
> >>
> >
>

Re: Broken/Flaky Tests

[Lars Volker]

Hi Thomas,

Can you give an update on where we are with the builds?

We currently have ~15 changes with a +2:
https://gerrit.cloudera.org/#/q/status:open+project:Impala-ASF+branch:master+label:Code-Review%253D2

Thanks, Lars

On Fri, May 25, 2018 at 5:20 PM, Henry Robinson  wrote:

> +1 - thanks for worrying about build health.
>
> On 25 May 2018 at 17:18, Jim Apple  wrote:
>
> > Sounds good to me. Thanks for taking ownership!
> >
> > On Fri, May 25, 2018 at 5:10 PM Thomas Tauber-Marshall <
> > tmarsh...@cloudera.com> wrote:
> >
> > > Hey Impala community,
> > >
> > > There seems to have been an unusually large number of flaky or broken
> > tests
> > > <
> > > https://issues.apache.org/jira/browse/IMPALA-7073?jql=
> > project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%
> > 22In%20Progress%22%2C%20Reopened)%20AND%20labels%
> > 20in%20(flaky%2C%20broken-build)
> > > >
> > > cropping up in the last few weeks. I'd like to suggest that we hold off
> > on
> > > merging new changes that aren't related to fixing those testing issues
> > for
> > > at least a few days until things become more stable.
> > >
> > > Does anyone have any objections? If not, I'll send out another email
> when
> > > more of the issues have been addressed.
> > >
> > > Thanks,
> > > Thomas Tauber-Marshall
> > >
> >
>

Re: [VOTE] 3.0.0 release candidate 1

[Lars Volker]

+1 (binding)

* Downloaded the files and validated the sha512 signature and GPG checksum
* Built Impala from the tarball on the machine under my desk
* Started impala, ran some queries
* Observed that the shell reports a valid git hash in its version: Server
version: impalad version 3.0.0-RELEASE DEBUG (build
671de258f29d31d280defe989af261fdbd9182ae)
* Convinced myself that the git hash reported by impalad is correct

Thanks for putting up the RC, Sailesh!


On Wed, May 2, 2018 at 9:17 PM, Jim Apple  wrote:

> +1, binding
>
> I looked at the output of the release testing job that Sailesh appears to
> have run on jenkins.impala.io.
>
> On Tue, May 1, 2018 at 4:34 PM Sailesh Mukil  wrote:
>
> > This is a vote to release Impala 3.0.0
> >
> > The artifacts for testing can be downloaded from:
> > https://dist.apache.org/repos/dist/dev/impala/3.0.0/RC1/
> > Git tag: 3.0.0-rc1
> > Tree hash: d5bf1fdeb20ac332ce48c6a2b0588bcaf7ac9cb0
> >
> > Please vote +1 or -1. -1 votes should be accompanied by an explanation of
> > the reason. Only PMC members have binding votes, but other
> > community members are encouraged to cast non-binding votes. This vote
> will
> > pass if there are 3 binding +1 votes and more binding +1 votes than -1
> > votes.
> >
> > Please also note that this release is a new major release, and does not
> > override the recently released 2.12.0 as the latest release. Rather,
> 2.12.0
> > is the latest release from the 2.x branch and 3.0.0 is the first and
> latest
> > release from the 3.x branch.
> >
> > This wiki page describes how to check the release before you vote:
> > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> >
> > The vote will be open until 04:30 PM PST on Friday, May 4
> >
>

Re: Unsupported major.minor version 52.0

[Lars Volker]

This has bitten me before, too. I pushed a change to issue an additional
warning if the minicluster fails to start and the environment points to
Java 7 and Hadoop 3: https://gerrit.cloudera.org/#/c/10222/

Most people will have seen it by now, but maybe it's not too late to safe
someone else from the trouble. :)

On Thu, Apr 26, 2018 at 9:40 AM, Gabor Kaszab 
wrote:

> Hey Jim,
> I got the same error the other day. Moving to Java8 solved it for me.
>
> Gabor
>
> On Thu, Apr 26, 2018 at 6:37 PM, Philip Zeyliger 
> wrote:
>
> > Hi Jim,
> >
> > Are you using JDK7? Hadoop3 supports only JDK8+.
> >
> > -- Philip
> >
> > On Thu, Apr 26, 2018 at 9:35 AM, Jim Apple  wrote:
> >
> > > testdata/bin/run-all.sh fails for me with the following error
> > >
> > > Exception in thread "main" java.lang.UnsupportedClassVersionError:
> > > org/apache/hadoop/hdfs/server/namenode/NameNode : Unsupported
> > major.minor
> > > version 52.0
> > >
> > > This is at gerrit HEAD, at tree hash
> > >
> > > 6e88e1b26423badb4012506c10a98f32f600dbdb IMPALA-6892:
> > > CheckHashAndDecrypt()
> > > includes file and host
> > >
> > > aka commit
> > >
> > > 518bcd3e148caa8b42011de11e971c2978fb6f3b
> > >
> > > This occurs even after a successful buildall.sh
> > >
> >
>

Re: [VOTE] 2.12.0 release candidate 1

[Lars Volker]

+1 (binding)

* Validated that the sha512 file has been fixed
* Validated the GPG signature
* Ran the release test job on jenkins.impala.io, result is here

* Unpacked and compiled the tarball on my development machine, made sure
the daemon starts, and ran some queries.

I noticed that the build identifies itself as "impalad version
2.12.0-RELEASE DEBUG (build Could not obtain git hash)".
This is tracked in IMPALA-4744
 and we're released with
this before. Should we require a subsequent RC, we can manually change
GIT_HASH in bin/save-version.sh.

Thanks for creating the RC, Sailesh!


On Thu, Apr 19, 2018 at 4:57 PM, Sailesh Mukil  wrote:

> I've already updated that file to not have an absolute path. Could you
> download that and try again?
>
> On Thu, Apr 19, 2018, 4:15 PM Jim Apple  wrote:
>
> > -1:
> >
> > /tmp/foo:$ sha512sum --check apache-impala-2.12.0.tar.gz.sha512
> > sha512sum: /tmp/apache-impala-2.12.0.tar.gz: No such file or directory
> > /tmp/apache-impala-2.12.0.tar.gz: FAILED open or read
> > sha512sum: WARNING: 1 listed file could not be read
> >
> >
> > On Thu, Apr 19, 2018 at 2:40 PM, Sailesh Mukil 
> > wrote:
> >
> > > This is a vote to release Impala 2.12.0
> > >
> > > The artifacts for testing can be downloaded from:
> > > https://dist.apache.org/repos/dist/dev/impala/2.12.0/RC1/
> > > Git tag: 2.12.0-rc1
> > > Tree hash: 07f18b8539a044c506bb3ff5a118f3b7b5ea1cf7
> > >
> > > Please vote +1 or -1. -1 votes should be accompanied by an explanation
> of
> > > the reason. Only PMC members have binding votes, but other
> > > community members are encouraged to cast non-binding votes. This vote
> > will
> > > pass if there are 3 binding +1 votes and more binding +1 votes than -1
> > > votes.
> > >
> > > This wiki page describes how to check the release before you vote:
> > > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> > >
> > > The vote will be open until 3:00PM PST on Tuesday, April 24
> > >
> >
>

Re: Adding to Apache JIRA as contributor

[Lars Volker]

Done.

On Thu, Apr 19, 2018 at 6:00 PM, Adrian Ng  wrote:

> Hi,
>
> Can someone please add me as a contributor on Apache JIRA for Impala?
>
> Thanks,
> Adrian
>

Re: Please hold off on merging changes that don't fix builds

[Lars Volker]

A bunch of fixes made it into master and the builds look better. If you are
working on a broken-build issue, please keep looking into it. Thank you all
for the help.

Let's open the branch for commits again.

Cheers, Lars

On Fri, Mar 2, 2018 at 3:55 PM, Lars Volker <l...@cloudera.com> wrote:

> Our builds are still in a bad shape, in particular all downstream
> exhaustive builds are currently failing. Tim will push a change to revert
> IMPALA-4835 to address some of the issue.
>
> Please hold off on any commits that don't address current build issues
> until our builds are reliably green.
>
> Thanks, Lars
>
> On Mon, Feb 26, 2018 at 11:22 AM, Tim Armstrong <tarmstr...@cloudera.com>
> wrote:
>
>> Things are in a bit better shape - we have changes successfully being
>> cherry-picked to 2.x. CentOS 6 builds are running to completion and we're
>> able to run S3 and LocalFS builds that use the snapshot loading path.
>>
>> There are still a lot of isolated test failures though and flaky tests
>> causing a lot of failed builds:
>> https://issues.apache.org/jira/issues/?jql=project%20%3D%
>> 20IMPALA%20AND%20status%20in%20(Open%2C%20%22In%20Progress%
>> 22%2C%20Reopened)%20AND%20labels%20in%20(Flaky%2C%
>> 20flaky%2C%20broken-build)%20ORDER%20BY%20updated%20DESC
>>
>> It seems ok to open things up to smaller bug fixes, but please hold off on
>> any large commits for now until builds are reliably green.
>>
>> On Fri, Feb 23, 2018 at 3:25 PM, Tim Armstrong <tarmstr...@cloudera.com>
>> wrote:
>>
>> > I'm triaging a bunch of builds now that have broken with the influx of
>> > large changes this week.
>> >
>> > I'll continue to file JIRAs for issues are breaking builds, but for now
>> > let's avoid merging anything that might make the situation worse.
>> >
>> > I'll send out an email once things are healthier.
>> >
>> > - Tim
>> >
>>
>
>

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

[Lars Volker]

Yeah, I missed that. We set it per column, so all other types could keep
TypeDefinedOrder and floats could have something like NanAwareDoubleOrder.

On Fri, Feb 16, 2018 at 9:18 AM, Tim Armstrong <tarmstr...@cloudera.com>
wrote:

> We wouldn't need to rev the whole TypeDefinedOrder thing right? Couldn't we
> just define a special order for floats? Essentially it would be a tag for
> writers to say "hey I know about this total order thing".
>
> On Fri, Feb 16, 2018 at 9:14 AM, Lars Volker <l...@cloudera.com> wrote:
>
> > I think one idea behind the column order fields was that if a reader does
> > not recognize a value there, it needs to ignore the stats. If I remember
> > correctly, that was intended to allow us to add new orderings for
> > collations, but it also seems useful to address gaps in the spec or known
> > broken readers. In this case we would need to deprecate the default
> > "TypeDefinedOrder" and replace it with something like
> > "TypeDefinedOrderWithCorrectOrderingForDoubles". We could also count up,
> > like TypeDefinedOrderV2 and so on.
> >
> > An alternative would be to list all writers that are known to have
> written
> > incorrect stats. However that will not prevent old implementations to
> > misinterpret correct stats - which I think was the main reason why we
> added
> > new stats fields.
> >
> >
> >
> > On Fri, Feb 16, 2018 at 9:03 AM, Alexander Behm <alex.b...@cloudera.com>
> > wrote:
> >
> > > I hope the common cases is that data files do not contain these special
> > > float values. As the simplest solution, how about writers refrain from
> > > populating the stats if a special value is encountered?
> > >
> > > That fix does not preclude a more thorough solution in the future, but
> it
> > > addresses the common case quickly.
> > >
> > > For existing data files we could check the writer version ignore
> filters
> > on
> > > float/double. I don't know whether min/max filtering is common on
> > > float/double, but I suspect it's not.
> > >
> > > On Fri, Feb 16, 2018 at 8:38 AM, Tim Armstrong <
> tarmstr...@cloudera.com>
> > > wrote:
> > >
> > > > There is an extensibility mechanism with the ColumnOrder union - I
> > think
> > > > that was meant to avoid the need to add new stat fields?
> > > >
> > > > Given that the bug was in the Parquet spec, we'll need to make a spec
> > > > change anyway, so we could add a new ColumnOrder -
> > > FloatingPointTotalOrder?
> > > > at the same time as fixing the gap in the spec.
> > > >
> > > > It could make sense to declare that the default ordering for
> > > floats/doubles
> > > > is not NaN-aware (i.e. the reader should assume that NaN was
> > arbitrarily
> > > > ordered) and readers should either implement the required logic to
> > handle
> > > > that correctly (I had some ideas here:
> > > > https://issues.apache.org/jira/browse/IMPALA-6527?
> > > > focusedCommentId=16366106=com.atlassian.jira.
> > > > plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16366106)
> > > > or ignore the stats.
> > > >
> > > > On Fri, Feb 16, 2018 at 8:15 AM, Jim Apple <jbap...@cloudera.com>
> > wrote:
> > > >
> > > > > > We could have a similar problem
> > > > > > with not finding +0.0 values because a -0.0 is written to the
> > > max_value
> > > > > > field by some component that considers them the same.
> > > > >
> > > > > My hope is that the filtering would behave sanely, since -0.0 ==
> +0.0
> > > > > under the real-number-inspired ordering, which is distinguished
> from
> > > > > total Ordering, and which is also what you get when you use the
> > > > > default C/C++ operators <, >, <=, ==, and so on.
> > > > >
> > > > > You can distinguish between -0.0 and +0.0 without using total
> > ordering
> > > > > by taking their reciprocal: 1.0/-0.0 is -inf. There are some other
> > > > > ways to distinguish, I suspect, but that's the simplest one I
> recall
> > > > > at the moment.
> > > > >
> > > >
> > >
> >
>

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

[Lars Volker]

I think one idea behind the column order fields was that if a reader does
not recognize a value there, it needs to ignore the stats. If I remember
correctly, that was intended to allow us to add new orderings for
collations, but it also seems useful to address gaps in the spec or known
broken readers. In this case we would need to deprecate the default
"TypeDefinedOrder" and replace it with something like
"TypeDefinedOrderWithCorrectOrderingForDoubles". We could also count up,
like TypeDefinedOrderV2 and so on.

An alternative would be to list all writers that are known to have written
incorrect stats. However that will not prevent old implementations to
misinterpret correct stats - which I think was the main reason why we added
new stats fields.



On Fri, Feb 16, 2018 at 9:03 AM, Alexander Behm 
wrote:

> I hope the common cases is that data files do not contain these special
> float values. As the simplest solution, how about writers refrain from
> populating the stats if a special value is encountered?
>
> That fix does not preclude a more thorough solution in the future, but it
> addresses the common case quickly.
>
> For existing data files we could check the writer version ignore filters on
> float/double. I don't know whether min/max filtering is common on
> float/double, but I suspect it's not.
>
> On Fri, Feb 16, 2018 at 8:38 AM, Tim Armstrong 
> wrote:
>
> > There is an extensibility mechanism with the ColumnOrder union - I think
> > that was meant to avoid the need to add new stat fields?
> >
> > Given that the bug was in the Parquet spec, we'll need to make a spec
> > change anyway, so we could add a new ColumnOrder -
> FloatingPointTotalOrder?
> > at the same time as fixing the gap in the spec.
> >
> > It could make sense to declare that the default ordering for
> floats/doubles
> > is not NaN-aware (i.e. the reader should assume that NaN was arbitrarily
> > ordered) and readers should either implement the required logic to handle
> > that correctly (I had some ideas here:
> > https://issues.apache.org/jira/browse/IMPALA-6527?
> > focusedCommentId=16366106=com.atlassian.jira.
> > plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16366106)
> > or ignore the stats.
> >
> > On Fri, Feb 16, 2018 at 8:15 AM, Jim Apple  wrote:
> >
> > > > We could have a similar problem
> > > > with not finding +0.0 values because a -0.0 is written to the
> max_value
> > > > field by some component that considers them the same.
> > >
> > > My hope is that the filtering would behave sanely, since -0.0 == +0.0
> > > under the real-number-inspired ordering, which is distinguished from
> > > total Ordering, and which is also what you get when you use the
> > > default C/C++ operators <, >, <=, ==, and so on.
> > >
> > > You can distinguish between -0.0 and +0.0 without using total ordering
> > > by taking their reciprocal: 1.0/-0.0 is -inf. There are some other
> > > ways to distinguish, I suspect, but that's the simplest one I recall
> > > at the moment.
> > >
> >
>

Re: Jenkins.impala.io back up

[Lars Volker]

Thank you for applying the updates, Zach!

On Wed, Feb 14, 2018 at 11:17 AM, Zachary Amsden 
wrote:

> Everything went as planned and Jenkins is now upgraded.
>
>  - Zach
>

TEST_START_CLUSTER_ARGS are not applied in custom cluster tests

[Lars Volker]

Hi All,

Mike noticed in a review that TEST_START_CLUSTER_ARGS are not applied in
custom cluster tests and we wondered whether that is on purpose or a bug.
Can someone with historic knowledge shed some light on it? The relevant
code is here:

https://github.com/apache/impala/blob/5f7599687748b1e1ce0a5a34d38cc49a4c4cd9f5/tests/common/custom_cluster_test_suite.py#L124

Thanks, Lars

Re: Freezing jenkins.impala.io Wed 2018-02-07 (today) at 6pm PST

[Lars Volker]

Thank you for applying these updates, Alex!

On Feb 7, 2018 22:57, "Alexander Behm"  wrote:

Jenkins and a few plugins were updated. The service is back now at
jenkins.impala.io

Warning: I updated the "Pipeline: Supporting APIs" plugin to version 2.16
due to a severe security vulnerability. The new version stated incompatible
changes, so please let me know if you see something strange.

On Wed, Feb 7, 2018 at 2:53 PM, Alexander Behm 
wrote:

> Jenkins will be upgraded today.
>
> Timeline:
> 6pm Wed 2018-02-07: Jenkins will stop accepting new jobs
> 10pm Wed 2018-02-07: Jenkins will restart
>
> If all goes well Jenkins should be back online the morning of Wed
> 2018-02-08 (PST)
>
>
>

Re: Impala 3.x (and 2.x)

[Lars Volker]

e as follows:
> > >> > >
> > >> > > # This created the branch on gerrit.cloudera.org
> > >> > > $ssh -p 29418 ph...@gerrit.cloudera.org gerrit create-branch
> > >> Impala-ASF
> > >> > > 2.x master
> > >> > >
> > >> > > # This created the branch on the Apache git server:
> > >> > > $git push apache asf-gerrit/2.x:refs/heads/2.x
> > >> > > Username for 'https://git-wip-us.apache.org': philz
> > >> > > Password for 'https://ph...@git-wip-us.apache.org':
> > >> > > Total 0 (delta 0), reused 0 (delta 0)
> > >> > > To https://git-wip-us.apache.org/repos/asf/impala.git
> > >> > >  * [new branch]  asf-gerrit/2.x -> 2.x
> > >> > >
> > >> > > # Double-check:
> > >> > > $git-ls-remote apache | grep 2.x; git-ls-remote asf-gerrit | grep
> > 2.x
> > >> > > 6cc76d72016b8d5672676e8e8a979b0807803ad9refs/heads/2.x
> > >> > > 6cc76d72016b8d5672676e8e8a979b0807803ad9refs/heads/2.x
> > >> > >
> > >> > > # push_to_asf learned of the branch "magically"
> > >> > > $bin/push_to_asf.py
> > >> > > INFO:root:Fetching from remote 'asf-gerrit'...
> > >> > > INFO:root:done
> > >> > > Branch '2.x':up to date
> > >> > > Branch 'asf-site':   up to date
> > >> > > Branch 'branch-2.10.0':  found on Apache but not in gerrit
> > >> > > Branch 'branch-2.11.0':  found on Apache but not in gerrit
> > >> > > Branch 'branch-2.7.0':   found on Apache but not in gerrit
> > >> > > Branch 'branch-2.8.0':   found on Apache but not in gerrit
> > >> > > Branch 'branch-2.9.0':   found on Apache but not in gerrit
> > >> > > Branch 'hadoop-next':up to date
> > >> > > Branch 'master': up to date
> > >> > >
> > >> > > -- Philip
> > >> > >
> > >> > > On Tue, Jan 16, 2018 at 5:29 PM, Alexander Behm <
> > >> alex.b...@cloudera.com>
> > >> > > wrote:
> > >> > >
> > >> > >> +1
> > >> > >>
> > >> > >> On Tue, Jan 16, 2018 at 5:27 PM, Taras Bobrovytsky <
> > >> taras...@apache.org
> > >> > >
> > >> > >> wrote:
> > >> > >>
> > >> > >> > +1
> > >> > >> >
> > >> > >> > On Tue, Jan 16, 2018 at 1:34 PM, Lars Volker <l...@cloudera.com>
> > >> wrote:
> > >> > >> >
> > >> > >> > > +1
> > >> > >> > >
> > >> > >> > > On Tue, Jan 16, 2018 at 1:29 PM, Philip Zeyliger <
> > >> > phi...@cloudera.com
> > >> > >> >
> > >> > >> > > wrote:
> > >> > >> > >
> > >> > >> > > > Hi folks!
> > >> > >> > > >
> > >> > >> > > > It sounds like there haven't been objections to having
> > master be
> > >> > >> "3.0"
> > >> > >> > > and
> > >> > >> > > > introducing a 2.x branch. Would folks be alright if I
> started
> > >> > making
> > >> > >> > > > changes in that direction?
> > >> > >> > > >
> > >> > >> > > > Thanks!
> > >> > >> > > >
> > >> > >> > > > -- Philip
> > >> > >> > > >
> > >> > >> > > > On Fri, Jan 12, 2018 at 4:10 PM, Philip Zeyliger <
> > >> > >> phi...@cloudera.com>
> > >> > >> > > > wrote:
> > >> > >> > > >
> > >> > >> > > > >
> > >> > >> > > > >
> > >> > >> > > > > On Fri, Jan 12, 2018 at 4:04 PM, Jim Apple <
> > >> > jbap...@cloudera.com>
> > >> > >> > > wrote:
> > >> > >> > > > >
> > >> > >> > > > >> This makes sense to me.
> > >> > >> > > > >>
> > >> > >> > >

Re: [ANNOUNCE] Apache Impala 2.11.0 release

[Lars Volker]

Done: https://twitter.com/ApacheImpala/status/954061402070704129

On Thu, Jan 18, 2018 at 10:31 AM, Thomas Tauber-Marshall <
tmarsh...@cloudera.com> wrote:

> -everyone but dev@
>
> Could someone with access to the Impala Twitter account (according to Henry
> that should be anyone on the PMC) announce this on Twitter as well? Thanks
>
> On Thu, Jan 18, 2018 at 10:17 AM Thomas Tauber-Marshall <
> tmarsh...@cloudera.com> wrote:
>
> > The Apache Impala team is pleased to announce the release of Impala
> 2.11.0
> >
> > Impala is a high-performance distributed SQL engine.
> >
> > The release is available at: https://impala.apache.org/downloads.html
> >
> > Thanks,
> >
> > The Apache Impala team
> >
>

Re: IMPALA-5440 Request for assignment

[Lars Volker]

Hi Xinran,

I added you to the Contributors group and assigned the Jira to you. Thank
you for working on Impala.

Cheers, Lars

On Sun, Jan 7, 2018 at 3:48 AM, Xinran Yu Tinney 
wrote:

> Hi, Impala dev community,
>I was wondering if I could get assigned for IMPALA-5440
> . My Jira username :
> xyu2017.
>
> Thanks!
>
> Xinran
>
> Software Engineer
> Cloudera
>

Re: [VOTE] 2.11.0 release candidate 1

[Lars Volker]

+1 (binding)

Here's what I did:

* Read the instructions on the wiki and fixed the link to the KEYs file
* Downloaded all files from the RC1 download link
* Manually verified md5, sha512 sum, and asc signature
* Verified git hashes
* Verified tarball contents
* Ran ./buildall.sh -skiptests -notests -noclean
* Started a local minicluster and ran some queries.
* Manually downloaded Apache Rat and ran it as described
in bin/check-rat-report.py

I noticed that the wiki page seems to have a different/older version of the
verification script used by Jenkins. It probably wouldn't hurt to move it
into git and use if from both.

On Thu, Dec 21, 2017 at 4:07 AM, Jim Apple  wrote:

> +1 (binding), ran the same job and inspected its output to make sure
> it was checking all of the requirements.
>
> On Wed, Dec 20, 2017 at 2:45 PM, Bharath Vissapragada
>  wrote:
> > +1 (binding)
> >
> > - Verified via https://jenkins.impala.io/job/release-test/27/
> > - Downloaded the tarball locally, verified checksums, compiled it from
> > source, started a mini cluster and tested some basic queries. Everything
> > looked fine.
> >
> >
> >
> > On Tue, Dec 19, 2017 at 9:07 PM, Thomas Tauber-Marshall <
> > tmarsh...@cloudera.com> wrote:
> >
> >> This is a vote to release Impala 2.11.0
> >>
> >> The artifacts for testing can be downloaded from:
> >> https://dist.apache.org/repos/dist/dev/impala/2.11.0/RC1/
> >> Git tag: 2.11.0-rc1
> >> Tree hash: 9618d0f833cf826252d2c1b2a720eaabe1bef689
> >>
> >> Please vote +1 or -1. -1 votes should be accompanied by an explanation
> of
> >> the reason. Only PMC members have binding votes, but other
> >> community members are encouraged to cast non-binding votes. This vote
> will
> >> pass if there are 3 binding +1 votes and more binding +1 votes than -1
> >> votes.
> >>
> >> This wiki page describes how to check the release before you vote:
> >> https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> >>
> >> The vote will be open until 9:00PM PST on Friday, December 22
> >>
>

Re: [DISCUSS] 2.11.0 release

[Lars Volker]

+1

Thank you for stepping up, Thomas! It'll be great to have a new release so
close after our graduation.

On Thu, Nov 30, 2017 at 1:32 PM, Taras Bobrovytsky 
wrote:

> +1
>
> I agree. This is a good time to release. Thanks for volunteering!
>
> On Thu, Nov 30, 2017 at 1:22 PM, Sailesh Mukil 
> wrote:
>
> > +1
> >
> > Thanks for volunteering.
> >
> > On Thu, Nov 30, 2017 at 1:02 PM, Jim Apple  wrote:
> >
> > > +1
> > >
> > > On Thu, Nov 30, 2017 at 12:46 PM, Thomas Tauber-Marshall <
> > > tmarsh...@cloudera.com> wrote:
> > >
> > > > Folks,
> > > >
> > > > It has been over 2 months since we released Apache Impala 2.10.0 and
> > > there
> > > > have been new feature improvements and a good number of bug fixes
> > checked
> > > > in since then.
> > > >
> > > > I propose that we release 2.11.0 soon and I volunteer to be its
> release
> > > > manager. Please speak up and let the community know if anyone has any
> > > > objections to this.
> > > >
> > > > Thanks,
> > > > Thomas
> > > >
> > >
> >
>

Re: Impala minidump dump_breakpad_symbols

[Lars Volker]

Yeah, this line indicates that you didn't have the right symbols in your
syms folder:

0x0040 - 0x0283efff  impalad  ???  (main)  (WARNING: No symbols,
impalad, 21AA452CEF0F8C034C7E78A639190ED40)

You may want to try running dump_breakpad_symbols.py again to get the
correct symbols.

On Tue, Nov 28, 2017 at 9:01 PM, Vincent Tran <vtt...@cloudera.com> wrote:

> Hi Sandish,
>
> Did you get the minidump from the minicluster of this private build? It
> looks like the hash of your private build symbols and the one from the md
> do not match.
>
> You can turn the md into a core to see the original binary version that
> generated it:
>
> /bin/minidump-2-core
> 11850fc8-c7ec-dd15-15ba7314-1926301a.dmp >
> 11850fc8-c7ec-dd15-15ba7314-1926301a.core
> If you open the resulting core file in gdb, what does the last line say?
>
>
>
>
>
> On Nov 28, 2017 11:36 PM, "sanysand...@gmail.com" <sanysand...@gmail.com>
> wrote:
>
> >
> >
> > On 2017-11-28 21:01, Lars Volker <l...@cloudera.com> wrote:
> > > The resolved minidump looks truncated, as does the output of minidump
> > > stackwalk. This could indicate that the minidump got corrupted. How did
> > you
> > > generate the minidump?
> > >
> > > On Nov 28, 2017 02:20, "sanysand...@gmail.com" <sanysand...@gmail.com>
> > > wrote:
> > >
> > > >
> > > > Thank you, Lars Volker, for quick response's
> > > >
> > > > The end content of /tmp/resolved.txt  file:
> > > >
> > > > [ec2-user@jenkins tmp]$ tail -20f /tmp/resolved.txt
> > > > rbp = 0x7fd7d4fdf220   rsp = 0x7fc10d6ad270
> > > >  r8 = 0x7fd7d4fdf220r9 = 0x
> > > > r10 = 0x7fc10d6ad3d0   r11 = 0x0202
> > > > r12 = 0x0006   r13 = 0x7fc10d6ad3d0
> > > > r14 = 0xff92   r15 = 0x7fc10d6ad400
> > > > rip = 0x7fdb5e2ffa82
> > > > Found by: given as instruction pointer in context
> > > >
> > > > Thread 960
> > > >  0  0x7fdb5e2ffa82
> > > > rax = 0xfdfc   rdx = 0x0001
> > > > rcx = 0x   rbx = 0x10d33e20
> > > > rsi = 0x0189   rdi = 0x10d33e4c
> > > > rbp = 0x7fc0f3e7a740   rsp = 0x7fc0f3e7a6b0
> > > >  r8 = 0x10d33e20r9 = 0x
> > > > r10 = 0x7fc0f3e7a740   r11 = 0x0202
> > > > r12 = 0x0001   r13 = 0x7fc0f3e7a740
> > > > r14 = 0xff92   r15 = 0x22217000
> > > > rip = 0x7fdb5e2ffa82
> > > > Found by: given as instruction pointer in context
> > > >
> > > > Output of find syms/impalad:
> > > > [ec2-user@jenkins tmp]$ find syms/impalad
> > > > syms/impalad
> > > > syms/impalad/73318DB4729C07530B7338E67EA255AB0
> > > > syms/impalad/73318DB4729C07530B7338E67EA255AB0/impalad.sym
> > > >
> > > > These are the steps I had followed:
> > > > Step-1:
> > > > I followed exact steps from this https://cwiki.apache.org/confl
> > > > uence/display/IMPALA/Building+Impala and also followed "build
> > > > native-toolchain from scratch" https://cwiki.apache.org/confl
> > > > uence/display/IMPALA/Building+native-toolchain+from+scratch+
> > > > and+using+with+Impala
> > > >
> > > > Step-2:
> > > > Env:
> > > > export IMPALA_HOME=/home/ec2-user/impala-setup/Impala
> > > > export PATH=$PATH:$IMPALA_HOME/bin
> > > > export KUDU_IS_SUPPORTED=false
> > > > export KUDU_CLIENT_DIR=/opt/kudu/
> > > >
> > > > Get breakpad symbols:
> > > > dump_breakpad_symbols.py -b be/build/ --dump_syms
> > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345
> > 7e55f1aecb60-p3/bin/dump_syms
> > > > -d /tmp/syms
> > > >
> > > > rm /tmp/resolved.txt
> > > >
> > > > Run minidump_stackwalk:
> > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345
> > > > 7e55f1aecb60-p3/bin/minidump_stackwalk
> /home/ec2-user/64811560-8bae-a
> > 67f-22f961c1-45bf52a3.dmp
> > > > /tmp/syms > /tmp/resolved.txt
> > > >
> > > >
> > > > Please let me know If I have missed any steps??
> > > >
> > > >

Re: Impala minidump dump_breakpad_symbols

[Lars Volker]

The resolved minidump looks truncated, as does the output of minidump
stackwalk. This could indicate that the minidump got corrupted. How did you
generate the minidump?

On Nov 28, 2017 02:20, "sanysand...@gmail.com" <sanysand...@gmail.com>
wrote:

>
> Thank you, Lars Volker, for quick response's
>
> The end content of /tmp/resolved.txt  file:
>
> [ec2-user@jenkins tmp]$ tail -20f /tmp/resolved.txt
> rbp = 0x7fd7d4fdf220   rsp = 0x7fc10d6ad270
>  r8 = 0x7fd7d4fdf220r9 = 0x
> r10 = 0x7fc10d6ad3d0   r11 = 0x0202
> r12 = 0x0006   r13 = 0x7fc10d6ad3d0
> r14 = 0xff92   r15 = 0x7fc10d6ad400
> rip = 0x7fdb5e2ffa82
> Found by: given as instruction pointer in context
>
> Thread 960
>  0  0x7fdb5e2ffa82
> rax = 0xfdfc   rdx = 0x0001
> rcx = 0x   rbx = 0x10d33e20
> rsi = 0x0189   rdi = 0x10d33e4c
> rbp = 0x7fc0f3e7a740   rsp = 0x7fc0f3e7a6b0
>  r8 = 0x10d33e20r9 = 0x
> r10 = 0x7fc0f3e7a740   r11 = 0x0202
> r12 = 0x0001   r13 = 0x7fc0f3e7a740
> r14 = 0xff92   r15 = 0x22217000
> rip = 0x7fdb5e2ffa82
> Found by: given as instruction pointer in context
>
> Output of find syms/impalad:
> [ec2-user@jenkins tmp]$ find syms/impalad
> syms/impalad
> syms/impalad/73318DB4729C07530B7338E67EA255AB0
> syms/impalad/73318DB4729C07530B7338E67EA255AB0/impalad.sym
>
> These are the steps I had followed:
> Step-1:
> I followed exact steps from this https://cwiki.apache.org/confl
> uence/display/IMPALA/Building+Impala and also followed "build
> native-toolchain from scratch" https://cwiki.apache.org/confl
> uence/display/IMPALA/Building+native-toolchain+from+scratch+
> and+using+with+Impala
>
> Step-2:
> Env:
> export IMPALA_HOME=/home/ec2-user/impala-setup/Impala
> export PATH=$PATH:$IMPALA_HOME/bin
> export KUDU_IS_SUPPORTED=false
> export KUDU_CLIENT_DIR=/opt/kudu/
>
> Get breakpad symbols:
> dump_breakpad_symbols.py -b be/build/ --dump_syms
> $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e613457e55f1aecb60-p3/bin/dump_syms
> -d /tmp/syms
>
> rm /tmp/resolved.txt
>
> Run minidump_stackwalk:
> $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345
> 7e55f1aecb60-p3/bin/minidump_stackwalk 
> /home/ec2-user/64811560-8bae-a67f-22f961c1-45bf52a3.dmp
> /tmp/syms > /tmp/resolved.txt
>
>
> Please let me know If I have missed any steps??
>
>
>

Re: Impala minidump dump_breakpad_symbols

[Lars Volker]

Thanks, Sandish,

All the way at the bottom of the resolved minidump you should find a list
of modules that breakpad tried to access when resolving the symbols. If you
don't have the symbols for the right binary, you'll find a line like this:

0x0040 - 0x02759fff  impalad  ???  (main)  (WARNING: No symbols,
impalad, 80F4346224A2670E0C8632FE78B405510)

If your symbols match, the (WARNING...) part will not be there. That hash
in the warning identifies the binary, and your syms/impalad folder needs to
have symbols for that hash:

$ find syms/impalad
syms/impalad
syms/impalad/1975CC59C4F6B26C3BDDF2B51A9694500
syms/impalad/1975CC59C4F6B26C3BDDF2B51A9694500/impalad.sym
syms/impalad/80F4346224A2670E0C8632FE78B405510
syms/impalad/80F4346224A2670E0C8632FE78B405510/impalad.sym

Can you double check that you have matching symbols for your minidump? If
that doesn't get you anywhere, can you please describe step-by-step what
you did?

Cheers, Lars

On Nov 27, 2017 22:44, "sanysand...@gmail.com" <sanysand...@gmail.com>
wrote:

>
>
> On 2017-11-28 12:01, Lars Volker <l...@cloudera.com> wrote:
> > Can you post the output of dump_breakpad_symbols.py and stderr of
> > minidump_stackwalk?
> >
> > Cheers, Lars
> >
> > On Mon, Nov 27, 2017 at 10:24 PM, sanysand...@gmail.com <
> > sanysand...@gmail.com> wrote:
> >
> > >
> > >
> > > On 2017-11-28 10:03, Sailesh Mukil <sail...@cloudera.com> wrote:
> > > > Hi Sandish,
> > > >
> > > > Have you tried the steps outlined here?
> > > > https://cwiki.apache.org/confluence/display/IMPALA/
> > > Debugging+Impala+Minidumps
> > > >
> > > > If it still doesn't work, as Brock mentioned, please paste portions
> of
> > > > errors or any other helpful output that you see.
> > > >
> > > > - Sailesh
> > > >
> > > >
> > > >
> > > > On Mon, Nov 27, 2017 at 8:23 PM, Brock Noland <br...@phdata.io>
> wrote:
> > > >
> > > > > Hi Sandish,
> > > > >
> > > > > I don't know how to help you, but I think it'd be useful if you
> shared
> > > a
> > > > > portion of the results file for folks to look at. Note that
> attachments
> > > > > don't work well so I'd just paste a portion of it inline.
> > > > >
> > > > > On Mon, Nov 27, 2017 at 9:15 PM, sanysand...@gmail.com <
> > > > > sanysand...@gmail.com> wrote:
> > > > >
> > > > > > Hi Team,
> > > > > >
> > > > > > I'm trying to get minidump_stackwalk results with dump breakpad
> > > symbols,
> > > > > > but symbols are not showing in results file, below are the steps
> > > which I
> > > > > > followed. Can someone verify or give me steps to get symbols in
> > > > > > minidump_stackwalk results ??
> > > > > >
> > > > > >
> > > > > > Steps:
> > > > > > dump_breakpad_symbols.py -b be/build/ --dump_syms
> > > > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e613457e
> > > > > 55f1aecb60-p3/bin/dump_syms
> > > > > > -d /tmp/syms
> > > > > >
> > > > > > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345
> > > > > > 7e55f1aecb60-p3/bin/minidump_stackwalk .dmp
> > > /tmp/syms
> > > > > > > /tmp/resolved.txt
> > > > > >
> > > > >
> > > >
> > >
> > > Brock, here is the content of /tmp/resolved.txt file
> > > -
> > > Operating system: Linux
> > >   0.0.0 Linux 3.10.0-514.21.1.el7.x86_64 #1 SMP Sat
> Apr 22
> > > 02:41:35 EDT 2017 x86_64
> > > CPU: amd64
> > >  family 6 model 63 stepping 2
> > >  36 CPUs
> > >
> > > GPU: UNKNOWN
> > >
> > > Crash reason:  SIGSEGV
> > > Crash address: 0x10
> > > Process uptime: not available
> > >
> > > Thread 72 (crashed)
> > >  0  0x7fdb5e2fdbd0
> > > rax = 0x1e17e560   rdx = 0x17382a10
> > > rcx = 0x   rbx = 0x
> > > rsi = 0x044520c0   rdi = 0x
> > > rbp = 0x7fd914150880   rsp = 0x7fdaef212a78
> > >  r8 = 0x7fdb5df7b938r9 = 0x0020
> > > r10 = 0x0019