Re: New PMC member: Laszlo Gaal

[Sailesh Mukil]

Congratulations, Laszlo!

On Mon, Jan 24, 2022 at 8:02 AM Laszlo Gaal 
wrote:

> Thank you all very much!
> It is great, and truly rewarding to belong to such a superb community.
>
>   - Laszlo
>
> On Mon, Jan 24, 2022 at 9:56 AM Gabor Kaszab 
> wrote:
>
> > Congrats, Laci!
> >
> > On Mon, Jan 24, 2022 at 9:04 AM Dinesh Garg 
> wrote:
> >
> > > Well deserved , long overdue . Congratulations Laszlo
> > >
> > > On Mon, Jan 24, 2022 at 12:26 PM skyyws  wrote:
> > >
> > > > Congratulations Laszlo!
> > > >
> > > >
> > > > | |
> > > > skyyws
> > > > |
> > > > |
> > > > sky...@163.com
> > > > |
> > > > 签名由网易邮箱大师定制
> > > >
> > > >
> > > > On 01/22/2022 03:46，Zoltán Borók-Nagy wrote：
> > > > The Project Management Committee (PMC) for Apache Impala has invited
> > > Laszlo
> > > > Gaal to become a PMC member and we are pleased to announce that they
> > have
> > > > accepted.
> > > >
> > > > Congratulations and welcome, Laszlo!
> > > >
> > > --
> > > Sent from Gmail Mobile
> > >
> >
>

Re: expr-benchmark

[Sailesh Mukil]

The benchmarks weren't being actively maintained last I saw (6 months back
or so), and most of them aren't run regularly as well. So a lot of the
dependent code may have changed causing them to break. So, there's no
guarantee that they will work.

On Sat, Sep 8, 2018 at 10:08 AM Jim Apple  wrote:

> I can't get exp-benchmark working near HEAD. Has anyone else seen and
> solved this? We don't run benchmarks in pre-commit testing, so I'm not
> sure if this is just me, perhaps something abut it not being hermetic,
> or if this has been broken for a while.
>
> 18/09/08 10:00:40 INFO util.JvmPauseMonitor: Starting JVM pause monitor
> Wrote minidump to
> /tmp/minidumps/expr-benchmark/11ed5026-eff4-4214-639a048a-0389c5c4.dmp
> Wrote minidump to
> /tmp/minidumps/expr-benchmark/11ed5026-eff4-4214-639a048a-0389c5c4.dmp
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> #  SIGSEGV (0xb) at pc=0x00e99791, pid=30887,
> tid=0x7fb47665f8c0
> #
> # JRE version: Java(TM) SE Runtime Environment (8.0_131-b11) (build
> 1.8.0_131-b11)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.131-b11 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # C  [expr-benchmark+0xa99791]
> impala::DescriptorTbl::GetSlotDescriptor(int) const+0x1
> #
> # Failed to write core dump. Core dumps have been disabled. To enable
> core dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # /home/jbapple/Impala/logs/hs_err_pid30887.log
> #
> # If you would like to submit a bug report, please visit:
> #   http://bugreport.java.com/bugreport/crash.jsp
> # The crash happened outside the Java Virtual Machine in native code.
> # See problematic frame for where to report the bug.
> #
>

Re: #pragma once?

[Sailesh Mukil]

An advantage of using #pragma once is potential improved compilation
speeds. However, a con is that it's non-standard and therefore, its
behavior can change at any point and can also vary across compilers,
potentially making the code even less portable.

That being said, since Kudu has been using it for a while and has had no
issues, we can do the same since the potential benefits outweigh the cons.

On Wed, Aug 1, 2018 at 11:48 AM, Tim Armstrong <
tarmstr...@cloudera.com.invalid> wrote:

> Todd brought up our include guards on a code review, asking why we don't
> use #pragma once instead: https://gerrit.cloudera.org/#/c/10988/5 . It
> sounds like Kudu has switched to it
>
> #pragma once does seem cleaner and our GCC and Clang versions are modern
> enough to support it.
>
> What do people think about switching to that as the preferred way of
> including headers only once?
>
> - Tim
>

Re: Order of error messages printed

[Sailesh Mukil]

The ErrorLogMap's key is "TErrorCode::type" which is an Enum. I don't think
we can derive any inherent meaning from the order of the errors listed in
the enum (other than OK=0, which wouldn't be in this map anyway). I'm not
aware of any clients relying on this order, and even if there were, it
would probably be incorrect to do so. So, unless anyone else chimes in with
a good reason to keep this as a map instead of an unordered_map, I'm all
for changing it.

On Tue, Jul 24, 2018 at 12:12 PM, Michael Ho 
wrote:

> Hi,
>
> While making some changes for be/src/util/error-util.cc, I stumbled upon
> the question of whether we need a map instead of an unordered map for
> ErrorLogMap (used for tracking error messages for each error code). This
> affects the order in which error messages are printed in PrintErrorMap().
> Looking through the code, it doesn't appear to me that there is any
> ordering requirement. However, it's unclear to me if any client of Impala
> makes assumption about the ordering of the output in PrintErrorMap(). So,
> sending this email to the list in case anyone knows anything.
>
> --
> Thanks,
> Michael
>

Re: impalad running as superuser in tests

[Sailesh Mukil]

On Wed, Jul 18, 2018 at 10:21 AM, Todd Lipcon 
wrote:

> On Tue, Jul 17, 2018 at 5:27 PM, Sailesh Mukil
>  > wrote:
>
> > On Tue, Jul 17, 2018 at 2:47 PM, Todd Lipcon 
> > wrote:
> >
> > > Hey folks,
> > >
> > > I'm working on a regression test for IMPALA-7311 and found something
> > > interesting. It appears that in our normal minicluster setup, impalad
> > runs
> > > as the same username as the namenode (namely, the username of the
> > > developer, in my case 'todd').
> > >
> > > This means that the NN treats impala as a superuser, and therefore
> > doesn't
> > > actually enforce permissions. So, tests about the behavior of Impala on
> > > files that it doesn't have access to are somewhat tricky to write.
> > >
> > >
> > What kind of files do you specifically mean? Something that the daemon
> > tries to access directly (Eg: keytab file, log files, etc.) ? I'm
> guessing
> > it's not this since you mentioned the NN.
> >
> > Or files that belong to a table/partition in HDFS? If it's this case, we
> > would go through Sentry before accessing files that belong to a table,
> and
> > access would be determined by Sentry on the "session user" (not the
> impalad
> > user) before Impala even tries to access HDFS. (Eg:
> > tests/authorization/test_authorization.py)
> >
>
> Right, files on HDFS. I mean that, in cases where Sentry is not enabled or
> set up, and even in some cases where it is set up but not synchronized with
> HDFS, it's possible that the user can point table metadata at files or
> directories that aren't writable to the 'impala' user on HDFS. For example,
> I can do:
>
> CREATE EXTERNAL TABLE foo (...) LOCATION '/user/todd/my-dir';
>
> and it's likely that 'my-dir' is not writable by 'impala' on a real
> cluster. Thus, if I try to insert into it, I get an error because "impala"
> does not have HDFS permissions to access this directory.
>
> Currently, the frontend does some checks here to try to produce a nice
> error. But, those checks are based on cached metadata which could be in
> accurate. In the case that it's inaccurate, the error will be thrown from
> the backend when it tries to create a file in a non-writable location.
>
> In the minicluster environment, it's impossible to test this case (actual
> permissions enforced by the NN causing an error) because the backend is
> running as an HDFS superuser. That is to say, it has full permissions
> everywhere. That's due to the special case behavior that HDFS has: it
> determines the name of the superuser to be the username that is running the
> NN. Since in the minicluster, both impala and the NN run as 'todd' in my
> case, impala acts as superuser. In a real cluster (even with security
> disabled) impala typically runs as 'impala' whereas the NN runs as 'hdfs'
> and thus impala does not have superuser privileges.
>

This makes sense, thanks for the explanation. The 'HADOOP_USER_NAME'
approach seems like a good way to go, but as Phil said, might cause issues
with other components (or not).


> -Todd
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: impalad running as superuser in tests

[Sailesh Mukil]

On Tue, Jul 17, 2018 at 2:47 PM, Todd Lipcon 
wrote:

> Hey folks,
>
> I'm working on a regression test for IMPALA-7311 and found something
> interesting. It appears that in our normal minicluster setup, impalad runs
> as the same username as the namenode (namely, the username of the
> developer, in my case 'todd').
>
> This means that the NN treats impala as a superuser, and therefore doesn't
> actually enforce permissions. So, tests about the behavior of Impala on
> files that it doesn't have access to are somewhat tricky to write.
>
>
What kind of files do you specifically mean? Something that the daemon
tries to access directly (Eg: keytab file, log files, etc.) ? I'm guessing
it's not this since you mentioned the NN.

Or files that belong to a table/partition in HDFS? If it's this case, we
would go through Sentry before accessing files that belong to a table, and
access would be determined by Sentry on the "session user" (not the impalad
user) before Impala even tries to access HDFS. (Eg:
tests/authorization/test_authorization.py)

If you're describing a different scenario than the above 2 I mentioned,
then I'd be interested to hear that, so it'll be easier to understand why
this change is necessary.

Has anyone run into this before? Should we consider running either the
> impalad or the namenode as a different spoofed username so that the
> minicluster environment is more authentic to true cluster environments? We
> can do this easily by setting the HADOOP_USER_NAME environment variable or
> system property.
>
> -Todd
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Re: Removing IMPALA_MINICLUSTER_PROFILE=2 from master

[Sailesh Mukil]

+1 for removing it.

I'm all for reduced complexity if not a lot of users are benefitting from
it.

On Wed, Jul 11, 2018 at 9:29 AM, Fredy Wijaya 
wrote:

> Hi all,
>
> In the master branch, we support both Hadoop 2 and Hadoop 3 via
> IMPALA_MINICLUSTER_PROFILE environment variable. When Impala transitioned
> from Hadoop 2 to Hadoop 3, we introduced IMPALA_MINICLUSTER_PROFILE as a
> way to easily switch between Hadoop 2 and Hadoop 3. It made sense at that
> time. However, I believe the IMPALA_MINICLUSTER_PROFILE=2 has outlived its
> usefulness and has grown more complex to maintain given that we need to
> introduce shims, feature flags, additional Jenkins jobs, etc. Given that we
> already have 2.x branch for Impala with Hadoop 2 support. I'm proposing
> that we remove IMPALA_MINICLUSTER_PROFILE=2 in the master branch so that we
> can focus our effort on supporting Hadoop 3 on the master branch.
>
> What do others think?
>

Re: boost::scoped_ptr vs std::unique_ptr

[Sailesh Mukil]

I'm in favor.

Since the main distinction is that a unique_ptr is moveable, whereas a
scoped_ptr is not, we should just make sure that we do our due diligence
during code reviews so that we catch those cases.

Also, making a unique_ptr const disallows moving it, since the move
constructor takes a non-const unique_ptr container. It probably won't work
in all places, but we could enforce that in certain parts of the code.

On Thu, Jul 5, 2018 at 4:49 PM, Thomas Tauber-Marshall <
tmarsh...@cloudera.com.invalid> wrote:

> I'm definitely in favor of using more standard c++ to reduce both confusion
> and our reliance on boost, especially as I suspect a lot of people (eg. me)
> don't know the subtle difference between scoped_ptr and unique_ptr off the
> top of their head anyways.
>
> Fwiw, I was under the impression from talking with people in the past that
> we were already trying to make this move, and the
> PartitionedAggregationNode refactor that just went in made the switch to
> unique_ptr, though no one commented on it in the review.
>
> On Thu, Jul 5, 2018 at 4:39 PM Tim Armstrong
>  wrote:
>
> > I was just talking with Michael Ho on a review about this
> > https://gerrit.cloudera.org/#/c/10810/7/be/src/exec/scan-node.h@271
> >
> > For a while we've continued using scoped_ptr in some places because it
> > supports a smaller set of operators and implies that the pointer isn't
> > movable. See
> >
> > https://cwiki.apache.org/confluence/display/IMPALA/
> Resource+Management+Best+Practices+in+Impala
> > .
> >
> > I don't think we're consistently following this pattern and it seems to
> > cause some confusion about what the best practice is, particularly for
> > people coming from other code bases. I personally like the distinction,
> but
> > I don't feel that strongly about it.
> >
> > What do people think? Should we continue using scoped_ptr or move away
> from
> > it. There is already a JIRA to make the change but we haven't done it
> > because of the above reasons:
> > https://issues.apache.org/jira/browse/IMPALA-3444
> >
> > - Tim
> >
>

Re: Automatically rebase changes before GVO?

[Sailesh Mukil]

+1

On Mon, Jun 11, 2018 at 3:02 PM, Jim Apple  wrote:

> No objection from me.
>
> On Mon, Jun 11, 2018 at 12:06 PM, Tim Armstrong 
> wrote:
>
> > > On nit: as GVD gets more complex, it becomes harder for new people to
> > understand the messages and +Ns applied to their patches. That doesn't
> mean
> > we shouldn't do this, only that it's something to keep an eye on.
> >
> > I think this helps with that problem in net by removing the manual rebase
> > step that people have to remember to do.
> >
> >
> > On Mon, Jun 11, 2018 at 12:04 PM, Tim Armstrong  >
> > wrote:
> >
> > > I've tried my job a few times and it's working as expected. Any
> > objections
> > > to me switching over gerrit-verify-dryrun to my approach?
> > >
> > > On Thu, Jun 7, 2018 at 2:42 PM, Tim Armstrong  >
> > > wrote:
> > >
> > >> Ok, I was able to put together a test job that does the automatic
> rebase
> > >> and carries a +2 here: https://jenkins.impala.io/job/
> > >> gerrit-verify-dryrun-tarmstrong/
> > >>
> > >> The diff in job config required to get it to work is here:
> > >> https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstron
> > >> g/jobConfigHistory/showDiffFiles?timestamp1=2018-06-07_20-
> > >> 41-18=2018-06-07_21-38-58
> > >>
> > >> I tested by creating some private drafts, adding "Impala Public
> Jenkins"
> > >> as a reviewer and testing the various scenarios.
> > >>
> > >> On Thu, Jun 7, 2018 at 2:26 PM, Jim Apple 
> wrote:
> > >>
> > >>> I agree with both of you.
> > >>>
> > >>> On nit: as GVD gets more complex, it becomes harder for new people to
> > >>> understand the messages and +Ns applied to their patches. That
> doesn't
> > >>> mean
> > >>> we shouldn't do this, only that it's something to keep an eye on.
> > >>>
> > >>> On Thu, Jun 7, 2018 at 1:28 PM, Philip Zeyliger  >
> > >>> wrote:
> > >>>
> > >>> > Seems fine, especially since we do the rebase as our submission
> > >>> strategy
> > >>> > anyway, so we're already accepting/testing something that's likely
> to
> > >>> get
> > >>> > rebased, and we may as well minimize that window.
> > >>> >
> > >>> > I'd be in favor of the bot also carrying the votes.
> > >>> >
> > >>> >
> > >>> >
> > >>> > On Thu, Jun 7, 2018 at 1:24 PM, Tim Armstrong <
> > tarmstr...@cloudera.com
> > >>> >
> > >>> > wrote:
> > >>> >
> > >>> > > One annoyance with our precommit job is the requirement to
> manually
> > >>> > rebase
> > >>> > > the change before starting the merge. Failure to do so either
> leads
> > >>> to
> > >>> > > false positives or false negatives - builds that failed because
> > they
> > >>> were
> > >>> > > missing a flaky/broken test fix and builds that succeeded despite
> > >>> > > interacting badly with a previous fix.
> > >>> > >
> > >>> > > What do people think about modifying gerrit-verify-dryrun to
> > >>> > automatically
> > >>> > > rebase the patch (by the programmatic equivalent of hitting the
> > >>> "Rebase"
> > >>> > > button) at the start of the job? The patch author would still
> have
> > to
> > >>> > carry
> > >>> > > the +2 but this might make our lives a bit easier.
> > >>> > >
> > >>> > > - Tim
> > >>> > >
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>

Re: New message on your code reviews when cherry-picking your change fails.

[Sailesh Mukil]

Thanks for doing this!

On Fri, May 11, 2018 at 12:17 PM, Tim Armstrong <tarmstr...@cloudera.com>
wrote:

> Thanks for setting this up, this should make things a lot easier.
>
> On Fri, May 11, 2018 at 12:02 PM, Philip Zeyliger <phi...@cloudera.com>
> wrote:
>
> > Hey folks,
> >
> > I just updated https://jenkins.impala.io/job/cherrypick-2.x-and-test/ to
> > comment on a code review the first time a change causes the
> cherry-picking
> > job to fail. A sample e-mail is forwarded below, with the message
> > highlighted by me. I've used Gerrit to send e-mail, which, frankly, was
> > easier to implement for me, but it does end up possibly sending to more
> > people than a more targeted approach.
> >
> > Hopefully this helps get failed cherrypicks noticed and handled
> > expediently.
> >
> > Please holler if you either notice this being buggy or if you hate it. I
> > figure we can try it for a week and rip it up if we don't like it.
> >
> > -- Philip
> >
> > -- Forwarded message --
> > From: Impala Public Jenkins (Code Review) <ger...@cloudera.org>
> > Date: Fri, May 11, 2018 at 11:58 AM
> > Subject: [Impala-ASF-CR] Update version to 3.1.0-SNAPSHOT
> > To: Sailesh Mukil <sail...@cloudera.com>, impala...@cloudera.com,
> > revi...@impala.incubator.apache.org
> >
> >
> > Impala Public Jenkins *posted comments* on this change.
> >
> > View Change <http://gerrit.cloudera.org:8080/10360>
> >
> > Patch set 2:
> >
> > This change did not cherrypick successfully into branch 2.x. To resolve
> > this, please do the cherry-pick manually and submit it to Gerrit at
> > refs/for/2.x or add an exception to the branch 2.x copy of
> > bin/ignored_commits.json. Thanks, your friendly bot at
> > https://jenkins.impala.io/job/cherrypick-2.x-and-test/479/ .
> >
> >
> > To view, visit change 10360 <http://gerrit.cloudera.org:8080/10360>. To
> > unsubscribe, visit settings <http://gerrit.cloudera.org:8080/settings>.
> > Gerrit-Project: Impala-ASF
> > Gerrit-Branch: master
> > Gerrit-MessageType: comment
> > Gerrit-Change-Id: Ia2e2df73d27fa332d17fec4e9aa469ea91b14167
> > Gerrit-Change-Number: 10360
> > Gerrit-PatchSet: 2
> > Gerrit-Owner: Sailesh Mukil <sail...@cloudera.com>
> > Gerrit-Reviewer: Impala Public Jenkins <impala-public-jenkins@
> cloudera.com
> > >
> > Gerrit-Reviewer: Michael Brown <mi...@cloudera.com>
> > Gerrit-Reviewer: Philip Zeyliger <phi...@cloudera.com>
> > Gerrit-Comment-Date: Fri, 11 May 2018 18:58:13 +
> > Gerrit-HasComments: No
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "impala-cr" group.
> > To unsubscribe from this group and stop receiving emails from it, send an
> > email to impala-cr+unsubscr...@cloudera.com.
> > For more options, visit https://groups.google.com/a/
> cloudera.com/d/optout.
> >
>

Re: [ANNOUNCE] Apache Impala 3.0.0 release

[Sailesh Mukil]

My pleasure!

On Thu, May 10, 2018 at 9:16 AM, Tim Armstrong <tarmstr...@cloudera.com>
wrote:

> Thanks Sailesh!
>
> On Wed, May 9, 2018 at 10:09 PM, Jim Apple <jbap...@cloudera.com> wrote:
>
> > Thanks to Sailesh for managing the release, and thanks to the many, many
> > others who contributed to creating this new branch and maintaining the
> > relationship between the two branches.
> >
> > On Wed, May 9, 2018 at 1:37 PM, Sailesh Mukil <sail...@apache.org>
> wrote:
> >
> > > The Apache Impala team is pleased to announce the release of Impala
> > 3.0.0.
> > >
> > > Impala is a high-performance distributed SQL engine.
> > >
> > > The release is available at: https://impala.apache.org/downloads.html
> > >
> > > Thanks,
> > >
> > > The Apache Impala team
> > >
> > >
> > >
> > >
> >
>

Re: Jenkins Upgrade

[Sailesh Mukil]

Jenkins is back up.

On Wed, May 9, 2018 at 9:47 PM, Sailesh Mukil <sail...@cloudera.com> wrote:

> I'll be performing a Jenkins upgrade shortly, and will have it in the
> "Prepare for restart" mode. I'll send in a mail when the upgrade is done.
>

Jenkins Upgrade

[Sailesh Mukil]

I'll be performing a Jenkins upgrade shortly, and will have it in the
"Prepare for restart" mode. I'll send in a mail when the upgrade is done.

Re: Maintaining Impala Documentation Sets for Releases

[Sailesh Mukil]

#3 is best. I'm doing something similar for the maintaining the latest
releases for 2.x and 3.x in the "Downloads" page.

Patch: https://gerrit.cloudera.org/#/c/10333/

On Mon, May 7, 2018 at 4:18 PM, Lars Volker  wrote:

> I'm in favor of #3, too. Ideally we'd have proper pages for both releases,
> including docs, and downloads.
>
> On Mon, May 7, 2018 at 4:14 PM, Jim Apple  wrote:
>
> > I'd advocate #3, at least until we are no longer regularly making new 2.x
> > releases.
> >
> > On Mon, May 7, 2018 at 3:06 PM, Alexandra Rodoni 
> > wrote:
> >
> > > I would like to get your feedback on how to manage Impala doc site when
> > we
> > > have 2 parallel release streams, 2.x and 3.x.
> > >
> > > Currently, https://impala.apache.org/impala-docs.html has the HTML
> > version
> > > of the latest version doc set. For older versions, only the PDF
> versions
> > > are available.
> > >
> > > Options are:
> > >
> > > 1. Have the latest HTML version doc available. This can look confusing,
> > for
> > > example, when 2.13 shows as the latest, but 3.0 is listed in the Older
> > > Release section.
> > >
> > > 2. Have the highest release number as the latest and offer the HTML for
> > > that version. So, 3.0 will still be the latest release when 2.13 comes
> > out.
> > > And 2.13 docs will be only available in PDF when released.
> > >
> > > 3. Revamp the doc page and the workflow so that we can have both 2.x
> and
> > > 3.x docs in HTML.
> > >
> > > Let me know what you think is the best option.
> > >
> > > Thanks.
> > > alex rodoni
> > >
> >
>

[RESULT] Vote on Impala 3.0.0 RC1

[Sailesh Mukil]

https://lists.apache.org/thread.html/c1be7c5d3b69986c854b7f17d616c7c454cac77a9d13bb1041b371ce@%3Cdev.impala.apache.org%3E

The vote has passed with the following tally:

+1 (binding):
- Jim Apple
- Thomas Tauber-Marshall
- Sailesh Mukil
- Lars Volker
- Philip Zeyliger

-1:  None


Thank you for participating in the vote!

- Sailesh

Re: 2.x branch divergence, its implications and what you can do.

[Sailesh Mukil]

On Thu, May 3, 2018 at 12:17 PM, Lars Volker <l...@cloudera.com> wrote:

> Should we add a check to the master GVO that makes sure that a change
> either applies to 2.x, or has a "not for 2.x" label?
>
>
I like this idea. Let me open a JIRA for it.


> On Thu, May 3, 2018 at 12:12 PM, Sailesh Mukil <sail...@cloudera.com>
> wrote:
>
> > It's been a while since we started maintaining 2 active branches, master
> > and 2.x, and the divergence between them has grown quite a bit.
> >
> > The implications of this (as Michael Brown brought to my attention) are
> > that a patch that goes into 'master' may not cleanly apply to '2.x',
> which
> > would fail the cherrypick-2.x-and-test(
> > https://jenkins.impala.io/job/cherrypick-2.x-and-test/) job, and if
> > unnoticed, can hold up all future cherry-picks from 'master' to '2.x'
> until
> > this is resolved.
> >
> > If this does happen, the original author of the patch needs to manually
> > cherry-pick the patch from 'master' to '2.x', resolve the conflicts and
> > push a change for review to the '2.x' branch with the same Change-Id.
> >
> > To avoid these types of failures, I advise authors to locally cherry pick
> > their patch to '2.x' to see if it applies cleanly before running GVO.
> >
> > Lastly, if a patch is meant only for 'master', and not for '2.x', please
> > remember to add the "Cherry-picks: not for 2.x" line to your commit
> > message. There have been a few patches in which this was not included,
> > which unfortunately requires manual work to resolve later on.
> >
> > Thank you for your time!
> >
> > - Sailesh
> >
>

Re: [VOTE] 3.0.0 release candidate 1

[Sailesh Mukil]

+1 (binding)

- Validated the sha512 file
- Validated the GPG signature
- Ran the release test job:
https://jenkins.impala.io/view/Utility/job/release-test-ub1604/12/

On Wed, May 2, 2018 at 9:17 PM, Jim Apple <jbap...@cloudera.com> wrote:

> +1, binding
>
> I looked at the output of the release testing job that Sailesh appears to
> have run on jenkins.impala.io.
>
> On Tue, May 1, 2018 at 4:34 PM Sailesh Mukil <sail...@cloudera.com> wrote:
>
> > This is a vote to release Impala 3.0.0
> >
> > The artifacts for testing can be downloaded from:
> > https://dist.apache.org/repos/dist/dev/impala/3.0.0/RC1/
> > Git tag: 3.0.0-rc1
> > Tree hash: d5bf1fdeb20ac332ce48c6a2b0588bcaf7ac9cb0
> >
> > Please vote +1 or -1. -1 votes should be accompanied by an explanation of
> > the reason. Only PMC members have binding votes, but other
> > community members are encouraged to cast non-binding votes. This vote
> will
> > pass if there are 3 binding +1 votes and more binding +1 votes than -1
> > votes.
> >
> > Please also note that this release is a new major release, and does not
> > override the recently released 2.12.0 as the latest release. Rather,
> 2.12.0
> > is the latest release from the 2.x branch and 3.0.0 is the first and
> latest
> > release from the 3.x branch.
> >
> > This wiki page describes how to check the release before you vote:
> > https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#
> >
> > The vote will be open until 04:30 PM PST on Friday, May 4
> >
>

[VOTE] 3.0.0 release candidate 1

[Sailesh Mukil]

This is a vote to release Impala 3.0.0

The artifacts for testing can be downloaded from:
https://dist.apache.org/repos/dist/dev/impala/3.0.0/RC1/
Git tag: 3.0.0-rc1
Tree hash: d5bf1fdeb20ac332ce48c6a2b0588bcaf7ac9cb0

Please vote +1 or -1. -1 votes should be accompanied by an explanation of
the reason. Only PMC members have binding votes, but other
community members are encouraged to cast non-binding votes. This vote will
pass if there are 3 binding +1 votes and more binding +1 votes than -1
votes.

Please also note that this release is a new major release, and does not
override the recently released 2.12.0 as the latest release. Rather, 2.12.0
is the latest release from the 2.x branch and 3.0.0 is the first and latest
release from the 3.x branch.

This wiki page describes how to check the release before you vote:
https://cwiki.apache.org/confluence/display/IMPALA/How+to+Release#

The vote will be open until 04:30 PM PST on Friday, May 4

Re: Re: Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

[Sailesh Mukil]

An update:

Impala 2.12.0 has been released.

All remaining tasks for Impala 3.0.0 have been completed. The release is
now in preparation and a vote will commence soon.

On Fri, Apr 13, 2018 at 11:51 AM, Sailesh Mukil <sail...@cloudera.com>
wrote:

> Thank you all.
>
> I've triaged all the remaining open JIRAs for Impala 2.12.0 and Impala 3.0
> and we have 2 more breaking changes that need to go in for Impala 3.0:
> https://issues.apache.org/jira/browse/IMPALA-6340
> https://issues.apache.org/jira/browse/IMPALA-5893
>
> There are also a couple of patches that are very close to completion that
> will make it in:
> https://issues.apache.org/jira/browse/IMPALA-6790
> https://issues.apache.org/jira/browse/IMPALA-6793
>
> And lastly, there are a few docs JIRAs that are being worked on for Impala
> 3.0 and Impala 2.12.0.
>
> IMPALA-6340 is probably the most important change at this point, since
> DECIMAL_V2 is enabled by default in Impala 3.0 and this is a blocker bug
> that needs to be fixed to allow the feature to be used reliably.
>
> All the above JIRAs are being worked on as top priorities and we should
> have them completed soon. I will start the remaining formalities as soon as
> those changes are in.
>
> Full list of open 2.12.0 JIRAs:
> https://issues.apache.org/jira/browse/IMPALA-6793?jql=
> project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%22%20%3D%20%
> 22Impala%202.12.0%22
>
> Full list of open 3.0 JIRAs:
> https://issues.apache.org/jira/browse/IMPALA-6793?jql=
> project%20%3D%20IMPALA%20AND%20status%20in%20(Open%2C%20%
> 22In%20Progress%22%2C%20Reopened)%20AND%20%22Target%20Version%22%20%3D%20%
> 22Impala%203.0%22
>
> Thanks to all the authors and reviewers of the above JIRAs for working on
> them in a timely manner.
>
> - Sailesh
>
> On Tue, Apr 10, 2018 at 11:28 AM, Tim Armstrong <tarmstr...@cloudera.com>
> wrote:
>
>> We also just merged a small breaking change to pave the way for Jinchul
>> Kim's patch: https://gerrit.cloudera.org/#/c/9957/ (otherwise we might
>> have
>> to wait for the next major release to merge that).
>>
>> On Mon, Apr 2, 2018 at 8:33 PM, Quanlong Huang <huang_quanl...@126.com>
>> wrote:
>>
>> > Thanks, Tim and Jim! Then I have no other suggestions.+1
>> >
>> > At 2018-04-03 10:57:58, "Jim Apple" <jbap...@cloudera.com> wrote:
>> > >Additionally, because this patch doesn't break any backwards
>> > compatibility,
>> > >it's fine to merge it for 2.13 and 3.1 when those releases are cut. And
>> > >since there are no minimum waiting period between releases, we could
>> cut
>> > >those right after that patch lands if we like.
>> > >
>> > >On Mon, Apr 2, 2018 at 4:05 PM, Tim Armstrong <tarmstr...@cloudera.com
>> >
>> > >wrote:
>> > >
>> > >> I don't think we should hold the release for the ORC scanner patch -
>> > it's a
>> > >> great feature, but there are many other equally important features
>> and
>> > >> improvements that have been waiting for months to be released.
>> > >>
>> > >> Generally, there are always going to be big changes that are "nearly
>> > there"
>> > >> and I think we're best off releasing frequently rather than trying to
>> > hold
>> > >> releases for specific features. This is especially true for large
>> > features
>> > >> where it sometimes take a bit of bake time to flush out all of the
>> > issues
>> > >> with the initial patch.
>> > >>
>> > >> - Tim
>> > >>
>> > >> On Mon, Apr 2, 2018 at 3:49 PM, Quanlong Huang <
>> huang_quanl...@126.com>
>> > >> wrote:
>> > >>
>> > >> > Hi friends,
>> > >> >
>> > >> >
>> > >> > Is it possible to wait until IMPALA-5717 (
>> > https://gerrit.cloudera.org/#
>> > >> > /c/9134/) is merged? We're eager to upgrade our Impala cluster with
>> > the
>> > >> > latest ORC-support. I've heard from the community that many other
>> > >> companies
>> > >> > are waiting for it too. As it's an experimental feature so far, we
>> can
>> > >> get
>> > >> > more feedback on this release.
>> > >> >
>> > >> >
>> > >> > As Tim said, this patch is closed to be merged. Could you wait
>> until
>> > it's
>> > >> > merged?
>> > >> >
>> > >> >
>> > >> > Thanks,
>> > >> > Quanlong
>> > >> >
>> > >> > At 2018-04-03 05:56:11, "Jim Apple" <jbap...@cloudera.com> wrote:
>> > >> > >agreed.
>> > >> > >
>> > >> > >On Mon, Apr 2, 2018 at 2:50 PM, Philip Zeyliger <
>> phi...@cloudera.com
>> > >
>> > >> > wrote:
>> > >> > >
>> > >> > >> Sounds like a good plan to me!
>> > >> > >>
>> > >> > >> -- Philip
>> > >> > >>
>> > >> >
>> > >>
>> >
>>
>
>

Re: [ANNOUNCE] Apache Impala 2.12.0 release

[Sailesh Mukil]

My pleasure!

On Mon, Apr 30, 2018 at 11:19 AM, Jim Apple <jbap...@cloudera.com> wrote:

> -user, -announce
>
> Thanks for driving this, Sailesh!
>
> On Mon, Apr 30, 2018 at 10:03 AM, Sailesh Mukil <sail...@apache.org>
> wrote:
>
> > The Apache Impala team is pleased to announce the release of Impala
> 2.12.0.
> >
> > Impala is a high-performance distributed SQL engine.
> >
> > The release is available at: https://impala.apache.org/downloads.html
> >
> > Thanks,
> >
> > The Apache Impala team
> >
> >
> >
>

[ANNOUNCE] Apache Impala 2.12.0 release

[Sailesh Mukil]

The Apache Impala team is pleased to announce the release of Impala 2.12.0.

Impala is a high-performance distributed SQL engine.

The release is available at: https://impala.apache.org/downloads.html

Thanks,

The Apache Impala team

[RESULT] Vote on Impala 2.12.0 RC 1

[Sailesh Mukil]

https://lists.apache.org/thread.html/bdbb9d74834a5db53ee6565958044f3af1d4160f45cf0024393a4fe2@%3Cdev.impala.apache.org%3E

The vote has passed with the following tally:

+1 (binding):
* Lars Volker
* Thomas Tauber-Marshall
* Jim Apple
* Sailesh Mukil

-1 (binding) - None
(There was one which was changed to a +1 in the same vote)

0 - None

+1 (non-binding):
* 张启凡

Thank you everyone for testing and participating in the vote.

- Sailesh

Re: JIRA necromancy

[Sailesh Mukil]

Yes to this.

On Tue, Apr 17, 2018 at 11:26 AM, Alexander Behm 
wrote:

> I agree. Reopening can be very confusing.
>
> On Tue, Apr 17, 2018 at 11:20 AM, Jim Apple  wrote:
>
> > I'm convinced.
> >
> > On Tue, Apr 17, 2018 at 10:29 AM, Tim Armstrong  >
> > wrote:
> > > I noticed that there's been a trend recently towards reopening old
> issues
> > > instead of filing new issues. Not trying to pick on anyone but it seems
> > > like its worth having a discussion about best practices.
> > >
> > > Personally I think reopening JIRAs is often a bad thing for a several
> > > reasons:
> > >
> > > * We don't tend to properly triage the issue to determine if it is
> > actually
> > > has same root cause as the old one. E.g. the same test fails for two
> > > completely different reasons.
> > > * People are tempted to skimp on including diagnostic information.
> > > * It gets confusing trying to figure out which version the issue was
> > fixed
> > > in, particularly if the new thing turns out to be a separate issue.
> > > * The target version, fix version, priority, etc is wrong
> > > * It automatically ends up on the plate of whoever last fixed it,
> rather
> > > than whoever currently has bandwidth. This is particularly bad for
> anyone
> > > who has fixed or tried to fix a lot of flaky tests over the last year
> or
> > > two (e.g. me).
> > >
> > > I'd prefer if we opened new issues by default unless we're really
> > confident
> > > that it's the same issue. It's much easier to mark issues as duplicates
> > > than it is to separate out two distinct issues tracked by one JIRA.
> Even
> > if
> > > we're pretty sure it's the same thing, I think we should think
> carefully
> > > before re-opening issues from previous releases.
> > >
> > > Anyway, this is just my opinion. Do others agree or disagree?
> > >
> > > - Tim
> >
>

[DISCUSS] Apache Impala 2.12.0 and 3.0 releases

[Sailesh Mukil]

Folks,

It's been about 3 months since our last release and there have been a good
number of bug fixes, features added and performance improvements since then.

I propose that we have a 2.12.0 release soon and a 3.0 release in parallel.
I, as a PMC of Apache Impala volunteer to mange both these releases.

Why have 2 releases in parallel?

We've been maintaining 2 branches in parallel for a while now. A 2.x branch
which works with Hadoop 2.0 and friends, and a master branch, which now
works with Hadoop 3.0 and friends.
We've also made sure to not push any incompatible changes into our 2.x
branch, but only to the master branch, and we have a good number of these
incompatible changes on master now that justifies having a 3.0 release.

There have been 351 commits on master since the last release, Apache Impala
2.11.0. I've gone through them and outlined the commits that are either
considered breaking changes or support a breaking change:

IMPALA-4277: Switch to using Hadoop 3 and friends by default.

IMPALA-6510: [DOCS] Remove refresh_after_connect

IMPALA-6647: Add CREATE fine-grained privilege

IMPALA-6230, IMPALA-6468: Fix the output type of round() and related fns

IMPALA-4277: Support multiple versions of Hadoop ecosystem

IMPALA-6415: [DOCS] fix invalid 

IMPALA-6643: Add REFRESH fine-grained privilege

IMPALA-5191, IMPALA-6415: [DOCS] Document breaking change of alias and
ordinal substitution

IMPALA-3916: [DOCS] Reserved keywords updated for Impala 3.0

IMPALA-6240: [DOCS] Document PARQUET_ARRAY_RESOLUTION query option

[DOCS] Removed the obsolete Llama options files

IMPALA-6077: remove Parquet BIT_PACKED def level support

IMPALA-3271: organise and warn on removed startup flags

IMPALA-5037: Default PARQUET_ARRAY_RESOLUTION=THREE_LEVEL

IMPALA-5293: Turn insert clustering on by default

IMPALA-4924 addendum: Change result type to decimal in a TPCH query

IMPALA-3916: Reserve SQL:2016 reserved words   (Disabled on 2.x with
IMPALA-3916: Change reserved_words_version to 2.11.0)

IMPALA-4319: remove some deprecated query options

IMPALA-4924: Enable Decimal V2 by default

IMPALA-5191: Standardize column alias behavior

IMPALA-3998: Remove refresh_after_connect option from shell

The 3.0 release would come off the master branch and the 2.12.0 release
would come off the 2.x branch.

Please speak up and let the community know if anyone has any objections to
this.

P.S: This is not a VOTE thread but only a DISCUSS thread.

Thanks,
Sailesh

Re: Help with fixing clang-diagnostic-over-aligned

[Sailesh Mukil]

Thanks for the explanation Todd and Dan! I had introduced a new
CACHELINE_ALIGNED object into the class which caused this problem. Making
the parent classes CACHELINE_ALIGNED too fixed the problem.

On Wed, Feb 14, 2018 at 12:58 PM, Todd Lipcon <t...@cloudera.com> wrote:

> This probably means that QueryExecMgr (or one of its descendent members) is
> annotated with __attribute__((aligned(64)), perhaps through something like
> a CACHELINE_ALIGNED macro.
>
> The issue is that, as the warning says, normal libc malloc doesn't make any
> hard guarantees about the alignment of allocations, and the way that
> 'operator new' is implemented by default only gets the size and not the
> desired allocation. I believe C++17 is adding support for an 'operator new'
> overload that also passes the desired alignment, so that it can call down
> into posix_memalign and ensure the appropriate allocation.
>
> The "correct" fix prior to C++17 is probably to do something like overload
> 'operator new' for the type, or to use posix_memalign manually to allocate
> aligned memory, and then placement-new it into that memory.
>
> Another reasonable fix would probably be to add a CHECK in the constructor
> that 'this' is aligned as desired so that, if you ever run on an allocator
> that isn't guaranteeing what you think it is, you'll know immediately
> rather than getting a surprising perf or atomicity regression due to
> cross-cacheline loads/stores.
>
> -Todd
>
>
> On Wed, Feb 14, 2018 at 12:33 PM, Sailesh Mukil <sail...@cloudera.com>
> wrote:
>
> > Does anyone have experience with fixing the following type of
> > clang-tidy warning?
> >
> >
> > /home/ubuntu/Impala/be/src/runtime/exec-env.cc:159:21: warning: type
> > 'impala::QueryExecMgr' requires 64 bytes of alignment and the default
> > allocator only guarantees 16 bytes [clang-diagnostic-over-aligned]
> > query_exec_mgr_(new QueryExecMgr()),
> >
> >
> > /home/ubuntu/Impala/be/src/service/impalad-main.cc:80:49: warning:
> > type 'impala::ImpalaServer' requires 64 bytes of alignment and the
> > default allocator only guarantees 16 bytes
> > [clang-diagnostic-over-aligned]
> >   boost::shared_ptr impala_server(new
> > ImpalaServer(_env));
> >
>
>
>
> --
> Todd Lipcon
> Software Engineer, Cloudera
>

Help with fixing clang-diagnostic-over-aligned

[Sailesh Mukil]

Does anyone have experience with fixing the following type of
clang-tidy warning?


/home/ubuntu/Impala/be/src/runtime/exec-env.cc:159:21: warning: type
'impala::QueryExecMgr' requires 64 bytes of alignment and the default
allocator only guarantees 16 bytes [clang-diagnostic-over-aligned]
query_exec_mgr_(new QueryExecMgr()),


/home/ubuntu/Impala/be/src/service/impalad-main.cc:80:49: warning:
type 'impala::ImpalaServer' requires 64 bytes of alignment and the
default allocator only guarantees 16 bytes
[clang-diagnostic-over-aligned]
  boost::shared_ptr impala_server(new ImpalaServer(_env));

Re: thrift-server-test

[Sailesh Mukil]

Your principal isn't getting cached for some reason. The same on my machine
looks like:

$ klist /tmp/krb5cc_impala_internal

Ticket cache: FILE:/tmp/krb5cc_impala_internal

Default principal: impala/localh...@krbtest.com


Valid starting   Expires  Service principal

01/08/2018 15:49:30  01/09/2018 15:49:30  krbtgt/krbtest@krbtest.com

renew until 01/15/2018 15:49:30

01/08/2018 15:49:30  01/09/2018 15:49:30  impala/localhost@

renew until 01/15/2018 15:49:30

01/08/2018 15:49:30  01/09/2018 15:49:30  impala/localh...@krbtest.com

renew until 01/15/2018 15:49:30

Did you check if your user has appropriate permissions on the file? Looks
like only the MiniKDC was able to write to it, and maybe your 'kinit'
couldn't and silently failed (which should be a bug if it didn't throw an
error) ?

On Mon, Jan 8, 2018 at 3:40 PM, Philip Zeyliger <phi...@cloudera.com> wrote:

> Hi Sailiesh,
>
> Is this what you'd expect?
>
> $klist /tmp/krb5cc_impala_internal
> Ticket cache: FILE:/tmp/krb5cc_impala_internal
> Default principal: impala/localh...@krbtest.com
>
> Valid starting   Expires  Service principal
> 01/08/2018 15:39:23  01/09/2018 15:39:23  krbtgt/krbtest@krbtest.com
> renew until 01/15/2018 15:39:23
>
> Thanks!
>
>
> On Mon, Jan 8, 2018 at 12:20 PM, Sailesh Mukil <sail...@cloudera.com>
> wrote:
>
> > Can you run the test again, and klist the contents of the credential
> cache
> > and post the error logs again? Looks like "impala/localhost" might not be
> > stored as expected in the cache on your machine.
> >
> > On Wed, Dec 13, 2017 at 2:47 PM, Philip Zeyliger <phi...@cloudera.com>
> > wrote:
> >
> > > The KDC in this case is the "minikdc" from
> > > https://github.com/apache/impala/blob/master/be/src/
> > > kudu/security/test/mini_kdc.cc.
> > > I see evidence of it, and have been able to look at its configuration
> by,
> > > um, adding --gtest_break_on_failure. (The feature actually doesn't
> work,
> > > presumably because of an interaction with breakpad, but a temporary
> > > directory is left on my filesystem, so that's nice.)
> > >
> > > -- Philip
> > >
> > > On Tue, Dec 12, 2017 at 4:08 PM, Evo Eftimov <evo.efti...@isecc.com>
> > > wrote:
> > >
> > > > Is your cluster Kerberized at all, especially the Impala daemon - it
> > > > doesn’t seem to be enrolled in the KDC at all
> > > >
> > > > You / your personal account/principal is definitely enrolled though
> > > >
> > > > And there is definetly a KDC in your environment
> > > >
> > > > -Original Message-
> > > > From: Philip Zeyliger [mailto:phi...@cloudera.com]
> > > > Sent: Tuesday, December 12, 2017 11:26 PM
> > > > To: dev@impala.apache.org
> > > > Subject: thrift-server-test
> > > >
> > > > Hi folks,
> > > >
> > > > I've been running into issues with thrift-server-test and Kerberos.
> > Below
> > > > is an excerpt of "KRB5_TRACE=/dev/stderr be/build/debug/rpc/thrift-
> > > server-test";
> > > > both SslConnectivity/1 and
> > > > SslConnectivity/2 fail the same way.
> > > >
> > > > I'm running Ubuntu16.04. I've seen this both on my host, as well as
> > > inside
> > > > of an Ubuntu 16.04 Docker container.
> > > >
> > > > Does this ring any bells?
> > > >
> > > > Thanks!
> > > >
> > > > -- Philip
> > > >
> > > >
> > > > [ RUN  ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> > > SslConnectivity/2
> > > > Loading random data
> > > > Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for
> > realm
> > > '
> > > > KRBTEST.COM',
> > > > master key name 'K/m...@krbtest.com'
> > > > [31585] 1513120922.459517: Retrieving K/m...@krbtest.com from
> > > > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> > with
> > > > result: 0/Success
> > > > [31586] 1513120922.472314: Retrieving K/m...@krbtest.com from
> > > > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0)
> > with
> > > > result: 0/Success
> > > > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > > setting
> > > > up network...
> > > > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> &

Re: thrift-server-test

[Sailesh Mukil]

Can you run the test again, and klist the contents of the credential cache
and post the error logs again? Looks like "impala/localhost" might not be
stored as expected in the cache on your machine.

On Wed, Dec 13, 2017 at 2:47 PM, Philip Zeyliger 
wrote:

> The KDC in this case is the "minikdc" from
> https://github.com/apache/impala/blob/master/be/src/
> kudu/security/test/mini_kdc.cc.
> I see evidence of it, and have been able to look at its configuration by,
> um, adding --gtest_break_on_failure. (The feature actually doesn't work,
> presumably because of an interaction with breakpad, but a temporary
> directory is left on my filesystem, so that's nice.)
>
> -- Philip
>
> On Tue, Dec 12, 2017 at 4:08 PM, Evo Eftimov 
> wrote:
>
> > Is your cluster Kerberized at all, especially the Impala daemon - it
> > doesn’t seem to be enrolled in the KDC at all
> >
> > You / your personal account/principal is definitely enrolled though
> >
> > And there is definetly a KDC in your environment
> >
> > -Original Message-
> > From: Philip Zeyliger [mailto:phi...@cloudera.com]
> > Sent: Tuesday, December 12, 2017 11:26 PM
> > To: dev@impala.apache.org
> > Subject: thrift-server-test
> >
> > Hi folks,
> >
> > I've been running into issues with thrift-server-test and Kerberos. Below
> > is an excerpt of "KRB5_TRACE=/dev/stderr be/build/debug/rpc/thrift-
> server-test";
> > both SslConnectivity/1 and
> > SslConnectivity/2 fail the same way.
> >
> > I'm running Ubuntu16.04. I've seen this both on my host, as well as
> inside
> > of an Ubuntu 16.04 Docker container.
> >
> > Does this ring any bells?
> >
> > Thanks!
> >
> > -- Philip
> >
> >
> > [ RUN  ] KerberosOnAndOff/ThriftKerberizedParamsTest.
> SslConnectivity/2
> > Loading random data
> > Initializing database '7abf-cef9-113e-eae3/krb5kdc/principal' for realm
> '
> > KRBTEST.COM',
> > master key name 'K/m...@krbtest.com'
> > [31585] 1513120922.459517: Retrieving K/m...@krbtest.com from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > [31586] 1513120922.472314: Retrieving K/m...@krbtest.com from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> setting
> > up network...
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > listening on fd 11: udp 0.0.0.0.51781 (pktinfo)
> > krb5kdc: setsockopt(12,IPV6_V6ONLY,1) worked Dec 12 15:22:02
> > philip-dev.gce.cloudera.com krb5kdc[31586](info): listening on fd 12:
> udp
> > ::.51781 (pktinfo) Dec 12 15:22:02 philip-dev.gce.cloudera.com
> > krb5kdc[31586](info): set up 2 sockets Dec 12 15:22:02
> > philip-dev.gce.cloudera.com krb5kdc[31586](info):
> > commencing operation
> > krb5kdc: starting...
> > Authenticating as principal philip/ad...@krbtest.com with password.
> > [31589] 1513120922.498913: Retrieving K/m...@krbtest.com from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > WARNING: no policy specified for impala/localh...@krbtest.com;
> defaulting
> > to no policy Principal "impala/localh...@krbtest.com" created.
> > Authenticating as principal philip/ad...@krbtest.com with password.
> > [31590] 1513120922.508777: Retrieving K/m...@krbtest.com from
> > FILE:7abf-cef9-113e-eae3/krb5kdc/.k5.KRBTEST.COM (vno 0, enctype 0) with
> > result: 0/Success
> > Entry for principal impala/localhost with kvno 2, encryption type
> > aes256-cts-hmac-sha1-96 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > aes128-cts-hmac-sha1-96 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > des3-cbc-sha1 added to keytab
> > WRFILE:7abf-cef9-113e-eae3/krb5kdc/impala_localhost.keytab.
> > Entry for principal impala/localhost with kvno 2, encryption type
> > arcfour-hmac added to keytab WRFILE:7abf-cef9-113e-eae3/
> > krb5kdc/impala_localhost.keytab.
> > Dec 12 15:22:02 philip-dev.gce.cloudera.com krb5kdc[31586](info): AS_REQ
> > (6 etypes {18 17 16 23 25 26}) 127.0.0.1: ISSUE: authtime 1513120922,
> > etypes
> > {rep=18 tkt=18 ses=18}, impala/localh...@krbtest.com for krbtgt/
> > krbtest@krbtest.com [31476] 1513120922.532304: ccselect can't find
> > appropriate cache for server principal impala@localhost [31476]
> > 1513120922.532347: Getting credentials impala/localh...@krbtest.com
> > -> impala@localhost using ccache FILE:/tmp/krb5cc_impala_internal
> > [31476] 1513120922.532382: Retrieving impala/localh...@krbtest.com ->
> > impala@localhost from FILE:/tmp/krb5cc_impala_internal with result:
> > -1765328243/Matching credential not found [31476] 1513120922.532407:
> > Retrieving impala/localh...@krbtest.com -> krbtgt/localhost@localhost
> > 

Re: build broken

[Sailesh Mukil]

On Wed, Dec 6, 2017 at 9:16 AM, Tim Armstrong <tarmstr...@cloudera.com>
wrote:

> I wonder if the "Rebase always" merge strategy would help with this
> scenario:
> https://gerrit-review.googlesource.com/Documentation/project-
> configuration.html#submit_type
>
>
"Rebase Always" seems like it would fix the above scenario. If the child
isn't merged, it wouldn't merge both the patches and it would fall back to
the client having to manually merge both the patches.

I recall doing a similar thing quite a while back (i.e. GVO-ing 2 patches
in one go), and the behavior was similar to "Rebase Always", so I had to
manually merge the patches after the GVO succeeded, which seems reasonable
as long as the GVO passes. Not sure if our gerrit settings changed at some
point.


> On Wed, Dec 6, 2017 at 9:04 AM, Daniel Hecht <dhe...@cloudera.com> wrote:
>
> > Sailesh, please keep that in mind with the ServicePool change -- you'll
> > need to explicitly merge part 1 first.
> >
>

Yes, I'll make sure to do that. Thanks.


> > On Wed, Dec 6, 2017 at 8:41 AM, Tim Armstrong <tarmstr...@cloudera.com>
> > wrote:
> >
> > > I think that's a consequence of the "cherry pick" merge strategy. It
> does
> > > seem like a flaw in our merge process. It would be nice if we could
> > > configure gerrit so that it didn't merge patches where the parent
> isn't a
> > > commit on the master branch.
> > >
> > > On Tue, Dec 5, 2017 at 9:31 PM, Sailesh Mukil <sail...@cloudera.com>
> > > wrote:
> > >
> > > > Sorry about that. I thought they would both be merged together, but
> it
> > > > looks like that wasn't the case.
> > > >
> > > > On Tue, Dec 5, 2017 at 6:22 PM, Michael Ho <k...@cloudera.com>
> wrote:
> > > >
> > > > > Thanks Tim for fixing it and Jin Chul for investigating the
> problem.
> > > > Sorry
> > > > > for missing that during code review.
> > > > >
> > > > > On Tue, Dec 5, 2017 at 6:00 PM, Tim Armstrong <
> > tarmstr...@cloudera.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Thanks for investigating Jin Chul. I came to the same conclusion
> > and
> > > > > merged
> > > > > > the change.
> > > > > >
> > > > > > On Tue, Dec 5, 2017 at 5:57 PM, Jin Chul Kim <jinc...@gmail.com>
> > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > Currently Impala build is broken because the child change was
> > > merged
> > > > > > > without the merge of the parent change.
> > > > > > >
> > > > > > > (parent) https://gerrit.cloudera.org/#/c/8760/: [security]
> Make
> > > the
> > > > > > > kerberos principal configurable for Kudu servers
> > > > > > >
> > > > > > > (child) https://gerrit.cloudera.org/#/c/8761/: IMPALA-6256:
> > > > Incorrect
> > > > > > > principal will be used for internal connections if
> > > FLAGS_be_principal
> > > > > is
> > > > > > > set
> > > > > > >
> > > > > > > A workaround: cherry-pick the parent change.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Jinchul
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks,
> > > > > Michael
> > > > >
> > > >
> > >
> >
>

Re: build broken

[Sailesh Mukil]

Sorry about that. I thought they would both be merged together, but it
looks like that wasn't the case.

On Tue, Dec 5, 2017 at 6:22 PM, Michael Ho  wrote:

> Thanks Tim for fixing it and Jin Chul for investigating the problem. Sorry
> for missing that during code review.
>
> On Tue, Dec 5, 2017 at 6:00 PM, Tim Armstrong 
> wrote:
>
> > Thanks for investigating Jin Chul. I came to the same conclusion and
> merged
> > the change.
> >
> > On Tue, Dec 5, 2017 at 5:57 PM, Jin Chul Kim  wrote:
> >
> > > Hi,
> > >
> > > Currently Impala build is broken because the child change was merged
> > > without the merge of the parent change.
> > >
> > > (parent) https://gerrit.cloudera.org/#/c/8760/: [security] Make the
> > > kerberos principal configurable for Kudu servers
> > >
> > > (child) https://gerrit.cloudera.org/#/c/8761/: IMPALA-6256: Incorrect
> > > principal will be used for internal connections if FLAGS_be_principal
> is
> > > set
> > >
> > > A workaround: cherry-pick the parent change.
> > >
> > > Best regards,
> > > Jinchul
> > >
> >
>
>
>
> --
> Thanks,
> Michael
>

Re: [DISCUSS] 2.11.0 release

[Sailesh Mukil]

+1

Thanks for volunteering.

On Thu, Nov 30, 2017 at 1:02 PM, Jim Apple  wrote:

> +1
>
> On Thu, Nov 30, 2017 at 12:46 PM, Thomas Tauber-Marshall <
> tmarsh...@cloudera.com> wrote:
>
> > Folks,
> >
> > It has been over 2 months since we released Apache Impala 2.10.0 and
> there
> > have been new feature improvements and a good number of bug fixes checked
> > in since then.
> >
> > I propose that we release 2.11.0 soon and I volunteer to be its release
> > manager. Please speak up and let the community know if anyone has any
> > objections to this.
> >
> > Thanks,
> > Thomas
> >
>

Re: Impala minidump dump_breakpad_symbols

[Sailesh Mukil]

Hi Sandish,

Have you tried the steps outlined here?
https://cwiki.apache.org/confluence/display/IMPALA/Debugging+Impala+Minidumps

If it still doesn't work, as Brock mentioned, please paste portions of
errors or any other helpful output that you see.

- Sailesh



On Mon, Nov 27, 2017 at 8:23 PM, Brock Noland  wrote:

> Hi Sandish,
>
> I don't know how to help you, but I think it'd be useful if you shared a
> portion of the results file for folks to look at. Note that attachments
> don't work well so I'd just paste a portion of it inline.
>
> On Mon, Nov 27, 2017 at 9:15 PM, sanysand...@gmail.com <
> sanysand...@gmail.com> wrote:
>
> > Hi Team,
> >
> > I'm trying to get minidump_stackwalk results with dump breakpad symbols,
> > but symbols are not showing in results file, below are the steps which I
> > followed. Can someone verify or give me steps to get symbols in
> > minidump_stackwalk results ??
> >
> >
> > Steps:
> > dump_breakpad_symbols.py -b be/build/ --dump_syms
> > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e613457e
> 55f1aecb60-p3/bin/dump_syms
> > -d /tmp/syms
> >
> > $IMPALA_HOME/toolchain/breakpad-1b704857f1e78a864e6942e61345
> > 7e55f1aecb60-p3/bin/minidump_stackwalk .dmp /tmp/syms
> > > /tmp/resolved.txt
> >
>