Re: Impalad Process Crash

2017-11-22 Thread Tim Armstrong
Crashes in tcmalloc::CentralFreeList::FetchFromSpans() or in TCMalloc functions in general usually indicate a heap corruption bug (e.g. a double-free or a free of an invalid pointer). The stack trace often isn't useful because the corruption happened on a different thread. These are among the

Re: A couple of questions

2017-12-15 Thread Tim Armstrong
1. I think you're right that many of the shell tests don't inherently require to be executed serially. Some of them would require work to execute in parallel, particularly the ones that inspect files like .impalahistory and tests that check the values of global impala daemon metrics. 2. Yes, Java

Re: Switch gerrit merge strategy to "rebase always"?

2017-12-12 Thread Tim Armstrong
Ok, I went ahead and changed it in gerrit. Let me know if you have any problems with it. On Mon, Dec 11, 2017 at 11:58 AM, Philip Zeyliger <phi...@cloudera.com> wrote: > Seems like it's the right thing to do. > > On Mon, Dec 11, 2017 at 11:43 AM, Tim Armstrong <tarmstr...@clo

New Impala Contributors: IMPALA-6296

2017-12-12 Thread Tim Armstrong
If you'd like to contribute a patch to Impala, but aren't sure what you want to work on, you can look at Impala's newbie issues: https://issues.apache.org/jira/issues/?filter=12341668. You can find detailed instructions on submitting patches at

Re: [DISCUSS] 2.11.0 release

2017-12-07 Thread Tim Armstrong
Actually it looks like we have a new blocker that Taras filed: https://issues.apache.org/jira/browse/IMPALA-6292 On Thu, Dec 7, 2017 at 10:03 AM, Tim Armstrong <tarmstr...@cloudera.com> wrote: > I think that makes sense. We'll have to go through the fix versions of > recent JIRAs a

Re: build broken

2017-12-05 Thread Tim Armstrong
Thanks for investigating Jin Chul. I came to the same conclusion and merged the change. On Tue, Dec 5, 2017 at 5:57 PM, Jin Chul Kim wrote: > Hi, > > Currently Impala build is broken because the child change was merged > without the merge of the parent change. > > (parent)

Re: [DISCUSS] 2.11.0 release

2017-12-07 Thread Tim Armstrong
It looks like https://issues.apache.org/jira/browse/IMPALA-5017 caused https://issues.apache.org/jira/browse/IMPALA-6284, so maybe we should not include IMPALA-5017 in the 2.11 release also. On Thu, Dec 7, 2017 at 9:43 AM, Jim Apple wrote: > I think it would be great to

Re: Re: How to control max memory usage in JNI

2017-12-11 Thread Tim Armstrong
The catalog is cached on all Impala daemons so you also need to increase the impalad heap size too. There's a brief explanation in the docs: https://impala.apache.org/docs/build/html/topics/impala_scalability.html (see the "Impact of Many Tables or Partitions on Impala Catalog Performance and

Re: build broken

2017-12-06 Thread Tim Armstrong
Chul for investigating the problem. > Sorry > > for missing that during code review. > > > > On Tue, Dec 5, 2017 at 6:00 PM, Tim Armstrong <tarmstr...@cloudera.com> > > wrote: > > > > > Thanks for investigating Jin Chul. I came to the same conclusion and

New committer - Zoltán Borók-Nagy

2018-05-04 Thread Tim Armstrong
The Project Management Committee (PMC) for Apache Impala has invited Zoltán Borók-Nagy to become a committer and we are pleased to announce that they have accepted. Congratulations and welcome, Zoltán!

Re: [ANNOUNCE] Apache Impala 3.0.0 release

2018-05-10 Thread Tim Armstrong
Thanks Sailesh! On Wed, May 9, 2018 at 10:09 PM, Jim Apple wrote: > Thanks to Sailesh for managing the release, and thanks to the many, many > others who contributed to creating this new branch and maintaining the > relationship between the two branches. > > On Wed, May 9,

Build error with sqlparse-0.1.14

2018-05-11 Thread Tim Armstrong
Just an FYI, since everyone else is probably going to run into this. The sqlparse version change breaks buildall.sh if you run it on a non-clean tree. Creating an egg for /home/tarmstrong/Impala/incubator-impala/shell/ext-py/sqlparse-0.1.14 python: can't open file 'setup.py': [Errno 2] No such

Re: New message on your code reviews when cherry-picking your change fails.

2018-05-11 Thread Tim Armstrong
Thanks for setting this up, this should make things a lot easier. On Fri, May 11, 2018 at 12:02 PM, Philip Zeyliger wrote: > Hey folks, > > I just updated https://jenkins.impala.io/job/cherrypick-2.x-and-test/ to > comment on a code review the first time a change causes the

New committers - Gabor Kaszab and Attila Jeges

2018-05-18 Thread Tim Armstrong
The Project Management Committee (PMC) for Apache Impala has invited Attila and Gabor to become committers and we are pleased to announce that they have accepted. Congratulations and welcome!

Re: jenkins.impala.io's ubuntu-16.04-from-scratch will now have Junit test results

2018-05-15 Thread Tim Armstrong
I think it's trying to interpret our docs as test results: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2186/#showFailuresLink - - Test Result (250 failures

Re: Automatically rebase changes before GVO?

2018-06-11 Thread Tim Armstrong
pala.io/job/gerrit-verify-dryrun/jobConfigHistory/ > showDiffFiles?timestamp1=2018-06-11_22-09-04= > 2018-06-12_02-47-10 > > On Mon, Jun 11, 2018 at 3:09 PM, Tim Armstrong > wrote: > > > Ok, I applied the changes. Let me know if you run into any issues. > > > &

Re: Broken/Flaky Tests

2018-06-07 Thread Tim Armstrong
that it won't break any other configurations (S3, Local, etc). On Wed, Jun 6, 2018 at 2:44 PM, Tim Armstrong wrote: > We ran into some test issues cherry-picking the latest set of changes to > 2.x. I pushed out a fix and I'm merging now. Once that is done the main > build fixes should be

Re: Automatically rebase changes before GVO?

2018-06-07 Thread Tim Armstrong
gt; > Seems fine, especially since we do the rebase as our submission strategy > > anyway, so we're already accepting/testing something that's likely to get > > rebased, and we may as well minimize that window. > > > > I'd be in favor of the bot also carrying the votes. >

Automatically rebase changes before GVO?

2018-06-07 Thread Tim Armstrong
One annoyance with our precommit job is the requirement to manually rebase the change before starting the merge. Failure to do so either leads to false positives or false negatives - builds that failed because they were missing a flaky/broken test fix and builds that succeeded despite interacting

Re: Branch 2.x

2018-06-18 Thread Tim Armstrong
This just came up - do we want to backport bug or test fixes to 2.x or nothing at all? https://gerrit.cloudera.org/#/c/10722/ On Mon, Jun 18, 2018 at 10:48 AM, Tim Armstrong wrote: > Thanks Phil, I think it will be good not to have to maintain this. > > On Mon, Jun 18, 2018 at 9:40 A

Re: Branch 2.x

2018-06-18 Thread Tim Armstrong
Thanks Phil, I think it will be good not to have to maintain this. On Mon, Jun 18, 2018 at 9:40 AM, Philip Zeyliger < phi...@cloudera.com.invalid> wrote: > I've not heard anything, so I've removed the "trigger" for job > https://jenkins.impala.io/job/cherrypick-2.x-and-test/. It used to "Poll >

Future of unsupported formats?

2018-06-18 Thread Tim Armstrong
For a few years now we've had write support for Sequence, Avro and compressed text hidden behind a query option. We haven't really made any progress on turning it into a supported feature, so I'm wondering if we should remove the code and save some overhead of building, testing and code

Re: Configuring Impala to exit on OutOfMemoryError

2018-06-14 Thread Tim Armstrong
It definitely seems like it would result in more predictable behaviour. I think the other reasonable alternative is that we actually test the behaviour of the system when it gets into an OOM state and make sure that it can recover (which might not be possible with a reasonable amount of effort).

Apache jira-bot stopped working

2018-06-15 Thread Tim Armstrong
Just FYI, it decided to take a day off, in case you were waiting for a comment on your JIRA: https://issues.apache.org/jira/browse/INFRA-16651 Hopefully it's back up now.

Re: Broken/Flaky Tests

2018-06-14 Thread Tim Armstrong
I'm concerned that for the last couple of days we've been finding new issues faster than we're fixing them. I'll start pushing back on some higher-risk changes (e.g. in race-prone parts of the code) until we've cleared up some of the issues. On Tue, Jun 12, 2018 at 1:31 PM, Tim Armstrong wrote

Re: Automatically rebase changes before GVO?

2018-06-11 Thread Tim Armstrong
I've tried my job a few times and it's working as expected. Any objections to me switching over gerrit-verify-dryrun to my approach? On Thu, Jun 7, 2018 at 2:42 PM, Tim Armstrong wrote: > Ok, I was able to put together a test job that does the automatic rebase > and carries a +2 here:

Re: Automatically rebase changes before GVO?

2018-06-11 Thread Tim Armstrong
me. > > On Mon, Jun 11, 2018 at 12:06 PM, Tim Armstrong > wrote: > > > > On nit: as GVD gets more complex, it becomes harder for new people to > > understand the messages and +Ns applied to their patches. That doesn't > mean > > we shouldn't do this, only that it's

Re: Automatically rebase changes before GVO?

2018-06-11 Thread Tim Armstrong
Ok, I applied the changes. Let me know if you run into any issues. On Mon, Jun 11, 2018 at 3:05 PM, Sailesh Mukil wrote: > +1 > > On Mon, Jun 11, 2018 at 3:02 PM, Jim Apple wrote: > > > No objection from me. > > > > On Mon, Jun 11, 2018 at 12:06

Re: Future of unsupported formats?

2018-06-19 Thread Tim Armstrong
rth taking a survey wider than dev@. User@ or the customers > of Impala packagers might be good places to start. > > On Mon, Jun 18, 2018 at 5:10 PM Tim Armstrong > wrote: > > > For a few years now we've had write support for Sequence, Avro and > > compressed text hidd

Re: Future of unsupported formats?

2018-06-19 Thread Tim Armstrong
Lars, Even if people were using it, would that affect our decision if there's noone to maintain it? I don't think we were ambiguous about whether writing those formats was supported or not. On Tue, Jun 19, 2018 at 10:04 AM, Tim Armstrong wrote: > Hi Edward, > I was talking about

Re: Future of unsupported formats?

2018-06-19 Thread Tim Armstrong
r > > communication channels they might have access to. > > > > Cheers, Lars > > > > On Tue, Jun 19, 2018 at 8:26 AM Tim Armstrong > > wrote: > > > > > I don't think we need to bump a major version to remove something that >

Re: Future of unsupported formats?

2018-06-19 Thread Tim Armstrong
ge: > > It seems like we should get it out of this in-between state - either delete > > the code or get it to the point where it's supported and tested. If we > > delete it, it's always possible for someone to resurrect it later. > > > Cheers, Lars > > > > On Tue, Jun

Re: Broken/Flaky Tests

2018-06-05 Thread Tim Armstrong
Things are starting to look healthier now. I went through the broken-build JIRAs and downgraded some of the infrequent infrastructure issues to critical so we have a clearer idea of what's actually breaking the build now versus what's an occasional infra issue:

Re: Broken/Flaky Tests

2018-06-05 Thread Tim Armstrong
know and I can considering whether to merge them. On Tue, Jun 5, 2018 at 10:11 AM, Tim Armstrong wrote: > Things are starting to look healthier now. > > I went through the broken-build JIRAs and downgraded some of the > infrequent infrastructure issues to critical so we have a

Re: UDA debugging, was Re: Broken/Flaky Tests

2018-06-05 Thread Tim Armstrong
so 2/3 of those fixes are merged and the other is being merged. > > > > We still have a long list of flaky issues but I went through and we've > > either mitigated them or we're blocked on being able to repro them. > > > > I'll see how things look tomorrow, but if you have

Re: Broken/Flaky Tests

2018-06-06 Thread Tim Armstrong
We ran into some test issues cherry-picking the latest set of changes to 2.x. I pushed out a fix and I'm merging now. Once that is done the main build fixes should be on both 2.x and master. On Tue, Jun 5, 2018 at 6:08 PM, Tim Armstrong wrote: > Ok, so 2/3 of those fixes are mer

Re: New Committer: Csaba Ringhofer

2018-05-29 Thread Tim Armstrong
Congratulations Csaba! On Tue, May 29, 2018 at 9:03 AM, Lars Volker wrote: > The Project Management Committee (PMC) for Apache Impala has invited > Csaba Ringhofer to become a committer and we are pleased to announce that > he has accepted. > > Congratulations and welcome, Csaba! >

Jenkins upgrade

2018-06-04 Thread Tim Armstrong
I'll be upgrading Jenkins today in response to a security advisory. There was only one job running and I've talked to mikeb about it already. I'll reply to this thread once it's back up.

Re: Jenkins upgrade

2018-06-04 Thread Tim Armstrong
And we're back up and running. Let me know if you see any issues. On Mon, Jun 4, 2018 at 10:21 AM, Tim Armstrong wrote: > I'll be upgrading Jenkins today in response to a security advisory. There > was only one job running and I've talked to mikeb about it already. I'll > reply to th

Need reviewer for SHUTDOWN patch

2018-07-02 Thread Tim Armstrong
Does anyone have time to review this patch? It's a pretty big step towards being able to shrink an Impala cluster without disrupting workloads. https://gerrit.cloudera.org/#/c/10744/

Re: 2.x branch divergence, its implications and what you can do.

2018-05-03 Thread Tim Armstrong
Maybe we should just start with posting back a warning to the code review if the change doesn't cleanly apply? On Thu, May 3, 2018 at 12:34 PM, Tim Armstrong <tarmstr...@cloudera.com> wrote: > It would be good to encourage people to proactively do cherry-picks but I > think it need

Re: 2.x branch divergence, its implications and what you can do.

2018-05-03 Thread Tim Armstrong
It would be good to encourage people to proactively do cherry-picks but I think it needs a bit more thought before automating it or adding more barriers. That could potentially cause hold-ups if there's a dependency on a previous patch. E.g. if I'm merging two patches B and C that depend on each

Re: Build error

2018-01-09 Thread Tim Armstrong
The error is probably higher up in your console output. The lines you have there look like the side-effects of parallel build tasks being killed off after an earlier failure. The below line hints that the failure may have been in building the frontend Java: fe/CMakeFiles/fe.dir/build.make:57:

Re: Contributor bit for Alex Rodoni?

2018-01-16 Thread Tim Armstrong
I added her as a contributor in JIRA admin and tested assigning her IMPALA-3330. Seems to work. On Tue, Jan 16, 2018 at 12:04 PM, John Russell wrote: > Hi, > > Alex Rodoni will be contributing to the upstream Apache Impala docs. I > can see her user profile here: > >

Re: Re: Cancellation logic in HdfsScanners

2018-01-16 Thread Tim Armstrong
Would you be able to file a JIRA? On Tue, Jan 16, 2018 at 3:00 PM, Tim Armstrong <tarmstr...@cloudera.com> wrote: > Looks to me like you found a bug. I think the scanners should be checking > both cancellation conditions, i.e. RuntimeState::is_cancelled_ for MT and >

Re: Re: Cancellation logic in HdfsScanners

2018-01-16 Thread Tim Armstrong
--+ > > When errors happen in F00, cancellation rpc will be sent to F01. However, the > hdfs scanner in F01 does not notice it in time and pass up all the row > batches. Then the DataStreamSender will try to send these r

Re: Cancellation logic in HdfsScanners

2018-01-16 Thread Tim Armstrong
ScannerContext::cancelled() == true means that the scan has completed, either because it has returned enough rows, because the query is cancelled, or because it hit an error. RuntimeState::cancelled() == true means that the query is cancelled. So there are cases where ScannerContext::cancelled()

Re: [Ready for Review] IMPALA-5717: Support reading from ORC format files

2018-01-26 Thread Tim Armstrong
Thank you! I had few higher-level questions or thoughts: * Assuming we end up using the ORC C++ library, we probably want to manage it in the same way that we do Avro by building it externally and then linking against it (we use the native-toolchain project for convenience). Importing the code

Re: Re: Re: Cancellation logic in HdfsScanners

2018-01-16 Thread Tim Armstrong
don't they just check > about RuntimeState::is_cancelled()? Are there any reasons that they > should go ahead until HdfsScanNode::done()? > > At 2018-01-17 07:00:51, "Tim Armstrong" <tarmstr...@cloudera.com> wrote: > > Looks to me like you found a bug. I think the

Re: .test file format

2018-01-12 Thread Tim Armstrong
It is possible to get the test infra to generate .test files based on the actual output. For planner tests, they automatically get written to logs/fe_tests/PlannerTest/. If you're running end-to-end tests with impala-py.test, you can use the --update_results flag to get it to generate similar

Re: Accumulo Integration [SEC=UNOFFICIAL]

2018-01-12 Thread Tim Armstrong
I agree that the HBase integration would be a reasonable starting point to look at. The Kudu integration is also interesting because there are also similarities and the integration is generally more optimised and featureful. I think the devil is in the details - like Jim mentioned, auth can be

Re: Query compilation approach

2018-01-30 Thread Tim Armstrong
Thanks for sharing. I think we have some of the infrastructure required to do something similar - we have an interpreted path already and we could swap in compiled versions of functions by updating function pointers. Michael Ho and I have talked about doing async codegen before or even having a

Re: ERROR: NotImplementedException: MT_DOP not supported

2018-02-05 Thread Tim Armstrong
o through, I was not aware > that. basically, I was trying to insert some values in a partitioned table > and trying to find a way set a specific row's numDVs and the error > happened. > > 2018-02-05 15:24 GMT-06:00 Tim Armstrong <tarmstr...@cloudera.com>: > > > We don

Re: Re: ORC scanner - points for discussion

2018-02-12 Thread Tim Armstrong
Putting it behind a flag sounds good to me too. Hopefully we can get feedback from Hulu and other users of Impala that will try out the experimental version. On Mon, Feb 12, 2018 at 10:26 AM, Dimitris Tsirogiannis < dtsirogian...@cloudera.com> wrote: > Does the patch also implement an ORC

Re: Re: Re: ORC scanner - points for discussion

2018-02-12 Thread Tim Armstrong
Maybe it would make sense to create an Epic in JIRA for ORC scanner enhancements, following on from the initial implementation. I don't really feel strongly as long as the related JIRAs are linked together somehow. On Mon, Feb 12, 2018 at 1:42 PM, Quanlong Huang wrote: >

Re: Any design docs about the buffer pool

2018-02-09 Thread Tim Armstrong
I put together this summary to explain to reviewers the overall scope of the work: https://docs.google.com/presentation/d/1zSVATeUmhAmA6JvIWH4ODbhjsnx554WLeupigmUwi9g/edit#slide=id.p This was the top-level JIRA that tracked all of the work: https://issues.apache.org/jira/browse/IMPALA-3200 On

ORC scanner - points for discussion

2018-02-09 Thread Tim Armstrong
Quanlong has done a bunch of work implementing an ORC scanner. I've been playing around with it and it works pretty nicely - I can load and run TPC-H with no problem! It's a big addition to Impala and the integration with the external library has caused some implementation challenges, so I wanted

Re: Re: Re: Build fail in native-toolchain project

2018-02-14 Thread Tim Armstrong
There are. I tried to document the list at some point here: https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala I believe that covers most of them, but it may be slightly stale. Please feel free to update the list or let us know if

Re: ERROR: NotImplementedException: MT_DOP not supported

2018-02-05 Thread Tim Armstrong
We don't support multithreaded execution for all plans currently. I think the docs do a good job of explaining the current limitations: https://impala.apache.org/docs/build/html/topics/impala_mt_dop.html On Mon, Feb 5, 2018 at 1:05 PM, Xinran Yu Tinney wrote: > Hi,

Re: minicluster with SSL

2018-02-22 Thread Tim Armstrong
If you figured this out, it would be helpful to add it to the developer wiki: https://cwiki.apache.org/confluence/display/IMPALA/Impala+Home . I gave you permissions to edit pages on it. On Wed, Feb 21, 2018 at 6:36 PM, Vincent Tran wrote: > Please disregard. It turns out

Please hold off on merging changes that don't fix builds

2018-02-23 Thread Tim Armstrong
I'm triaging a bunch of builds now that have broken with the influx of large changes this week. I'll continue to file JIRAs for issues are breaking builds, but for now let's avoid merging anything that might make the situation worse. I'll send out an email once things are healthier. - Tim

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-16 Thread Tim Armstrong
There is an extensibility mechanism with the ColumnOrder union - I think that was meant to avoid the need to add new stat fields? Given that the bug was in the Parquet spec, we'll need to make a spec change anyway, so we could add a new ColumnOrder - FloatingPointTotalOrder? at the same time as

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-16 Thread Tim Armstrong
> > On Fri, Feb 16, 2018 at 8:38 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: > > > There is an extensibility mechanism with the ColumnOrder union - I think > > that was meant to avoid the need to add new stat fields? > > > > Given that the bug was

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-16 Thread Tim Armstrong
t; > > > That fix does not preclude a more thorough solution in the future, but it > > addresses the common case quickly. > > > > For existing data files we could check the writer version ignore filters > on > > float/double. I don't know whether min/max

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-16 Thread Tim Armstrong
; Yeah, I missed that. We set it per column, so all other types could keep > TypeDefinedOrder and floats could have something like NanAwareDoubleOrder. > > On Fri, Feb 16, 2018 at 9:18 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: > > > We wouldn't need to rev th

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-15 Thread Tim Armstrong
We could also consider treating NaN similar to NULL and having a separate piece of information with a count of NaN values (or just a bit indicating presence/absence of NaN). I'm not sure if that is easier or harder to implement than a total order. On Thu, Feb 15, 2018 at 9:12 AM, Laszlo Gaal

Working on cherry-picks to 2.x

2018-02-23 Thread Tim Armstrong
Just an FYI, there were some merge conflicts causing a logjam with the cherry-pick job. Alex fixed the first set, now i'm working on cherry-picking the remaining patches.

Re: test_semi_joins_exhaustive error?

2018-02-23 Thread Tim Armstrong
Looks like https://issues.apache.org/jira/browse/IMPALA-4857 . If there are rows with duplicate keys on the right side of a hash join, we have to be able to fit all rows for a key in memory to process it. I think we could avoid the problem by modifying the test so that it's joining on a column

Re: Setting up ASF Git Bot?

2018-02-21 Thread Tim Armstrong
Perhaps file an INFRA jira again? > > Thanks! > > -- Philip > > > > On Wed, Feb 21, 2018 at 9:52 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: > > > What happened to git bot? I noticed that it stopped posting on JIRAs. > E.g. > > I had to co

Re: Setting up ASF Git Bot?

2018-02-21 Thread Tim Armstrong
What happened to git bot? I noticed that it stopped posting on JIRAs. E.g. I had to copy and paste the commit message manually on this one: https://issues.apache.org/jira/browse/IMPALA-6497 On Thu, Feb 1, 2018 at 10:17 AM, Philip Zeyliger wrote: > I filed

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-19 Thread Tim Armstrong
We could drop NaNs and require that -0 be normalised to +0 when writing out stats. That would remove any degrees of freedom from the writer and then straightforward comparison with =, <, >, >=, <=, != would work as expected. On Mon, Feb 19, 2018 at 8:04 AM, Zoltan Ivanfi

Re: Joining Impala mailing list

2018-01-03 Thread Tim Armstrong
Hi Xinran, That looks like a good JIRA to start with. I wasn't able to figure out your Apache JIRA username but please feel free to assign it to yourself (or let us know if you need permissions to do so) Cheers, Tim On Tue, Jan 2, 2018 at 4:30 PM, kenneth mcfarland <

Re: Could anybody help to explain why there is such a big gap in 'ProbeTime' and How to fix this gap?

2018-01-03 Thread Tim Armstrong
That's a tricky one. I have a couple of ideas but it's a bit difficult to confirm since the profile isn't really designed to easily answer questions like this. ProbeTime measures wall-clock time rather than actual time spent executing on the CPU. My first guess is that it's because the Kudu scan

Re: Joining Impala mailing list

2018-01-03 Thread Tim Armstrong
and corner of the > "Contributors" box. A text box will appear to the right of the > contributor box. > > 3. Type the name of the person in that box to attempt to find their > Apache JIRA username. > > 4. Once you have it, press "Update" > > > On We

Re: Please hold off on merging changes that don't fix builds

2018-02-26 Thread Tim Armstrong
, but please hold off on any large commits for now until builds are reliably green. On Fri, Feb 23, 2018 at 3:25 PM, Tim Armstrong <tarmstr...@cloudera.com> wrote: > I'm triaging a bunch of builds now that have broken with the influx of > large changes this week. > > I'll cont

Re: Re: Future of unsupported formats?

2018-06-21 Thread Tim Armstrong
cing your work on fixing the recent > broken/flaky tests. > > > > Thanks for your hard working on this wonderful product! > > > > > > -- Quanlong > > > > > > At 2018-06-20 04:12:38, "Jim Apple" > wrote: > > >I'm convinced. Th

Re: Enabling automatic code review precommit job

2018-07-30 Thread Tim Armstrong
omatically does some checks on the diff and posts code review comments. I started off with Python flake8 comments. Let me know if you see any problems or if it turns out to be too noisy. On Mon, Jul 23, 2018 at 11:55 AM, Tim Armstrong wrote: > Hi All, > I'm enabling an automatic precom

Re: Enabling automatic code review precommit job

2018-07-31 Thread Tim Armstrong
e backslash is redundant between brackets > 368 : E302 expected 2 blank lines, found 1 >1295 : E128 continuation line under-indented for visual indent > > Maybe worth just disabling some of the indentation-related ones to start? > > > On Mon, Jul 30, 2018 at 4:09 PM

Re: Enabling automatic code review precommit job

2018-07-31 Thread Tim Armstrong
lready 4 spaces in at the start of the region, you don't want to revert > back to 0 or 2 spaces in. That said, it can still be helpful. I don't think > this is as sophisticated as clang-format.el. > > On Tue, Jul 31, 2018 at 9:09 AM, Todd Lipcon > wrote: > > > On Tu

#pragma once?

2018-08-01 Thread Tim Armstrong
Todd brought up our include guards on a code review, asking why we don't use #pragma once instead: https://gerrit.cloudera.org/#/c/10988/5 . It sounds like Kudu has switched to it #pragma once does seem cleaner and our GCC and Clang versions are modern enough to support it. What do people think

Re: Enabling automatic code review precommit job

2018-08-01 Thread Tim Armstrong
:00 AM, Tim Armstrong wrote: > Todd pointed out a bug where it was posting flake8 comments that didn't > align with the diff. I figured out the issue but will keep the job in a > silent mode for a bit while I monitor it. > > On Tue, Jul 31, 2018 at 9:16 AM, Jim Apple > wrote: &g

Re: Assign JIRA

2018-08-06 Thread Tim Armstrong
It looks like you're already an admin. I think you need to add the person as a "Contributor". I added "asherman" as a contributor. On Mon, Aug 6, 2018 at 12:54 PM, Thomas Tauber-Marshall < tmarsh...@cloudera.com.invalid> wrote: > I'm trying to assign a JIRA to a new contributor (Andrew Sherman),

Re: #pragma once?

2018-08-06 Thread Tim Armstrong
; > > potentially making the code even less portable. > > > > > > That being said, since Kudu has been using it for a while and has had > no > > > issues, we can do the same since the potential benefits outweigh the > > cons. > > > > > > On Wed, Aug 1,

Re: Enabling automatic code review precommit job

2018-08-16 Thread Tim Armstrong
un > which almost certainly do these checks as well, but if it didn't I don't > want to commit something that might break gerrit-core-review-checks for > subsequent changes. > > Thanks, > Zoltan > > > > On Wed, Aug 1, 2018 at 10:56 PM Tim Armstrong > wrote: &g

Re: Enabling automatic code review precommit job

2018-08-16 Thread Tim Armstrong
you can do it here: https://jenkins.impala.io/ > gerrit_manual_trigger/ > " > > Zoltan > > > On Thu, Aug 16, 2018 at 7:19 PM Tim Armstrong > wrote: > > > Thanks for letting me know and sorry for the confusion. It's fine to go > > ahead and do a gerrit-verify-d

New Impala committer - Quanlong Huang

2018-08-17 Thread Tim Armstrong
The Project Management Committee (PMC) for Apache Impala has invited Quanlong Huang to become a committer and we are pleased to announce that they have accepted. Congratulations and welcome, Quanlong Huang!

New Impala PMC member: Joe McDonnell

2018-08-21 Thread Tim Armstrong
The Project Management Committee (PMC) for Apache Impala has invited Joe McDonnell to become a PMC member and we are pleased to announce that they have accepted. Congratulations and welcome, Joe!

Re: Improving Kudu Build Support

2018-08-21 Thread Tim Armstrong
Is there a path to building a version of Kudu locally for an arbitrary linux distro? Personally I am less concerned about 14.04 support and more concerned about what the path to upgrading to 18.04. It would also be nice for it to be at least possible to develop on RedHat-derived distros even if

Re: Improving latency of catalog update propagation?

2018-08-21 Thread Tim Armstrong
ency or doing a "push" to the statestore, > etc. > > I > > > probably won't work on this in the near term to avoid complicating the > > > ongoing changes with catalog. > > > > > > -Todd > > > > > > On Tue, Aug 21, 2018 at 10

Re: Improving latency of catalog update propagation?

2018-08-21 Thread Tim Armstrong
This is somewhat relevant for admission control too - I had thought about some of these issues in that context, because reducing the latency of admission controls state propagation helps avoid overadmission but having a very low statestore frequency is very inefficient and doesn't scale well to

Re: thrift dependency error - pip

2018-07-19 Thread Tim Armstrong
It's weird that it thinks that thrift_sasl-0.1.0.tar.gz is a version of thrift. Do you have a thrift tarball in infra/python/deps/? You could try rerunning ./infra/python/deps/download_requirements On Thu, Jul 19, 2018 at 5:30 PM, Bharath Vissapragada < bhara...@cloudera.com.invalid> wrote: >

Enabling automatic code review precommit job

2018-07-23 Thread Tim Armstrong
Hi All, I'm enabling an automatic precommit job for code reviews uploaded to gerrit that will run RAT, clang-tidy and a GCC debug compilation. This is to provide faster feedback on code reviews: https://issues.apache.org/jira/browse/IMPALA-7317 . I'll add some more checks but I'm wanting to test

Re: JSON support: Hive compatibility or ANSI SQL standard

2018-09-10 Thread Tim Armstrong
I think I agree with Lars. Having the (mostly) Hive-compatible version is useful if there are shared views between Hive/Impala and for people migrating queries from Hive or some of the Impala JSON UDFs that I've seen floating around. On Mon, Sep 10, 2018 at 8:20 AM, Lars Volker wrote: > Thanks

boost::scoped_ptr vs std::unique_ptr

2018-07-05 Thread Tim Armstrong
I was just talking with Michael Ho on a review about this https://gerrit.cloudera.org/#/c/10810/7/be/src/exec/scan-node.h@271 For a while we've continued using scoped_ptr in some places because it supports a smaller set of operators and implies that the pointer isn't movable. See

Re: boost::scoped_ptr vs std::unique_ptr

2018-07-05 Thread Tim Armstrong
the subtle difference between scoped_ptr and unique_ptr > off > > > the > > > > top of their head anyways. > > > > > > > > Fwiw, I was under the impression from talking with people in the past > > > that > > > > we were already

Re: Impala OS and Java version support

2018-07-06 Thread Tim Armstrong
Sounds like a good idea to me. Java 8 makes sense to me too since the major Java implementations (OpenJDK and Oracle) no longer support the older JDKs. On Fri, Jul 6, 2018 at 9:58 AM, Sailesh Mukil wrote: > I'm in favor of the minimum OS version support proposal. From 3.x, I don't > see any

Re: boost::scoped_ptr vs std::unique_ptr

2018-07-06 Thread Tim Armstrong
gt; the globe weigh in. > > BR, > Zoltan > > > On Fri, Jul 6, 2018 at 3:31 AM Jim Apple > wrote: > > > SGTM > > > > On Thu, Jul 5, 2018 at 6:13 PM, Tim Armstrong < > > tarmstr...@cloudera.com.invalid> wrote: > > > > > Sounds like un

Re: Inconsistent handling of schema in Avro tables

2018-07-11 Thread Tim Armstrong
The behaviour of Avro schemas in all these cases has always been rather mysterious to me. Before you wrote this email I would have assumed that Impala's behaviour would be like Hive's behaviour. I agree with the principle that the creation of a partition without changes to table metadata

impala-4835 gerrit branch

2018-03-15 Thread Tim Armstrong
You might see some gerrit notifications about commits to this branch. I created this temporary branch to make it easier to incrementally review fixes to my original IMPALA-4835 patches. I won't merge the changes to master without consultation with others. - Tim

Re: Re: Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

2018-04-10 Thread Tim Armstrong
ses are cut. And > >since there are no minimum waiting period between releases, we could cut > >those right after that patch lands if we like. > > > >On Mon, Apr 2, 2018 at 4:05 PM, Tim Armstrong <tarmstr...@cloudera.com> > >wrote: > > > >>

  1   2   >