Re: Impalad Process Crash

2017-11-22 Thread Tim Armstrong
Crashes in tcmalloc::CentralFreeList::FetchFromSpans() or in TCMalloc functions in general usually indicate a heap corruption bug (e.g. a double-free or a free of an invalid pointer). The stack trace often isn't useful because the corruption happened on a different thread. These are among the

Re: A couple of questions

2017-12-15 Thread Tim Armstrong
1. I think you're right that many of the shell tests don't inherently require to be executed serially. Some of them would require work to execute in parallel, particularly the ones that inspect files like .impalahistory and tests that check the values of global impala daemon metrics. 2. Yes, Java

Re: Switch gerrit merge strategy to "rebase always"?

2017-12-12 Thread Tim Armstrong
Ok, I went ahead and changed it in gerrit. Let me know if you have any problems with it. On Mon, Dec 11, 2017 at 11:58 AM, Philip Zeyliger <phi...@cloudera.com> wrote: > Seems like it's the right thing to do. > > On Mon, Dec 11, 2017 at 11:43 AM, Tim Armstrong <tarmstr...@clo

New Impala Contributors: IMPALA-6296

2017-12-12 Thread Tim Armstrong
If you'd like to contribute a patch to Impala, but aren't sure what you want to work on, you can look at Impala's newbie issues: https://issues.apache.org/jira/issues/?filter=12341668. You can find detailed instructions on submitting patches at

Re: [DISCUSS] 2.11.0 release

2017-12-07 Thread Tim Armstrong
Actually it looks like we have a new blocker that Taras filed: https://issues.apache.org/jira/browse/IMPALA-6292 On Thu, Dec 7, 2017 at 10:03 AM, Tim Armstrong <tarmstr...@cloudera.com> wrote: > I think that makes sense. We'll have to go through the fix versions of > recent JIRAs a

Re: build broken

2017-12-05 Thread Tim Armstrong
Thanks for investigating Jin Chul. I came to the same conclusion and merged the change. On Tue, Dec 5, 2017 at 5:57 PM, Jin Chul Kim wrote: > Hi, > > Currently Impala build is broken because the child change was merged > without the merge of the parent change. > > (parent)

Re: [DISCUSS] 2.11.0 release

2017-12-07 Thread Tim Armstrong
I think that makes sense. We'll have to go through the fix versions of recent JIRAs and make sure that they weren't set to 2.12 though. On Thu, Dec 7, 2017 at 9:33 AM, Thomas Tauber-Marshall < tmarsh...@cloudera.com> wrote: > Since the response from the community has been good, and now that all

Re: [DISCUSS] 2.11.0 release

2017-12-07 Thread Tim Armstrong
It looks like https://issues.apache.org/jira/browse/IMPALA-5017 caused https://issues.apache.org/jira/browse/IMPALA-6284, so maybe we should not include IMPALA-5017 in the 2.11 release also. On Thu, Dec 7, 2017 at 9:43 AM, Jim Apple wrote: > I think it would be great to

Re: Re: How to control max memory usage in JNI

2017-12-11 Thread Tim Armstrong
The catalog is cached on all Impala daemons so you also need to increase the impalad heap size too. There's a brief explanation in the docs: https://impala.apache.org/docs/build/html/topics/impala_scalability.html (see the "Impact of Many Tables or Partitions on Impala Catalog Performance and

Re: build broken

2017-12-06 Thread Tim Armstrong
e keep that in mind with the ServicePool change -- you'll > need to explicitly merge part 1 first. > > On Wed, Dec 6, 2017 at 8:41 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: > > > I think that's a consequence of the "cherry pick" merge strategy. It does > > seem

Re: build broken

2017-12-06 Thread Tim Armstrong
Chul for investigating the problem. > Sorry > > for missing that during code review. > > > > On Tue, Dec 5, 2017 at 6:00 PM, Tim Armstrong <tarmstr...@cloudera.com> > > wrote: > > > > > Thanks for investigating Jin Chul. I came to the same conclusion and

New committer - Zoltán Borók-Nagy

2018-05-04 Thread Tim Armstrong
The Project Management Committee (PMC) for Apache Impala has invited Zoltán Borók-Nagy to become a committer and we are pleased to announce that they have accepted. Congratulations and welcome, Zoltán!

Re: [ANNOUNCE] Apache Impala 3.0.0 release

2018-05-10 Thread Tim Armstrong
Thanks Sailesh! On Wed, May 9, 2018 at 10:09 PM, Jim Apple wrote: > Thanks to Sailesh for managing the release, and thanks to the many, many > others who contributed to creating this new branch and maintaining the > relationship between the two branches. > > On Wed, May 9,

Build error with sqlparse-0.1.14

2018-05-11 Thread Tim Armstrong
Just an FYI, since everyone else is probably going to run into this. The sqlparse version change breaks buildall.sh if you run it on a non-clean tree. Creating an egg for /home/tarmstrong/Impala/incubator-impala/shell/ext-py/sqlparse-0.1.14 python: can't open file 'setup.py': [Errno 2] No such

Re: New message on your code reviews when cherry-picking your change fails.

2018-05-11 Thread Tim Armstrong
Thanks for setting this up, this should make things a lot easier. On Fri, May 11, 2018 at 12:02 PM, Philip Zeyliger wrote: > Hey folks, > > I just updated https://jenkins.impala.io/job/cherrypick-2.x-and-test/ to > comment on a code review the first time a change causes the

New committers - Gabor Kaszab and Attila Jeges

2018-05-18 Thread Tim Armstrong
The Project Management Committee (PMC) for Apache Impala has invited Attila and Gabor to become committers and we are pleased to announce that they have accepted. Congratulations and welcome!

Re: jenkins.impala.io's ubuntu-16.04-from-scratch will now have Junit test results

2018-05-15 Thread Tim Armstrong
I think it's trying to interpret our docs as test results: https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/2186/#showFailuresLink - - Test Result (250 failures

Re: Automatically rebase changes before GVO?

2018-06-11 Thread Tim Armstrong
pala.io/job/gerrit-verify-dryrun/jobConfigHistory/ > showDiffFiles?timestamp1=2018-06-11_22-09-04= > 2018-06-12_02-47-10 > > On Mon, Jun 11, 2018 at 3:09 PM, Tim Armstrong > wrote: > > > Ok, I applied the changes. Let me know if you run into any issues. > > > &

Re: Broken/Flaky Tests

2018-06-07 Thread Tim Armstrong
that it won't break any other configurations (S3, Local, etc). On Wed, Jun 6, 2018 at 2:44 PM, Tim Armstrong wrote: > We ran into some test issues cherry-picking the latest set of changes to > 2.x. I pushed out a fix and I'm merging now. Once that is done the main > build fixes should be

Re: Automatically rebase changes before GVO?

2018-06-07 Thread Tim Armstrong
gt; > Seems fine, especially since we do the rebase as our submission strategy > > anyway, so we're already accepting/testing something that's likely to get > > rebased, and we may as well minimize that window. > > > > I'd be in favor of the bot also carrying the votes. >

Automatically rebase changes before GVO?

2018-06-07 Thread Tim Armstrong
One annoyance with our precommit job is the requirement to manually rebase the change before starting the merge. Failure to do so either leads to false positives or false negatives - builds that failed because they were missing a flaky/broken test fix and builds that succeeded despite interacting

Re: Branch 2.x

2018-06-18 Thread Tim Armstrong
This just came up - do we want to backport bug or test fixes to 2.x or nothing at all? https://gerrit.cloudera.org/#/c/10722/ On Mon, Jun 18, 2018 at 10:48 AM, Tim Armstrong wrote: > Thanks Phil, I think it will be good not to have to maintain this. > > On Mon, Jun 18, 2018 at 9:40 A

Re: Branch 2.x

2018-06-18 Thread Tim Armstrong
Thanks Phil, I think it will be good not to have to maintain this. On Mon, Jun 18, 2018 at 9:40 AM, Philip Zeyliger < phi...@cloudera.com.invalid> wrote: > I've not heard anything, so I've removed the "trigger" for job > https://jenkins.impala.io/job/cherrypick-2.x-and-test/. It used to "Poll >

Future of unsupported formats?

2018-06-18 Thread Tim Armstrong
For a few years now we've had write support for Sequence, Avro and compressed text hidden behind a query option. We haven't really made any progress on turning it into a supported feature, so I'm wondering if we should remove the code and save some overhead of building, testing and code

Re: Configuring Impala to exit on OutOfMemoryError

2018-06-14 Thread Tim Armstrong
It definitely seems like it would result in more predictable behaviour. I think the other reasonable alternative is that we actually test the behaviour of the system when it gets into an OOM state and make sure that it can recover (which might not be possible with a reasonable amount of effort).

Apache jira-bot stopped working

2018-06-15 Thread Tim Armstrong
Just FYI, it decided to take a day off, in case you were waiting for a comment on your JIRA: https://issues.apache.org/jira/browse/INFRA-16651 Hopefully it's back up now.

Re: Broken/Flaky Tests

2018-06-14 Thread Tim Armstrong
I'm concerned that for the last couple of days we've been finding new issues faster than we're fixing them. I'll start pushing back on some higher-risk changes (e.g. in race-prone parts of the code) until we've cleared up some of the issues. On Tue, Jun 12, 2018 at 1:31 PM, Tim Armstrong wrote

Re: Automatically rebase changes before GVO?

2018-06-11 Thread Tim Armstrong
I've tried my job a few times and it's working as expected. Any objections to me switching over gerrit-verify-dryrun to my approach? On Thu, Jun 7, 2018 at 2:42 PM, Tim Armstrong wrote: > Ok, I was able to put together a test job that does the automatic rebase > and carries a +2 here:

Re: Automatically rebase changes before GVO?

2018-06-11 Thread Tim Armstrong
me. > > On Mon, Jun 11, 2018 at 12:06 PM, Tim Armstrong > wrote: > > > > On nit: as GVD gets more complex, it becomes harder for new people to > > understand the messages and +Ns applied to their patches. That doesn't > mean > > we shouldn't do this, only that it's

Re: Automatically rebase changes before GVO?

2018-06-11 Thread Tim Armstrong
Ok, I applied the changes. Let me know if you run into any issues. On Mon, Jun 11, 2018 at 3:05 PM, Sailesh Mukil wrote: > +1 > > On Mon, Jun 11, 2018 at 3:02 PM, Jim Apple wrote: > > > No objection from me. > > > > On Mon, Jun 11, 2018 at 12:06

Re: Future of unsupported formats?

2018-06-19 Thread Tim Armstrong
rth taking a survey wider than dev@. User@ or the customers > of Impala packagers might be good places to start. > > On Mon, Jun 18, 2018 at 5:10 PM Tim Armstrong > wrote: > > > For a few years now we've had write support for Sequence, Avro and > > compressed text hidd

Re: Future of unsupported formats?

2018-06-19 Thread Tim Armstrong
Lars, Even if people were using it, would that affect our decision if there's noone to maintain it? I don't think we were ambiguous about whether writing those formats was supported or not. On Tue, Jun 19, 2018 at 10:04 AM, Tim Armstrong wrote: > Hi Edward, > I was talking about

Re: Future of unsupported formats?

2018-06-19 Thread Tim Armstrong
r > > communication channels they might have access to. > > > > Cheers, Lars > > > > On Tue, Jun 19, 2018 at 8:26 AM Tim Armstrong > > wrote: > > > > > I don't think we need to bump a major version to remove something that >

Re: Future of unsupported formats?

2018-06-19 Thread Tim Armstrong
ge: > > It seems like we should get it out of this in-between state - either delete > > the code or get it to the point where it's supported and tested. If we > > delete it, it's always possible for someone to resurrect it later. > > > Cheers, Lars > > > > On Tue, Jun

Re: Broken/Flaky Tests

2018-06-05 Thread Tim Armstrong
Things are starting to look healthier now. I went through the broken-build JIRAs and downgraded some of the infrequent infrastructure issues to critical so we have a clearer idea of what's actually breaking the build now versus what's an occasional infra issue:

Re: Broken/Flaky Tests

2018-06-05 Thread Tim Armstrong
know and I can considering whether to merge them. On Tue, Jun 5, 2018 at 10:11 AM, Tim Armstrong wrote: > Things are starting to look healthier now. > > I went through the broken-build JIRAs and downgraded some of the > infrequent infrastructure issues to critical so we have a

Re: UDA debugging, was Re: Broken/Flaky Tests

2018-06-05 Thread Tim Armstrong
so 2/3 of those fixes are merged and the other is being merged. > > > > We still have a long list of flaky issues but I went through and we've > > either mitigated them or we're blocked on being able to repro them. > > > > I'll see how things look tomorrow, but if you have

Re: Broken/Flaky Tests

2018-06-06 Thread Tim Armstrong
We ran into some test issues cherry-picking the latest set of changes to 2.x. I pushed out a fix and I'm merging now. Once that is done the main build fixes should be on both 2.x and master. On Tue, Jun 5, 2018 at 6:08 PM, Tim Armstrong wrote: > Ok, so 2/3 of those fixes are mer

Re: New Committer: Csaba Ringhofer

2018-05-29 Thread Tim Armstrong
Congratulations Csaba! On Tue, May 29, 2018 at 9:03 AM, Lars Volker wrote: > The Project Management Committee (PMC) for Apache Impala has invited > Csaba Ringhofer to become a committer and we are pleased to announce that > he has accepted. > > Congratulations and welcome, Csaba! >

Jenkins upgrade

2018-06-04 Thread Tim Armstrong
I'll be upgrading Jenkins today in response to a security advisory. There was only one job running and I've talked to mikeb about it already. I'll reply to this thread once it's back up.

Re: Jenkins upgrade

2018-06-04 Thread Tim Armstrong
And we're back up and running. Let me know if you see any issues. On Mon, Jun 4, 2018 at 10:21 AM, Tim Armstrong wrote: > I'll be upgrading Jenkins today in response to a security advisory. There > was only one job running and I've talked to mikeb about it already. I'll > reply to th

Re: Broken/Flaky Tests

2018-07-02 Thread Tim Armstrong
aside from IMPALA-6910 <https://issues.apache.org/jira/browse/IMPALA-6910> so we can continue merging as normal, but let's not get complacent. On Thu, Jun 14, 2018 at 3:35 PM, Tim Armstrong wrote: > I'm concerned that for the last couple of days we've been finding new > issues faste

Need reviewer for SHUTDOWN patch

2018-07-02 Thread Tim Armstrong
Does anyone have time to review this patch? It's a pretty big step towards being able to shrink an Impala cluster without disrupting workloads. https://gerrit.cloudera.org/#/c/10744/

Re: 2.x branch divergence, its implications and what you can do.

2018-05-03 Thread Tim Armstrong
Maybe we should just start with posting back a warning to the code review if the change doesn't cleanly apply? On Thu, May 3, 2018 at 12:34 PM, Tim Armstrong <tarmstr...@cloudera.com> wrote: > It would be good to encourage people to proactively do cherry-picks but I > think it need

Re: 2.x branch divergence, its implications and what you can do.

2018-05-03 Thread Tim Armstrong
It would be good to encourage people to proactively do cherry-picks but I think it needs a bit more thought before automating it or adding more barriers. That could potentially cause hold-ups if there's a dependency on a previous patch. E.g. if I'm merging two patches B and C that depend on each

Re: Build error

2018-01-09 Thread Tim Armstrong
The error is probably higher up in your console output. The lines you have there look like the side-effects of parallel build tasks being killed off after an earlier failure. The below line hints that the failure may have been in building the frontend Java: fe/CMakeFiles/fe.dir/build.make:57:

Re: Contributor bit for Alex Rodoni?

2018-01-16 Thread Tim Armstrong
I added her as a contributor in JIRA admin and tested assigning her IMPALA-3330. Seems to work. On Tue, Jan 16, 2018 at 12:04 PM, John Russell wrote: > Hi, > > Alex Rodoni will be contributing to the upstream Apache Impala docs. I > can see her user profile here: > >

Re: Re: Cancellation logic in HdfsScanners

2018-01-16 Thread Tim Armstrong
Would you be able to file a JIRA? On Tue, Jan 16, 2018 at 3:00 PM, Tim Armstrong <tarmstr...@cloudera.com> wrote: > Looks to me like you found a bug. I think the scanners should be checking > both cancellation conditions, i.e. RuntimeState::is_cancelled_ for MT and >

Re: Re: Cancellation logic in HdfsScanners

2018-01-16 Thread Tim Armstrong
--+ > > When errors happen in F00, cancellation rpc will be sent to F01. However, the > hdfs scanner in F01 does not notice it in time and pass up all the row > batches. Then the DataStreamSender will try to send these r

Re: Cancellation logic in HdfsScanners

2018-01-16 Thread Tim Armstrong
ScannerContext::cancelled() == true means that the scan has completed, either because it has returned enough rows, because the query is cancelled, or because it hit an error. RuntimeState::cancelled() == true means that the query is cancelled. So there are cases where ScannerContext::cancelled()

Re: [Ready for Review] IMPALA-5717: Support reading from ORC format files

2018-01-26 Thread Tim Armstrong
Thank you! I had few higher-level questions or thoughts: * Assuming we end up using the ORC C++ library, we probably want to manage it in the same way that we do Avro by building it externally and then linking against it (we use the native-toolchain project for convenience). Importing the code

Re: Re: Re: Cancellation logic in HdfsScanners

2018-01-16 Thread Tim Armstrong
don't they just check > about RuntimeState::is_cancelled()? Are there any reasons that they > should go ahead until HdfsScanNode::done()? > > At 2018-01-17 07:00:51, "Tim Armstrong" <tarmstr...@cloudera.com> wrote: > > Looks to me like you found a bug. I think the

Re: .test file format

2018-01-12 Thread Tim Armstrong
It is possible to get the test infra to generate .test files based on the actual output. For planner tests, they automatically get written to logs/fe_tests/PlannerTest/. If you're running end-to-end tests with impala-py.test, you can use the --update_results flag to get it to generate similar

Re: Accumulo Integration [SEC=UNOFFICIAL]

2018-01-12 Thread Tim Armstrong
I agree that the HBase integration would be a reasonable starting point to look at. The Kudu integration is also interesting because there are also similarities and the integration is generally more optimised and featureful. I think the devil is in the details - like Jim mentioned, auth can be

Re: Query compilation approach

2018-01-30 Thread Tim Armstrong
Thanks for sharing. I think we have some of the infrastructure required to do something similar - we have an interpreted path already and we could swap in compiled versions of functions by updating function pointers. Michael Ho and I have talked about doing async codegen before or even having a

Re: ERROR: NotImplementedException: MT_DOP not supported

2018-02-05 Thread Tim Armstrong
o through, I was not aware > that. basically, I was trying to insert some values in a partitioned table > and trying to find a way set a specific row's numDVs and the error > happened. > > 2018-02-05 15:24 GMT-06:00 Tim Armstrong <tarmstr...@cloudera.com>: > > > We don

Re: Re: ORC scanner - points for discussion

2018-02-12 Thread Tim Armstrong
Putting it behind a flag sounds good to me too. Hopefully we can get feedback from Hulu and other users of Impala that will try out the experimental version. On Mon, Feb 12, 2018 at 10:26 AM, Dimitris Tsirogiannis < dtsirogian...@cloudera.com> wrote: > Does the patch also implement an ORC

Re: Re: Re: ORC scanner - points for discussion

2018-02-12 Thread Tim Armstrong
Maybe it would make sense to create an Epic in JIRA for ORC scanner enhancements, following on from the initial implementation. I don't really feel strongly as long as the related JIRAs are linked together somehow. On Mon, Feb 12, 2018 at 1:42 PM, Quanlong Huang wrote: >

Re: Any design docs about the buffer pool

2018-02-09 Thread Tim Armstrong
I put together this summary to explain to reviewers the overall scope of the work: https://docs.google.com/presentation/d/1zSVATeUmhAmA6JvIWH4ODbhjsnx554WLeupigmUwi9g/edit#slide=id.p This was the top-level JIRA that tracked all of the work: https://issues.apache.org/jira/browse/IMPALA-3200 On

ORC scanner - points for discussion

2018-02-09 Thread Tim Armstrong
Quanlong has done a bunch of work implementing an ORC scanner. I've been playing around with it and it works pretty nicely - I can load and run TPC-H with no problem! It's a big addition to Impala and the integration with the external library has caused some implementation challenges, so I wanted

Re: Re: Re: Build fail in native-toolchain project

2018-02-14 Thread Tim Armstrong
There are. I tried to document the list at some point here: https://cwiki.apache.org/confluence/display/IMPALA/Building+native-toolchain+from+scratch+and+using+with+Impala I believe that covers most of them, but it may be slightly stale. Please feel free to update the list or let us know if

Re: ERROR: NotImplementedException: MT_DOP not supported

2018-02-05 Thread Tim Armstrong
We don't support multithreaded execution for all plans currently. I think the docs do a good job of explaining the current limitations: https://impala.apache.org/docs/build/html/topics/impala_mt_dop.html On Mon, Feb 5, 2018 at 1:05 PM, Xinran Yu Tinney wrote: > Hi,

Re: minicluster with SSL

2018-02-22 Thread Tim Armstrong
If you figured this out, it would be helpful to add it to the developer wiki: https://cwiki.apache.org/confluence/display/IMPALA/Impala+Home . I gave you permissions to edit pages on it. On Wed, Feb 21, 2018 at 6:36 PM, Vincent Tran wrote: > Please disregard. It turns out

Please hold off on merging changes that don't fix builds

2018-02-23 Thread Tim Armstrong
I'm triaging a bunch of builds now that have broken with the influx of large changes this week. I'll continue to file JIRAs for issues are breaking builds, but for now let's avoid merging anything that might make the situation worse. I'll send out an email once things are healthier. - Tim

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-16 Thread Tim Armstrong
There is an extensibility mechanism with the ColumnOrder union - I think that was meant to avoid the need to add new stat fields? Given that the bug was in the Parquet spec, we'll need to make a spec change anyway, so we could add a new ColumnOrder - FloatingPointTotalOrder? at the same time as

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-16 Thread Tim Armstrong
> > On Fri, Feb 16, 2018 at 8:38 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: > > > There is an extensibility mechanism with the ColumnOrder union - I think > > that was meant to avoid the need to add new stat fields? > > > > Given that the bug was

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-16 Thread Tim Armstrong
t; > > > That fix does not preclude a more thorough solution in the future, but it > > addresses the common case quickly. > > > > For existing data files we could check the writer version ignore filters > on > > float/double. I don't know whether min/max

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-16 Thread Tim Armstrong
; Yeah, I missed that. We set it per column, so all other types could keep > TypeDefinedOrder and floats could have something like NanAwareDoubleOrder. > > On Fri, Feb 16, 2018 at 9:18 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: > > > We wouldn't need to rev th

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-15 Thread Tim Armstrong
We could also consider treating NaN similar to NULL and having a separate piece of information with a count of NaN values (or just a bit indicating presence/absence of NaN). I'm not sure if that is easier or harder to implement than a total order. On Thu, Feb 15, 2018 at 9:12 AM, Laszlo Gaal

Working on cherry-picks to 2.x

2018-02-23 Thread Tim Armstrong
Just an FYI, there were some merge conflicts causing a logjam with the cherry-pick job. Alex fixed the first set, now i'm working on cherry-picking the remaining patches.

Re: test_semi_joins_exhaustive error?

2018-02-23 Thread Tim Armstrong
Looks like https://issues.apache.org/jira/browse/IMPALA-4857 . If there are rows with duplicate keys on the right side of a hash join, we have to be able to fit all rows for a key in memory to process it. I think we could avoid the problem by modifying the test so that it's joining on a column

Re: Setting up ASF Git Bot?

2018-02-21 Thread Tim Armstrong
Perhaps file an INFRA jira again? > > Thanks! > > -- Philip > > > > On Wed, Feb 21, 2018 at 9:52 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: > > > What happened to git bot? I noticed that it stopped posting on JIRAs. > E.g. > > I had to co

Re: Setting up ASF Git Bot?

2018-02-21 Thread Tim Armstrong
What happened to git bot? I noticed that it stopped posting on JIRAs. E.g. I had to copy and paste the commit message manually on this one: https://issues.apache.org/jira/browse/IMPALA-6497 On Thu, Feb 1, 2018 at 10:17 AM, Philip Zeyliger wrote: > I filed

Re: Inconsistent float/double sort order in spec and implementations can lead to incorrect results

2018-02-19 Thread Tim Armstrong
We could drop NaNs and require that -0 be normalised to +0 when writing out stats. That would remove any degrees of freedom from the writer and then straightforward comparison with =, <, >, >=, <=, != would work as expected. On Mon, Feb 19, 2018 at 8:04 AM, Zoltan Ivanfi

Re: Joining Impala mailing list

2018-01-03 Thread Tim Armstrong
Hi Xinran, That looks like a good JIRA to start with. I wasn't able to figure out your Apache JIRA username but please feel free to assign it to yourself (or let us know if you need permissions to do so) Cheers, Tim On Tue, Jan 2, 2018 at 4:30 PM, kenneth mcfarland <

Re: Could anybody help to explain why there is such a big gap in 'ProbeTime' and How to fix this gap?

2018-01-03 Thread Tim Armstrong
That's a tricky one. I have a couple of ideas but it's a bit difficult to confirm since the profile isn't really designed to easily answer questions like this. ProbeTime measures wall-clock time rather than actual time spent executing on the CPU. My first guess is that it's because the Kudu scan

Re: Joining Impala mailing list

2018-01-03 Thread Tim Armstrong
and corner of the > "Contributors" box. A text box will appear to the right of the > contributor box. > > 3. Type the name of the person in that box to attempt to find their > Apache JIRA username. > > 4. Once you have it, press "Update" > > > On We

Re: Please hold off on merging changes that don't fix builds

2018-02-26 Thread Tim Armstrong
, but please hold off on any large commits for now until builds are reliably green. On Fri, Feb 23, 2018 at 3:25 PM, Tim Armstrong <tarmstr...@cloudera.com> wrote: > I'm triaging a bunch of builds now that have broken with the influx of > large changes this week. > > I'll cont

impala-4835 gerrit branch

2018-03-15 Thread Tim Armstrong
You might see some gerrit notifications about commits to this branch. I created this temporary branch to make it easier to incrementally review fixes to my original IMPALA-4835 patches. I won't merge the changes to master without consultation with others. - Tim

Re: Re: Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

2018-04-10 Thread Tim Armstrong
ses are cut. And > >since there are no minimum waiting period between releases, we could cut > >those right after that patch lands if we like. > > > >On Mon, Apr 2, 2018 at 4:05 PM, Tim Armstrong <tarmstr...@cloudera.com> > >wrote: > > > >>

Re: Maintenance for jenkins.impala.io today

2018-04-11 Thread Tim Armstrong
Is it possible that the upgrade broke some of our builds? I'm seeing some weird errors like the below, e.g. on this build https://jenkins.impala.io/job/clang-tidy-ub1604/1327/consoleFull *17:38:46* FATAL: java.lang.RuntimeException: Failed to serialize hudson.model.Actionable#actions for class

Re: Help with task: Allow fully qualified table names to start with a number

2018-04-13 Thread Tim Armstrong
Why is it incompatible? On Fri, Apr 13, 2018 at 9:05 AM, Alexander Behm <alex.b...@cloudera.com> wrote: > IMPALA-452 is not a good candidate because it's an incompatible change, > > On Fri, Apr 13, 2018 at 9:00 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: &g

Re: Help with task: Allow fully qualified table names to start with a number

2018-04-13 Thread Tim Armstrong
ariant and go with it. > > Determining the variant later during compilation requires deeper changes. > > On Fri, Apr 13, 2018 at 9:55 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: > > > Why is it incompatible? > > > > > > On Fri, Apr 13, 20

Re: Help with task: Allow fully qualified table names to start with a number

2018-04-13 Thread Tim Armstrong
Here are a few JIRAs that might be good starters: https://issues.apache.org/jira/browse/IMPALA-6844 (bug in builtin function) https://issues.apache.org/jira/browse/IMPALA-6393 (adding a small improvement to impala-shell) https://issues.apache.org/jira/browse/IMPALA-6234 (fixing a bug in query

Re: Reminder to merge commits to the ASF repo

2018-04-23 Thread Tim Armstrong
One tip is that the ASF JIRA bot doesn't post on JIRAs until the commits are pushed to ASF. So if you're resolving a JIRA and you don't see comments from the JIRA bot, you probably have forgotten to push your commits. - Tim On Mon, Apr 23, 2018 at 10:49 AM, Henry Robinson

Re: JIRA necromancy

2018-04-17 Thread Tim Armstrong
> wrote: > I agree, a JIRA shouldn't be reopened unless it's high confidence that the > original root cause wasn't actually addressed. > > On Tue, Apr 17, 2018 at 10:29 AM, Tim Armstrong <tarmstr...@cloudera.com> > wrote: > > > I noticed that there's been a trend rec

JIRA necromancy

2018-04-17 Thread Tim Armstrong
I noticed that there's been a trend recently towards reopening old issues instead of filing new issues. Not trying to pick on anyone but it seems like its worth having a discussion about best practices. Personally I think reopening JIRAs is often a bad thing for a several reasons: * We don't

Re: Re: Re: ORC scanner - points for discussion

2018-03-28 Thread Tim Armstrong
-existing issues as a result of the testing done on this patch: https://issues.apache.org/jira/issues/?jql=project%20%3D%20ORC%20AND%20reporter%20in%20(stiga-huang) On Mon, Feb 12, 2018 at 1:52 PM, Tim Armstrong <tarmstr...@cloudera.com> wrote: > Maybe it would make sense to create an Epi

Re: Re: [DISCUSS] Apache Impala 2.12.0 and 3.0 releases

2018-04-02 Thread Tim Armstrong
I don't think we should hold the release for the ORC scanner patch - it's a great feature, but there are many other equally important features and improvements that have been waiting for months to be released. Generally, there are always going to be big changes that are "nearly there" and I think

Re: questions about resource management of impala

2018-03-16 Thread Tim Armstrong
Hi, It's extremely common to run YARN and Impala on the same cluster. Currently we recommend setting up the cluster to divide CPU and memory resources between YARN and Impala statically. The idea is to set Impala's and YARN's memory limits to share the available memory, and then optionally to use

Re: Re: Future of unsupported formats?

2018-06-21 Thread Tim Armstrong
cing your work on fixing the recent > broken/flaky tests. > > > > Thanks for your hard working on this wonderful product! > > > > > > -- Quanlong > > > > > > At 2018-06-20 04:12:38, "Jim Apple" > wrote: > > >I'm convinced. Th

boost::scoped_ptr vs std::unique_ptr

2018-07-05 Thread Tim Armstrong
I was just talking with Michael Ho on a review about this https://gerrit.cloudera.org/#/c/10810/7/be/src/exec/scan-node.h@271 For a while we've continued using scoped_ptr in some places because it supports a smaller set of operators and implies that the pointer isn't movable. See

Re: boost::scoped_ptr vs std::unique_ptr

2018-07-05 Thread Tim Armstrong
the subtle difference between scoped_ptr and unique_ptr > off > > > the > > > > top of their head anyways. > > > > > > > > Fwiw, I was under the impression from talking with people in the past > > > that > > > > we were already

Re: Impala OS and Java version support

2018-07-06 Thread Tim Armstrong
Sounds like a good idea to me. Java 8 makes sense to me too since the major Java implementations (OpenJDK and Oracle) no longer support the older JDKs. On Fri, Jul 6, 2018 at 9:58 AM, Sailesh Mukil wrote: > I'm in favor of the minimum OS version support proposal. From 3.x, I don't > see any

Re: boost::scoped_ptr vs std::unique_ptr

2018-07-06 Thread Tim Armstrong
gt; the globe weigh in. > > BR, > Zoltan > > > On Fri, Jul 6, 2018 at 3:31 AM Jim Apple > wrote: > > > SGTM > > > > On Thu, Jul 5, 2018 at 6:13 PM, Tim Armstrong < > > tarmstr...@cloudera.com.invalid> wrote: > > > > > Sounds like un

Re: Inconsistent handling of schema in Avro tables

2018-07-11 Thread Tim Armstrong
The behaviour of Avro schemas in all these cases has always been rather mysterious to me. Before you wrote this email I would have assumed that Impala's behaviour would be like Hive's behaviour. I agree with the principle that the creation of a partition without changes to table metadata