impala git commit: IMPALA-7035: Configure jceks.key.serialFilter for KMS.

2018-05-17 Thread philz
Repository: impala Updated Branches: refs/heads/master ccf19f9f8 -> 5b824408a IMPALA-7035: Configure jceks.key.serialFilter for KMS. Configures a Java property for KMS to account for JDK 8u171's security fixes. I was seeing impala-py.test tests/metadata/test_hdfs_encryption.py fail with the fo

[1/2] impala git commit: IMPALA-7035: Configure jceks.key.serialFilter for KMS.

2018-05-18 Thread philz
Repository: impala Updated Branches: refs/heads/2.x 5f9641043 -> fe57b2009 IMPALA-7035: Configure jceks.key.serialFilter for KMS. Configures a Java property for KMS to account for JDK 8u171's security fixes. I was seeing impala-py.test tests/metadata/test_hdfs_encryption.py fail with the follo

[1/2] impala git commit: Moving default sanitizer options into init.cc from shell scripts.

2018-05-18 Thread philz
Repository: impala Updated Branches: refs/heads/master ab75dd12e -> 85ed7ae88 Moving default sanitizer options into init.cc from shell scripts. When running tests with ASAN, you need to set ASAN_OPTIONS explicitly, to avoid various failures. In particular, backend tests fail complaining about

[2/2] impala git commit: IMPALA-6070: Adding ASAN, --tail to test-with-docker.

2018-05-18 Thread philz
IMPALA-6070: Adding ASAN, --tail to test-with-docker. * Adds -ASAN suites to test-with-docker. * Adds --tail flag, which starts a tail subprocess. This isn't pretty (there's potential for overlap), but it's a dead simple way to keep an eye on what's going on. * Fixes a bug wherein I could call

[2/2] impala git commit: test-with-docker: work with git worktree

2018-05-18 Thread philz
test-with-docker: work with git worktree This commit adds a little of git-wrangling to allow test-with-docker to work when invoked from git directories managed by "git worktree". These are different in that they reference another git directory elsewhere on the file system, which also needs to be m

[4/5] impala git commit: IMPALA-7019: Schedule EC as remote & disable failed tests

2018-05-21 Thread philz
IMPALA-7019: Schedule EC as remote & disable failed tests This patch schedules HDFS EC files without considering locality. Failed tests are disabled and a jenkins build should succeed with export ERASURE_COINDG=true. Testing: It passes core tests. Cherry-picks: not for 2.x. Change-Id: I138738d3

[5/5] impala git commit: IMPALA-7051: Serialize Maven invocations.

2018-05-21 Thread philz
IMPALA-7051: Serialize Maven invocations. I've observed some rare cases where Impala fails to build. I believe it's because two Maven targets (yarn-extras and ext-data-source) are being executed simultaneously. Maven's handling of ~/.m2/repository, for example, is known to be not safe. This patch

[3/5] impala git commit: IMPALA-7011: Simplify PlanRootSink control logic

2018-05-21 Thread philz
IMPALA-7011: Simplify PlanRootSink control logic 1) The eos_ and sender_done_ bits really encode three possible states that the sender can be in. Make this explicit using an enum with three values. 2) The purpose of CloseConsumer() has changed over time and we can clean this up now: a)

[1/5] impala git commit: IMPALA-6317: Add -cmake_only option to buildall.sh

2018-05-21 Thread philz
Repository: impala Updated Branches: refs/heads/master 5c7d3b12e -> 23e11dc72 IMPALA-6317: Add -cmake_only option to buildall.sh It's sometimes useful to be able to build a complete Impala dev environment without necessarily building the Impala binary itself -- e.g., when one wants to use the

[2/5] impala git commit: [DOCS] Fixed misleading documentation on Impala + HDFS caching

2018-05-21 Thread philz
[DOCS] Fixed misleading documentation on Impala + HDFS caching Change-Id: I63cd1ff7b885a094a4a3e91c31101d25414b4db7 Reviewed-on: http://gerrit.cloudera.org:8080/10454 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit:

[3/4] impala git commit: IMPALA-6070: Adding ASAN, --tail to test-with-docker.

2018-05-21 Thread philz
IMPALA-6070: Adding ASAN, --tail to test-with-docker. * Adds -ASAN suites to test-with-docker. * Adds --tail flag, which starts a tail subprocess. This isn't pretty (there's potential for overlap), but it's a dead simple way to keep an eye on what's going on. * Fixes a bug wherein I could call

[4/4] impala git commit: IMPALA-3833: Fix invalid data handling in Sequence and RCFile scanners

2018-05-21 Thread philz
IMPALA-3833: Fix invalid data handling in Sequence and RCFile scanners Introduced new error message when scanning a corrupt Sequence or RCFile. Added new checks to detect buffer overrun while handling Sequence or RCFile. Testing: a) Made changes to fuzz test for RCFile/Sequence file, ran fuzz t

[2/4] impala git commit: Moving default sanitizer options into init.cc from shell scripts.

2018-05-21 Thread philz
Moving default sanitizer options into init.cc from shell scripts. When running tests with ASAN, you need to set ASAN_OPTIONS explicitly, to avoid various failures. In particular, backend tests fail complaining about memory leaks and tests that use the parquet-reader binary complain similarly. It t

[1/4] impala git commit: IMPALA-6998: test_bloom_wait_time fails due to late arrival of filters on Isilon

2018-05-21 Thread philz
Repository: impala Updated Branches: refs/heads/2.x 75d19c874 -> fb876f7e3 IMPALA-6998: test_bloom_wait_time fails due to late arrival of filters on Isilon This test has been failing on Isilon runs, most likely due to timing issues which makes it a test issue rather than a product bug. This p

[1/2] impala git commit: IMPALA-4993: extend dictionary filtering to collections

2018-01-19 Thread philz
Repository: impala Updated Branches: refs/heads/2.x 579e33207 -> 4afabd4e3 IMPALA-4993: extend dictionary filtering to collections Currently, top-level scalar columns in parquet files can be used at runtime to prune row-groups by evaluating certain conjuncts over the column's dictionary (if av

[2/2] impala git commit: IMPALA-5310: [DOCS] Reserve 'repeatable' keyword from TABLESAMPLE clause

2018-01-19 Thread philz
IMPALA-5310: [DOCS] Reserve 'repeatable' keyword from TABLESAMPLE clause Overlooked the new keyword when the clause was originally introduced. Change-Id: Ie8e6713fb97ced279f0aedfe8f42c09a7e6edae9 Reviewed-on: http://gerrit.cloudera.org:8080/9066 Reviewed-by: Alex Behm Tested-by: Impala Public Je

[2/5] impala git commit: IMPALA-6427: fix QUERYOPTIONS in planner test output

2018-01-22 Thread philz
IMPALA-6427: fix QUERYOPTIONS in planner test output Testing: Ran planner tests. Confirmed that the generated union.test included the QUERYOPTIONS section and that test files without the QUERYOPTIONS section did not have QUERYOPTIONS sections added. Change-Id: I2971588f977f72b0f869370803b089d5630

[3/5] impala git commit: IMPALA-6422: Use ldexp() instead of powf() in HLL.

2018-01-22 Thread philz
IMPALA-6422: Use ldexp() instead of powf() in HLL. Using ldexp() to compute a floating point power of two is over 10x faster than powf(). This change is particularly helpful for speeding up COMPUTE STATS TABLESAMPLE which has many calls to HllFinalEstimate() where floating point power of two comp

[5/5] impala git commit: IMPALA-6420: Fix TestCharFormats for local filesystem tests

2018-01-22 Thread philz
IMPALA-6420: Fix TestCharFormats for local filesystem tests Tl;dr - running tests was covering up a bug in the tests, or alternatively, not running tests exposes a bug. Local tests were failing because we failed to prepend the local filesystem prefix. This was covered up because the snapshot cre

[1/5] impala git commit: IMPALA-5654: Disallow setting Kudu table name in CREATE TABLE

2018-01-22 Thread philz
Repository: impala Updated Branches: refs/heads/2.x 4afabd4e3 -> 1bb3547e4 IMPALA-5654: Disallow setting Kudu table name in CREATE TABLE This change disallows explicitly setting the Kudu table name property for managed Kudu tables in a CREATE TABLE statement. The Kudu table name property gets

[4/5] impala git commit: IMPALA-6092: run flaky test serially (temporary).

2018-01-22 Thread philz
IMPALA-6092: run flaky test serially (temporary). The reason for the flake is described in IMPALA-6215. This patch runs the test serially to help reduce the recent increase of flakes on gvo. Change-Id: I4a4165f0e82f270df2c8a3af087a9a6ec63fd086 Reviewed-on: http://gerrit.cloudera.org:8080/9080 Rev

[1/6] impala git commit: IMPALA-6422: Use ldexp() instead of powf() in HLL.

2018-01-22 Thread philz
Repository: impala Updated Branches: refs/heads/master 4afabd4e3 -> 7f652ce65 IMPALA-6422: Use ldexp() instead of powf() in HLL. Using ldexp() to compute a floating point power of two is over 10x faster than powf(). This change is particularly helpful for speeding up COMPUTE STATS TABLESAMPLE

[5/6] impala git commit: IMPALA-6092: run flaky test serially (temporary).

2018-01-22 Thread philz
IMPALA-6092: run flaky test serially (temporary). The reason for the flake is described in IMPALA-6215. This patch runs the test serially to help reduce the recent increase of flakes on gvo. Change-Id: I4a4165f0e82f270df2c8a3af087a9a6ec63fd086 Reviewed-on: http://gerrit.cloudera.org:8080/9080 Rev

[3/6] impala git commit: IMPALA-6427: fix QUERYOPTIONS in planner test output

2018-01-22 Thread philz
IMPALA-6427: fix QUERYOPTIONS in planner test output Testing: Ran planner tests. Confirmed that the generated union.test included the QUERYOPTIONS section and that test files without the QUERYOPTIONS section did not have QUERYOPTIONS sections added. Change-Id: I2971588f977f72b0f869370803b089d5630

[4/6] impala git commit: Bumping version to 3.0.

2018-01-22 Thread philz
Bumping version to 3.0. This changes the version that Impala presents as to 3.0. We are simultaneously introducing a 2.x branch that continues to identify itself as 2.x (where x=11 right now). Cherry-picks: not for 2.x. Change-Id: Id39f9648cb9b40b67b1029fa8c4132cd04c1d0c6 Reviewed-on: http://gerr

[6/6] impala git commit: IMPALA-5654: Disallow setting Kudu table name in CREATE TABLE

2018-01-22 Thread philz
IMPALA-5654: Disallow setting Kudu table name in CREATE TABLE This change disallows explicitly setting the Kudu table name property for managed Kudu tables in a CREATE TABLE statement. The Kudu table name property gets a generated value as the following: 'impala::db_name.table_name' where table_na

[2/6] impala git commit: IMPALA-6420: Fix TestCharFormats for local filesystem tests

2018-01-22 Thread philz
IMPALA-6420: Fix TestCharFormats for local filesystem tests Tl;dr - running tests was covering up a bug in the tests, or alternatively, not running tests exposes a bug. Local tests were failing because we failed to prepend the local filesystem prefix. This was covered up because the snapshot cre

[18/19] impala git commit: IMPALA-3562: support column restriction for compute stats

2018-02-02 Thread philz
IMPALA-3562: support column restriction for compute stats The 'compute stats' statement currently computes column-level statistics for all columns of a table. This adds potentially unneeded work for columns whose stats are not needed by queries. It can be especially costly for very wide tables and

[09/19] impala git commit: IMPALA-6303: [DOCS] Fix incorrect mention of DataNodes

2018-02-02 Thread philz
IMPALA-6303: [DOCS] Fix incorrect mention of DataNodes The documentation on Impala Components (https://impala.apache.org/ docs/build/html/topics/impala_components.html) incorrectly states "... catalog service relays the metadata changes from Impala SQL statements to all the DataNodes in a cluster

[11/19] impala git commit: IMPALA-6455: unique tmpdirs for test_partition_metadata_compatibility

2018-02-02 Thread philz
IMPALA-6455: unique tmpdirs for test_partition_metadata_compatibility Concurrent hive statements running in local mode can race to modify the contents of temporary directories - see IMPALA-6108. This applies the workaround for IMPALA-6108 to the run_stmt_in_hive() utility function, which is used b

[02/19] impala git commit: IMPALA-5528: Bump total thread cache size when KRPC is enabled

2018-02-02 Thread philz
IMPALA-5528: Bump total thread cache size when KRPC is enabled KRPC in general tends to put more pressure on the thread caches due to allocations of more small objects (i.e. <1MB). While some of them are being addressed in KUDU-1865, it's shown that the following TCMalloc workarounds will provide

[08/19] impala git commit: IMPALA-5054: [SECURITY] Enable KRPC w/ TLS in Impala

2018-02-02 Thread philz
IMPALA-5054: [SECURITY] Enable KRPC w/ TLS in Impala KRPC has some flags that turn on TLS. This patch sets those to enable TLS communication. Tests are added to rpc-mgr-test. TODO: Kudu kerberos testing is disabled. Will re-enable as part of IMPALA-6448. Change-Id: I9a14a44fdea9ab668f3714eb69fd

[04/19] impala git commit: IMPALA-6454: CTAS into Kudu fails with mixed-case partition or primary key column names.

2018-02-02 Thread philz
IMPALA-6454: CTAS into Kudu fails with mixed-case partition or primary key column names. CTAS into Kudu fails if the primary key and/or the partition column names are not specified in lower case.The problem is that we pass in the primary key column names directly from the parser instead we shou

[01/19] impala git commit: IMPALA-4835: prerequisite buffer pool changes

2018-02-02 Thread philz
Repository: impala Updated Branches: refs/heads/2.x cefc212c5 -> 7097ee88d IMPALA-4835: prerequisite buffer pool changes The scanner/buffer pool changes will have different scanner threads sharing the same buffer pool client. This requires that the AllocateBuffer() API is safe to call concurre

[14/19] impala git commit: IMPALA-6450: fix EventSequence::Start()

2018-02-02 Thread philz
IMPALA-6450: fix EventSequence::Start() It looks like this newly-added DCHECK is being hit because of the same underlying issue as IMPALA-4631. This patch loosens the DCHECK to accept time going backward 1 tick, the same as the original workaround for IMPALA-4631. 'offset_' also isn't being used,

[3/3] impala git commit: IMPALA-6346: Potential deadlock in KrpcDataStreamMgr

2018-02-02 Thread philz
IMPALA-6346: Potential deadlock in KrpcDataStreamMgr In KrpcDataStreamMgr::CreateRecvr() we take the lock_ and then call recvr->TakeOverEarlySender() for all contexts. recvr->TakeOverEarlySender() then calls recvr_->mgr_->EnqueueDeserializeTask((), which can block if the deserialize pool queue is

[07/19] impala git commit: IMPALA-6447: remove Python 2.7 dictionary comprehensions

2018-02-02 Thread philz
IMPALA-6447: remove Python 2.7 dictionary comprehensions In the fix for IMPALA-6441, we began importing from the stress test (concurrent_select.py). The import fails on some downstream environments that use Python 2.6. The failure is due to the fact that concurrent_select.py uses a few dictionary

[2/3] impala git commit: IMPALA-3916: Reserve SQL:2016 reserved words

2018-02-02 Thread philz
IMPALA-3916: Reserve SQL:2016 reserved words This patch reserves SQL:2016 reserved words, excluding: 1. Impala builtin function names. 2. Time unit words(year, month, etc.). 3. An exception list based on a discussion. Some test cases are modified to avoid these words. A impalad and catalogd start

[15/19] impala git commit: IMPALA-6430: Log relevant debug pages if wait_for_metric_value times out

2018-02-02 Thread philz
IMPALA-6430: Log relevant debug pages if wait_for_metric_value times out Log the memz, metrics and query page if the method wait_for_metric_value times out. This would help us understand the state of the defaulting impalad when the time out happens. Change-Id: I069dad48ede709c4114f4d7175861f98321

[05/19] impala git commit: IMPALA-6242: Change runtime-profile-test into using the same clock

2018-02-02 Thread philz
IMPALA-6242: Change runtime-profile-test into using the same clock In runtime-profile-test, both MonotonicStopWatch::Now() and MonotonicNanos() are used. The former may use CLOCK_MONOTONIC_COARSE or CLOCK_MONOTONIC while the latter always uses CLOCK_MONOTONIC. This may contribute to the flakiness

[03/19] impala git commit: IMPALA-6356: Reduce amount of logging from RpczStore::LogTrace()

2018-02-02 Thread philz
IMPALA-6356: Reduce amount of logging from RpczStore::LogTrace() This change bumps the threshold of RPC duration above which a RPC is logged. It's increased from 1 second to 2 minutes which is a conservative value in order to reduce the amount of logging from RpczStore::LogTrace() when an Impala d

[06/19] impala git commit: IMPALA-6215: Removes race when using LibCache.

2018-02-02 Thread philz
IMPALA-6215: Removes race when using LibCache. LibCache's api to provide access to locally cached files has a race. Currently, the client of the cache accesses the locally cached path as a string, but nothing guarantees that the associated file is not removed before the client is done using it. Th

[1/3] impala git commit: IMPALA-3916: Reserve SQL:2016 reserved words

2018-02-02 Thread philz
Repository: impala Updated Branches: refs/heads/master 4bd7cc8db -> ff86feaa6 http://git-wip-us.apache.org/repos/asf/impala/blob/f0b3d9d1/testdata/workloads/functional-query/queries/QueryTest/partition-col-types.test -- diff --

[10/19] impala git commit: IMPALA-6024: Min sample bytes for COMPUTE STATS TABLESAMPLE

2018-02-02 Thread philz
IMPALA-6024: Min sample bytes for COMPUTE STATS TABLESAMPLE Adds a new query option COMPUTE_STATS_MIN_SAMPLE_SIZE which is the minimum number of bytes that will be scanned in COMPUTE STATS TABLESAMPLE, regardless of the user-supplied sampling percent. The motivation is to prevent sampling for ver

[12/19] impala git commit: IMPALA-2642: Fix a potential deadlock in statestore

2018-02-02 Thread philz
IMPALA-2642: Fix a potential deadlock in statestore The statestored can deadlock if the number of subscribers has reached STATESTORE_MAX_SUBSCRIBERS, because the DoSubscriberUpdate() method calls OfferUpdate(), while holding subscribers_lock_, which also tries to take the same lock in this situati

[19/19] impala git commit: IMPALA-6429: Fix decimal division

2018-02-02 Thread philz
IMPALA-6429: Fix decimal division Before this patch, it was possible for an overflow to not be detected when doing a decimal division. When scaling up the dividend before doing the division, we do not check for overflow. This is ok if the we are scaling up by 10^38 or less because the result is gu

[17/19] impala git commit: IMPALA-6441 addendum: fix reading rows from HS2 via Impyla

2018-02-02 Thread philz
IMPALA-6441 addendum: fix reading rows from HS2 via Impyla When fetching explain output from HS2 using Impyla, rows come back in lists of 1-tuples. This patch exhibits the need to do end-to-end testing when the case warrants. In this case, although the unit test for http://gerrit.cloudera.org:808

[16/19] impala git commit: IMPALA-6193: Track memory of incoming data streams

2018-02-02 Thread philz
IMPALA-6193: Track memory of incoming data streams This change adds memory tracking to incoming transmit data RPCs when using KRPC. We track memory against a global tracker called "Data Stream Service" until it is handed over to the stream manager. There we track it in a global tracker called "Dat

[13/19] impala git commit: IMPALA-3282: Adds regexp_escape built-in function

2018-02-02 Thread philz
IMPALA-3282: Adds regexp_escape built-in function Escapes the following special characters in RE2 library: .\+*?[^]$(){}=!<>|:- Testing: Add some unit tests into ExprTest.StringRegexpFunctions Add some E2E tests into exprs.test Change-Id: I84c3e0ded26f6eb20794c38b75be9b25cd111e4b Reviewed-on: ht

[10/12] impala git commit: IMPALA-6731: Move execnet Python dependency to stage 2

2018-03-31 Thread philz
IMPALA-6731: Move execnet Python dependency to stage 2 It seems that execnet also cannot be installed together with setuptools-scm if only a local mirror and index are available (similar to https://github.com/pywebhdfs/pywebhdfs/issues/52). Testing: Observed that execnet failed to install during

[02/12] impala git commit: IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and KrpcDataStreamSender

2018-03-31 Thread philz
IMPALA-6685: Improve profiles in KrpcDataStreamRecvr and KrpcDataStreamSender This change implements a couple of improvements to the profiles of KrpcDataStreamRecvr and KrpcDataStreamSender: - track pending number of deferred row batches over time in KrpcDataStreamRecvr - track the number of byte

[03/12] impala git commit: IMPALA-6759: align stress test memory estimation parse pattern

2018-03-31 Thread philz
IMPALA-6759: align stress test memory estimation parse pattern The stress test never expected to see memory estimates on the order of PB. Apparently it can happen with TPC DS 1, so update the pattern. It's not clear how to quickly write a test to catch this, because it involves crossing langu

[08/12] impala git commit: IMPALA-5384, part 1: introduce DmlExecState

2018-03-31 Thread philz
IMPALA-5384, part 1: introduce DmlExecState This change is based on a patch by Marcel Kornacker. Move data structures that collect DML operation stats from the RuntimeState and Coordinator into a new InsertExecState class, which has it's own lock. This removes a dependency on the coordinator's l

[06/12] impala git commit: IMPALA-6747: Automate diagnostics collection.

2018-03-31 Thread philz
IMPALA-6747: Automate diagnostics collection. This commit adds the necessary tooling to automate diagnostics collection for Impala daemons. Following diagnostics are supported. 1. Native core dump (+ shared libs) 2. GDB/Java thread dump (pstack + jstack) 3. Java heap dump (jmap) 4. Minidumps (usi

[07/12] impala git commit: IMPALA-5384, part 1: introduce DmlExecState

2018-03-31 Thread philz
http://git-wip-us.apache.org/repos/asf/impala/blob/e882cbb9/be/src/service/impala-server.cc -- diff --git a/be/src/service/impala-server.cc b/be/src/service/impala-server.cc index ea88d73..9b1c2f5 100644 --- a/be/src/service/impala

[11/12] impala git commit: IMPALA-6760: Fix for py2.7-ism in run-tests.py.

2018-03-31 Thread philz
IMPALA-6760: Fix for py2.7-ism in run-tests.py. A set-literal snuck into run-tests.py in a recent change. We wish to avoid these to be able to run on py2.6. Change-Id: I81928d1880a493b91abb13b3a8149568c9789f66 Reviewed-on: http://gerrit.cloudera.org:8080/9843 Reviewed-by: Philip Zeyliger Tested-

[09/12] impala git commit: Revert "IMPALA-6389: Make '\0' delimited text files work"

2018-03-31 Thread philz
Revert "IMPALA-6389: Make '\0' delimited text files work" This reverts commit c2bdaf8af4cf35d3462595c2a341ed84dcf5d960. An ASAN issue and potentially other problem have been found; reverting to unbreak the build and tests. Change-Id: If581311033de8c26e33316b19192c4579594f261 Reviewed-on: http://

[05/12] impala git commit: KUDU-2385: Fix typo in KinitContext::DoRenewal()

2018-03-31 Thread philz
KUDU-2385: Fix typo in KinitContext::DoRenewal() On platforms without krb5_get_init_creds_opt_set_out_ccache(), krb5_cc_store_cred() is called to insert the newly acquired credential into the ccache. However, there was a typo in the code which resulted in inserting the old credential into ccache.

[01/12] impala git commit: IMPALA-6694: fix "buffer pool" child profile order

2018-03-31 Thread philz
Repository: impala Updated Branches: refs/heads/2.x fc0af7f2a -> 0197b17fa IMPALA-6694: fix "buffer pool" child profile order The bug is that child profiles can be re-ordered when being sent between an executor and a coordinator. This occurs if child profile A is present in one update, then an

[04/12] impala git commit: IMPALA-5721, IMPALA-6717, IMPALA-6738: improve stress test binary search

2018-03-31 Thread philz
IMPALA-5721,IMPALA-6717,IMPALA-6738: improve stress test binary search IMPALA-5721: - Save profiles of queries at the end of both the spilling and non-spilling binary search. These were not being saved before. Note these profiles won't have ExecSummary until IMPALA-6640 is addressed. - Save t

[12/12] impala git commit: Revert "IMPALA-6747: Automate diagnostics collection."

2018-03-31 Thread philz
Revert "IMPALA-6747: Automate diagnostics collection." A couple of things donot work in python2.6 -- Multiple with statements in the same context -- shutil.make_archive() I need a little more time to test the fix with python2.6. Meanwhile, reverting this to unblock others. I'll resubmit the fix

[2/7] impala git commit: Loosen hive-exec.jar glob pattern in copy-udfs-udas.sh.

2018-04-06 Thread philz
Loosen hive-exec.jar glob pattern in copy-udfs-udas.sh. This commit slightly loosens the coupling between IMPALA_HIVE_VERSION and "hive.version" in the Maven sense. Cherry-picks: not for 2.x Change-Id: Ifbe6f5208b4ad0ffc9cbfe4e93d712ce698beb23 Reviewed-on: http://gerrit.cloudera.org:8080/9925 Re

[7/7] impala git commit: IMPALA-6070: Expose using Docker to run tests faster.

2018-04-06 Thread philz
IMPALA-6070: Expose using Docker to run tests faster. Allows running the tests that make up the "core" suite in about 2 hours. By comparison, https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/buildTimeTrend tends to run in about 3.5 hours. This commit: * Adds "echo" statements in a few pla

[4/7] impala git commit: IMPALA-6807: [DOCS] Update the known issue for HDFS-12528

2018-04-06 Thread philz
IMPALA-6807: [DOCS] Update the known issue for HDFS-12528 Added a new recommendation for the new setting with the fix version of HDFS, 2.10 and higher. Change-Id: If51cb111a9ddc67be4a1cf42502a8a021486b7e4 Reviewed-on: http://gerrit.cloudera.org:8080/9929 Reviewed-by: Joe McDonnell Tested-by: Imp

[3/7] impala git commit: IMPALA-6389: Make '\0' delimited text files work

2018-04-06 Thread philz
IMPALA-6389: Make '\0' delimited text files work Initially I didn't want to fully implement this, as the metadata for these tables can't even be fully stored in Postgres; however after digging into some older documentation, it appears that the ASCII NUL character actually has been used as a field

[5/7] impala git commit: [DOCS] Removed old files no longer in use

2018-04-06 Thread philz
[DOCS] Removed old files no longer in use Change-Id: Ia0aaa756bf0fc0092186ebc293543408b403aa5e Reviewed-on: http://gerrit.cloudera.org:8080/9938 Reviewed-by: John Russell Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit: http://git-wip-us.apac

[1/7] impala git commit: IMPALA-6801: Cleanup request_pool

2018-04-06 Thread philz
Repository: impala Updated Branches: refs/heads/master 4d6b07f0e -> 2896b8d12 IMPALA-6801: Cleanup request_pool Eliminate the copy of this field in the QuerySchedule. Instead, just set it directly in the TQueryCtx early on. Then, the TQueryCtx doesn't need to be copied by the coordinator. Cha

[6/7] impala git commit: KUDU-2401: External TLS certificate with Intermediate CA in server cert file fails

2018-04-06 Thread philz
KUDU-2401: External TLS certificate with Intermediate CA in server cert file fails Take 2 certificate files: cert.pem and truststore.pem cert.pem has 2 certificates in it: A cert for that node (with CN="hostname", and signed by CN=CertToolkitIntCA) And the intermediate CA cert (with CN=CertToolk

impala git commit: IMPALA-6811: Increase timeout on S3 for test_exchange_delays

2018-04-12 Thread philz
Repository: impala Updated Branches: refs/heads/2.x 867fc82b8 -> 04bc6162b IMPALA-6811: Increase timeout on S3 for test_exchange_delays The exchange-delays-zero-rows portion of test_exchange_delays tests a RPC timeout when sending an EOS with no rows. In order to send the EOS, the executor mus

[6/6] impala git commit: IMPALA-6822: Add a query option to control shuffling by distinct exprs

2018-04-12 Thread philz
IMPALA-6822: Add a query option to control shuffling by distinct exprs IMPALA-4794 changed the distinct aggregation behavior to shuffling by both grouping exprs and the distinct expr. It's slower in queries where the NDVs of grouping exprs are high and data are uniformly distributed among groups.

[2/6] impala git commit: IMPALA-6480: [DOCS] DESCRIBE respects column-level privilege

2018-04-12 Thread philz
IMPALA-6480: [DOCS] DESCRIBE respects column-level privilege Cherry-picks: not for 2.x. Change-Id: I094e00c2a4e8b19226e06afd8cf67968265edc4d Reviewed-on: http://gerrit.cloudera.org:8080/9996 Reviewed-by: Alex Behm Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/as

[4/6] impala git commit: IMPALA-5814: Remove startup flag to disable admission control

2018-04-12 Thread philz
IMPALA-5814: Remove startup flag to disable admission control Remove "--disable admission control" startup flag and its related functionality and usage. Cherry-picks: not for 2.x Change-Id: I9bf4087ce03ca63f82fd27c6d94b578881b85d42 Reviewed-on: http://gerrit.cloudera.org:8080/9964 Reviewed-by: T

[1/6] impala git commit: IMPALA-6842: [DOCS] Remove disable_admission_control startup flag

2018-04-12 Thread philz
Repository: impala Updated Branches: refs/heads/master 15b388c5f -> 9a751f00b IMPALA-6842: [DOCS] Remove disable_admission_control startup flag Change-Id: Idbd15823308dbce5d2d00e79607e5ebbdab3e38f Reviewed-on: http://gerrit.cloudera.org:8080/10046 Reviewed-by: Alex Rodoni Tested-by: Impala Pu

[3/6] impala git commit: IMPALA-2717: fix output of formatted unicode to non-TTY

2018-04-12 Thread philz
IMPALA-2717: fix output of formatted unicode to non-TTY The bug is that PrettyOutputFormatter.format() returned a unicode object, and Python cannot automatically write unicode objects to output streams where there is no default encoding. The fix is to convert to UTF-8 encoded in a regular string,

[5/6] impala git commit: Bump Kudu version to a954418

2018-04-12 Thread philz
Bump Kudu version to a954418 Change-Id: Ib06c0fb3c24a8cee1dd4f34a221cf41a711a5359 Reviewed-on: http://gerrit.cloudera.org:8080/9982 Reviewed-by: Alex Behm Tested-by: Impala Public Jenkins Project: http://git-wip-us.apache.org/repos/asf/impala/repo Commit: http://git-wip-us.apache.org/repos/asf

impala git commit: IMPALA-6353: Fix crash in snappy decompressor

2018-01-16 Thread philz
Repository: impala Updated Branches: refs/heads/master f8b406222 -> 6cc76d720 IMPALA-6353: Fix crash in snappy decompressor SnappyDecompressor::MaxOutputLen assumes the input pointer to be non-null. It's not true when the parquet file is corrupted and the compressed_page_size field in a page h

[impala] Git Push Summary

2018-01-17 Thread philz
Repository: impala Updated Branches: refs/heads/2.x [created] 6cc76d720

[2/2] impala git commit: test_recover_partitions.py had asserts that were always true.

2018-07-19 Thread philz
test_recover_partitions.py had asserts that were always true. Running "python -m compileall" discovered some assertions that were always true. I've re-instated them to their true spirit. Change-Id: Id49171304b853f15c43c8cfca066b6694c4a669f Reviewed-on: http://gerrit.cloudera.org:8080/10993 Review

[1/2] impala git commit: IMPALA-7259: Improve Impala shell performance

2018-07-19 Thread philz
Repository: impala Updated Branches: refs/heads/master 70e2d57fc -> 2a40e8f2a IMPALA-7259: Improve Impala shell performance This patch fixes the slow performance in Impala shell, especially for large queries by replacing all calls to sqlparse.format(sql_string, strip_comments=True) with the cu

impala git commit: IMPALA-7455: Exclude another log4j artifact.

2018-08-16 Thread philz
Repository: impala Updated Branches: refs/heads/master 864be3e60 -> 59435fe0a IMPALA-7455: Exclude another log4j artifact. log4j-1.2-api-2.8.2.jar is the "log4j2 bridge for log4j 1.2 api", and, if it is included on the classpath before log4j-1.2.17.jar, frontend logging doesn't work. This resu

[4/4] impala git commit: Fix python2.6 itertools.count() usage

2018-09-10 Thread philz
Fix python2.6 itertools.count() usage In python 2.6, itertools.count() can't take keyword arguments. Change-Id: I979e532101c9f0b76b149f80ad9aa390edd3f646 Reviewed-on: http://gerrit.cloudera.org:8080/11408 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong Project: http://git-wip-us.a

[3/4] impala git commit: IMPALA-7542: fix find-fragment-instances to find all "root threads"

2018-09-10 Thread philz
IMPALA-7542: fix find-fragment-instances to find all "root threads" find-fragment-instances didn't show all the threads that worked on some fragment instance. It missed the top-level "root threads" that started working on the fragment instances. I modified the get_fragment_instances() function to

[2/4] impala git commit: IMPALA-5031: Allow UBSAN to be set on codegen

2018-09-10 Thread philz
IMPALA-5031: Allow UBSAN to be set on codegen This adds a new buildall options, -full_ubsan, that is stronger than just -ubsan, in that it forces code generated by cross compilation to LLVM IR to use the undefined behavior sanitizer as well. Because this slows down testing significantly, it is not

[1/4] impala git commit: IMPALA-7522: Fix overflow in milliseconds_add()

2018-09-10 Thread philz
Repository: impala Updated Branches: refs/heads/master 1c6058fa1 -> 9cfa228c2 IMPALA-7522: Fix overflow in milliseconds_add() Adding milliseconds to a timestamp lead to overflow if the interval is very large. This hit a DCHECK in debug builds and returned incorrect results in release builds.

[impala] 02/05: IMPALA-7934: Switch to java.util.Base64 implementation

2019-02-01 Thread philz
This is an automated email from the ASF dual-hosted git repository. philz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/impala.git commit b0942296ab5f24660473abc218d45978fc402d81 Author: Fredy Wijaya AuthorDate: Tue Jan 22 13:29:34 2019 -0800 IMPALA

[impala] branch master updated (1c2778a -> a8e3050)

2019-02-01 Thread philz
This is an automated email from the ASF dual-hosted git repository. philz pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/impala.git. from 1c2778a IMPALA-8111: [DOCS] Take 2: Removed the Fix Version for KUDU-2198 new 79e735a IMPALA-8102: update

[impala] 03/05: IMPALA-7867 (Part 5): Collection cleanup in analyzer

2019-02-01 Thread philz
This is an automated email from the ASF dual-hosted git repository. philz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/impala.git commit 396f542eda32dd92e80edbeb216a4cdeb7fe0ace Author: paul-rogers AuthorDate: Wed Dec 26 23:02:00 2018 -0800 IMPALA-7867

[impala] 04/05: IMPALA-8137: [DOCS] Order By does not happens on one node

2019-02-01 Thread philz
This is an automated email from the ASF dual-hosted git repository. philz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/impala.git commit 6291d6063fe4ff9c483b60b8d9fc254298a51473 Author: Alex Rodoni AuthorDate: Thu Jan 31 17:05:19 2019 -0800 IMPALA-8137

[impala] branch 2.x updated: IMPALA-6479: Update DESCRIBE to respect column privileges

2019-02-01 Thread philz
This is an automated email from the ASF dual-hosted git repository. philz pushed a commit to branch 2.x in repository https://gitbox.apache.org/repos/asf/impala.git The following commit(s) were added to refs/heads/2.x by this push: new b795a2c IMPALA-6479: Update DESCRIBE to respect

[impala] 01/05: IMPALA-8102: update Impala/HBase docs

2019-02-01 Thread philz
This is an automated email from the ASF dual-hosted git repository. philz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/impala.git commit 79e735a46df258395ea518a5cf6e22e851a91119 Author: Tim Armstrong AuthorDate: Wed Jan 30 12:54:09 2019 -0800 IMPALA

[impala] 05/05: IMPALA-7980: Fix spinning because of buggy num_unqueued_files_.

2019-02-01 Thread philz
This is an automated email from the ASF dual-hosted git repository. philz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/impala.git commit a8e30506aafef14646d95a56fb87cf7c28d259d6 Author: Philip Zeyliger AuthorDate: Fri Dec 14 09:28:26 2018 -0800 IMPALA