[5/5] incubator-impala git commit: IMPALA-4927: Impala should handle invalid input from Sentry

2017-11-22 Thread tarmstrong
IMPALA-4927: Impala should handle invalid input from Sentry Impala requests a list of roles from Sentry and then asks for privileges for each role. If Sentry returns a non existent role in the first step, then there will be a Java exception in Impala in the second step and the communication with

[2/5] incubator-impala git commit: IMPALA-4985: use parquet stats of nested types for dynamic pruning

2017-11-22 Thread tarmstrong
IMPALA-4985: use parquet stats of nested types for dynamic pruning Currently, parquet row-groups can be pruned at run-time using min/max stats when predicates (in, binary) are specified for column scalar types. This patch extends pruning to nested types for the same class of predicates. A nested

[3/5] incubator-impala git commit: IMPALA-2250: Make multiple COUNT(DISTINCT) message state workarounds

2017-11-22 Thread tarmstrong
IMPALA-2250: Make multiple COUNT(DISTINCT) message state workarounds Change-Id: I5084be10946d68f3ec0760c2b7e698635df26a89 Reviewed-on: http://gerrit.cloudera.org:8080/8614 Reviewed-by: Alex Behm Tested-by: Impala Public Jenkins Project:

[1/5] incubator-impala git commit: IMPALA-6206: Fix data load failure with -notests

2017-11-22 Thread tarmstrong
Repository: incubator-impala Updated Branches: refs/heads/master 32baa695f -> c65618182 IMPALA-6206: Fix data load failure with -notests When tests are not built, specifically with -notests, instead of just -skiptests, the be-test target is omitted by cmake, and since nothing in impalad

[4/5] incubator-impala git commit: IMPALA-4964: Fix Decimal modulo overflow

2017-11-22 Thread tarmstrong
IMPALA-4964: Fix Decimal modulo overflow The modulo operation between two decimals should never overflow. Before this patch, there would be an overflow if the scale difference between the two decimals was large. We would try to scale up the one with the smaller scale, so that the scales matched,

incubator-impala git commit: IMPALA-6227: deflake admission stress tests

2017-11-23 Thread tarmstrong
Repository: incubator-impala Updated Branches: refs/heads/master c65618182 -> 1a7b0d0bd IMPALA-6227: deflake admission stress tests The problem was that, during the initial admission decision phase, some queries were initially queued then dequeued once memory came available. All of the

[7/7] impala git commit: IMPALA-6187: Fix missing conjuncts evaluation with empty projection

2017-11-28 Thread tarmstrong
IMPALA-6187: Fix missing conjuncts evaluation with empty projection Previously, scanners will assume that there are no conjuncts associated with a scan node for queries with no materialized slots (e.g. count(*)). This is not necessarily the case as one can write queries such as select count(*)

[5/7] impala git commit: IMPALA-1144: Fix exception when cancelling query in Impala-shell with CTRL-C

2017-11-28 Thread tarmstrong
IMPALA-1144: Fix exception when cancelling query in Impala-shell with CTRL-C Issue 1: When query is cancelled via CTRL-C while being executed in Impala-shell then an exception is thrown from Impala backend saying 'Invalid query handle'. This is because one ImpalaClient was making RPC's while

[1/7] impala git commit: IMPALA-5936: operator '%' overflows on large decimals

2017-11-28 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master d3afe3075 -> 63f17e9ce IMPALA-5936: operator '%' overflows on large decimals Suppose we have a large decimal number, which is greater than INT_MAX. We want to calculate the modulo of this number by 3: BIG_DECIMAL % 3 The result of this

[6/7] impala git commit: IMPALA-5146: Fix inconsitent results at FROM_UNIXTIME()

2017-11-28 Thread tarmstrong
IMPALA-5146: Fix inconsitent results at FROM_UNIXTIME() The FROM_UNIXTIME(epoch) and FROM_UNIXTIME(epoch, format) produce different results when epoch is out of range of TimestampValue. The former produces an empty string, while the latter gives NULL. The fix is to harmonize the results to NULL.

impala git commit: Update incubator-impala -> impala URLs

2017-11-29 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 63f17e9ce -> 72ed4fc88 Update incubator-impala -> impala URLs This fixes push_to_asf.py and various other scripts that had the Apache repo location hard-coded. Also fixed the location of the github mirror and mailing list archives.

impala git commit: IMPALA-5987: LZ4 Codec silently produces bogus compressed data for large inputs

2017-12-04 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master d428c16f1 -> 7e368b8f0 IMPALA-5987: LZ4 Codec silently produces bogus compressed data for large inputs When Lz4Compressor::MaxOutputLen returns 0, it means that the input is too large to compress. When invoked Lz4Compressor::ProcessBlock

[1/3] impala git commit: IMPALA-6222: Add details to error msg on failure to get min reservation

2017-12-15 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 3cbbaf3b3 -> bfbcd1fe8 IMPALA-6222: Add details to error msg on failure to get min reservation This patch adds the following details to the error message encountered on failure to get minimum memory reservation: - which ReservationTracker

[2/3] impala git commit: IMPALA-6114: Require type equality for NumericLiteral::localEquals().

2017-12-15 Thread tarmstrong
IMPALA-6114: Require type equality for NumericLiteral::localEquals(). This patch fixes a regression introduced as part of IMPALA-1788, where an expression like 'CAST(0 AS DECIMAL(14))' is rewritten as a NumericLiteral expression of type DECIMAL(14,0). The query had another NumericLiteral of type

[2/4] impala git commit: IMPALA-6245: Tolerate column indenting from Hive

2017-12-12 Thread tarmstrong
IMPALA-6245: Tolerate column indenting from Hive The fix for HIVE-3140 started indenting multi-line comments, which breaks Impala testing when run against Hive 2.1.1. To test this using the pure test runner proved difficult since it would require extensive changes to support both row_regexes

[1/4] impala git commit: IMPALA-6298: Skip test_profile_fragment_instances on local filesystem

2017-12-12 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 12fa20d1a -> d2fe9f437 IMPALA-6298: Skip test_profile_fragment_instances on local filesystem test_profile_fragment_instances was recently added to verify that the final runtime profile for a query has the expected fragments and exec

[4/4] impala git commit: IMPALA-6270: create Impala parent pom

2017-12-12 Thread tarmstrong
IMPALA-6270: create Impala parent pom This commit links together all the individual pom.xml files to have a new "impala-parent" pom as the parent. This enables de-duplicating all the repository configuration. I ran the build to test this. Change-Id: Id744e4357ee4d8e4be4e5490b2159bb76a2192f0

[3/4] impala git commit: Add 'lsof' to bootstrap_system.

2017-12-12 Thread tarmstrong
Add 'lsof' to bootstrap_system. "be/src/kudu/security/test/mini_kdc.cc" uses lsof, which doesn't exist on the base ubuntu:16.04 Docker image; adding it in. Change-Id: I6a458f2ef0313b2d08d6dd21290f8a38fa6d07f7 Reviewed-on: http://gerrit.cloudera.org:8080/8813 Reviewed-by: Tim Armstrong

[5/8] impala git commit: IMPALA-6270: remove redundant version properties

2017-12-14 Thread tarmstrong
IMPALA-6270: remove redundant version properties Removes properties that are already defined in the impala-parent pom. I ran the tests. Change-Id: I6812e11bb41716450ef29bb523773479e9f76eec Reviewed-on: http://gerrit.cloudera.org:8080/8827 Reviewed-by: Zach Amsden

[4/8] impala git commit: IMPALA-5754: Improve randomness of rand()/random()

2017-12-14 Thread tarmstrong
IMPALA-5754: Improve randomness of rand()/random() Currently implementation of rand/random built-in functions use rand_r of C library. We recognized its randomness was poor. pcg32 of third party library shows better randomness than rand_r. Testing: Revise unit test in expr-test Add E2E test to

[3/8] impala git commit: IMPALA-5754: Improve randomness of rand()/random()

2017-12-14 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/4feb4f3a/be/src/thirdparty/pcg-cpp-0.98/include/pcg_random.hpp -- diff --git a/be/src/thirdparty/pcg-cpp-0.98/include/pcg_random.hpp

[7/8] impala git commit: IMPALA-6070: Parallelize another bit of data load.

2017-12-14 Thread tarmstrong
IMPALA-6070: Parallelize another bit of data load. The two Kudu loads and Hive UDFs can all run in parallel. This should shave about 4 minutes off of the data load. (Current timings are 3.5, 4, and 0.6 minutes, see below.) I've run dataload with this change many times. Loading Kudu

[1/8] impala git commit: IMPALA-5848: Account for TCMalloc overhead in MemTracker

2017-12-14 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 0936e3296 -> 3cbbaf3b3 IMPALA-5848: Account for TCMalloc overhead in MemTracker This patch adds a new MemTracker under the Process MemTracker called "TCMalloc Overhead" which accounts for different cache freelists maintained by TCMalloc.

[2/8] impala git commit: IMPALA-5754: Improve randomness of rand()/random()

2017-12-14 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/4feb4f3a/bin/run_clang_tidy.sh -- diff --git a/bin/run_clang_tidy.sh b/bin/run_clang_tidy.sh index e879b35..36d46ce 100755 --- a/bin/run_clang_tidy.sh +++ b/bin/run_clang_tidy.sh

[6/8] impala git commit: IMPALA-6301: Fix test failures when username or group name contains dots

2017-12-14 Thread tarmstrong
IMPALA-6301: Fix test failures when username or group name contains dots Some tests use the local user's group name to construct SQLs, which may lead to syntax errors when group name contains dots. We need to quote the group names in SQL to avoid this error. Besides, a test in

[2/5] incubator-impala git commit: IMPALA-3613: Avoid topic updates to unregistered subscriber instances

2017-11-16 Thread tarmstrong
IMPALA-3613: Avoid topic updates to unregistered subscriber instances Bug: Without this patch, when a subscriber repeatedly reconnects to the statestore, the latter queues the initial heartbeat message and a bunch of topic updates to every instance of the registered subscriber. These queued

[1/5] incubator-impala git commit: IMPALA-2235: Fix current db when shell auto-reconnects

2017-11-16 Thread tarmstrong
Repository: incubator-impala Updated Branches: refs/heads/master 87fc463e0 -> 6769220e2 IMPALA-2235: Fix current db when shell auto-reconnects The ImpalaShell didn't issue the 'USE ' command after reconnecting to the Impala daemon. Therefore the client session used the default DB after

[3/5] incubator-impala git commit: IMPALA-6188: make test_top_n_reclaim less flaky

2017-11-16 Thread tarmstrong
IMPALA-6188: make test_top_n_reclaim less flaky Testing: Previously I needed ~20 iterations to get the test to fail on my local machine. After these changes I haven't been able to reproduce the failure Change-Id: I2bea7b0f770dec362a6df075da4e340402bd1d5d Reviewed-on:

[2/2] incubator-impala git commit: IMPALA-6217: fix DCHECK in Parquet fuzz test

2017-11-20 Thread tarmstrong
IMPALA-6217: fix DCHECK in Parquet fuzz test The IMPALA-4177 change accidentally removed a Status check that could be hit with a corrupt parquet file. Testing: Ran TestScannersFuzzing in a loop for 2 days. Change-Id: I6ceca7de31f602b75d744dacbdf37afa75983344 Reviewed-on:

[1/3] incubator-impala git commit: Remove unused/defunct Maven repositories.

2017-11-16 Thread tarmstrong
Repository: incubator-impala Updated Branches: refs/heads/master 6769220e2 -> ae116b5bf Remove unused/defunct Maven repositories. Removes three Maven repositories. davidtrott and codehaus both don't exist any more, so they're not doing anyone any good. (We had previously cleaned up Codehaus

[3/3] incubator-impala git commit: IMPALA-4177, IMPALA-6039: batched bit reading and rle decoding

2017-11-16 Thread tarmstrong
IMPALA-4177,IMPALA-6039: batched bit reading and rle decoding Switch the decoders to using more batch-oriented interfaces. As an intermediate step this doesn't make the interfaces of LevelDecoder or DictDecoder batch-oriented, only the lower-level utility classes. The next step would be to

[2/3] incubator-impala git commit: IMPALA-4177, IMPALA-6039: batched bit reading and rle decoding

2017-11-16 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/ae116b5b/be/src/util/parquet-reader.cc -- diff --git a/be/src/util/parquet-reader.cc b/be/src/util/parquet-reader.cc index d5b0d01..3c83c23 100644 ---

[13/16] incubator-impala git commit: IMPALA-4835 (prep only): create io subfolder and namespace

2017-11-17 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/b840137c/be/src/runtime/disk-io-mgr.cc -- diff --git a/be/src/runtime/disk-io-mgr.cc b/be/src/runtime/disk-io-mgr.cc deleted file mode 100644 index d614ac7..000 ---

[15/16] incubator-impala git commit: IMPALA-4835 (prep only): create io subfolder and namespace

2017-11-17 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/incubator-impala/blob/b840137c/be/src/runtime/disk-io-mgr-reader-context.h -- diff --git a/be/src/runtime/disk-io-mgr-reader-context.h b/be/src/runtime/disk-io-mgr-reader-context.h deleted

[1/2] impala git commit: IMPALA-4671: (part-2) Replace kudu::ServicePool with one that uses Impala threads

2017-12-07 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master a94d6068c -> f3fa3e017 IMPALA-4671: (part-2) Replace kudu::ServicePool with one that uses Impala threads The KuduRPC subsystem uses kudu::ServicePool to service all incoming RPCs. Since this lives inside the Kudu codebase, all

[2/2] impala git commit: IMPALA-6081: Fix test_basic_filters runtime profile failure

2017-12-07 Thread tarmstrong
IMPALA-6081: Fix test_basic_filters runtime profile failure test_basic_filters has been occasionally failing due to a line missing from a runtime profile for a particular query. The problem is that the query returns all of its results before all of its fragment instances are finished executing

[1/8] impala git commit: IMPALA-6068: Scale back fixing functional-types

2017-12-05 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 7e368b8f0 -> c505a8159 IMPALA-6068: Scale back fixing functional-types I re-created the original patch for IMPALA-6068, but only performed what I believe to be the limited legal transformation of data load: DEPENDENT_LOAD ->

[6/8] impala git commit: IMPALA-6256: Incorrect principal will be used for internal connections if FLAGS_be_principal is set

2017-12-05 Thread tarmstrong
IMPALA-6256: Incorrect principal will be used for internal connections if FLAGS_be_principal is set In Impala, we have FLAGS_principal and FLAGS_be_principal flags. If only FLAGS_principal is set, we use it as both the internal and external principals. If both FLAGS_principal and

[2/8] impala git commit: IMPALA-6238: Enhance TErrorCode::DATASTREAM_SENDER_TIMEOUT message

2017-12-05 Thread tarmstrong
IMPALA-6238: Enhance TErrorCode::DATASTREAM_SENDER_TIMEOUT message This change augments the message of TErrorCode::DATASTREAM_SENDER_TIMEOUT to include the source address when KRPC is enabled. The source address is not readily available in Thrift. The new message includes the destination plan

[5/8] impala git commit: IMPALA-6268: KerberosOnAndOff/RpcMgrKerberizedTest.MultipleServices failing

2017-12-05 Thread tarmstrong
IMPALA-6268: KerberosOnAndOff/RpcMgrKerberizedTest.MultipleServices failing This patch just disables the failing test to unblock builds. We will investigate in parallel the root cause for these failures and post a real fix. Change-Id: I6c750850ff916617a06e3cfac330072d8e2179e8 Reviewed-on:

[4/8] impala git commit: IMPALA-6265 Query cancellation test enhancements

2017-12-05 Thread tarmstrong
IMPALA-6265 Query cancellation test enhancements In the query cancellation tests it is essential to wait until the query gets to a desired state (waiting_to_finish, fetching) and then cancel it. Apparently, ASAN query execution happens slower than on a Release build. As a result a hard coded

[3/8] impala git commit: IMPALA-6232: Disable file handle cache by default

2017-12-05 Thread tarmstrong
IMPALA-6232: Disable file handle cache by default There are scenarios where HDFS file appends or HDFS file overwrites can lead to HDFS disabling short circuit reads. Since this can be a performance regression, this changes the default value for max_cached_file_handles to 0 to disable the file

impala git commit: [security] Make the kerberos principal configurable for Kudu servers

2017-12-05 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master c505a8159 -> 9303b0aed [security] Make the kerberos principal configurable for Kudu servers The Kudu security library currently sources the kerberos principal directly from FLAGS_principal. Since this is a library, we'd rather move this

[3/4] impala git commit: IMPALA-6291: disable AVX512 codegen in LLVM

2017-12-08 Thread tarmstrong
IMPALA-6291: disable AVX512 codegen in LLVM Adds a whitelist of LLVM CPU attributes that I know that we routinely test Impala with. This excludes the problematic AVX512 attributes as well as some other flags we don't test with - e.g. AMD-only instructions, NVM-related instructions, etc. We're

[1/4] impala git commit: IMPALA-6280: Materialize TupleIsNullPredicate for insert sorts

2017-12-08 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 11497c2aa -> 12fa20d1a IMPALA-6280: Materialize TupleIsNullPredicate for insert sorts When a sort is inserted into a plan for an INSERT due to either the target table being a Kudu table or the use of the 'clustered' hint, and a

[4/4] impala git commit: IMPALA-6242: Reduce flakiness in TimerCounterTest

2017-12-08 Thread tarmstrong
IMPALA-6242: Reduce flakiness in TimerCounterTest The error threshold in TimerCounterTest is 15ms, which is still not enough in some rare cases. This patch increases it to 30ms. Change-Id: Ifc038908857060ccbabfe30c46e72fd93907f412 Reviewed-on: http://gerrit.cloudera.org:8080/8670 Reviewed-by:

[2/4] impala git commit: IMPALA-6067: Enable S3 access via IAM roles for EC2 VMs

2017-12-08 Thread tarmstrong
IMPALA-6067: Enable S3 access via IAM roles for EC2 VMs For some time Impala in a production environment has been able to access data stored in Amazon S3 buckets using credentials specified in a number of ways: - storing Amazon access keys in environment variables or in core-site.xml. - using

[7/8] impala git commit: IMPALA-4123 (prep): Parquet column reader cleanup

2018-05-04 Thread tarmstrong
IMPALA-4123 (prep): Parquet column reader cleanup Some miscellaneous cleanup to make it easier to understand and make future changes to the Parquet scanner. A lot of the refactoring is about more cleanly separating functions so that they have clearer purpose, e.g.: * Functions that strictly do

[5/8] impala git commit: IMPALA-6954: Fix problems with CTAS into Kudu with an expr rewrite

2018-05-04 Thread tarmstrong
IMPALA-6954: Fix problems with CTAS into Kudu with an expr rewrite This patch fixes two problems: - Previously a CTAS into a Kudu table where an expr rewrite occurred would create an unpartitioned table, due to the partition info being reset in TableDataLayout and then never reconstructed.

[1/2] impala git commit: IMPALA-6507: remove --disable_mem_pools debug feature

2018-05-04 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 1eedafed6 -> d6dad9cdf IMPALA-6507: remove --disable_mem_pools debug feature Save some maintenance overhead by simplifying memory allocation code paths. ASAN poisoning provides the same general functionality and is on by default.

[8/8] impala git commit: IMPALA-6961: [DOCS] Doc --enable_minidump flag to disable minidumps

2018-05-04 Thread tarmstrong
IMPALA-6961: [DOCS] Doc --enable_minidump flag to disable minidumps Change-Id: I3412e36272cda0c1502d4643afcdbad01e9548a5 Reviewed-on: http://gerrit.cloudera.org:8080/10285 Reviewed-by: Lars Volker Tested-by: Impala Public Jenkins Project:

[1/8] impala git commit: IMPALA-6959: [DOCS] Update to HAProxy configuration sample code

2018-05-04 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/2.x ae9c47bb5 -> dd6f28747 IMPALA-6959: [DOCS] Update to HAProxy configuration sample code - Changed to deprecated timeouts: contimeout, clitimeout, srvtimeout - Changed the sample timeout values to more realistic values - Added a note that

[2/2] impala git commit: IMPALA-6968: Fix TestBlockVerification flakiness

2018-05-04 Thread tarmstrong
IMPALA-6968: Fix TestBlockVerification flakiness The bug is that the byte in the encrypted data is '?' around 1/256 runs of the test. Instead, flip a bit in the original data so that it's always different from the input. Change-Id: Ibdf063ff32848035af667c7cd2a1268f5b785cfe Reviewed-on:

[2/8] impala git commit: IMPALA-6882: prevent instr. hoist from CpuInfo::IsSupported()

2018-05-04 Thread tarmstrong
IMPALA-6882: prevent instr. hoist from CpuInfo::IsSupported() Marking the __asm__ with __volatile__ *should* prevent the compiler from speculatively executing the instruction before the branch. Testing: Added a regression test that tries to emulate the problematic pattern, but I was unable to

[4/8] impala git commit: IMPALA-6931: reduces races in query expiration tests

2018-05-04 Thread tarmstrong
IMPALA-6931: reduces races in query expiration tests Recent tests ran into flakiness when testing query expiration. This change makes two changes: 1) query state is retrieved earlier; a flaky test skipped the expected state. 2) bump the timing; a flaky test had queries expire before it could

[6/8] impala git commit: Fix minor perf bug in BufferedTupleStream

2018-05-04 Thread tarmstrong
Fix minor perf bug in BufferedTupleStream The code referenced a member variable when the intent was clearly to use the template argument. Change-Id: I05bae0018f79aec9c6014ae228ff28621b548860 Reviewed-on: http://gerrit.cloudera.org:8080/10239 Reviewed-by: Alex Behm

[3/8] impala git commit: Speed up Python dependencies.

2018-05-04 Thread tarmstrong
Speed up Python dependencies. This parallelizes downloading some Python libraries, giving a speedup of $IMPALA_HOME/infra/python/deps/download_requirements. I've seen this take from 7-15 seconds before and from 2-5 seconds after. I also checked that we always have at least Python 2.6 when

[1/5] impala git commit: IMPALA-5969: [DOCS] Adds --auth_creds_ok_in_clear to shell options

2018-05-08 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master e2e7c103a -> 96c9dac28 IMPALA-5969: [DOCS] Adds --auth_creds_ok_in_clear to shell options This patch adds --auth_creds_ok_in_clear to the impala_shell_options documentation xml Change-Id: I19450ebd839b84a85598d283c04a77662fa5e44e

impala git commit: IMPALA-6908: IsConnResetTException() should include ECONNRESET

2018-05-08 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/2.x ffac1ab48 -> 33dbdd0bb IMPALA-6908: IsConnResetTException() should include ECONNRESET The utility function IsConnResetTException() attempted to match error strings from RPCs that fail due to the remote end resetting the connection for any

[4/5] impala git commit: IMPALA-6227: reduce window of metric inconsistency

2018-05-08 Thread tarmstrong
IMPALA-6227: reduce window of metric inconsistency The admission controller test fetches multiple metrics relating to the admission controller. Before this patch it fetched the whole metrics list for each metric, meaning there was a substantial window for the metrics to be inconsistent for a

[3/5] impala git commit: IMPALA-4850: [DOCS] COMMENT should come after PARTITIONED BY

2018-05-08 Thread tarmstrong
IMPALA-4850: [DOCS] COMMENT should come after PARTITIONED BY Change-Id: I03fd4a308981955bb52ca79772fe2f7c01b5894f Reviewed-on: http://gerrit.cloudera.org:8080/10316 Reviewed-by: Joe McDonnell Tested-by: Impala Public Jenkins

[5/5] impala git commit: IMPALA-6974: Use CMAKE_POSITION_INDEPENDENT_CODE in backend

2018-05-08 Thread tarmstrong
IMPALA-6974: Use CMAKE_POSITION_INDEPENDENT_CODE in backend Compilation of individual c++ files are only slightly different between static and shared compilation. First, CMake adds -D${LIBRARY_NAME}_EXPORTS to each compilation. Second, CMake sets CMAKE_POSITION_INDEPENDENT_CODE, which adds an

[2/5] impala git commit: [DOCS] Removed the references to YARN as Impala does not support YARN

2018-05-08 Thread tarmstrong
[DOCS] Removed the references to YARN as Impala does not support YARN Change-Id: Ifcea49b5859a2afbbbe99197e7818c30c7ba6d67 Reviewed-on: http://gerrit.cloudera.org:8080/10346 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins

[8/8] impala git commit: IMPALA-6916: Implement COMMENT ON DATABASE

2018-05-15 Thread tarmstrong
IMPALA-6916: Implement COMMENT ON DATABASE This patch implements updating comment on a database. Syntax: COMMENT ON DATABASE db IS 'comment' Testing: - Added new front-end tests - Ran all front-end tests - Added new end-to-end tests - Ran end-to-end DDL tests Change-Id:

[6/8] impala git commit: IMPALA-3813: [DOCS] How to create a Kudu table with a replication factor

2018-05-15 Thread tarmstrong
IMPALA-3813: [DOCS] How to create a Kudu table with a replication factor Described how to create a Kudu table with a replication factor that is not the default value of 3. Change-Id: I9dc68dcd395fcd0bd31563ea46229a12553482dc Reviewed-on: http://gerrit.cloudera.org:8080/10401 Reviewed-by: Thomas

[1/8] impala git commit: IMPALA-4464: Remove /bin/remote_data_load.py

2018-05-15 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 19bcc3099 -> 97ecc154b IMPALA-4464: Remove /bin/remote_data_load.py This file was started before the ASF project was set up, and committed as-is. However, it relies on some internal resources not generally available to the external Apache

impala git commit: IMPALA-5384, part 2: Simplify Coordinator locking and clarify state

2018-05-15 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/2.x 22244bb07 -> da329442a IMPALA-5384, part 2: Simplify Coordinator locking and clarify state The is the final change to clarify and break up the Coordinator's lock. The state machine for the coordinator is made explicit, distinguishing between

[4/8] impala git commit: IMPALA-7024: Convert Coordinator::wait_lock_ to SpinLock

2018-05-15 Thread tarmstrong
IMPALA-7024: Convert Coordinator::wait_lock_ to SpinLock For consistency with the other locks in this class, use SpinLock rather than boost::mutex. We expect SpinLock to work okay for locks that block since it is adaptive. This came up in the code review for IMPALA-5384, but I wanted to make

[5/8] impala git commit: Remove IMPALA_THRIFT_JAVA_VERSION and untested Darwin Thrift versions.

2018-05-15 Thread tarmstrong
Remove IMPALA_THRIFT_JAVA_VERSION and untested Darwin Thrift versions. The singular use of IMPALA_THRIFT_JAVA_VERSION was in impala-parent/pom.xml. We can reduce complexity by just inlining the version there, like we do with several other Java dependencies. Meanwhile, with the upgrade to Thrift

[3/8] impala git commit: Add a missing PrintId()

2018-05-15 Thread tarmstrong
Add a missing PrintId() For consistency, add a PrintId() around a query_id() used in a stream, which was missing from this commit: IMPALA-5384, part 2: Simplify Coordinator locking and clarify state This change was put into the cherry-pick for 2.x, so: Cherry-picks: not for 2.x Change-Id:

[2/8] impala git commit: impala-6233: [DOCS] Documented the COMMENT clause for CREATE VIEW

2018-05-15 Thread tarmstrong
impala-6233: [DOCS] Documented the COMMENT clause for CREATE VIEW Change-Id: I176d525925c8dc5c5b83612da43b349049764d2b Reviewed-on: http://gerrit.cloudera.org:8080/10312 Reviewed-by: Alex Behm Tested-by: Impala Public Jenkins

[7/8] impala git commit: IMPALA-7018: fix spill-to-disk encryption err handling

2018-05-15 Thread tarmstrong
IMPALA-7018: fix spill-to-disk encryption err handling The EVP_CIPHER_CTX_ctrl() function was being misused: 1. It was called before initialising the context 2. Errors were not being handled (including the error from #1) Testing: Added some checks to assert that the OpenSSL error queue is empty.

[3/3] impala git commit: IMPALA-6946: handle negative counts in RLE decoder

2018-05-07 Thread tarmstrong
IMPALA-6946: handle negative counts in RLE decoder This improves the handling of out-of-range values to avoid hitting various DCHECKs, including the one in the JIRA. repeat_count_ and literal_count_ are int32_ts. Avoid setting them to a negative value directly or by integer overflow. Switch to

[1/3] impala git commit: test-with-docker: exit properly on failures

2018-05-07 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master f20415755 -> 10187bfff test-with-docker: exit properly on failures If the build was failing, test-with-docker wouldn't recognize it and continue with the script; this fixes that. The bash puzzle I learned here is that bash -c "set -e;

[2/3] impala git commit: Move admission-controller and catalog metrics into own groups

2018-05-07 Thread tarmstrong
Move admission-controller and catalog metrics into own groups This makes using these metrics a lot easier since they're not mixed into the big impala-metrics group. Testing: Checked /metrics debug page to see that all metrics were now in a separate section Change-Id:

impala git commit: IMPALA-6980: [DOCS] You can add or change column comment in ALTER TABLE

2018-05-09 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 19752af1f -> e34f6aa5a IMPALA-6980: [DOCS] You can add or change column comment in ALTER TABLE Change-Id: Ia317a4b74d96aa064d375f6afc114f2cc8d30bf4 Reviewed-on: http://gerrit.cloudera.org:8080/10317 Reviewed-by: Fredy Wijaya

[4/4] impala git commit: IMPALA-6920: fix inconsistencies with scanner thread tokens

2018-04-27 Thread tarmstrong
IMPALA-6920: fix inconsistencies with scanner thread tokens The first scanner thread to start now takes a "required" token, which always succeeds. Only additional threads try to get "optional" tokens, which can fail. Previously threads always requested optional tokens, which could fail and leave

[3/4] impala git commit: IMPALA-6340, IMPALA-6518: Check that decimal types are compatible in FE

2018-04-27 Thread tarmstrong
IMPALA-6340,IMPALA-6518: Check that decimal types are compatible in FE In this patch we implement strict decimal type checking in the FE in various situations when DECIMAL_V2 is enabled. What is affected: - Union. If we union two decimals and it is not possible to come up with a decimal that

[2/2] impala git commit: IMPALA-6934: Wrong results with EXISTS subquery containing ORDER BY, LIMIT, and OFFSET

2018-04-27 Thread tarmstrong
IMPALA-6934: Wrong results with EXISTS subquery containing ORDER BY, LIMIT, and OFFSET Queries may return wrong results if an EXISTS subquery has an ORDER BY with a LIMIT and OFFSET clause. The EXISTS subquery may incorrectly evaluate to TRUE even though it is FALSE. The bug was found during

[1/2] impala git commit: IMPALA-6885: handle empty s3 dirs in recover_partitions test

2018-04-27 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/2.x 2c893f46b -> 9c32594f7 IMPALA-6885: handle empty s3 dirs in recover_partitions test Two tests (test_duplicate_partitions, test_support_all_types) added only directories, which is a no-op on s3. This results in s3 test failures. Another test

[1/4] impala git commit: Warn about Hadoop / Java version incompatibility

2018-04-27 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 87be63e32 -> 789c5aac2 Warn about Hadoop / Java version incompatibility Running Hadoop 3 with Java 7 can result in some obscure error messages. This change adds a warning to impala-config.sh when using Hadoop 3 with Java 7. Your

[4/7] impala git commit: IMPALA-6957: calc thread resource requirement in planner

2018-05-11 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/e12ee485/testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test -- diff --git

[8/8] impala git commit: IMPALA-6999: Upgrade to sqlparse-0.1.19 for Impala shell

2018-05-11 Thread tarmstrong
IMPALA-6999: Upgrade to sqlparse-0.1.19 for Impala shell sqlparse-0.1.19 is the last version of sqlparse that supports Python 2.6. Testing: - Ran all end-to-end tests Change-Id: Ide51ef3ac52d25a96b0fa832e29b6535197d23cb Reviewed-on: http://gerrit.cloudera.org:8080/10354 Reviewed-by: David Knupp

[1/7] impala git commit: IMPALA-5384, part 2: Simplify Coordinator locking and clarify state

2018-05-11 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master 25c13bfdd -> e12ee485c IMPALA-5384, part 2: Simplify Coordinator locking and clarify state The is the final change to clarify and break up the Coordinator's lock. The state machine for the coordinator is made explicit, distinguishing

[7/7] impala git commit: IMPALA-6957: calc thread resource requirement in planner

2018-05-11 Thread tarmstrong
IMPALA-6957: calc thread resource requirement in planner This only factors in fragment execution threads. E.g. this does *not* try to account for the number of threads on the old Thrift RPC code path if that is enabled. This is loosely related to the old VCores estimate, but is different in that

[7/8] impala git commit: IMPALA-6999: Upgrade to sqlparse-0.1.19 for Impala shell

2018-05-11 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/417bc8c8/shell/ext-py/sqlparse-0.1.14/sqlparse/filters.py -- diff --git a/shell/ext-py/sqlparse-0.1.14/sqlparse/filters.py b/shell/ext-py/sqlparse-0.1.14/sqlparse/filters.py

[6/8] impala git commit: IMPALA-6999: Upgrade to sqlparse-0.1.19 for Impala shell

2018-05-11 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/417bc8c8/shell/ext-py/sqlparse-0.1.14/sqlparse/utils.py -- diff --git a/shell/ext-py/sqlparse-0.1.14/sqlparse/utils.py b/shell/ext-py/sqlparse-0.1.14/sqlparse/utils.py deleted

[3/8] impala git commit: IMPALA-6999: Upgrade to sqlparse-0.1.19 for Impala shell

2018-05-11 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/417bc8c8/shell/ext-py/sqlparse-0.1.19/sqlparse/tokens.py -- diff --git a/shell/ext-py/sqlparse-0.1.19/sqlparse/tokens.py b/shell/ext-py/sqlparse-0.1.19/sqlparse/tokens.py new

[5/8] impala git commit: IMPALA-6999: Upgrade to sqlparse-0.1.19 for Impala shell

2018-05-11 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/417bc8c8/shell/ext-py/sqlparse-0.1.14/tests/test_tokenize.py -- diff --git a/shell/ext-py/sqlparse-0.1.14/tests/test_tokenize.py

[1/8] impala git commit: IMPALA-6966: sort table memory by size in catalogd web UI

2018-05-11 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/2.x 381f50ad0 -> 7b8bd6a19 IMPALA-6966: sort table memory by size in catalogd web UI This patch fix the sorting order in "Top-K Tables with Highest Memory Requirements" in which "Estimated memory" column is sorted as strings. Values got from

[2/7] impala git commit: IMPALA-6957: calc thread resource requirement in planner

2018-05-11 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/e12ee485/testdata/workloads/functional-query/queries/QueryTest/explain-level2.test -- diff --git a/testdata/workloads/functional-query/queries/QueryTest/explain-level2.test

[2/8] impala git commit: IMPALA-6999: Upgrade to sqlparse-0.1.19 for Impala shell

2018-05-11 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/417bc8c8/shell/ext-py/sqlparse-0.1.19/tests/test_regressions.py -- diff --git a/shell/ext-py/sqlparse-0.1.19/tests/test_regressions.py

[4/8] impala git commit: IMPALA-6999: Upgrade to sqlparse-0.1.19 for Impala shell

2018-05-11 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/417bc8c8/shell/ext-py/sqlparse-0.1.19/sqlparse/filters.py


[5/7] impala git commit: IMPALA-6957: calc thread resource requirement in planner

2018-05-11 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/e12ee485/testdata/workloads/functional-planner/queries/PlannerTest/parquet-filtering.test -- diff --git

[6/7] impala git commit: IMPALA-6957: calc thread resource requirement in planner

2018-05-11 Thread tarmstrong
http://git-wip-us.apache.org/repos/asf/impala/blob/e12ee485/testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test -- diff --git

impala git commit: IMPALA-6907: Close stale connections to removed cluster members

2018-05-12 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master e12ee485c -> 47f95f14b IMPALA-6907: Close stale connections to removed cluster members Previously, ImpalaServer::MembershipCallback() is used by each Impala backend node to update cluster membership. It also removes stale connections to

impala git commit: IMPALA-7010: don't run memory usage tests on non-HDFS

2018-05-12 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/2.x 7b8bd6a19 -> 22244bb07 IMPALA-7010: don't run memory usage tests on non-HDFS Moved a number of tests with tuned mem_limits. In some cases this required separating the tests from non-tuned functional tests. TestQueryMemLimit used very high

impala git commit: IMPALA-7010: don't run memory usage tests on non-HDFS

2018-05-11 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master ea4715fd7 -> 25c13bfdd IMPALA-7010: don't run memory usage tests on non-HDFS Moved a number of tests with tuned mem_limits. In some cases this required separating the tests from non-tuned functional tests. TestQueryMemLimit used very

[2/4] impala git commit: IMPALA-6968: Fix TestBlockVerification flakiness

2018-05-07 Thread tarmstrong
IMPALA-6968: Fix TestBlockVerification flakiness The bug is that the byte in the encrypted data is '?' around 1/256 runs of the test. Instead, flip a bit in the original data so that it's always different from the input. Change-Id: Ibdf063ff32848035af667c7cd2a1268f5b785cfe Reviewed-on:

[1/3] impala git commit: IMPALA-6866: Rework timeouts for test_exchange_delays.py

2018-05-07 Thread tarmstrong
Repository: impala Updated Branches: refs/heads/master d6dad9cdf -> f13abdca6 IMPALA-6866: Rework timeouts for test_exchange_delays.py Isilon has been failing on the exchange-delays-zero-rows test case due to slow scans. Running this part of the test with a larger value for

  1   2   3   4   5   6   7   8   9   10   >