[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9238 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 2: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1910/


--
To view, visit http://gerrit.cloudera.org:8080/9238
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia188a0c5b74e4a22fb30f8c12f65e0469eb75f6b
Gerrit-Change-Number: 9238
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Fri, 09 Feb 2018 07:18:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3271: organise and warn on removed startup flags

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9173 )

Change subject: IMPALA-3271: organise and warn on removed startup flags
..


Patch Set 5: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1911/


--
To view, visit http://gerrit.cloudera.org:8080/9173
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic9bb4792e1b18840aa85aa5d755a09f2b5461f34
Gerrit-Change-Number: 9173
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 09 Feb 2018 07:17:47 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6396: Exchange node's memory usage should include its receiver's

[Sailesh Mukil (Code Review)]

Sailesh Mukil has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9202 )

Change subject: IMPALA-6396: Exchange node's memory usage should include its 
receiver's
..


Patch Set 3: Code-Review+2

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9202/2//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9202/2//COMMIT_MSG@16
PS2, Line 16: This makes it
: easier to identify the peak memory usage of the receivers
: of different exchange nodes in the runtime profile and
: query summary.
> Yes, they use different variants of the MemTracker constructors. The exchan
Ok thanks for the explanation. Could you add that line to the commit message?

I was worried about the SortedRunMerger, but I see that's in the 
DataStreamRecvr. I previously erroneously thought it was part of the exchange 
node itself. Given that's the case, I don't think it's a critical necessity to 
track the peak mem usage of the recvr separately.



--
To view, visit http://gerrit.cloudera.org:8080/9202
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8ca3c47d87bfcd221d34565eda1878f3c15d5c45
Gerrit-Change-Number: 9202
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Comment-Date: Fri, 09 Feb 2018 06:50:15 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6070: Expose using Docker to run tests faster.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/9085 )

Change subject: IMPALA-6070: Expose using Docker to run tests faster.
..

IMPALA-6070: Expose using Docker to run tests faster.

Allows running the tests that make up the "core" suite in about 2 hours.
By comparison, 
https://jenkins.impala.io/job/ubuntu-16.04-from-scratch/buildTimeTrend
tends to run in about 3.5 hours.

This commit:
* Adds "echo" statements in a few places, to facilitate timing.
* Adds --skip-parallel/--skip-serial flags to run-tests.py,
  and exposes them in run-all-tests.sh.
* Marks TestRuntimeFilters as a serial test. This test runs
  queries that need > 1GB of memory, and, combined with
  other tests running in parallel, can kill the parallel test
  suite.
* Adds "test-with-docker.py", which runs a full build, data load,
  and executes tests inside of Docker containers, generating
  a timeline at the end. In short, one container is used
  to do the build and data load, and then this container is
  re-used to run various tests in parallel. All logs are
  left on the host system.

Besides the obvious win of getting test results more quickly, this
commit serves as an example of how to get various bits of Impala
development working inside of Docker containers. For example, Kudu
relies on atomic rename of directories, which isn't available in most
Docker filesystems, and entrypoint.sh works around it.

In addition, the timeline generated by the build suggests where further
optimizations can be made. Most obviously, dataload eats up a precious
~30-50 minutes, on a largely idle machine.

This work is significantly CPU and memory hungry. It was developed on a
32-core, 120GB RAM Google Compute Engine machine. I've worked out
parallelism configurations such that it runs nicely on 60GB of RAM
(c4.8xlarge) as over 100GB (eg., m4.10xlarge, which has 160GB). There is
some simple logic to guess at some knobs, and there are knobs.  By and
large, EC2 and GCE price machines linearly, so, if CPU usage can be kept
up, it's not wasteful to run on bigger machines.

I've run into some fragility with the tests. Specifically,
thrift-server-test fails consistently, as it does on my development
Ubuntu 16.04 machine as well. I've filed IMPALA-6494 to look at
it more. It's skipped in the current implementation.

Change-Id: I82052ef31979564968effef13a3c6af0d5c62767
---
M bin/bootstrap_system.sh
M bin/run-all-tests.sh
M bin/start-impala-cluster.py
M buildall.sh
A docker/annotate.pl
A docker/annotate.py
A docker/entrypoint.sh
A docker/monitor.py
A docker/test-with-docker.py
A docker/timeline.html.template
M testdata/bin/run-all.sh
M tests/query_test/test_runtime_filters.py
M tests/run-tests.py
13 files changed, 1,429 insertions(+), 17 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/85/9085/3
--
To view, visit http://gerrit.cloudera.org:8080/9085
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I82052ef31979564968effef13a3c6af0d5c62767
Gerrit-Change-Number: 9085
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Philip Zeyliger 

[Impala-ASF-CR] Remove Yarn from minicluster.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9264 )

Change subject: Remove Yarn from minicluster.
..


Patch Set 1:

https://jenkins.impala.io/job/gerrit-verify-dryrun-external/78/ is running this.


--
To view, visit http://gerrit.cloudera.org:8080/9264
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5504cc40b89e3c6d53fac0b7aa4b395fa63e8d79
Gerrit-Change-Number: 9264
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Fri, 09 Feb 2018 04:29:36 +
Gerrit-HasComments: No

[Impala-ASF-CR] Remove Yarn from minicluster.

[Philip Zeyliger (Code Review)]

Philip Zeyliger has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9264


Change subject: Remove Yarn from minicluster.
..

Remove Yarn from minicluster.

Removes vestiges of Yarn from the minicluster.

Turns out that we start Yarn as part of the minicluster, but we never
use it. (HiveServer2 is configured to run MR jobs "locally" in process.)
Likely, this Yarn integration is a vestige of Yarn/Llama integration.
We can save memory by not starting it.

The minimal change here was removing "yarn" from testdata/cluster/admin
in the list of services. I went a little bit farther and removed
vestiges of Yarn configuration as well.

I tested by running core tests. I did not test the kerberized
minicluster.

Change-Id: I5504cc40b89e3c6d53fac0b7aa4b395fa63e8d79
---
M bin/create-test-configuration.sh
M testdata/cluster/admin
D testdata/cluster/node_templates/cdh5/etc/hadoop/conf/yarn-site.xml.tmpl
D testdata/cluster/node_templates/common/etc/hadoop/conf/yarn-site.xml.tmpl
D testdata/cluster/node_templates/common/etc/init.d/yarn-common
D testdata/cluster/node_templates/common/etc/init.d/yarn-nodemanager
D testdata/cluster/node_templates/common/etc/init.d/yarn-resourcemanager
7 files changed, 4 insertions(+), 330 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/64/9264/1
--
To view, visit http://gerrit.cloudera.org:8080/9264
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5504cc40b89e3c6d53fac0b7aa4b395fa63e8d79
Gerrit-Change-Number: 9264
Gerrit-PatchSet: 1
Gerrit-Owner: Philip Zeyliger 

[Impala-ASF-CR] IMPALA-3271: organise and warn on removed startup flags

[Tim Armstrong (Code Review)]

Tim Armstrong has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/9173 )

Change subject: IMPALA-3271: organise and warn on removed startup flags
..

IMPALA-3271: organise and warn on removed startup flags

Gather all the removed flags in a single place and establish consistent
behaviour.

If the flag is set, a warning is logged. GFlags are initialised before
logging, so the warning goes to stderr instead of the WARNING log.

The affected flags are:
be_service_threads, cgroup_hierarchy_path, enable_accept_queue_server,
enable_partitioned_aggregation, enable_partitioned_hash_join,
enable_phj_probe_side_filtering, enable_rm, llama_addresses,
llama_callback_port, llama_host, llama_max_request_attempts,
llama_port, llama_registration_timeout_secs,
llama_registration_wait_secs, local_nodemanager_url,
resource_broker_cnxn_attempts, resource_broker_cnxn_retry_interval_ms,
resource_broker_recv_timeout, resource_broker_send_timeout,
rm_always_use_defaults, rm_default_cpu_vcores, rm_default_memory,
rpc_cnxn_attempts, rpc_cnxn_retry_interval_ms, staging_cgroup,
suppress_unknown_disk_id_warnings, use_statestore,

Testing:
Ran "start-impala-cluster.py --impalad_args=--enable_rm=true"
and verified that the warning appeared in
logs/cluster/impalad-error.log

Cherry-picks: not for 2.x

Change-Id: Ic9bb4792e1b18840aa85aa5d755a09f2b5461f34
---
M be/src/common/global-flags.cc
M be/src/exec/exec-node.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/rpc/thrift-server.cc
M be/src/runtime/exec-env.cc
M be/src/scheduling/query-schedule.cc
M be/src/service/impala-server.cc
M be/src/service/impalad-main.cc
9 files changed, 54 insertions(+), 47 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/73/9173/5
--
To view, visit http://gerrit.cloudera.org:8080/9173
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ic9bb4792e1b18840aa85aa5d755a09f2b5461f34
Gerrit-Change-Number: 9173
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-3271: organise and warn on removed startup flags

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9173 )

Change subject: IMPALA-3271: organise and warn on removed startup flags
..


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1911/


--
To view, visit http://gerrit.cloudera.org:8080/9173
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ic9bb4792e1b18840aa85aa5d755a09f2b5461f34
Gerrit-Change-Number: 9173
Gerrit-PatchSet: 5
Gerrit-Owner: Tim Armstrong 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 09 Feb 2018 04:19:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9238 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 2: Code-Review+2

Tests passed on CentOS 6


--
To view, visit http://gerrit.cloudera.org:8080/9238
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia188a0c5b74e4a22fb30f8c12f65e0469eb75f6b
Gerrit-Change-Number: 9238
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Fri, 09 Feb 2018 04:10:02 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6219: Use AES-GCM for spill-to-disk encryption

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9238 )

Change subject: IMPALA-6219: Use AES-GCM for spill-to-disk encryption
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1910/


--
To view, visit http://gerrit.cloudera.org:8080/9238
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia188a0c5b74e4a22fb30f8c12f65e0469eb75f6b
Gerrit-Change-Number: 9238
Gerrit-PatchSet: 2
Gerrit-Owner: Xianda Ke 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Xianda Ke 
Gerrit-Comment-Date: Fri, 09 Feb 2018 04:10:11 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6372: Go parallel for Hive dataload

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8894 )

Change subject: IMPALA-6372: Go parallel for Hive dataload
..


Patch Set 7:

I ran tests with this change and observed data loading go from 53 minutes to 37 
minutes (on identical virtual hardware).


--
To view, visit http://gerrit.cloudera.org:8080/8894
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I34b71e6df3c8f23a5a31451280e35f4dc015a2fd
Gerrit-Change-Number: 8894
Gerrit-PatchSet: 7
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Fri, 09 Feb 2018 03:32:45 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6477: Disable rpc-mgr-kerberized-test

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9256 )

Change subject: IMPALA-6477: Disable rpc-mgr-kerberized-test
..

IMPALA-6477: Disable rpc-mgr-kerberized-test

This test is breaking builds on CentOS 6.4. Disable it until we
find out the root cause.

Change-Id: I5654d8c41b130544a8f9bc38f69433fcc171eca0
Reviewed-on: http://gerrit.cloudera.org:8080/9256
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins
---
M be/src/rpc/rpc-mgr-kerberized-test.cc
1 file changed, 3 insertions(+), 2 deletions(-)

Approvals:
  Alex Behm: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9256
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I5654d8c41b130544a8f9bc38f69433fcc171eca0
Gerrit-Change-Number: 9256
Gerrit-PatchSet: 2
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-6477: Disable rpc-mgr-kerberized-test

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9256 )

Change subject: IMPALA-6477: Disable rpc-mgr-kerberized-test
..


Patch Set 1: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9256
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5654d8c41b130544a8f9bc38f69433fcc171eca0
Gerrit-Change-Number: 9256
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Fri, 09 Feb 2018 02:15:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3193: Show table's comment on show tables

[Kim Jin Chul (Code Review)]

Kim Jin Chul has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8851 )

Change subject: IMPALA-3193: Show table's comment on show tables
..


Patch Set 13:

Applied the update.


--
To view, visit http://gerrit.cloudera.org:8080/8851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I44f814af05db6f3c027718ade9f474f8b8153bcd
Gerrit-Change-Number: 8851
Gerrit-PatchSet: 13
Gerrit-Owner: Kim Jin Chul 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kim Jin Chul 
Gerrit-Comment-Date: Fri, 09 Feb 2018 00:32:03 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3193: Show table's comment on show tables

[Kim Jin Chul (Code Review)]

Hello Dimitris Tsirogiannis, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8851

to look at the new patch set (#14).

Change subject: IMPALA-3193: Show table's comment on show tables
..

IMPALA-3193: Show table's comment on show tables

Currently SHOW TABLES command does not display given comment
when creating a table. DESC  shows column's comment and
SHOW DATABASES also shows database's comment.

On this change, table's comment is displayed if the table is
loaded. If the table is not loaded, an empty comment is returned.

Testing:
Adds E2E test: TestShowTables.test_table_comment_visibility

Change-Id: I44f814af05db6f3c027718ade9f474f8b8153bcd
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M testdata/workloads/functional-query/queries/QueryTest/show.test
M tests/metadata/test_metadata_query_statements.py
M www/catalog.tmpl
19 files changed, 296 insertions(+), 206 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/8851/14
--
To view, visit http://gerrit.cloudera.org:8080/8851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44f814af05db6f3c027718ade9f474f8b8153bcd
Gerrit-Change-Number: 8851
Gerrit-PatchSet: 14
Gerrit-Owner: Kim Jin Chul 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kim Jin Chul 

[Impala-ASF-CR] IMPALA-5993: Fix the file offset in value parsing error

[Kim Jin Chul (Code Review)]

Kim Jin Chul has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8747 )

Change subject: IMPALA-5993: Fix the file offset in value parsing error
..


Patch Set 7:

Hi Lars,

You may be busy. It would be great if you review this change again. Thanks!

Regards,
Jinchul


--
To view, visit http://gerrit.cloudera.org:8080/8747
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4a4ac3199ffe12fcce0bf792b3e6ce529b9af61f
Gerrit-Change-Number: 8747
Gerrit-PatchSet: 7
Gerrit-Owner: Kim Jin Chul 
Gerrit-Reviewer: Kim Jin Chul 
Gerrit-Reviewer: Lars Volker 
Gerrit-Comment-Date: Fri, 09 Feb 2018 00:06:38 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6477: Disable rpc-mgr-kerberized-test

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9256 )

Change subject: IMPALA-6477: Disable rpc-mgr-kerberized-test
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1909/


--
To view, visit http://gerrit.cloudera.org:8080/9256
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5654d8c41b130544a8f9bc38f69433fcc171eca0
Gerrit-Change-Number: 9256
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 08 Feb 2018 22:32:48 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6477: Disable rpc-mgr-kerberized-test

[Alex Behm (Code Review)]

Alex Behm has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9256 )

Change subject: IMPALA-6477: Disable rpc-mgr-kerberized-test
..


Patch Set 1: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9256
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I5654d8c41b130544a8f9bc38f69433fcc171eca0
Gerrit-Change-Number: 9256
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Alex Behm 
Gerrit-Comment-Date: Thu, 08 Feb 2018 22:32:16 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5801: Clean up codegen GetType() interface

[Csaba Ringhofer (Code Review)]

Csaba Ringhofer has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9063 )

Change subject: IMPALA-5801: Clean up codegen GetType() interface
..


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/9063/1/be/src/codegen/llvm-codegen.h
File be/src/codegen/llvm-codegen.h:

http://gerrit.cloudera.org:8080/#/c/9063/1/be/src/codegen/llvm-codegen.h@263
PS1, Line 263: GetType
> I'm not sure that the templated version would improve readability enough to
GetNamed(Ptr)Type is ok for me, even if it it a bit longer than I would prefer.

I was curious about the error message, so I created an implementation and added 
a call with parameter int (which should lead to an error, as it doesn't have 
LLVM_CLASS_NAME).

The error message contains some useful clues (first half) but also some strange 
stuff (second half), so I must agree that it could lead to confusion:

In file included from be/src/codegen/llvm-codegen.cc:18:0:
be/src/codegen/llvm-codegen.h: In instantiation of ‘llvm::Type* 
impala::LlvmCodeGen::GetType() [with T = int]’:
be/src/codegen/llvm-codegen.cc:484:33:   required from here
be/src/codegen/llvm-codegen.h:259:65: error: ‘LLVM_CLASS_NAME’ is not a member 
of ‘int’
   llvm::Type* GetType() { return GetNamedType(T::LLVM_CLASS_NAME); }
 ^
be/src/codegen/llvm-codegen.h: In member function ‘llvm::Type* 
impala::LlvmCodeGen::GetType() [with T = int]’:
be/src/codegen/llvm-codegen.h:259:68: error: control reaches end of non-void 
function [-Werror=return-type]
   llvm::Type* GetType() { return GetNamedType(T::LLVM_CLASS_NAME); }
^
be/src/codegen/llvm-codegen.cc: In function ‘void boost::throw_exception(const 
std::exception&)’:
be/src/codegen/llvm-codegen.cc:1719:1: error: ‘noreturn’ function does return 
[-Werror]
 }


http://gerrit.cloudera.org:8080/#/c/9063/1/be/src/codegen/llvm-codegen.h@497
PS1, Line 497:   llvm::Type* i8_type() { return 
GetInternalRepresentation(TYPE_TINYINT); }
> Changing the interface first makes sense, we should switch over the impleme
Done



--
To view, visit http://gerrit.cloudera.org:8080/9063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib146ca21af82023b0b460f813eae3435b4b2eb77
Gerrit-Change-Number: 9063
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 08 Feb 2018 22:12:32 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6486: Fix INVALIDATE METADATA hang after statestore restart

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9232 )

Change subject: IMPALA-6486: Fix INVALIDATE METADATA hang after statestore 
restart
..


Patch Set 2: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9232
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Icf3ee31c28884ed79c5dbc700ccd4ef761015c68
Gerrit-Change-Number: 9232
Gerrit-PatchSet: 2
Gerrit-Owner: Dimitris Tsirogiannis 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Comment-Date: Thu, 08 Feb 2018 21:42:56 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6486: Fix INVALIDATE METADATA hang after statestore restart

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9232 )

Change subject: IMPALA-6486: Fix INVALIDATE METADATA hang after statestore 
restart
..

IMPALA-6486: Fix INVALIDATE METADATA hang after statestore restart

This commit fixes an issue where an INVALIDATE METADATA statement will
hang if executed after a statestore restart. The issue was that the
CatalogObjectVersionQueue was not properly reset when a non-delta
catalog topic update was received in an impalad, causing it to record
duplicate catalog versions and, hence, not properly advancing the minimum
catalog object watermark. The latter is used by INVALIDATE METADATA to
determine when the effects of that operation have been applied to the
local impalad catalog cache.

Change-Id: Icf3ee31c28884ed79c5dbc700ccd4ef761015c68
Reviewed-on: http://gerrit.cloudera.org:8080/9232
Reviewed-by: Dimitris Tsirogiannis 
Tested-by: Impala Public Jenkins
---
M fe/src/main/java/org/apache/impala/catalog/CatalogObjectVersionQueue.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
2 files changed, 5 insertions(+), 4 deletions(-)

Approvals:
  Dimitris Tsirogiannis: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9232
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Icf3ee31c28884ed79c5dbc700ccd4ef761015c68
Gerrit-Change-Number: 9232
Gerrit-PatchSet: 3
Gerrit-Owner: Dimitris Tsirogiannis 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Vuk Ercegovac 

[Impala-ASF-CR] IMPALA-5690: Upgrade thrift to 0.9.3-p3

[Tianyi Wang (Code Review)]

Tianyi Wang has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/9168 )

Change subject: IMPALA-5690: Upgrade thrift to 0.9.3-p3
..

IMPALA-5690: Upgrade thrift to 0.9.3-p3

Dependency changes:
- BE and python use thrift 0.9.3-p3 from native-toolchain.
- FE uses thrift 0.9.3 from apache maven repo.
- Fb303 and http components dependencies are no longer needed in FE and
  are removed.
- The minimum openssl version requirement is increased to 1.0.1.

Notable code change:
- Thrift 0.9.3 implements "ostream& operator<<(ostream&, T)" for thrift
  data types. It conflicts with impala-defined functions. This patch
  removes these impala-defined functions and changes the call sites
  accordingly.

Configuration change:
- Thrift codegen option movable_type is enabled. New code no longer
  needs to use std::swap to avoid copying.

Change-Id: I9c303997411237e988ef960157f781776f6fcb60
---
M CMakeLists.txt
M be/src/benchmarks/scheduler-benchmark.cc
M be/src/codegen/llvm-codegen.cc
M be/src/common/init.cc
M be/src/exec/exchange-node.cc
M be/src/exec/exec-node.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/parquet-column-readers.cc
M be/src/exec/parquet-metadata-utils.cc
M be/src/rpc/TAcceptQueueServer.cpp
M be/src/rpc/TAcceptQueueServer.h
M be/src/rpc/rpc-trace.cc
M be/src/rpc/thrift-client.cc
M be/src/rpc/thrift-server-test.cc
M be/src/rpc/thrift-server.cc
M be/src/rpc/thrift-server.h
M be/src/rpc/thrift-thread.h
M be/src/rpc/thrift-util.cc
M be/src/rpc/thrift-util.h
M be/src/runtime/client-cache.cc
M be/src/runtime/coordinator-backend-state.cc
M be/src/runtime/coordinator.cc
M be/src/runtime/data-stream-mgr.cc
M be/src/runtime/data-stream-recvr.cc
M be/src/runtime/data-stream-sender.cc
M be/src/runtime/fragment-instance-state.cc
M be/src/runtime/krpc-data-stream-mgr.cc
M be/src/runtime/krpc-data-stream-recvr.cc
M be/src/runtime/krpc-data-stream-sender.cc
M be/src/runtime/mem-tracker.cc
M be/src/runtime/query-exec-mgr.cc
M be/src/runtime/query-state.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/runtime-state.cc
M be/src/scheduling/admission-controller.cc
M be/src/scheduling/scheduler.cc
M be/src/service/child-query.cc
M be/src/service/client-request-state.cc
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-internal-service.cc
M be/src/service/impala-server.cc
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M be/src/service/query-result-set.cc
M be/src/statestore/statestore.cc
M be/src/util/collection-metrics.h
M be/src/util/debug-util.cc
M be/src/util/debug-util.h
M be/src/util/histogram-metric.h
M be/src/util/metrics.h
M be/src/util/network-util.cc
M be/src/util/network-util.h
M be/src/util/webserver.cc
M bin/impala-config.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M fe/pom.xml
M infra/python/deps/compiled-requirements.txt
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/shell/test_shell_commandline.py
63 files changed, 336 insertions(+), 443 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/68/9168/5
--
To view, visit http://gerrit.cloudera.org:8080/9168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9c303997411237e988ef960157f781776f6fcb60
Gerrit-Change-Number: 9168
Gerrit-PatchSet: 5
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tianyi Wang 

[Impala-ASF-CR] IMPALA-5801: Clean up codegen GetType() interface

[Csaba Ringhofer (Code Review)]

Hello Lars Volker, Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9063

to look at the new patch set (#4).

Change subject: IMPALA-5801: Clean up codegen GetType() interface
..

IMPALA-5801: Clean up codegen GetType() interface

Several functions that return llvm::(Pointer)Type were renamed
to make them shorter or indicate their roles more clearly.

The renamed functions can be found in llvm-codegen.h/cc. Changes
in other files contain only renamed calls with the same functinality.

Testing:
No new tests are necessery, as no functianility was changed.

Change-Id: Ib146ca21af82023b0b460f813eae3435b4b2eb77
---
M be/src/benchmarks/hash-benchmark.cc
M be/src/codegen/codegen-anyval.cc
M be/src/codegen/llvm-codegen-test.cc
M be/src/codegen/llvm-codegen.cc
M be/src/codegen/llvm-codegen.h
M be/src/exec/exec-node.cc
M be/src/exec/filter-context.cc
M be/src/exec/hash-table.cc
M be/src/exec/hdfs-avro-scanner.cc
M be/src/exec/hdfs-parquet-scanner.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/partitioned-aggregation-node.cc
M be/src/exec/partitioned-hash-join-builder.cc
M be/src/exec/partitioned-hash-join-node.cc
M be/src/exec/text-converter.cc
M be/src/exprs/compound-predicates.cc
M be/src/exprs/scalar-expr.cc
M be/src/exprs/scalar-fn-call.cc
M be/src/exprs/slot-ref.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/tuple.cc
M be/src/runtime/types.cc
M be/src/util/tuple-row-compare.cc
23 files changed, 325 insertions(+), 307 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/9063/4
--
To view, visit http://gerrit.cloudera.org:8080/9063
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib146ca21af82023b0b460f813eae3435b4b2eb77
Gerrit-Change-Number: 9063
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-5440 Add planner tests with extreme statistics values

[Anonymous Coward (Code Review)]

xyutin...@cloudera.com has uploaded a new patch set (#12). ( 
http://gerrit.cloudera.org:8080/9065 )

Change subject: IMPALA-5440 Add planner tests with extreme statistics values
..

IMPALA-5440 Add planner tests with extreme statistics values

This commit address some of the issues in JIRA: tests against the cardinality
overflowing from JOIN, UNION, CROSS JOIN, FULL OUTER JOIN,
0 row number and negative row number, as well as cardinality on Subplan node.

Change-Id: I86dec47cf1438882cafaec53e97864ccfcdff3cb
---
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTestBase.java
2 files changed, 142 insertions(+), 1 deletion(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/9065/12
--
To view, visit http://gerrit.cloudera.org:8080/9065
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I86dec47cf1438882cafaec53e97864ccfcdff3cb
Gerrit-Change-Number: 9065
Gerrit-PatchSet: 12
Gerrit-Owner: xyutin...@cloudera.com
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Vuk Ercegovac 
Gerrit-Reviewer: xyutin...@cloudera.com

[Impala-ASF-CR] IMPALA-6477: Fix flakiness with thrift-server-test

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/9247 )

Change subject: IMPALA-6477: Fix flakiness with thrift-server-test
..

IMPALA-6477: Fix flakiness with thrift-server-test

THe ThriftKerberizedParamsTest.SslConnectivity backend test verifies
that when kerberos and TLS are switched on, we cannot perform a
negotiation with a non TLS enabled client.

The test works as expected, however, the error messages can be
different in some scenarios. This patch just updates the test to
expect a different error message that has the same effect.

This change also links KRB5_REALM_OVERRIDE back with the
rpc-mgr-test, as one test run showed a similar failure as
IMPALA-6268. It's possible that this is due to IMPALA-6448.

Change-Id: I6294447416e864b842013487f4d93afadc6b
Reviewed-on: http://gerrit.cloudera.org:8080/9247
Reviewed-by: Alex Behm 
Tested-by: Impala Public Jenkins
---
M be/src/rpc/CMakeLists.txt
M be/src/rpc/thrift-server-test.cc
2 files changed, 9 insertions(+), 2 deletions(-)

Approvals:
  Alex Behm: Looks good to me, approved
  Impala Public Jenkins: Verified

--
To view, visit http://gerrit.cloudera.org:8080/9247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I6294447416e864b842013487f4d93afadc6b
Gerrit-Change-Number: 9247
Gerrit-PatchSet: 4
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-6477: Fix flakiness with thrift-server-test

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9247 )

Change subject: IMPALA-6477: Fix flakiness with thrift-server-test
..


Patch Set 3: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/9247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6294447416e864b842013487f4d93afadc6b
Gerrit-Change-Number: 9247
Gerrit-PatchSet: 3
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 08 Feb 2018 20:22:25 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3193: Show table's comment on show tables

[Dimitris Tsirogiannis (Code Review)]

Dimitris Tsirogiannis has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8851 )

Change subject: IMPALA-3193: Show table's comment on show tables
..


Patch Set 13:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8851/13/fe/src/main/java/org/apache/impala/service/MetadataOp.java
File fe/src/main/java/org/apache/impala/service/MetadataOp.java:

http://gerrit.cloudera.org:8080/#/c/8851/13/fe/src/main/java/org/apache/impala/service/MetadataOp.java@281
PS13, Line 281:   try {
  : table = catalog.getTable(db.getName(), tabName);
  :   } catch (TableLoadingException e) {
  : // Ignore exception (this table will be skipped).
  :   }
  :   if (table == null) continue;
> The removal of the code causes the timed out issue. The problem is infinite
Thanks for investigating this. I think that we may have some redundant calls 
(fe.getTables() L279 and getTable() L282). Feel free to revert some of these 
changes to avoid these redundancies.



--
To view, visit http://gerrit.cloudera.org:8080/8851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I44f814af05db6f3c027718ade9f474f8b8153bcd
Gerrit-Change-Number: 8851
Gerrit-PatchSet: 13
Gerrit-Owner: Kim Jin Chul 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kim Jin Chul 
Gerrit-Comment-Date: Thu, 08 Feb 2018 18:19:21 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-5690: Upgrade thrift to 0.9.3-p3

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9168 )

Change subject: IMPALA-5690: Upgrade thrift to 0.9.3-p3
..


Patch Set 4:

Thomas: could you give this a review?

Sailesh: please review the OpenSSL-related stuff!


--
To view, visit http://gerrit.cloudera.org:8080/9168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9c303997411237e988ef960157f781776f6fcb60
Gerrit-Change-Number: 9168
Gerrit-PatchSet: 4
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Thu, 08 Feb 2018 17:31:16 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-5690: Upgrade thrift to 0.9.3-p3

[Philip Zeyliger (Code Review)]

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9168 )

Change subject: IMPALA-5690: Upgrade thrift to 0.9.3-p3
..


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9168/4/be/src/common/init.cc
File be/src/common/init.cc:

http://gerrit.cloudera.org:8080/#/c/9168/4/be/src/common/init.cc@236
PS4, Line 236:   // There is no way to pass MSG_NOSIGNAL into OpenSSL so the 
signal must be disabled.
Tell us more about this? Who sends SIGPIPE?



--
To view, visit http://gerrit.cloudera.org:8080/9168
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9c303997411237e988ef960157f781776f6fcb60
Gerrit-Change-Number: 9168
Gerrit-PatchSet: 4
Gerrit-Owner: Tianyi Wang 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Sailesh Mukil 
Gerrit-Reviewer: Tianyi Wang 
Gerrit-Comment-Date: Thu, 08 Feb 2018 17:30:40 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6258: Uninitialized tuple pointers in row batch for empty rows

[Zoltan Borok-Nagy (Code Review)]

Zoltan Borok-Nagy has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9239 )

Change subject: IMPALA-6258: Uninitialized tuple pointers in row batch for 
empty rows
..


Patch Set 1:

(4 comments)

Thanks for the comments!

http://gerrit.cloudera.org:8080/#/c/9239/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9239/1//COMMIT_MSG@32
PS1, Line 32:
> It would be good to do a microbenchmark to make sure we haven't regressed p
Extended the commit with a measurement. Do you think we should develop a 
count(*) optimisation for kudu like we do for Parquet?


http://gerrit.cloudera.org:8080/#/c/9239/1/be/src/runtime/tuple.h
File be/src/runtime/tuple.h:

http://gerrit.cloudera.org:8080/#/c/9239/1/be/src/runtime/tuple.h@225
PS1, Line 225:   void SetNull(const NullIndicatorOffset& offset) {
> I'm not sure about adding DCHECKs to all these functions. They're definitel
Removed the DCHECKs, pointer value is 42 now, such a low address should 
segfault (it does segfault on my system).


http://gerrit.cloudera.org:8080/#/c/9239/1/be/src/runtime/tuple.cc
File be/src/runtime/tuple.cc:

http://gerrit.cloudera.org:8080/#/c/9239/1/be/src/runtime/tuple.cc@49
PS1, Line 49: Tuple* Tuple::Poison() {
> Can we avoid this function call indirection? MemPool just uses a bogus cons
Now it is just a pointer.


http://gerrit.cloudera.org:8080/#/c/9239/1/be/src/runtime/tuple.cc@87
PS1, Line 87:   if (UNLIKELY(this == Poison())) return Poison();
> We can skip this case. MemPool::Allocate() actually already returns a poiso
Yeah I saw that, just thought maybe it is better to keep our own poison (that 
sounds wrong:) ).



--
To view, visit http://gerrit.cloudera.org:8080/9239
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2981227e62eb5971508e0698e189519755de
Gerrit-Change-Number: 9239
Gerrit-PatchSet: 1
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: Zoltan Borok-Nagy 
Gerrit-Comment-Date: Thu, 08 Feb 2018 17:28:12 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6258: Uninitialized tuple pointers in row batch for empty rows

[Zoltan Borok-Nagy (Code Review)]

Hello Tim Armstrong,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9239

to look at the new patch set (#2).

Change subject: IMPALA-6258: Uninitialized tuple pointers in row batch for 
empty rows
..

IMPALA-6258: Uninitialized tuple pointers in row batch for empty rows

Tuple pointers in the generated row batches may not be initialized
if a tuple has byte size 0. There are some codes which compare these
uninitialized pointers against nullptr so having them uninitialized
may return wrong (and non-deterministic) results, e.g.:
impala::TupleIsNullPredicate::GetBooleanVal()

The following query produces non-deterministic results currently:

SELECT count(v.x) FROM functional.alltypestiny t3 LEFT OUTER JOIN (
  SELECT true AS x FROM functional.alltypestiny t1 LEFT OUTER JOIN
  functional.alltypestiny t2 ON (true)) v ON (v.x = t3.bool_col)
WHERE t3.bool_col = true;

The alltypestiny table has 8 records, 4 records of them has the true
boolean value for bool_col. Therefore, the above query is a fancy
way of saying "8 * 8 * 4", i.e. it should return 256.

The solution is that scanners initialize tuple pointers to a non-null
value when there are no materialized slots. This non-null value is
provided by the new static method called Tuple::Poison().

I extended QueryTest/scanners.test with the above query. This test
executes the query against all table formats.

This change has the biggest performance impact on count(*) queries on
large kudu tables. For my quick benchmark I chose tpch_kudu.lineitem
which has 6001215 rows.

Without this patch 'select count(*) from tpch_kudu.lineitem' runs
around 0.15s. With the patch applied, it runs around 0.24s. I ran the
query on my desktop PC.

Without this patch:
ExecSummary:
Operator   #Hosts   Avg Time   Max Time  #Rows  Est. #Rows   Peak Mem  Est. 
Peak Mem  Detail
-
03:AGGREGATE1  432.840us  432.840us  1   1   28.00 KB   
10.00 MB  FINALIZE
02:EXCHANGE 1   31.375ms   31.375ms  3   1  0   
   0  UNPARTITIONED
01:AGGREGATE35.013ms5.502ms  3   1   16.00 KB   
10.00 MB
00:SCAN KUDU3   16.581ms   20.199ms  6.00M   6.00M  512.00 KB   
   0  tpch_kudu.lineitem

With this patch:
ExecSummary:
Operator   #Hosts   Avg Time   Max Time  #Rows  Est. #Rows   Peak Mem  Est. 
Peak Mem  Detail
-
03:AGGREGATE1  472.898us  472.898us  1   1   28.00 KB   
10.00 MB  FINALIZE
02:EXCHANGE 1  122.700ms  122.700ms  3   1  0   
   0  UNPARTITIONED
01:AGGREGATE36.104ms6.151ms  3   1   16.00 KB   
10.00 MB
00:SCAN KUDU3   90.856ms  107.409ms  6.00M   6.00M  512.00 KB   
   0  tpch_kudu.lineitem

Change-Id: I2981227e62eb5971508e0698e189519755de
---
M be/src/exec/hdfs-scanner.cc
M be/src/exec/kudu-scanner.cc
M be/src/runtime/tuple.cc
M be/src/runtime/tuple.h
M testdata/workloads/functional-query/queries/QueryTest/scanners.test
5 files changed, 30 insertions(+), 3 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/9239/2
--
To view, visit http://gerrit.cloudera.org:8080/9239
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2981227e62eb5971508e0698e189519755de
Gerrit-Change-Number: 9239
Gerrit-PatchSet: 2
Gerrit-Owner: Zoltan Borok-Nagy 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-6477: Disable rpc-mgr-kerberized-test

[Sailesh Mukil (Code Review)]

Sailesh Mukil has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9256


Change subject: IMPALA-6477: Disable rpc-mgr-kerberized-test
..

IMPALA-6477: Disable rpc-mgr-kerberized-test

This test is breaking builds on CentOS 6.4. Disable it until we
find out the root cause.

Change-Id: I5654d8c41b130544a8f9bc38f69433fcc171eca0
---
M be/src/rpc/rpc-mgr-kerberized-test.cc
1 file changed, 3 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/56/9256/1
--
To view, visit http://gerrit.cloudera.org:8080/9256
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I5654d8c41b130544a8f9bc38f69433fcc171eca0
Gerrit-Change-Number: 9256
Gerrit-PatchSet: 1
Gerrit-Owner: Sailesh Mukil 

[Impala-ASF-CR] IMPALA-6477: Fix flakiness with thrift-server-test

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9247 )

Change subject: IMPALA-6477: Fix flakiness with thrift-server-test
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1907/


--
To view, visit http://gerrit.cloudera.org:8080/9247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6294447416e864b842013487f4d93afadc6b
Gerrit-Change-Number: 9247
Gerrit-PatchSet: 3
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 08 Feb 2018 16:42:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-3193: Show table's comment on show tables

[Kim Jin Chul (Code Review)]

Kim Jin Chul has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/8851 )

Change subject: IMPALA-3193: Show table's comment on show tables
..


Patch Set 13:

(1 comment)

The timed out issue at Jenkins should be resolved on this patch set.

http://gerrit.cloudera.org:8080/#/c/8851/13/fe/src/main/java/org/apache/impala/service/MetadataOp.java
File fe/src/main/java/org/apache/impala/service/MetadataOp.java:

http://gerrit.cloudera.org:8080/#/c/8851/13/fe/src/main/java/org/apache/impala/service/MetadataOp.java@281
PS13, Line 281:   try {
  : table = catalog.getTable(db.getName(), tabName);
  :   } catch (TableLoadingException e) {
  : // Ignore exception (this table will be skipped).
  :   }
  :   if (table == null) continue;
The removal of the code causes the timed out issue. The problem is infinite 
retries for not loaded table at Frontend.requestTblLoadAndWait.

I thought the code should be duplicate. By the way, there are some conditions 
to check that a table is loaded, table type is come from IncompleteType and 
TableLoadingException happens. I prefer to revive the code instead of copy of 
essential code snippet due to avoidance of any side effect. Here is the 
relevant code for the condition: 
https://github.com/apache/impala/blob/master/fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java#L250



--
To view, visit http://gerrit.cloudera.org:8080/8851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I44f814af05db6f3c027718ade9f474f8b8153bcd
Gerrit-Change-Number: 8851
Gerrit-PatchSet: 13
Gerrit-Owner: Kim Jin Chul 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kim Jin Chul 
Gerrit-Comment-Date: Thu, 08 Feb 2018 13:53:45 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-3193: Show table's comment on show tables

[Kim Jin Chul (Code Review)]

Hello Dimitris Tsirogiannis, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/8851

to look at the new patch set (#13).

Change subject: IMPALA-3193: Show table's comment on show tables
..

IMPALA-3193: Show table's comment on show tables

Currently SHOW TABLES command does not display given comment
when creating a table. DESC  shows column's comment and
SHOW DATABASES also shows database's comment.

On this change, table's comment is displayed if the table is
loaded. If the table is not loaded, an empty comment is returned.

Testing:
Adds E2E test: TestShowTables.test_table_comment_visibility

Change-Id: I44f814af05db6f3c027718ade9f474f8b8153bcd
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M common/thrift/Frontend.thrift
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M testdata/workloads/functional-query/queries/QueryTest/show.test
M tests/metadata/test_metadata_query_statements.py
M www/catalog.tmpl
19 files changed, 290 insertions(+), 200 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/8851/13
--
To view, visit http://gerrit.cloudera.org:8080/8851
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I44f814af05db6f3c027718ade9f474f8b8153bcd
Gerrit-Change-Number: 8851
Gerrit-PatchSet: 13
Gerrit-Owner: Kim Jin Chul 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Kim Jin Chul 

[Impala-ASF-CR] IMPALA-6477: Fix flakiness with thrift-server-test

[Impala Public Jenkins (Code Review)]

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9247 )

Change subject: IMPALA-6477: Fix flakiness with thrift-server-test
..


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/1906/


--
To view, visit http://gerrit.cloudera.org:8080/9247
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6294447416e864b842013487f4d93afadc6b
Gerrit-Change-Number: 9247
Gerrit-PatchSet: 3
Gerrit-Owner: Sailesh Mukil 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Comment-Date: Thu, 08 Feb 2018 10:44:56 +
Gerrit-HasComments: No