date:20190221

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12548 )

Change subject: IMPALA-7046: introduce "global" debug_actions
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12548/1/tests/custom_cluster/test_admission_controller.py
File tests/custom_cluster/test_admission_controller.py:

http://gerrit.cloudera.org:8080/#/c/12548/1/tests/custom_cluster/test_admission_controller.py@549
PS1, Line 549: "
> Done
Feel free to ignore these formatting changes - it may increase the changes of 
conflicts with later patches.



--
To view, visit http://gerrit.cloudera.org:8080/12548
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I77663a539be18711a4f12c470ffd7474e3d69388
Gerrit-Change-Number: 12548
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 22 Feb 2019 01:13:42 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8178: Disable file handle cache for HDFS erasure coded files

Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12552 )

Change subject: IMPALA-8178: Disable file handle cache for HDFS erasure coded 
files
..


Patch Set 2: Code-Review+2

Carrying +2


--
To view, visit http://gerrit.cloudera.org:8080/12552
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c761e08aacc952de0033a4c91e07f15c8ec96da
Gerrit-Change-Number: 12552
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Fri, 22 Feb 2019 01:10:28 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8178: Disable file handle cache for HDFS erasure coded files

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12552 )

Change subject: IMPALA-8178: Disable file handle cache for HDFS erasure coded 
files
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3817/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/12552
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c761e08aacc952de0033a4c91e07f15c8ec96da
Gerrit-Change-Number: 12552
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Fri, 22 Feb 2019 01:16:06 +
Gerrit-HasComments: No

[Impala-ASF-CR](2.x) IMPALA-7046: introduce "global" debug actions

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12548 )

Change subject: IMPALA-7046: introduce "global" debug_actions
..


Patch Set 2: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12548
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: 2.x
Gerrit-MessageType: comment
Gerrit-Change-Id: I77663a539be18711a4f12c470ffd7474e3d69388
Gerrit-Change-Number: 12548
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 22 Feb 2019 01:13:48 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8178: Disable file handle cache for HDFS erasure coded files

Hello Philip Zeyliger, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12552

to look at the new patch set (#2).

Change subject: IMPALA-8178: Disable file handle cache for HDFS erasure coded 
files
..

IMPALA-8178: Disable file handle cache for HDFS erasure coded files

Testing on an erasure coded minicluster has revealed that each
file handle for an erasure coded files uses about 3MB of native
memory. This shows up as "java.nio:type=BufferPool,name=direct"
in the /jmx endpoint (here showing the output when 608 handles
are open):

{
  "name": "java.nio:type=BufferPool,name=direct",
  "modelerType": "sun.management.ManagementFactoryHelper$1",
  "Name": "direct",
  "TotalCapacity": 1921048960,
  "MemoryUsed": 1921048961,
  "Count": 633,
  "ObjectName": "java.nio:type=BufferPool,name=direct"
}

The memory is not released or reduced by a call to unbuffer(),
so these file handles are not suitable for long term caching.
HDFS-14308 tracks the implementation of unbuffer() for
DFSStripedInputStream. This issue showed up when remote
file handle caching was enabled in IMPALA-7265, as erasure
coded files are always scheduled to be remote (IMPALA-7019).

This disables file handle caching for erasure coded files,
which requires plumbing through the information about which
ScanRanges are accessing erasure coded files.

With this change, core tests pass on an erasure coded system.

Change-Id: I8c761e08aacc952de0033a4c91e07f15c8ec96da
---
M be/src/exec/base-sequence-scanner.cc
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/scanner-context.cc
M be/src/runtime/io/disk-io-mgr-stress.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/scheduler.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
17 files changed, 62 insertions(+), 33 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/12552/2
--
To view, visit http://gerrit.cloudera.org:8080/12552
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8c761e08aacc952de0033a4c91e07f15c8ec96da
Gerrit-Change-Number: 12552
Gerrit-PatchSet: 2
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger

[Impala-ASF-CR] IMPALA-8178: Disable file handle cache for HDFS erasure coded files

Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12552 )

Change subject: IMPALA-8178: Disable file handle cache for HDFS erasure coded 
files
..


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/12552/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12552/1//COMMIT_MSG@12
PS1, Line 12: unbuffer(), so these file handles are not suitable for long
> Maybe mention that we used /jmx to get at at least part of the memory usage
Added info about what /jmx shows


http://gerrit.cloudera.org:8080/#/c/12552/1/be/src/runtime/io/scan-range.cc
File be/src/runtime/io/scan-range.cc:

http://gerrit.cloudera.org:8080/#/c/12552/1/be/src/runtime/io/scan-range.cc@463
PS1, Line 463:   DCHECK(buffer_opts.client_buffer_ == nullptr ||
> I think you could reasonably assert:
I added this assert. This is true due to IMPALA-7019, so I mention that here.



--
To view, visit http://gerrit.cloudera.org:8080/12552
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c761e08aacc952de0033a4c91e07f15c8ec96da
Gerrit-Change-Number: 12552
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Fri, 22 Feb 2019 01:09:12 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 7: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 7
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 21 Feb 2019 23:57:25 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8178: Disable file handle cache for HDFS erasure coded files

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12552 )

Change subject: IMPALA-8178: Disable file handle cache for HDFS erasure coded 
files
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2208/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12552
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c761e08aacc952de0033a4c91e07f15c8ec96da
Gerrit-Change-Number: 12552
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Fri, 22 Feb 2019 00:28:06 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 6:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2207/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 6
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Fri, 22 Feb 2019 00:24:22 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7975 : Improve supportability of the automatic invalidate feature

2019-02-21 Thread Bharath Vissapragada (Code Review)

Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12549 )

Change subject: IMPALA-7975 : Improve supportability of the automatic 
invalidate feature
..


Patch Set 2:

(15 comments)

Haven't looked at each metric in detail but have some high-level comments on 
the code flow. Will take a deeper look in the next pass.

http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/catalog/catalog-server.h
File be/src/catalog/catalog-server.h:

http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/catalog/catalog-server.h@212
PS2, Line 212:
Method doc


http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/catalog/catalog-server.cc
File be/src/catalog/catalog-server.cc:

http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/catalog/catalog-server.cc@399
PS2, Line 399:  TGetEventProcessorMetricsResponse eventProcessorMetricsResponse;
 : status = 
catalog_->GetEventProcessorMetrics();
 : if (!status.ok()) {
 :   LOG(ERROR) << "Error refreshing metastore event metrics: " 
<< status.GetDetail();
 :   continue;
 : }
merge this with the above call? Avoids an unncessary JNI call.


http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/catalog/catalog-server.cc@521
PS2, Line 521:  url handler for the metastore events page. It calls into 
JniCatalog to get the latest
 : // metastore event processor metrics and adds it to the document
Move to header.


http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/catalog/catalog.h
File be/src/catalog/catalog.h:

http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/catalog/catalog.h@112
PS2, Line 112:   /// Gets all functions in the catalog matching the parameters 
in the given
newline and method doc.


http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/util/event-metrics.h
File be/src/util/event-metrics.h:

http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/util/event-metrics.h@22
PS2, Line 22: // class which is used to fetch the metastore event related 
metrics from catalog
///


http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/util/event-metrics.h@28
PS2, Line 28:   static void refresh(TGetEventProcessorMetricsResponse* 
response);
///


http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/util/event-metrics.h@30
PS2, Line 30:   static IntCounter* NUM_EVENTS_RECEIVED_COUNTER;
Add inline comments for each of these? What they represent?


http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/util/event-metrics.cc
File be/src/util/event-metrics.cc:

http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/util/event-metrics.cc@56
PS2, Line 56:   MetricGroup* event_metrics = 
metric_group->GetOrCreateChildGroup("events");
If event processing is disabled, do we need to register them?


http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/util/event-metrics.cc@76
PS2, Line 76: // refresh all the metrics which are used to display on webui 
based on the given response
Move to header


http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/util/event-metrics.cc@82
PS2, Line 82: DCHECK(response->__isset.events_process_duration_mean);
if you are DCHECKing here why the if(__isset)s below?


http://gerrit.cloudera.org:8080/#/c/12549/2/be/src/util/event-metrics.cc@113
PS2, Line 113:   LOG(ERROR) << "Received a null response when trying to refresh 
metastore event metrics";
Move to the top?

if (!response) LOG.error() return;


http://gerrit.cloudera.org:8080/#/c/12549/2/common/thrift/JniCatalog.thrift
File common/thrift/JniCatalog.thrift:

http://gerrit.cloudera.org:8080/#/c/12549/2/common/thrift/JniCatalog.thrift@759
PS2, Line 759:   // status of event processor
Add new lines between each.


http://gerrit.cloudera.org:8080/#/c/12549/2/common/thrift/JniCatalog.thrift@775
PS2, Line 775:   // Summary of the event processor metrics which include status 
and more detailed metrics
It looks like there are a couple of ways this is used. One to get metrics 1-9, 
and two the summary.

Can we split those up into two different groups and merge the first group into 
TGetCatalogServerMetricsResponse? That way in RefreshMetrics() we have just 1 
JNI call and for the web end point, we could call the latter. That way we don't 
build the summary string and discard it every 2s.


http://gerrit.cloudera.org:8080/#/c/12549/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
File fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java:

http://gerrit.cloudera.org:8080/#/c/12549/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java@232
PS2, Line 232: protected final Metrics metrics_;
Curious why this would be in the Event class? I'd guess it should be something 
more global accessible by all events?


http://gerrit.cloudera.org:8080/#/c/12549/2/fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
File

[Impala-ASF-CR] IMPALA-8178: Disable file handle cache for HDFS erasure coded files

2019-02-21 Thread Philip Zeyliger (Code Review)

Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12552 )

Change subject: IMPALA-8178: Disable file handle cache for HDFS erasure coded 
files
..


Patch Set 1: Code-Review+2

(2 comments)

Thanks.

http://gerrit.cloudera.org:8080/#/c/12552/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/12552/1//COMMIT_MSG@12
PS1, Line 12: unbuffer(), so these file handles are not suitable for long
Maybe mention that we used /jmx to get at at least part of the memory usage?


http://gerrit.cloudera.org:8080/#/c/12552/1/be/src/runtime/io/scan-range.cc
File be/src/runtime/io/scan-range.cc:

http://gerrit.cloudera.org:8080/#/c/12552/1/be/src/runtime/io/scan-range.cc@463
PS1, Line 463:   DCHECK(buffer_opts.client_buffer_ == nullptr ||
I think you could reasonably assert:

DCHECK(!(expected_local && is_erasure_coded))

I.e., if something is expected_local, it can't possibly be erasure coded.



--
To view, visit http://gerrit.cloudera.org:8080/12552
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8c761e08aacc952de0033a4c91e07f15c8ec96da
Gerrit-Change-Number: 12552
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 21 Feb 2019 23:59:11 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 6: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 6
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 21 Feb 2019 23:51:51 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3816/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 7
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 21 Feb 2019 23:57:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/12543/5/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:

http://gerrit.cloudera.org:8080/#/c/12543/5/common/thrift/ImpalaService.thrift@61
PS5, Line 61:   // > 1: executes on at most that many nodes at any point in 
time (ie there can be
:   //  more nodes than numNodes with plan fragments for this 
query but at most
:   //  numNodes would be active at any point in time)
:   // Constants (NUM_NODES_ALL NUM_NODES_ALL_RACKS) are defined in 
JavaConstants.thrift.
> Nit: This change removes a bunch of commas in various comments. Maybe some
Thanks for catching this one, fixed it.


http://gerrit.cloudera.org:8080/#/c/12543/5/common/thrift/PlanNodes.thrift
File common/thrift/PlanNodes.thrift:

http://gerrit.cloudera.org:8080/#/c/12543/5/common/thrift/PlanNodes.thrift@75
PS5, Line 75:   // A floating point number in range [0.0 1.0] that gives the 
probability of denying
> Same comma missing thing here and other comments in this file.
Done



--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 5
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 21 Feb 2019 23:41:28 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8178: Disable file handle cache for HDFS erasure coded files

Joe McDonnell has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/12552


Change subject: IMPALA-8178: Disable file handle cache for HDFS erasure coded 
files
..

IMPALA-8178: Disable file handle cache for HDFS erasure coded files

Testing on an erasure coded minicluster has revealed that each
file handle for an erasure coded files uses about 3MB of native
memory. The memory is not released or reduced by a call to
unbuffer(), so these file handles are not suitable for long
term caching. HDFS-14308 tracks the implementation of unbuffer()
for DFSStripedInputStream. This issue showed up when remote
file handle caching was enabled in IMPALA-7265, causing high
memory requirements for ordinary core tests.

This disables file handle caching for erasure coded files,
which requires plumbing through the information about which
ScanRanges are on erasure coded files.

With this change, core tests pass on an erasure coded system.

Change-Id: I8c761e08aacc952de0033a4c91e07f15c8ec96da
---
M be/src/exec/base-sequence-scanner.cc
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/parquet-column-readers.cc
M be/src/exec/scanner-context.cc
M be/src/runtime/io/disk-io-mgr-stress.cc
M be/src/runtime/io/disk-io-mgr-test.cc
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
M be/src/runtime/tmp-file-mgr.cc
M be/src/scheduling/scheduler.cc
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
17 files changed, 60 insertions(+), 33 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/12552/1
--
To view, visit http://gerrit.cloudera.org:8080/12552
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8c761e08aacc952de0033a4c91e07f15c8ec96da
Gerrit-Change-Number: 12552
Gerrit-PatchSet: 1
Gerrit-Owner: Joe McDonnell

[Impala-ASF-CR] IMPALA-7560: Improve selectivity estimate for !=

Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/12427 )

Change subject: IMPALA-7560: Improve selectivity estimate for !=
..


Abandoned

Too risky to do now. Will revisit as part of a new patch later.
--
To view, visit http://gerrit.cloudera.org:8080/12427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I8f6013c9ef95a89d55d8b25f0b5433c81582a62f
Gerrit-Change-Number: 12427
Gerrit-PatchSet: 5
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

Hello Philip Zeyliger, Tim Armstrong, Joe McDonnell, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12543

to look at the new patch set (#6).

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..

IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

IMPALA-7694 added a field in the middle of the Metrics.TUnit enum, which
broke backwards compatibility with profiles that had been written by
older versions of Impala. This change fixes the ordering by moving the
field to the end of the enum.

Additionally, it adds a warning to the top of all Thrift files that are
part of the binary profile format, and an note of caution to the main
definition in RuntimeProfile.thrift.

This change also fixes the order of all enums in our Thrift files to
make errors like this less likely in the future.

Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
---
M common/thrift/BackendGflags.thrift
M common/thrift/CatalogObjects.thrift
M common/thrift/DataSinks.thrift
M common/thrift/ExecStats.thrift
M common/thrift/Exprs.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/LineageGraph.thrift
M common/thrift/Logging.thrift
M common/thrift/Metrics.thrift
M common/thrift/Partitions.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/RuntimeProfile.thrift
M common/thrift/StatestoreService.thrift
M common/thrift/Status.thrift
M common/thrift/Types.thrift
M common/thrift/beeswax.thrift
20 files changed, 482 insertions(+), 444 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/12543/6
--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 6
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

Alex Rodoni has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/12550 )

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..

[DOCS] Updated the description of the port_number parameter in SHUTDOWN

Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Reviewed-on: http://gerrit.cloudera.org:8080/12550
Tested-by: Impala Public Jenkins 
Reviewed-by: Bikramjeet Vig 
---
M docs/topics/impala_shutdown.xml
1 file changed, 28 insertions(+), 13 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Bikramjeet Vig: Looks good to me, approved

--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 4
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-7975 : Improve supportability of the automatic invalidate feature

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12549 )

Change subject: IMPALA-7975 : Improve supportability of the automatic 
invalidate feature
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2206/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12549
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I23cb3aa866879eca03c64ab881796eaa9caa0337
Gerrit-Change-Number: 12549
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 23:36:58 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

2019-02-21 Thread Bikramjeet Vig (Code Review)

Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12550 )

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..


Patch Set 3: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 23:37:13 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12550 )

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..


Patch Set 3:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/230/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 23:03:44 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7975 : Improve supportability of the automatic invalidate feature

2019-02-21 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/12549 )

Change subject: IMPALA-7975 : Improve supportability of the automatic 
invalidate feature
..

IMPALA-7975 : Improve supportability of the automatic invalidate feature

This patch adds supportability metrics to make it easier to find the
state of Metastore events processor. It adds the following metrics

1. Total number of events received so far
This is not super useful in itself but is helpful to see if there are
events being received or not. A more usable metric which publishes the
rate of such events received is also published. See details for it
below.

2. 1min, 5min and 15min rate of events received
Exponential moving average of number of events received in last 1min,
5min and 15min. This can be used to determine if there are spikes in
event processor activity during certain hours of the day

3. Total number of events skipped so far
Events can be skipped based on certain flags are table and database
level. This metric keeps a count of total number of events skipped so
far. Useful to make decisions like - If most of the events are being
skipped, may be just turn off the event processing. If most of the
events are not skipped, do we need to add flags on certain databases?

4. Status of events processor
Easy way to determine what is the current state of metastore events
processor. Possible states could be STOPPED, ACTIVE, ERROR,
NEEDS_INVALIDATE

5. Events fetch and process durations in seconds
Average duration to fetch a batch of events and process it.

Additionally, it also registers a url handler specific for events at
/events. The /events page provides a more detailed view of the metrics
of the event processor which include (min, max, mean, median) of the
durations and rate metrics for all the counters. In the future this page
can be enhanced to show the top-n tables with most number of invalidates
due to the events, etc.

Patch adds few tests to make sure the value of these event metrics are
as expected.

Change-Id: I23cb3aa866879eca03c64ab881796eaa9caa0337
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/catalog/catalogd-main.cc
M be/src/util/CMakeLists.txt
A be/src/util/event-metrics.cc
A be/src/util/event-metrics.h
M common/thrift/JniCatalog.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/common/Metrics.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A www/events.tmpl
19 files changed, 620 insertions(+), 71 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/12549/3
--
To view, visit http://gerrit.cloudera.org:8080/12549
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I23cb3aa866879eca03c64ab881796eaa9caa0337
Gerrit-Change-Number: 12549
Gerrit-PatchSet: 3
Gerrit-Owner: Vihang Karajgaonkar 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

2019-02-21 Thread Bikramjeet Vig (Code Review)

Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12550 )

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..


Patch Set 2:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/12550/2/docs/topics/impala_shutdown.xml
File docs/topics/impala_shutdown.xml:

http://gerrit.cloudera.org:8080/#/c/12550/2/docs/topics/impala_shutdown.xml@133
PS2, Line 133:   based communication.
nit: maybe mention 'inter-impala communication'
since the hs2 and beeswax are also thrift based servers hosted on separate ports


http://gerrit.cloudera.org:8080/#/c/12550/2/docs/topics/impala_shutdown.xml@197
PS2, Line 197: the grace period
oops, this is the 'deadline' not the grace period



--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 22:45:24 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert into

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12529 )

Change subject: IMPALA-7107: [DOCS] Updated the doc for storage formats impala 
cannot insert into
..


Patch Set 2: Verified+1

Build Successful

https://jenkins.impala.io/job/gerrit-docs-auto-test/229/ : Doc tests passed.


--
To view, visit http://gerrit.cloudera.org:8080/12529
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If154e24ade83e2cde2b08878e6f4f88e0048d657
Gerrit-Change-Number: 12529
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Balazs Jeszenszky 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zsombor Fedor (406)
Gerrit-Comment-Date: Thu, 21 Feb 2019 22:39:02 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12550 )

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..


Patch Set 2: Verified+1

Build Successful 

https://jenkins.impala.io/job/gerrit-docs-auto-test/228/ : Doc tests passed.


--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 22:38:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert into

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12529 )

Change subject: IMPALA-7107: [DOCS] Updated the doc for storage formats impala 
cannot insert into
..


Patch Set 2:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/229/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/12529
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If154e24ade83e2cde2b08878e6f4f88e0048d657
Gerrit-Change-Number: 12529
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Balazs Jeszenszky 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zsombor Fedor (406)
Gerrit-Comment-Date: Thu, 21 Feb 2019 22:26:14 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert into

Hello Zsombor Fedor, Balazs Jeszenszky, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12529

to look at the new patch set (#2).

Change subject: IMPALA-7107: [DOCS] Updated the doc for storage formats impala 
cannot insert into
..

IMPALA-7107: [DOCS] Updated the doc for storage formats impala cannot insert 
into

Change-Id: If154e24ade83e2cde2b08878e6f4f88e0048d657
---
M docs/topics/impala_file_formats.xml
1 file changed, 168 insertions(+), 101 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/12529/2
--
To view, visit http://gerrit.cloudera.org:8080/12529
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If154e24ade83e2cde2b08878e6f4f88e0048d657
Gerrit-Change-Number: 12529
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Balazs Jeszenszky 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Zsombor Fedor (406)

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12550 )

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..


Patch Set 2:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/228/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 22:16:19 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

Alex Rodoni has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12550 )

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..


Patch Set 1:

(9 comments)

http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml
File docs/topics/impala_shutdown.xml:

http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@90
PS1, Line 90: current impalad
> should we be more specific to clarify that the "current impalad" is the imp
Done


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@95
PS1, Line 95: backend
> nit: impalad
Done


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@116
PS1, Line 116: the same backend port as the current
> the current impalad's port on which the thrift based ImpalaInternalService
Done


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@117
PS1, Line 117: 25000
> 22000
Done


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@121
PS1, Line 121: the KRPC port for the current
> the current impalad's port on which the KRPC based ImpalaInternalService is
Done


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@131
PS1, Line 131: use the same impalad port.
> use the port on the impalad to be shut down on which the thrift based Impal
Done


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@135
PS1, Line 135: use the KRPC port for your impalad.
> use the port on the impalad to be shut down on which the KRPC based ImpalaI
Done


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@181
PS1, Line 181: another non-coordinator impalad
> any impalad
Done


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@194
PS1, Line 194: threshold time limit
> nit: maybe use the same term 'grace period' here to be consistent with the
Done



--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 22:16:20 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

Hello Bikramjeet Vig, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12550

to look at the new patch set (#2).

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..

[DOCS] Updated the description of the port_number parameter in SHUTDOWN

Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
---
M docs/topics/impala_shutdown.xml
1 file changed, 28 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/12550/2
--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 5:

(2 comments)

One small thing. Otherwise, I'm ready to approve

http://gerrit.cloudera.org:8080/#/c/12543/5/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:

http://gerrit.cloudera.org:8080/#/c/12543/5/common/thrift/ImpalaService.thrift@61
PS5, Line 61:   // > 1: executes on at most that many nodes at any point in 
time (ie there can be
:   //  more nodes than numNodes with plan fragments for this 
query but at most
:   //  numNodes would be active at any point in time)
:   // Constants (NUM_NODES_ALL NUM_NODES_ALL_RACKS) are defined in 
JavaConstants.thrift.
Nit: This change removes a bunch of commas in various comments. Maybe some 
extra commas got caught in a search-and-replace?


http://gerrit.cloudera.org:8080/#/c/12543/5/common/thrift/PlanNodes.thrift
File common/thrift/PlanNodes.thrift:

http://gerrit.cloudera.org:8080/#/c/12543/5/common/thrift/PlanNodes.thrift@75
PS5, Line 75:   // A floating point number in range [0.0 1.0] that gives the 
probability of denying
Same comma missing thing here and other comments in this file.



-- 
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 5
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 21 Feb 2019 22:09:42 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8185: Abstract out real/mock file system operations

2019-02-21 Thread Bharath Vissapragada (Code Review)

Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12437 )

Change subject: IMPALA-8185: Abstract out real/mock file system operations
..


Patch Set 5:

(4 comments)

The general separation of FS stuff looks good to me.

http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/4/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java@102
PS4, Line 102:   public ScanAllocation allocateScans(FeFsTable table, 
TScanRangeSpec scanRangeSpecs,
> Do you know what this is used for?
Per my understanding, we use this in cost formulae. Say, if you have a join on 
top of the scan, the lhs scan determines the number of nodes on which the join 
runs and we factor in the num nodes to determine the overall join cost (for ex: 
is inverted join any cheaper?)

Yes this can diverge with the scheduler implementation and ideally we need to 
consolidate. I remember asking Alex exactly this and he confirmed the behavior.


http://gerrit.cloudera.org:8080/#/c/12437/5/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/5/fe/src/main/java/org/apache/impala/analysis/HdfsFileSystemFacade.java@42
PS5, Line 42: new Configuration();
Use FileSystemUtil.getConfiguration(). I vaguely remember that building a new 
Conf object was expensive and I don't think we need to do it for every query.


http://gerrit.cloudera.org:8080/#/c/12437/5/fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java
File fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java:

http://gerrit.cloudera.org:8080/#/c/12437/5/fe/src/main/java/org/apache/impala/analysis/MockFileSystemFacade.java@40
PS5, Line 40:   public Path validatePath(Analyzer analyzer, Path path,
Add comments why we pass?


http://gerrit.cloudera.org:8080/#/c/12437/5/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test
File 
testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test:

http://gerrit.cloudera.org:8080/#/c/12437/5/testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test@222
PS5, Line 222: |  Per-Host Resources: mem-estimate=88.00MB 
mem-reservation=88.00MB thread-reservation=1
Any idea why the mem-estimates changed with this patch?



--
To view, visit http://gerrit.cloudera.org:8080/12437
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1a385923b64c9fb59cc6e700ee7ee14919398e6d
Gerrit-Change-Number: 12437
Gerrit-PatchSet: 5
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:56:03 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8181: Abbreviate row counts in EXPLAIN

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12438 )

Change subject: IMPALA-8181: Abbreviate row counts in EXPLAIN
..


Patch Set 9: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/3814/


--
To view, visit http://gerrit.cloudera.org:8080/12438
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c
Gerrit-Change-Number: 12438
Gerrit-PatchSet: 9
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:48:50 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

2019-02-21 Thread Bikramjeet Vig (Code Review)

Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12550 )

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..


Patch Set 1:

(9 comments)

http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml
File docs/topics/impala_shutdown.xml:

http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@90
PS1, Line 90: current impalad
should we be more specific to clarify that the "current impalad" is the impalad 
to whom the shutdown statement is submitted.


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@95
PS1, Line 95: backend
nit: impalad
To be consistent on how we address impalad/backend


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@116
PS1, Line 116: the same backend port as the current
the current impalad's port on which the thrift based ImpalaInternalService is 
exported


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@117
PS1, Line 117: 25000
22000


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@121
PS1, Line 121: the KRPC port for the current
the current impalad's port on which the KRPC based ImpalaInternalService is 
exported


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@131
PS1, Line 131: use the same impalad port.
use the port on the impalad to be shut down on which the thrift based 
ImpalaInternalService is exported


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@135
PS1, Line 135: use the KRPC port for your impalad.
use the port on the impalad to be shut down on which the KRPC based 
ImpalaInternalService is exported


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@181
PS1, Line 181: another non-coordinator impalad
any impalad


http://gerrit.cloudera.org:8080/#/c/12550/1/docs/topics/impala_shutdown.xml@194
PS1, Line 194: threshold time limit
nit: maybe use the same term 'grace period' here to be consistent with the 
first para.



--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:47:15 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] [DOCS] Noted IMPALA-8154 as fixed in Known Issues doc

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12551 )

Change subject: [DOCS] Noted IMPALA-8154 as fixed in Known Issues doc
..


Patch Set 1:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/227/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/12551
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4ecae87938cc2ee9ce5a2bed639d3719f9bd793e
Gerrit-Change-Number: 12551
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:30:49 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Noted IMPALA-8154 as fixed in Known Issues doc

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12551 )

Change subject: [DOCS] Noted IMPALA-8154 as fixed in Known Issues doc
..


Patch Set 1: Verified+1

Build Successful

https://jenkins.impala.io/job/gerrit-docs-auto-test/227/ : Doc tests passed.


--
To view, visit http://gerrit.cloudera.org:8080/12551
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4ecae87938cc2ee9ce5a2bed639d3719f9bd793e
Gerrit-Change-Number: 12551
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Michael Ho 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:33:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7917 (Part 2): Decouple Sentry from Impala

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12542 )

Change subject: IMPALA-7917 (Part 2): Decouple Sentry from Impala
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2205/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I051d06451fb14ba2ffb3f7e0d4aae37dee55ab86
Gerrit-Change-Number: 12542
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Austin Nobis (450)
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:32:49 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Noted IMPALA-8154 as fixed in Known Issues doc

Alex Rodoni has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/12551


Change subject: [DOCS] Noted IMPALA-8154 as fixed in Known Issues doc
..

[DOCS] Noted IMPALA-8154 as fixed in Known Issues doc

Change-Id: I4ecae87938cc2ee9ce5a2bed639d3719f9bd793e
---
M docs/topics/impala_known_issues.xml
1 file changed, 3 insertions(+), 1 deletion(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/51/12551/1
--
To view, visit http://gerrit.cloudera.org:8080/12551
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4ecae87938cc2ee9ce5a2bed639d3719f9bd793e
Gerrit-Change-Number: 12551
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12550 )

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..


Patch Set 1: Verified+1

Build Successful 

https://jenkins.impala.io/job/gerrit-docs-auto-test/226/ : Doc tests passed.


--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:28:06 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8233: Do not re-download Ranger if it is already downloaded

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/12541 )

Change subject: IMPALA-8233: Do not re-download Ranger if it is already 
downloaded
..

IMPALA-8233: Do not re-download Ranger if it is already downloaded

This patch updates the bootstrap_toolchain.py to not re-download Ranger
if it is already downloaded.

Testing: Manually tested it by running the boolstrap_toolchain.py.

Change-Id: Iec3b200bda11d00bba6a250461b37c599d8d1adf
Reviewed-on: http://gerrit.cloudera.org:8080/12541
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M bin/bootstrap_toolchain.py
1 file changed, 4 insertions(+), 1 deletion(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/12541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Iec3b200bda11d00bba6a250461b37c599d8d1adf
Gerrit-Change-Number: 12541
Gerrit-PatchSet: 5
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger

[native-toolchain-CR] Enable publishing on the docker-based build

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12423 )

Change subject: Enable publishing on the docker-based build
..


Patch Set 3:

I'm going to do a trial (non-docker) build on your stack of 4 patches and 
verify + submit them if it succeeds.


--
To view, visit http://gerrit.cloudera.org:8080/12423
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I32966bf5b5ad32a3caaf2aee0430ea3a67ecb5d6
Gerrit-Change-Number: 12423
Gerrit-PatchSet: 3
Gerrit-Owner: hector.aco...@cloudera.com 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: hector.aco...@cloudera.com 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:23:57 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8233: Do not re-download Ranger if it is already downloaded

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12541 )

Change subject: IMPALA-8233: Do not re-download Ranger if it is already 
downloaded
..


Patch Set 4: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/12541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iec3b200bda11d00bba6a250461b37c599d8d1adf
Gerrit-Change-Number: 12541
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:25:30 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12550 )

Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..


Patch Set 1:

Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/226/

Testing docs change - this change appears to modify docs/ and no code. This is 
experimental - please report any issues to tarmstr...@cloudera.com or on this 
JIRA: IMPALA-7317


--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:17:24 +
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Updated the description of the port number parameter in SHUTDOWN

Alex Rodoni has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/12550


Change subject: [DOCS] Updated the description of the port_number parameter in 
SHUTDOWN
..

[DOCS] Updated the description of the port_number parameter in SHUTDOWN

Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
---
M docs/topics/impala_shutdown.xml
1 file changed, 21 insertions(+), 9 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/12550/1
--
To view, visit http://gerrit.cloudera.org:8080/12550
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8b1e3e228059860b764885395c88cecd3033ff41
Gerrit-Change-Number: 12550
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni

[Impala-ASF-CR] IMPALA-8168: Disable Sentry HDFS sync on S3

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has submitted this change and it was merged. ( 
http://gerrit.cloudera.org:8080/12482 )

Change subject: IMPALA-8168: Disable Sentry HDFS sync on S3
..

IMPALA-8168: Disable Sentry HDFS sync on S3

This patch updates the build script to disable Sentry HDFS sync plugin
on S3 since Sentry HDFS sync is not supported and it may cause issues,
such as making the tests flaky in a non-supported environment.

This patch also updates the CDH_BUILD_NUMBER to 909265 to test the new
Sentry changes for batch insert of HMS paths in full snapshot. It also
bumps Kudu version to 1.10.0.

Testing:
- Ran core tests on both HDFS and S3.

Change-Id: Ifa12f163a32bb6e597323149fbe565a6f66f5e69
Reviewed-on: http://gerrit.cloudera.org:8080/12482
Reviewed-by: Impala Public Jenkins 
Tested-by: Impala Public Jenkins 
---
M bin/impala-config.sh
M fe/src/test/resources/sentry-site_no_oo.xml.template
M fe/src/test/resources/sentry-site_oo.xml.template
M fe/src/test/resources/sentry-site_oo_nogrant.xml.template
4 files changed, 12 insertions(+), 6 deletions(-)

Approvals:
  Impala Public Jenkins: Looks good to me, approved; Verified

--
To view, visit http://gerrit.cloudera.org:8080/12482
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ifa12f163a32bb6e597323149fbe565a6f66f5e69
Gerrit-Change-Number: 12482
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Reviewer: Philip Zeyliger

[Impala-ASF-CR] IMPALA-8168: Disable Sentry HDFS sync on S3

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12482 )

Change subject: IMPALA-8168: Disable Sentry HDFS sync on S3
..


Patch Set 3: Verified+1


--
To view, visit http://gerrit.cloudera.org:8080/12482
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ifa12f163a32bb6e597323149fbe565a6f66f5e69
Gerrit-Change-Number: 12482
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 21 Feb 2019 21:07:43 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7917 (Part 2): Decouple Sentry from Impala

Fredy Wijaya has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/12542 )

Change subject: IMPALA-7917 (Part 2): Decouple Sentry from Impala
..

IMPALA-7917 (Part 2): Decouple Sentry from Impala

This patch adds a new flag --authorization_provider and creates some
factories to make it easier to switch to a different authorization
provider. For backward compatability the default authorization provider
is still set to Sentry.

This patch has no functional change.

Testing:
- Ran all FE tests
- Ran all authorization E2E tests

Change-Id: I051d06451fb14ba2ffb3f7e0d4aae37dee55ab86
---
M be/src/service/frontend.cc
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/analysis/AdminFnStmt.java
M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/CollectionTableRef.java
M fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateFunctionStmtBase.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/DropFunctionStmt.java
M fe/src/main/java/org/apache/impala/analysis/HdfsUri.java
M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java
M fe/src/main/java/org/apache/impala/analysis/ResetMetadataStmt.java
A fe/src/main/java/org/apache/impala/authorization/AuthorizableColumn.java
A fe/src/main/java/org/apache/impala/authorization/AuthorizableDb.java
A fe/src/main/java/org/apache/impala/authorization/AuthorizableFactory.java
A fe/src/main/java/org/apache/impala/authorization/AuthorizableFn.java
A fe/src/main/java/org/apache/impala/authorization/AuthorizableServer.java
A fe/src/main/java/org/apache/impala/authorization/AuthorizableTable.java
A fe/src/main/java/org/apache/impala/authorization/AuthorizableUri.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationChecker.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationPolicy.java
M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
M fe/src/main/java/org/apache/impala/authorization/PrivilegeRequestBuilder.java
M 
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationChecker.java
M 
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeAuthStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java
M fe/src/test/java/org/apache/impala/common/FrontendFixture.java
M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
M fe/src/test/java/org/apache/impala/common/QueryFixture.java
M fe/src/test/java/org/apache/impala/testutil/CatalogServiceTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/ImpaladTestCatalog.java
M fe/src/test/java/org/apache/impala/testutil/PlannerTestCaseLoader.java
43 files changed, 806 insertions(+), 229 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/12542/3
--
To view, visit http://gerrit.cloudera.org:8080/12542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I051d06451fb14ba2ffb3f7e0d4aae37dee55ab86
Gerrit-Change-Number: 12542
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Austin Nobis (450)
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers

[Impala-ASF-CR] IMPALA-7917 (Part 2): Decouple Sentry from Impala

Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12542 )

Change subject: IMPALA-7917 (Part 2): Decouple Sentry from Impala
..


Patch Set 3:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2516
PS2, Line 2516:* the specified privilege.
> I think you can shorten this by moving the if statement inside the lambda
Done


http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2694
PS2, Line 2694: builder.grantOption();
> nit: The style is inconsistent with the rest of the file
Done


http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
File fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java:

http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java@121
PS2, Line 121: analyzer.registerPrivReq(builder ->
> nit: There are a few different lambda styles in different files. Wondering
I like the last one. Will update.


http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/authorization/AuthorizableServer.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableServer.java:

http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/authorization/AuthorizableServer.java@27
PS2, Line 27: serverName_ = serverName == null ? "server" : serverName;
> Any reason the `null` is handled here instead of being checked in the `Auth
Yeah that's how it works AFAIK. Basically if the name is not specified, it will 
use the default name. For example:

"grant select on server to role foo"
"grant select on server my_server to role foo"



--
To view, visit http://gerrit.cloudera.org:8080/12542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I051d06451fb14ba2ffb3f7e0d4aae37dee55ab86
Gerrit-Change-Number: 12542
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Austin Nobis (450)
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 20:59:33 +
Gerrit-HasComments: Yes

[native-toolchain-CR] Initial support for building the toolchain in docker

2019-02-21 Thread Laszlo Gaal (Code Review)

Laszlo Gaal has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12285 )

Change subject: Initial support for building the toolchain in docker
..


Patch Set 7: Code-Review+2

Tim is of course right about the Java versions. I guess I stared at config 
diffs a bit too much...


--
To view, visit http://gerrit.cloudera.org:8080/12285
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If42c9bc06a3d303642eb37dea784b61e2a1f5cc6
Gerrit-Change-Number: 12285
Gerrit-PatchSet: 7
Gerrit-Owner: hector.aco...@cloudera.com 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: hector.aco...@cloudera.com 
Gerrit-Comment-Date: Thu, 21 Feb 2019 20:40:53 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7975 : Improve supportability of the automatic invalidate feature

2019-02-21 Thread Vihang Karajgaonkar (Code Review)

Vihang Karajgaonkar has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/12549


Change subject: IMPALA-7975 : Improve supportability of the automatic 
invalidate feature
..

IMPALA-7975 : Improve supportability of the automatic invalidate feature

This patch adds supportability metrics to make it easier to find the
state of Metastore events processor. It adds the following metrics

1. Total number of events received so far
This is not super useful in itself but is helpful to see if there are
events being received or not. A more usable metric which publishes the
rate of such events received is also published. See details for it
below.

2. 1min, 5min and 15min rate of events received
Exponential moving average of number of events received in last 1min,
5min and 15min. This can be used to determine if there are spikes in
event processor activity during certain hours of the day

3. Total number of events skipped so far
Events can be skipped based on certain flags are table and database
level. This metric keeps a count of total number of events skipped so
far. Useful to make decisions like - If most of the events are being
skipped, may be just turn off the event processing. If most of the
events are not skipped, do we need to add flags on certain databases?

4. Status of events processor
Easy way to determine what is the current state of metastore events
processor. Possible states could be STOPPED, ACTIVE, ERROR,
NEEDS_INVALIDATE

5. Events fetch and process durations in seconds
Average duration to fetch a batch of events and process it.

Additionally, it also registers a url handler specific for events at
/events. The /events page provides a more detailed view of the metrics
of the event processor which include (min, max, mean, median) of the
durations and rate metrics for all the counters. In the future this page
can be enhanced to show the top-n tables with most number of invalidates
due to the events, etc.

Patch adds few tests to make sure the value of these event metrics are
as expected.

Change-Id: I23cb3aa866879eca03c64ab881796eaa9caa0337
---
M be/src/catalog/catalog-server.cc
M be/src/catalog/catalog-server.h
M be/src/catalog/catalog.cc
M be/src/catalog/catalog.h
M be/src/catalog/catalogd-main.cc
M be/src/util/CMakeLists.txt
A be/src/util/event-metrics.cc
A be/src/util/event-metrics.h
M common/thrift/JniCatalog.thrift
M common/thrift/metrics.json
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
M fe/src/main/java/org/apache/impala/catalog/events/ExternalEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/catalog/events/MetastoreEventsProcessor.java
M fe/src/main/java/org/apache/impala/catalog/events/NoOpEventProcessor.java
M fe/src/main/java/org/apache/impala/common/Metrics.java
M fe/src/main/java/org/apache/impala/service/JniCatalog.java
M 
fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java
A www/events.tmpl
19 files changed, 621 insertions(+), 66 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/12549/2
--
To view, visit http://gerrit.cloudera.org:8080/12549
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I23cb3aa866879eca03c64ab881796eaa9caa0337
Gerrit-Change-Number: 12549
Gerrit-PatchSet: 2
Gerrit-Owner: Vihang Karajgaonkar

[Impala-ASF-CR] IMPALA-7917 (Part 2): Decouple Sentry from Impala

2019-02-21 Thread Anonymous Coward (Code Review)

Anonymous Coward (450) has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12542 )

Change subject: IMPALA-7917 (Part 2): Decouple Sentry from Impala
..


Patch Set 2:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java
File fe/src/main/java/org/apache/impala/analysis/Analyzer.java:

http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2516
PS2, Line 2516: if (privilege == Privilege.ANY) {
I think you can shorten this by moving the if statement inside the lambda

registerPrivReq(builder -> {
if (requireGrantOption) {
builder.grantOption();
}
return (privelege == Privelege.ANY) ? 
build.any().onAnyColumn(dbName).build() :
builder.allOf(privilege).onDb(dbName).build();
}


http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/analysis/Analyzer.java@2694
PS2, Line 2694:   .allOf(priv);
nit: The style is inconsistent with the rest of the file


http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java
File fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java:

http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/analysis/CopyTestCaseStmt.java@121
PS2, Line 121: analyzer.registerPrivReq(builder ->
nit: There are a few different lambda styles in different files. Wondering what 
the goto should be.

Personally, I prefer the last one even if it is possible for everything to be 
on a single line.

registerPrivReq(builder -> builder
.allOf(priv).onAnyColumn(fqTableName.getDb(), 
fqTableName.getTbl()).build());

vs

analyzer.registerPrivReq(builder ->
builder.onFunction(fn_.dbName(), fn_.signatureString())
.allOf(Privilege.CREATE)
.build());

vs

analyzer.registerPrivReq(builder -> builder
.onTable(table.getDb().getName(), table.getName())
.allOf(Privilege.VIEW_METADATA)
.build());


http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/authorization/AuthorizableServer.java
File fe/src/main/java/org/apache/impala/authorization/AuthorizableServer.java:

http://gerrit.cloudera.org:8080/#/c/12542/2/fe/src/main/java/org/apache/impala/authorization/AuthorizableServer.java@27
PS2, Line 27: serverName_ = serverName == null ? "server" : serverName;
Any reason the `null` is handled here instead of being checked in the 
`AuthorizableFactory` like `AuthorizableDatabase`, `AuthorizableTable`, etc.



--
To view, visit http://gerrit.cloudera.org:8080/12542
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I051d06451fb14ba2ffb3f7e0d4aae37dee55ab86
Gerrit-Change-Number: 12542
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Anonymous Coward (450)
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 20:15:07 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7560: Improve selectivity estimate for !=

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12427 )

Change subject: IMPALA-7560: Improve selectivity estimate for !=
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2203/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f6013c9ef95a89d55d8b25f0b5433c81582a62f
Gerrit-Change-Number: 12427
Gerrit-PatchSet: 5
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 19:53:52 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7560: Improve selectivity estimate for !=

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12427 )

Change subject: IMPALA-7560: Improve selectivity estimate for !=
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2201/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f6013c9ef95a89d55d8b25f0b5433c81582a62f
Gerrit-Change-Number: 12427
Gerrit-PatchSet: 4
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 19:34:10 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2202/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 5
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 21 Feb 2019 19:34:05 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 4:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2200/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 4
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 21 Feb 2019 19:33:21 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7560: Improve selectivity estimate for !=

Paul Rogers has posted comments on this change. (
http://gerrit.cloudera.org:8080/12427 )

Change subject: IMPALA-7560: Improve selectivity estimate for !=
..

Patch Set 5:

(2 comments)

Rebased on master, again (twice). Addressed review comments.

Am re-running pre-review tests after all the recent rebases. Will send a note
when complete.

http://gerrit.cloudera.org:8080/#/c/12427/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:

http://gerrit.cloudera.org:8080/#/c/12427/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@216
PS3, Line 216: // Note: the following case statement is skeletal.
> nit: switch case indentation seems off.
This is Java-style case indents: case at same level as switch. Changed to C++
style used elsewhere in the file (case indented one level from switch.)

http://gerrit.cloudera.org:8080/#/c/12427/3/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@227
PS3, Line 227: }
> nit: I think you could refactor the whole thing like before to make the int
Could, and that is how I started. But, this shape looks ahead to the other
operators, some of which produce the same estimate regardless of the two
conditions. Also, there will be the non-single col case in some places. For
example, for <, <=, >, >=, the answer might always be 1/3 regardless of the
other factors. For =, we might guess 0.1 without an NDV.

I have deliberately left these as later patches so that we can evaluate each
operator and each condition one at a time. Too complex (i.e. too hard to
review) if I lump them all together.

Added a comment to clarify this intent.

--
To view, visit http://gerrit.cloudera.org:8080/12427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f6013c9ef95a89d55d8b25f0b5433c81582a62f
Gerrit-Change-Number: 12427
Gerrit-PatchSet: 5
Gerrit-Owner: Paul Rogers
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Paul Rogers
Gerrit-Comment-Date: Thu, 21 Feb 2019 19:25:51 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7560: Improve selectivity estimate for !=

Hello Bharath Vissapragada, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12427

to look at the new patch set (#5).

Change subject: IMPALA-7560: Improve selectivity estimate for !=
..

IMPALA-7560: Improve selectivity estimate for !=

Impala has historically used a generic selectivity of 0.1 for all
non-equality predicates. However, this can underestimate cardinality
in some cases. The correct value is

sel(c != x) = 1 - sel(c = x)

if c is a column, x is a constant and ndv(c) is known.

Adds the above for the != case and for IS DISTINCT FROM case. There are
many related issues that are left as separate patches.

Tests:
* Used the newly-added expression cardinality tests to highlight the
  change (some formerly broken tests now pass).
* Used the newly-added cardinality tests in PlannerTest to highlight the
  fix.
* Fixed a merge artifact by having the PlannerTest cardinality tests
  actually verify cardinality.

Change-Id: I8f6013c9ef95a89d55d8b25f0b5433c81582a62f
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test
13 files changed, 149 insertions(+), 122 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/12427/5
--
To view, visit http://gerrit.cloudera.org:8080/12427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8f6013c9ef95a89d55d8b25f0b5433c81582a62f
Gerrit-Change-Number: 12427
Gerrit-PatchSet: 5
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers

[Impala-ASF-CR] IMPALA-7560: Improve selectivity estimate for !=

Hello Bharath Vissapragada, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12427

to look at the new patch set (#4).

Change subject: IMPALA-7560: Improve selectivity estimate for !=
..

IMPALA-7560: Improve selectivity estimate for !=

Impala has historically used a generic selectivity of 0.1 for all
non-equality predicates. However, this can underestimate cardinality
in some cases. The correct value is

sel(c != x) = 1 - sel(c = x)

if c is a column, x is a constant and ndv(c) is known.

Adds the above for the != case and for IS DISTINCT FROM case. There are
many related issues that are left as separate patches.

Tests:
* Used the newly-added expression cardinality tests to highlight the
  change (some formerly broken tests now pass).
* Used the newly-added cardinality tests in PlannerTest to highlight the
  fix.
* Fixed a merge artifact by having the PlannerTest cardinality tests
  actually verify cardinality.

Change-Id: I8f6013c9ef95a89d55d8b25f0b5433c81582a62f
---
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/test/java/org/apache/impala/analysis/ExprCardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/card-scan.test
M testdata/workloads/functional-planner/queries/PlannerTest/hbase.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/inline-view-limit.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/mt-dop-validation.test
M 
testdata/workloads/functional-planner/queries/PlannerTest/predicate-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-all.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-nested.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-views.test
13 files changed, 158 insertions(+), 131 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/12427/4
--
To view, visit http://gerrit.cloudera.org:8080/12427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8f6013c9ef95a89d55d8b25f0b5433c81582a62f
Gerrit-Change-Number: 12427
Gerrit-PatchSet: 4
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers

[Impala-ASF-CR] IMPALA-6900: Fix the min version invariant for invalidate metadata

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12547 )

Change subject: IMPALA-6900: Fix the min version invariant for invalidate 
metadata
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12547
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I22ace3f5917b6167af63c3dfd2620e69ce8ec1cb
Gerrit-Change-Number: 12547
Gerrit-PatchSet: 4
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Thu, 21 Feb 2019 19:11:47 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12543/4/common/thrift/Types.thrift
File common/thrift/Types.thrift:

http://gerrit.cloudera.org:8080/#/c/12543/4/common/thrift/Types.thrift@107
PS4, Line 107:   TESTCASE = 4
> This was added recently for IMPALA-5872. Should we also move it to the end?
thanks for catching it, done.



--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 4
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 21 Feb 2019 19:02:31 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

Joe McDonnell has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 4:

(1 comment)

Thanks for working on this!

http://gerrit.cloudera.org:8080/#/c/12543/4/common/thrift/Types.thrift
File common/thrift/Types.thrift:

http://gerrit.cloudera.org:8080/#/c/12543/4/common/thrift/Types.thrift@107
PS4, Line 107:   TESTCASE = 4
This was added recently for IMPALA-5872. Should we also move it to the end?



--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 4
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 21 Feb 2019 18:59:49 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7560: Improve selectivity estimate for !=

2019-02-21 Thread Bharath Vissapragada (Code Review)

Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12427 )

Change subject: IMPALA-7560: Improve selectivity estimate for !=
..


Patch Set 4:

Gerrit still complaining about the merge conflict.


--
To view, visit http://gerrit.cloudera.org:8080/12427
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8f6013c9ef95a89d55d8b25f0b5433c81582a62f
Gerrit-Change-Number: 12427
Gerrit-PatchSet: 4
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 18:56:02 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6900: Fix the min version invariant for invalidate metadata

Paul Rogers has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12547 )

Change subject: IMPALA-6900: Fix the min version invariant for invalidate 
metadata
..


Patch Set 3: Code-Review+2

LGTM


--
To view, visit http://gerrit.cloudera.org:8080/12547
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I22ace3f5917b6167af63c3dfd2620e69ce8ec1cb
Gerrit-Change-Number: 12547
Gerrit-PatchSet: 3
Gerrit-Owner: Bharath Vissapragada 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Todd Lipcon 
Gerrit-Comment-Date: Thu, 21 Feb 2019 18:56:18 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

Lars Volker has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12543 )

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/12543/3/common/thrift/hive-1-api/TCLIService.thrift
File common/thrift/hive-1-api/TCLIService.thrift:

PS3:
> This is a copy of the Hive thrift so it might cause confusion if we diverge
Good point, I reverted it.



--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 4
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Comment-Date: Thu, 21 Feb 2019 18:50:22 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

Hello Philip Zeyliger, Tim Armstrong, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/12543

to look at the new patch set (#4).

Change subject: IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add 
warning
..

IMPALA-8234: Fix ordering of Thrift enum, fix enum values, add warning

IMPALA-7694 added a field in the middle of the Metrics.TUnit enum, which
broke backwards compatibility with profiles that had been written by
older versions of Impala. This change fixes the ordering by moving the
field to the end of the enum.

Additionally, it adds a warning to the top of all Thrift files that are
part of the binary profile format, and an note of caution to the main
definition in RuntimeProfile.thrift.

This change also fixes the order of all enums in our Thrift files to
make errors like this less likely in the future.

Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
---
M common/thrift/BackendGflags.thrift
M common/thrift/CatalogObjects.thrift
M common/thrift/DataSinks.thrift
M common/thrift/ExecStats.thrift
M common/thrift/Exprs.thrift
M common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/ImpalaInternalService.thrift
M common/thrift/ImpalaService.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/LineageGraph.thrift
M common/thrift/Logging.thrift
M common/thrift/Metrics.thrift
M common/thrift/Partitions.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/RuntimeProfile.thrift
M common/thrift/StatestoreService.thrift
M common/thrift/Status.thrift
M common/thrift/Types.thrift
M common/thrift/beeswax.thrift
20 files changed, 557 insertions(+), 519 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/12543/4
--
To view, visit http://gerrit.cloudera.org:8080/12543
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If215f16a636008757ceb439edbd6900a1be88c59
Gerrit-Change-Number: 12543
Gerrit-PatchSet: 4
Gerrit-Owner: Lars Volker 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong

[Impala-ASF-CR] IMPALA-8181: Abbreviate row counts in EXPLAIN

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12438 )

Change subject: IMPALA-8181: Abbreviate row counts in EXPLAIN
..


Patch Set 9: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12438
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c
Gerrit-Change-Number: 12438
Gerrit-PatchSet: 9
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 18:48:35 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8181: Abbreviate row counts in EXPLAIN

2019-02-21 Thread Bharath Vissapragada (Code Review)

Bharath Vissapragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12438 )

Change subject: IMPALA-8181: Abbreviate row counts in EXPLAIN
..


Patch Set 8: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12438
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c
Gerrit-Change-Number: 12438
Gerrit-PatchSet: 8
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 18:48:26 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8181: Abbreviate row counts in EXPLAIN

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12438 )

Change subject: IMPALA-8181: Abbreviate row counts in EXPLAIN
..


Patch Set 9:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3814/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/12438
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I08faaa9ad7b5ed42dcd7a15a333e8734bb45f10c
Gerrit-Change-Number: 12438
Gerrit-PatchSet: 9
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Comment-Date: Thu, 21 Feb 2019 18:48:36 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-7781: Clean up expr rewriter predicates

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/11875 )

Change subject: IMPALA-7781: Clean up expr rewriter predicates
..


Patch Set 3:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/11875/3/fe/src/test/java/org/apache/impala/testutil/TestUtils.java
File fe/src/test/java/org/apache/impala/testutil/TestUtils.java:

http://gerrit.cloudera.org:8080/#/c/11875/3/fe/src/test/java/org/apache/impala/testutil/TestUtils.java@293
PS3, Line 293: return createQueryContext(Catalog.DEFAULT_DB, 
System.getProperty("user.name"), options);
line too long (92 > 90)


http://gerrit.cloudera.org:8080/#/c/11875/3/fe/src/test/java/org/apache/impala/testutil/TestUtils.java@301
PS3, Line 301:   public static TQueryCtx createQueryContext(String defaultDb, 
String user, TQueryOptions options) {
line too long (100 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/11875
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I92d117145e427fcc5c71299096cf0e2d964ab63c
Gerrit-Change-Number: 11875
Gerrit-PatchSet: 3
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Thu, 21 Feb 2019 18:48:12 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-7781: Clean up expr rewriter predicates

Paul Rogers has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/11875


Change subject: IMPALA-7781: Clean up expr rewriter predicates
..

IMPALA-7781: Clean up expr rewriter predicates

The expression rewriter often checks if an expression is null or a
literal, but does so using ad-hoc "instance of" or similar checks.
Yet, the expression nodes provide predicates that do the same job.
This change uses the defined predicates in place of the ad-hoc tests.

Add new predicates for the most common cases: literal, null literal,
null value, non-null literal, and literal-value. Converted several Expr
class methods to predicates.

The predicates handle cases that the ad-hoc tests do not. For example,
when the constant folding rule produces a NULL result, it casts that
null to some type: CAST(NULL AS INT), say. Later, other rules want to
check if that result is null. A simple check for a null literal won't
work, we need the IS_NULL_VALUE predicate which checks for the CAST
case.

This change adds predicate and revises the rewrite rules to use the
new predicates.

Also adds a number of minor Java cleanups found while doing the fix.

One of which is the toSql() for numeric values: tests showed that
the form of zero printed depended on the precision and scale, with one
value printing as 0E-38. Modified the code to emit "0" for all zero
values (since, unlike Java, SQL attaches no meaning to 0.0 as oppposed
to just 0.)

Detailed testing revealed missed simplification opportunites, especially
in CASE. Added the missing logic. For CASE, this meant a wholesale
refactoring of the code in order to better deal with nuances such as

CASE null WHEN ... END --> NULL
CASE ... ELSE NULL END --> CASE ... END
CASE ... WHEN ... THEN NULL END CASE ... END

The constant folding rule always wrapped the result in a cast,
even if the result was of the correct type. Skipped the unnecessary
cast to avoid repeated passes through this rule.

Testing:

* Refactored conditional tests into a new class, broken down
by type expression type for easier debugging.
* Created a base class for the existing and new expression test
classes.
* Created a test fixture to manage analysis tests to make it easier
to grab and inspect various aspects of the plan, as well as to control
query and other options.
* Added new tests that failed without these changes, but which pass
with them.
* Some of the plans in PlannerTest changed: verified that the new
plans differ only in the added simplifications.

Change-Id: I92d117145e427fcc5c71299096cf0e2d964ab63c
---
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BetweenPredicate.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
M fe/src/main/java/org/apache/impala/analysis/CaseExpr.java
M fe/src/main/java/org/apache/impala/analysis/ColumnDef.java
M fe/src/main/java/org/apache/impala/analysis/CompoundPredicate.java
M fe/src/main/java/org/apache/impala/analysis/ExistsPredicate.java
M fe/src/main/java/org/apache/impala/analysis/Expr.java
M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java
M fe/src/main/java/org/apache/impala/analysis/LikePredicate.java
M fe/src/main/java/org/apache/impala/analysis/NumericLiteral.java
M fe/src/main/java/org/apache/impala/analysis/RangePartition.java
M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
M fe/src/main/java/org/apache/impala/common/TreeNode.java
M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/KuduScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/rewrite/BetweenToCompoundRule.java
M fe/src/main/java/org/apache/impala/rewrite/EqualityDisjunctsToInRule.java
M fe/src/main/java/org/apache/impala/rewrite/FoldConstantsRule.java
M fe/src/main/java/org/apache/impala/rewrite/NormalizeBinaryPredicatesRule.java
M fe/src/main/java/org/apache/impala/rewrite/NormalizeCountStarRule.java
M fe/src/main/java/org/apache/impala/rewrite/NormalizeExprsRule.java
M fe/src/main/java/org/apache/impala/rewrite/RemoveRedundantStringCast.java
M fe/src/main/java/org/apache/impala/rewrite/SimplifyConditionalsRule.java
M fe/src/main/java/org/apache/impala/rewrite/SimplifyDistinctFromRule.java
A fe/src/test/java/org/apache/impala/analysis/AnalysisFixture.java
A fe/src/test/java/org/apache/impala/analysis/BaseRewriteRulesTest.java
M fe/src/test/java/org/apache/impala/analysis/ExprRewriteRulesTest.java
A fe/src/test/java/org/apache/impala/analysis/ExtendedRewriteTest.java
A fe/src/test/java/org/apache/impala/analysis/SimplifyConditionalsRuleTest.java
M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M

[Impala-ASF-CR] IMPALA-7781: Clean up expr rewriter predicates

Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/11875 )

Change subject: IMPALA-7781: Clean up expr rewriter predicates
..


Abandoned

Draft
--
To view, visit http://gerrit.cloudera.org:8080/11875
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I92d117145e427fcc5c71299096cf0e2d964ab63c
Gerrit-Change-Number: 11875
Gerrit-PatchSet: 3
Gerrit-Owner: Paul Rogers

[Impala-ASF-CR] IMPALA-7968, Part 1: JSON serialization framework

Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/12079 )

Change subject: IMPALA-7968, Part 1: JSON serialization framework
..


Abandoned

Was meant to help with testing, but moving away from that effort.
--
To view, visit http://gerrit.cloudera.org:8080/12079
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ie3101c2708bf6cf4bec61af83a5db9b6d70ddd9c
Gerrit-Change-Number: 12079
Gerrit-PatchSet: 8
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers 
Gerrit-Reviewer: Philip Zeyliger

[Impala-ASF-CR] IMPALA-7866: Predicates, helpers for implicit casts, slot refs

Paul Rogers has abandoned this change. ( http://gerrit.cloudera.org:8080/11953 )

Change subject: IMPALA-7866: Predicates, helpers for implicit casts, slot refs
..


Abandoned

Lower priority, not worth keeping this open for many months.
--
To view, visit http://gerrit.cloudera.org:8080/11953
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ieaa0aee1b9015e0aed521f2038bf44513d7f8613
Gerrit-Change-Number: 11953
Gerrit-PatchSet: 3
Gerrit-Owner: Paul Rogers 
Gerrit-Reviewer: Bharath Vissapragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Paul Rogers

[native-toolchain-CR] Initial support for building the toolchain in docker

Tim Armstrong has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12285 )

Change subject: Initial support for building the toolchain in docker
..


Patch Set 7: Code-Review+1


--
To view, visit http://gerrit.cloudera.org:8080/12285
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: If42c9bc06a3d303642eb37dea784b61e2a1f5cc6
Gerrit-Change-Number: 12285
Gerrit-PatchSet: 7
Gerrit-Owner: hector.aco...@cloudera.com 
Gerrit-Reviewer: Lars Volker 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-Reviewer: hector.aco...@cloudera.com 
Gerrit-Comment-Date: Thu, 21 Feb 2019 18:33:02 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8233: Do not re-download Ranger if it is already downloaded

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12541 )

Change subject: IMPALA-8233: Do not re-download Ranger if it is already 
downloaded
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/2198/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/12541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iec3b200bda11d00bba6a250461b37c599d8d1adf
Gerrit-Change-Number: 12541
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 21 Feb 2019 17:50:28 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8233: Do not re-download Ranger if it is already downloaded

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12541 )

Change subject: IMPALA-8233: Do not re-download Ranger if it is already 
downloaded
..


Patch Set 4:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3813/ 
DRY_RUN=false


--
To view, visit http://gerrit.cloudera.org:8080/12541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iec3b200bda11d00bba6a250461b37c599d8d1adf
Gerrit-Change-Number: 12541
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 21 Feb 2019 17:08:05 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8233: Do not re-download Ranger if it is already downloaded

2019-02-21 Thread Impala Public Jenkins (Code Review)

Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12541 )

Change subject: IMPALA-8233: Do not re-download Ranger if it is already 
downloaded
..


Patch Set 4: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/12541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iec3b200bda11d00bba6a250461b37c599d8d1adf
Gerrit-Change-Number: 12541
Gerrit-PatchSet: 4
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 21 Feb 2019 17:08:04 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-8233: Do not re-download Ranger if it is already downloaded

Fredy Wijaya has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/12541 )

Change subject: IMPALA-8233: Do not re-download Ranger if it is already 
downloaded
..


Patch Set 3: Code-Review+2

(2 comments)

Carry Phil's +2.

http://gerrit.cloudera.org:8080/#/c/12541/1/bin/bootstrap_toolchain.py
File bin/bootstrap_toolchain.py:

http://gerrit.cloudera.org:8080/#/c/12541/1/bin/bootstrap_toolchain.py@423
PS1, Line 423:   if not version:
> I think a comment here would be good, I was confused by the 'isdir' call, d
I thought it was already pretty clear from the variable name "pkg_directory" :) 
os.path.isdir will return False if a file already exists. I'll add a comment, 
though. Done.


http://gerrit.cloudera.org:8080/#/c/12541/1/bin/bootstrap_toolchain.py@425
PS1, Line 425: .format(env_var_version))
 :   pkg_directory = "{0}/ranger-{1}-admin".format(toolchain_root, 
version)
 :   if os.path.isdir(pkg_directory): return
> While you're here, do you want to move this to between line 421 and 422, so
Done



--
To view, visit http://gerrit.cloudera.org:8080/12541
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iec3b200bda11d00bba6a250461b37c599d8d1adf
Gerrit-Change-Number: 12541
Gerrit-PatchSet: 3
Gerrit-Owner: Fredy Wijaya 
Gerrit-Reviewer: Andrew Sherman 
Gerrit-Reviewer: Fredy Wijaya 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Thu, 21 Feb 2019 17:07:05 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-8233: Do not re-download Ranger if it is already downloaded