[Impala-ASF-CR] Remove Llama support.

[Henry Robinson (Code Review)]

Henry Robinson has posted comments on this change.

Change subject: Remove Llama support.
..


Patch Set 1:

Passed a core test run.

-- 
To view, visit http://gerrit.cloudera.org:8080/4445
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Icfb14209e31f6608bb7b8a33789e00411a6447ef
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson 
Gerrit-Reviewer: Henry Robinson 
Gerrit-HasComments: No

[Impala-ASF-CR] Remove Llama support.

[Henry Robinson (Code Review)]

Henry Robinson has uploaded a new change for review.

  http://gerrit.cloudera.org:8080/4445

Change subject: Remove Llama support.
..

Remove Llama support.

Alas, poor Llama! I knew him, Impala: a system
of infinite jest, of most excellent fancy: we hath
borne him on our back a thousand times; and now, how
abhorred in my imagination it is!

Done:

* Removed QueryResourceMgr, ResourceBroker, CGroupsMgr
* Removed untested 'offline' mode and NM failure detection from
  ImpalaServer
* Removed all Llama-related Thrift files
* Removed RM-related arguments to MemTracker constructors
* Deprecated all RM-related flags, printing a warning if enable_rm is
  set
* Removed expansion logic from MemTracker
* Removed VCore logic from QuerySchedule
* Removed all reservation-related logic from Scheduler
* Removed RM metric descriptions
* Various misc. small class changes

Not done:

* Remove RM flags (--enable_rm etc.)
* Remove RM query options
* Changes to RequestPoolService.

Change-Id: Icfb14209e31f6608bb7b8a33789e00411a6447ef
---
M be/CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/bufferpool/reservation-tracker-test.cc
M be/src/exec/blocking-join-node.cc
M be/src/exec/data-sink.cc
M be/src/exec/exec-node.cc
M be/src/exec/hash-join-node.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/kudu-scan-node-test.cc
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-table-sink-test.cc
M be/src/exprs/expr-test.cc
D be/src/resourcebroker/CMakeLists.txt
D be/src/resourcebroker/resource-broker.cc
D be/src/resourcebroker/resource-broker.h
M be/src/runtime/buffered-block-mgr-test.cc
M be/src/runtime/buffered-block-mgr.cc
M be/src/runtime/collection-value-builder-test.cc
M be/src/runtime/coordinator.cc
M be/src/runtime/data-stream-recvr.cc
M be/src/runtime/data-stream-test.cc
M be/src/runtime/disk-io-mgr-test.cc
M be/src/runtime/disk-io-mgr.cc
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/mem-pool-test.cc
M be/src/runtime/mem-tracker-test.cc
M be/src/runtime/mem-tracker.cc
M be/src/runtime/mem-tracker.h
M be/src/runtime/plan-fragment-executor.cc
M be/src/runtime/runtime-filter-bank.cc
M be/src/runtime/runtime-state.cc
M be/src/runtime/runtime-state.h
M be/src/runtime/test-env.cc
M be/src/scheduling/CMakeLists.txt
D be/src/scheduling/query-resource-mgr.cc
D be/src/scheduling/query-resource-mgr.h
M be/src/scheduling/query-schedule.cc
M be/src/scheduling/query-schedule.h
M be/src/scheduling/request-pool-service.cc
M be/src/scheduling/scheduler.h
M be/src/scheduling/simple-scheduler-test.cc
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/service/impalad-main.cc
M be/src/service/query-exec-state.cc
M be/src/util/CMakeLists.txt
D be/src/util/cgroups-mgr.cc
D be/src/util/cgroups-mgr.h
M be/src/util/debug-util.h
D be/src/util/llama-util.cc
D be/src/util/llama-util.h
M be/src/util/thread-pool.h
M be/src/util/thread.cc
M be/src/util/thread.h
M be/src/util/uid-util.h
M bin/bootstrap_toolchain.py
M bin/generate_minidump_collection_testdata.py
M bin/start-impala-cluster.py
M common/thrift/CMakeLists.txt
M common/thrift/ImpalaInternalService.thrift
D common/thrift/Llama.thrift
D common/thrift/ResourceBrokerService.thrift
M common/thrift/metrics.json
M testdata/cluster/admin
M testdata/cluster/node_templates/cdh5/etc/hadoop/conf/yarn-site.xml.tmpl
D testdata/cluster/node_templates/cdh5/etc/init.d/llama-application
D 
testdata/cluster/node_templates/cdh5/etc/llama/conf/llama-log4j.properties.tmpl
D testdata/cluster/node_templates/cdh5/etc/llama/conf/llama-site.xml.tmpl
M testdata/cluster/node_templates/common/etc/hadoop/conf/core-site.xml.tmpl
73 files changed, 117 insertions(+), 4,296 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/45/4445/1
-- 
To view, visit http://gerrit.cloudera.org:8080/4445
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newchange
Gerrit-Change-Id: Icfb14209e31f6608bb7b8a33789e00411a6447ef
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Henry Robinson 

[Impala-ASF-CR] IMPALA-1654: General partition exprs in DDL operations.

[Alex Behm (Code Review)]

Alex Behm has posted comments on this change.

Change subject: IMPALA-1654: General partition exprs in DDL operations.
..


Patch Set 14:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/3942/14/fe/src/main/java/com/cloudera/impala/analysis/ComputeStatsStmt.java
File fe/src/main/java/com/cloudera/impala/analysis/ComputeStatsStmt.java:

Line 251: table_ = analyzer.getTable(tableName_, Privilege.ALTER);
This is an example that can be cleaned similar to what I did in 
AlterTableStmt.java.

The current code has a strange pattern of first doing analyzer.getTable() and 
then maybe analyzing a TableRef. 
This could be problematic because there is a slight chance these two return 
different tables. It also seems like duplicate work/code.

I think we should just analyze a TableRef at the beginning, just like we now do 
in AnalyzeTableStmt.

Amos, can you please go through the other statements changed in this patch and 
apply that cleanup as well? Thanks!


-- 
To view, visit http://gerrit.cloudera.org:8080/3942
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2c9162fcf9d227b8daf4c2e761d57bab4e26408f
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Amos Bird 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Amos Bird 
Gerrit-Reviewer: Jim Apple 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-1654: General partition exprs in DDL operations.

[Alex Behm (Code Review)]

Alex Behm has posted comments on this change.

Change subject: IMPALA-1654: General partition exprs in DDL operations.
..


Patch Set 14:

I fixed a few more tests. Doing another test run now.

Based on that exercise I do have some additional comments on the code, I hope 
you can address them, Amos.

-- 
To view, visit http://gerrit.cloudera.org:8080/3942
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2c9162fcf9d227b8daf4c2e761d57bab4e26408f
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Amos Bird 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Amos Bird 
Gerrit-Reviewer: Jim Apple 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4026: Implement double-buffering for BlockingQueue

[Michael Ho (Code Review)]

Michael Ho has posted comments on this change.

Change subject: IMPALA-4026: Implement double-buffering for BlockingQueue
..


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/4350/3/be/src/util/blocking-queue.h
File be/src/util/blocking-queue.h:

Line 99:   bool BlockingPut(const T& val) {
> We discussed this a little offline and I feel like I understand why option 
The new patch reverted to using option(1). This doesn't provide as much 
performance in the non-selective scan but this may avoid some pathological case 
in thread scheduling.


-- 
To view, visit http://gerrit.cloudera.org:8080/4350
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: Ib9f4cf351455efefb0f3bb791cf9bc82d1421d54
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Chen Huang 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Mostafa Mokhtar 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4026: Implement double-buffering for BlockingQueue

[Michael Ho (Code Review)]

Michael Ho has uploaded a new patch set (#4).

Change subject: IMPALA-4026: Implement double-buffering for BlockingQueue
..

IMPALA-4026: Implement double-buffering for BlockingQueue

With recent changes to improve the parquet scanner's efficency,
row batches are produced more quickly, leading to higher contention
in the blocking queue shared between scanner threads and the scan
node. The contention happens between different producers (i.e.
the scanner threads) and also to a lesser extent, between the
scanner threads and the scan node.

This change addresses the contention between the scanner threads
and the scan node by splitting the queue into a 'read_list_' and
a 'write_list_'. The consumers will consume from 'read_list_' until
it's exhausted while the producers will enqueue into 'write_list_'
until it's full. When 'read_list_' is exhausted, the consumer will
atomically swap the 'read_list_' with 'write_list_'. This reduces
the contention/overhead: 'read_list_' and 'write_list_' are protected
by two different locks so consumer only contends for the
write lock when 'read_list_' is empty.

Change-Id: Ib9f4cf351455efefb0f3bb791cf9bc82d1421d54
---
M be/src/common/compiler-util.h
M be/src/exec/hdfs-scan-node.cc
M be/src/exec/hdfs-scan-node.h
M be/src/util/blocking-queue.h
A be/src/util/condition-variable.h
5 files changed, 184 insertions(+), 61 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/4350/4
-- 
To view, visit http://gerrit.cloudera.org:8080/4350
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib9f4cf351455efefb0f3bb791cf9bc82d1421d54
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Michael Ho 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Chen Huang 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Michael Ho 
Gerrit-Reviewer: Mostafa Mokhtar 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4110, IMPALA-3853: npm.js uses Artistic License 2.

[Internal Jenkins (Code Review)]

Internal Jenkins has posted comments on this change.

Change subject: IMPALA-4110, IMPALA-3853: npm.js uses Artistic License 2.
..


Patch Set 2: Verified+1

-- 
To view, visit http://gerrit.cloudera.org:8080/4438
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I4bcbeeb6e8552894803315967b95f365227d7505
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Jim Apple 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Henry Robinson 
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Taras Bobrovytsky 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4110, IMPALA-3853: npm.js uses Artistic License 2.

[Internal Jenkins (Code Review)]

Internal Jenkins has submitted this change and it was merged.

Change subject: IMPALA-4110, IMPALA-3853: npm.js uses Artistic License 2.
..


IMPALA-4110, IMPALA-3853: npm.js uses Artistic License 2.

The comments on https://issues.apache.org/jira/browse/LEGAL-64 seem to
indicate that Artistic License might not be allowed in ASF
projects. Bootstrap distributes it, and Bootstrap itself is
MIT-licensed, but that doesn't mean ASF projects can distribute
npm.js.

Change-Id: I4bcbeeb6e8552894803315967b95f365227d7505
Reviewed-on: http://gerrit.cloudera.org:8080/4438
Reviewed-by: Jim Apple 
Tested-by: Internal Jenkins
---
M LICENSE.txt
M bin/rat_exclude_files.txt
D tests/comparison/leopard/static/js/npm.js
M tests/comparison/leopard/templates/custom_run.template
M tests/comparison/leopard/templates/index.template
M tests/comparison/leopard/templates/report.template
D www/bootstrap/js/npm.js
7 files changed, 70 insertions(+), 31 deletions(-)

Approvals:
  Jim Apple: Looks good to me, approved
  Internal Jenkins: Verified



-- 
To view, visit http://gerrit.cloudera.org:8080/4438
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: merged
Gerrit-Change-Id: I4bcbeeb6e8552894803315967b95f365227d7505
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Jim Apple 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Henry Robinson 
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Michael Brown 
Gerrit-Reviewer: Taras Bobrovytsky