[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 22:

patch 22: "fixed some of the describe commands"
show create table metrics
describe formatted metrics

-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 22
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#22).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescribeTableStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/ShowCreateTableStmt.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/catalog/TableId.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
38 files changed, 1,091 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/22
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 22
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#21).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/catalog/TableId.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
36 files changed, 1,075 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/21
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 21
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 20:

(25 comments)

> We also need to fix some of the describe commands:
 > 
 > [tarmstrong-box.ca.cloudera.com:21000] > show create table metrics;
 > Query: show create table metrics
 > Query submitted at: 2016-09-07 16:53:20 (Coordinator:
 > http://0.0.0.0:25000)
 > ERROR: NullPointerException: null
 > 
 > [tarmstrong-box.ca.cloudera.com:21000] > describe formatted
 > metrics;
 > Query: describe formatted metrics
 > Query submitted at: 2016-09-07 16:53:40 (Coordinator:
 > http://0.0.0.0:25000)
 > ERROR: NullPointerException: null

not include in the 21 patch, discuss offline
discuss offline

http://gerrit.cloudera.org:8080/#/c/3863/20/be/src/exec/system-table-scan-node.cc
File be/src/exec/system-table-scan-node.cc:

Line 54:   DCHECK(table_name_ == TSystemTableName::type::METRICS);
> We prefer DCHECK_EQ
Done


Line 76: case TSystemTableName::QUERIES:
> We can remove this case (we already check that it's only METRICS above). Th
Done


Line 89:   scanner_->MaterializeNextTuple(tuple_pool, tuple, tuple_desc_);
> We need to check the return status here.
Done


Line 127: ++(scanner_->next_row_idx_);
> MaterializeNextTuple() should increment this inside the scanner.
Done


Line 128: scanner_->CheckEOS();
> This doesn't seem too necessary?
Done


http://gerrit.cloudera.org:8080/#/c/3863/20/be/src/exec/system-table-scan-node.h
File be/src/exec/system-table-scan-node.h:

Line 31: class SystemTableScanNode : public ScanNode {
> I think we should add a short class comment. E.g.
Done


Line 37:   /// Create  schema and columns to slots mapping.
> Extra space before schema.
Done


Line 40:   /// Start  scan.
> Extra space before scan.
Done


PS20, Line 43: metrics
> from the system table.
Done


PS20, Line 69: next_row_idx_
> next_row_idx was removed.
Done


http://gerrit.cloudera.org:8080/#/c/3863/20/be/src/exec/system-table-scanner.cc
File be/src/exec/system-table-scanner.cc:

Line 38: "SystemTableScanNode::$0() failed to allocate "
> The string should fit one one line.
Done


PS20, Line 107: group_name
> Should be metric_name.
Done


Line 116: return Status(Substitute("column error"));
> This shouldn't be possible, so let's make this a DCHECK. E.g.
Done


http://gerrit.cloudera.org:8080/#/c/3863/20/be/src/exec/system-table-scanner.h
File be/src/exec/system-table-scanner.h:

Line 47:   virtual void CheckEOS() = 0;
> I don't think this is necessary - MaterializeNextTuple should set eos_
Done


Line 50:   bool eos_;
> Let's make eos_ private and expose it with an accessor function.
Done


Line 53:   int next_row_idx_;
> I think this can live in the MetricScanner subclass for now.
Done


Line 73: if (next_row_idx_ >= metric_pool_.size()) eos_ = true;
> We can just do this at the end of MaterializeNextTuple()
Done


http://gerrit.cloudera.org:8080/#/c/3863/20/fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
File fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java:

Line 205:   Table table = getTable();
> You can undo this change - it doesn't look needed.
Done


http://gerrit.cloudera.org:8080/#/c/3863/20/fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
File fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java:

PS20, Line 32: matrics
> /// Table for all Impala daemon metrics.
Done


http://gerrit.cloudera.org:8080/#/c/3863/20/fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
File fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java:

PS20, Line 66: info schema 
> system table.
Done


http://gerrit.cloudera.org:8080/#/c/3863/20/fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
File fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java:

PS20, Line 2188: .getDb("system")
> Can we use getSystemDb() here?
Done


Line 2189: Table table= new SystemTable(
> Missing space before =
Done


Line 2218: // table doesn't exist
> This comment "table doesn't exist" doesn't seem necessary.
Done


Line 2258: Db system = catalog.getDb("system");
> Can we use getSystemDb() here?
Done


Line 2259: Table table= new SystemTable(
> Need space before =
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 20
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 20:

We also need to fix some of the describe commands:

[tarmstrong-box.ca.cloudera.com:21000] > show create table metrics;
Query: show create table metrics
Query submitted at: 2016-09-07 16:53:20 (Coordinator: http://0.0.0.0:25000)
ERROR: NullPointerException: null

[tarmstrong-box.ca.cloudera.com:21000] > describe formatted metrics;
Query: describe formatted metrics
Query submitted at: 2016-09-07 16:53:40 (Coordinator: http://0.0.0.0:25000)
ERROR: NullPointerException: null

-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 20
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 20:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/3863/20/be/src/exec/system-table-scanner.cc
File be/src/exec/system-table-scanner.cc:

PS20, Line 107: group_name
Should be metric_name.


-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 20
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 20:

(27 comments)

I think we can make a lot of small improvements but we're converging :)

http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scan-node.cc
File be/src/exec/system-table-scan-node.cc:

PS18, Line 49: tuple_id_(
> Done.
Looks like dead/old code in data-source-scan-node. It should be removed there 
too - feel free to create a patch to fix it there if you have any free time.


http://gerrit.cloudera.org:8080/#/c/3863/19/be/src/exec/system-table-scan-node.cc
File be/src/exec/system-table-scan-node.cc:

Line 93: Status SystemTableScanNode::GetNext(RuntimeState* state, RowBatch* 
row_batch, bool* eos) {
Need to check the return status here.


http://gerrit.cloudera.org:8080/#/c/3863/20/be/src/exec/system-table-scan-node.cc
File be/src/exec/system-table-scan-node.cc:

Line 54:   DCHECK(table_name_ == TSystemTableName::type::METRICS);
We prefer DCHECK_EQ


Line 76: case TSystemTableName::QUERIES:
We can remove this case (we already check that it's only METRICS above). This 
will be easy to add back in later.


Line 89:   scanner_->MaterializeNextTuple(tuple_pool, tuple, tuple_desc_);
We need to check the return status here.


Line 127: ++(scanner_->next_row_idx_);
MaterializeNextTuple() should increment this inside the scanner.


Line 128: scanner_->CheckEOS();
This doesn't seem too necessary?


http://gerrit.cloudera.org:8080/#/c/3863/20/be/src/exec/system-table-scan-node.h
File be/src/exec/system-table-scan-node.h:

Line 31: class SystemTableScanNode : public ScanNode {
I think we should add a short class comment. E.g.

/// A scan node that exposes Impala system state as a table.
///
/// Different SystemTableScanner subclasses gather data from different Impala 
subsystems and
/// materialize it into Tuples.


Line 37:   /// Create  schema and columns to slots mapping.
Extra space before schema.


Line 40:   /// Start  scan.
Extra space before scan.


PS20, Line 43: metrics
from the system table.


PS20, Line 69: next_row_idx_
next_row_idx was removed.


http://gerrit.cloudera.org:8080/#/c/3863/19/be/src/exec/system-table-scanner.cc
File be/src/exec/system-table-scanner.cc:

Line 63: enum MetricsColumn {
Let's put this in the MetricsScanner class in the header.


http://gerrit.cloudera.org:8080/#/c/3863/20/be/src/exec/system-table-scanner.cc
File be/src/exec/system-table-scanner.cc:

Line 38: "SystemTableScanNode::$0() failed to allocate "
The string should fit one one line.

Let's also add an empty line before and after.


Line 116: return Status(Substitute("column error"));
This shouldn't be possible, so let's make this a DCHECK. E.g.

  DCHECK(false) << "Unknown column position " << slot_desc->col_pos();


http://gerrit.cloudera.org:8080/#/c/3863/20/be/src/exec/system-table-scanner.h
File be/src/exec/system-table-scanner.h:

Line 47:   virtual void CheckEOS() = 0;
I don't think this is necessary - MaterializeNextTuple should set eos_


Line 50:   bool eos_;
Let's make eos_ private and expose it with an accessor function.


Line 53:   int next_row_idx_;
I think this can live in the MetricScanner subclass for now.


Line 73: if (next_row_idx_ >= metric_pool_.size()) eos_ = true;
We can just do this at the end of MaterializeNextTuple()


http://gerrit.cloudera.org:8080/#/c/3863/20/fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
File fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java:

Line 205:   Table table = getTable();
You can undo this change - it doesn't look needed.


http://gerrit.cloudera.org:8080/#/c/3863/20/fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
File fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java:

PS20, Line 32: matrics
/// Table for all Impala daemon metrics.


http://gerrit.cloudera.org:8080/#/c/3863/20/fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
File fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java:

PS20, Line 66: info schema 
system table.


http://gerrit.cloudera.org:8080/#/c/3863/20/fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
File fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java:

PS20, Line 2188: .getDb("system")
Can we use getSystemDb() here?


Line 2189: Table table= new SystemTable(
Missing space before =


Line 2218: // table doesn't exist
This comment "table doesn't exist" doesn't seem necessary.


Line 2258: Db system = catalog.getDb("system");
Can we use getSystemDb() here?


Line 2259: Table table= new SystemTable(
Need space before =


-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#19).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
M fe/src/main/java/com/cloudera/impala/catalog/Db.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/catalog/TableId.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
37 files changed, 1,106 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/19
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 19
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 19:

(4 comments)

1.Refactor code in system-table-scan-node
remove functions and variables that could be avoid.
2. revise Authz-test

http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scan-node.cc
File be/src/exec/system-table-scan-node.cc:

PS18, Line 94: 
> probably not.
Done


http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scanner.h
File be/src/exec/system-table-scanner.h:

Line 38: 
> I think I did batched GetNext()  (1024 rows per batch)
Done


Line 40:   virtual Status Open() = 0;
> But I do need size() to serve 1024 batch feature. to decide the size of nex
Done


http://gerrit.cloudera.org:8080/#/c/3863/18/fe/src/main/java/com/cloudera/impala/catalog/Db.java
File fe/src/main/java/com/cloudera/impala/catalog/Db.java:

Line 496:   @Override
> I don't think we need hashCode() and equals() do we?
Done


-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 19
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#20).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/catalog/TableId.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
36 files changed, 1,071 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/20
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 20
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#19).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
M fe/src/main/java/com/cloudera/impala/catalog/Db.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/catalog/TableId.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
37 files changed, 1,106 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/19
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 19
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 18:

(28 comments)

http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scan-node.cc
File be/src/exec/system-table-scan-node.cc:

PS18, Line 49: tuple_idx_
> We don't need tuple_idx_ because it's always zero. This is generally true f
Done.
I thought leaving it there would be consistent with other scan-node?


PS18, Line 94: idx
> I think we should move idx into the Metrics scanner. Maybe call it next_row
probably not.
idx is useful to decide whether we have next input batch.


Line 121: while (!ReachedLimit() && !row_batch->AtCapacity() && 
InputBatchHasNext()) {
> If we get rid of InputBatchHasNext(), GetNextInputBatch(), I think we can c
It follows the same style with data-source-scan-node. 

(1024 rows a batch)
But now i think  getNext() in data-source-scan-node.cc don't need two loops 
either. Two loops make getNext() deal with all the batches. if so getNext will 
only called once.

I fixed my getNext() here.


PS18, Line 137: MetricAtLimit
> Rename this?
how about 
AllItemsScanned()


Line 149: Status SystemTableScanNode::GetNextInputBatch() {
> I don't think we need this function, GetNextInputBatch(), or many of the ot
Since we process 1024 rows a batch,  these are served for this feature.

next_row_idx_ is the index inside the input_batch. So whenever we get a new 
input batch, we set the next_row_idx_ to 0.

items_size_ is the size of metrics_pool.  And this is related to another 
comment. you mentioned we may need to remove size() in system-table-scanner, 
since  for some of system table, we don't know it's size before we scan it.

I do need the item_size to know how many rows are left, and further how many 
rows are in next batch.


http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scan-node.h
File be/src/exec/system-table-scan-node.h:

PS18, Line 56: SystemTableScanner
> Let's make this a scoped_ptr so that we don't leak the 
Done


http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scanner.cc
File be/src/exec/system-table-scanner.cc:

Line 46: LOG(INFO) << "group name: " << group->name();
> We can remove this logging I think.
Done


PS18, Line 49: ChildGroupMap::value_type&
> I think it's easier to use auto here.
Done


Line 56:   metric_info.ip_address_ = 
ExecEnv::GetInstance()->webserver()->http_address();
> I think we should report the backend address rather than the webserver addr
Done


Line 84: size_t value_size = value[slot_desc->col_pos()].size();
> I think it would be just as simple and more efficient to avoid copying all 
Done.

But the code is longer now...
Is it ok if I create a enum like that. (not in thrift)


Line 85: char* buffer = 
reinterpret_cast(tuple_pool->TryAllocate(value_size));
> Need to check if buffer is NULL and return a MemLimitExceeded() error (look
Done


http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scanner.h
File be/src/exec/system-table-scanner.h:

PS18, Line 18: SYSTEMTABLESCANNER_H_
> Should be something like:
Done


Line 32: // For each table in "system" database, we should implement the 
scanner for scanning it
> maybe instead: "we should implement a scanner that can materialize the tabl
Done


Line 37:   virtual Status Open() = 0;
> Can you document Open() and MaterializeNextRow().
Done


Line 38:   virtual Status MaterializeNextRow(
> Maybe GetNextTuple()?
I think I did batched GetNext()  (1024 rows per batch)
but rows will be materialized one at a time, following the same design of other 
scan node.


Line 40:   int size() { return size_; }
> I think we maybe don't want to make size() part of the interface because it
But I do need size() to serve 1024 batch feature. to decide the size of next 
batch, whether it should be full 1024 batch, or the rows left.

maybe discuss offline


Line 46: struct MetricPackage {
> I mentioned in another comment - I think it's sufficient to save the Metric
Done


Line 54:   vector metric_pool;
> metric_pool should be private and have '_' on the end.
Done.
I thought metric_pool is better... it gives us a impression that it contains 
all of the metrics. ( "metrics" are used else where and doesn't have this 
meaning)


http://gerrit.cloudera.org:8080/#/c/3863/18/fe/src/main/java/com/cloudera/impala/catalog/Db.java
File fe/src/main/java/com/cloudera/impala/catalog/Db.java:

Line 496:   @Override
> I don't think we need hashCode() and equals() do we?
It is used when determine whether a list contains a certain db (system, 
used in authorizationTest() )

since db is a user defined object, we need to override hashCode() and equals() 
so that we could use list.Contains() function. I thought it could be useful 
later. So I add them on. But I could also compare the list by myself.

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 18:

(29 comments)

http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scan-node.cc
File be/src/exec/system-table-scan-node.cc:

PS18, Line 49: tuple_idx_
We don't need tuple_idx_ because it's always zero. This is generally true for 
scanners - they only produce a single tuple per row.


PS18, Line 94: idx
I think we should move idx into the Metrics scanner. Maybe call it 
next_row_idx_. That way each scanner can keep track of its position in the most 
convenient way.


PS18, Line 94: MaterializeNextRow
This should be MaterializeNextTuple() now that I think about it, because it's 
only filling in a Tuple.


Line 121: while (!ReachedLimit() && !row_batch->AtCapacity() && 
InputBatchHasNext()) {
If we get rid of InputBatchHasNext(), GetNextInputBatch(), I think we can 
combine the two loops.


PS18, Line 137: MetricAtLimit
Rename this?


Line 149: Status SystemTableScanNode::GetNextInputBatch() {
I don't think we need this function, GetNextInputBatch(), or many of the other 
variables: next_row_idx_, items_size_, etc, do we?


http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scan-node.h
File be/src/exec/system-table-scan-node.h:

PS18, Line 56: SystemTableScanner
Let's make this a scoped_ptr so that we don't leak the 
memory.


http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scanner.cc
File be/src/exec/system-table-scanner.cc:

Line 46: LOG(INFO) << "group name: " << group->name();
We can remove this logging I think.


PS18, Line 49: ChildGroupMap::value_type&
I think it's easier to use auto here.


Line 56:   metric_info.ip_address_ = 
ExecEnv::GetInstance()->webserver()->http_address();
I think we should report the backend address rather than the webserver address.


Line 84: size_t value_size = value[slot_desc->col_pos()].size();
I think it would be just as simple and more efficient to avoid copying all the 
strings by using a switch here.

e.g.
  const string& slot_value;
  switch (slot_desc->col_pos()) {
   case IMPALAD_ADDRESS:
RETURN_IF_ERROR(WriteStringSlot(impalad_address, tuple_pool, slot));
   ...
   case METRIC_GROUP:
RETURN_IF_ERROR(WriteStringSlot(metrics_[idx].group->Name(), tuple_pool, 
slot));
   ...
  }


Line 85: char* buffer = 
reinterpret_cast(tuple_pool->TryAllocate(value_size));
Need to check if buffer is NULL and return a MemLimitExceeded() error (look at 
other places we call TryAllocate() in scanners).


http://gerrit.cloudera.org:8080/#/c/3863/18/be/src/exec/system-table-scanner.h
File be/src/exec/system-table-scanner.h:

PS18, Line 18: SYSTEMTABLESCANNER_H_
Should be something like:

IMPALA_SYSTEM_TABLE_SCANNER_H

for consistency


Line 32: // For each table in "system" database, we should implement the 
scanner for scanning it
maybe instead: "we should implement a scanner that can materialize the table's 
rows.".


Line 37:   virtual Status Open() = 0;
Can you document Open() and MaterializeNextRow().


Line 38:   virtual Status MaterializeNextRow(
Maybe GetNextTuple()?

I think there's a design decision here about whether to have a batched 
GetNext() interface or a row-at-a-time interface. The batched interface would 
be faster but a little more complex.

I think it would be good to extend the class comment to say that the scanner 
materializes a tuple-at-a-time for simplicity because system tables are 
generally small and performance isn't critical.


Line 40:   int size() { return size_; }
I think we maybe don't want to make size() part of the interface because it's 
possible that we'll have scanners that don't know the # of rows before scanning.

I think add a bool member variable/method eos_ that is true when the scanner 
returned its last row. We use that pattern elsewhere.


Line 46: struct MetricPackage {
I mentioned in another comment - I think it's sufficient to save the Metric* 
and MetricGroup* here and then get the required strings when needed.


Line 54:   vector metric_pool;
metric_pool should be private and have '_' on the end.

Maybe just call it 'metrics_'?


http://gerrit.cloudera.org:8080/#/c/3863/18/fe/src/main/java/com/cloudera/impala/catalog/Db.java
File fe/src/main/java/com/cloudera/impala/catalog/Db.java:

Line 496:   @Override
I don't think we need hashCode() and equals() do we?


http://gerrit.cloudera.org:8080/#/c/3863/18/fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
File fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java:

Line 27:   private static final int SYSTEM_METRICS = -2;
I think this would be better in TableId, since it is something that it shared 
across all tables, not just system tables.


PS18, Line 31: this.
No "this" here and below (we generally avoid using it when it's not necessary).


PS18, Line 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 18:

(15 comments)

made change based on review.
1. assign table id to metrics table to make sure table descriptor i
2. add negative tests in authz test, also add test case for admin user.
3. add test for "show databases", check the existence of "system" database
3. small changes to further separate metrics related code out.

http://gerrit.cloudera.org:8080/#/c/3863/17/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:

Line 360: }
> Extra line.
Done


http://gerrit.cloudera.org:8080/#/c/3863/17/common/thrift/StatestoreService.thrift
File common/thrift/StatestoreService.thrift:

PS17, Line 67: sList {
> List?
Done


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
File fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java:

Line 469: if (dbName != null && checkSystemDbAccess(dbName, 
request.getPrivilege())) {
> I don't think this comment is necessary here - that logic is implemented in
Done


Line 527:*/
> "if authorization should be checked in the usual way."
Done


Line 538:   return false;
> Add a comment like:
Done


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
File fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java:

Line 169: if (table != null && !(table instanceof View))
> This doesn't seem right - we are referencing the system table and need to s
discuss offline


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
File fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java:

Line 204: if (getTable() != null && !(getTable() instanceof View)) {
> I think we do need to assign a table id for system tables, otherwise we won
discuss offline


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
File fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java:

Line 35:   private void AddMetricTable() {
> Remove "including impala metrics currently" so we don't have to remember to
Done


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
File fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java:

PS17, Line 110: 
> Do we set numRows_? I think we want to set it to something at least approxi
discussed offline


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
File fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java:

Line 80: 
> I don't think we need the throws annotation.
Done


Line 82:   scanRangeLocations.scan_range = new TScanRange();
> Extra line.
Done


Line 123:   String prefix, String detailPrefix, TExplainLevel detailLevel) {
> Remove comment?
Done


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/service/FeSupport.java
File fe/src/main/java/com/cloudera/impala/service/FeSupport.java:

PS17, Line 93: NativeGetBackends
> We seem to use "Backends" as the capitalization throughout. Let's do that f
Done


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
File fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java:

PS17, Line 105: pestiny')
> Remove- we do have select permissions.
Oops... I was about to say no INSERT permission…
removed.


Line 2175:   }
> We're missing a negative test where we don't have access to a system table.
discuss offline


-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 18
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#18).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
M fe/src/main/java/com/cloudera/impala/catalog/Db.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
36 files changed, 1,073 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/18
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 18
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#18).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
M fe/src/main/java/com/cloudera/impala/catalog/Db.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
36 files changed, 1,073 insertions(+), 22 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/18
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 18
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#18).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
M fe/src/main/java/com/cloudera/impala/catalog/Db.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
36 files changed, 1,069 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/18
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 18
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Tim Armstrong (Code Review)]

Tim Armstrong has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 17:

(16 comments)

Focused on the fe part. I think we need to sort out the TableId thing, 
otherwise mostly minor comments.

http://gerrit.cloudera.org:8080/#/c/3863/17/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:

Line 360: 
Extra line.


http://gerrit.cloudera.org:8080/#/c/3863/17/common/thrift/StatestoreService.thrift
File common/thrift/StatestoreService.thrift:

PS17, Line 67: sContaine
List?


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
File fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java:

Line 469: // For SELECT, we should always check auth policy.
I don't think this comment is necessary here - that logic is implemented in 
checkSystemDbAccess()


Line 527:* Return false if we need further authorization check.
"if authorization should be checked in the usual way."


Line 538:   return false;
Add a comment like:

// Check authorization for SELECT on system tables in the usual way.


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
File fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java:

Line 169: if (table != null && !(table instanceof View) && !(table 
instanceof SystemTable))
This doesn't seem right - we are referencing the system table and need to send 
the descriptor to the backend. Am I missing something?

This is probably the wrong way to fix the problem.


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
File fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java:

Line 204: if (getTable() != null && !(getTable() instanceof View)
I think we do need to assign a table id for system tables, otherwise we won't 
set up the backend descriptor table correctly when there are multiple tables 
per fragment.

I suspect if you joined two system tables you would run into problems.

It looks like automatically assigned table ids start at 0. Maybe we should 
assign hardcoded negative table ids to each system table. I.e. define constants 
in TableId like SYSTEM_METRICS = -1, etc.


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
File fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java:

Line 35:   "System database for Impala cluster, including impala metrics 
currently";
Remove "including impala metrics currently" so we don't have to remember to 
update the comment.


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
File fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java:

PS17, Line 110: numRows_
Do we set numRows_? I think we want to set it to something at least approximate 
so that the planner knows its a smallish table and has something to work with.


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
File fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java:

Line 70:   //  private void computeSingleScanRangeLocations(Analyzer analyzer) {
Old code


Line 80:* @throws InternalException
I don't think we need the throws annotation.


Line 82: 
Extra line.


Line 123: // tblDescripter
Remove comment?


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/main/java/com/cloudera/impala/service/FeSupport.java
File fe/src/main/java/com/cloudera/impala/service/FeSupport.java:

PS17, Line 93: NativeGetBackEnds
We seem to use "Backends" as the capitalization throughout. Let's do that for 
consistency.


http://gerrit.cloudera.org:8080/#/c/3863/17/fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
File fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java:

PS17, Line 105:  (no SELECT permissions)
Remove- we do have select permissions.


Line 2175: AuthzOk("select * from system.metrics");
We're missing a negative test where we don't have access to a system table. 
Maybe we could add a fake table to ImpaladTestCatalog?

Could you also add a AuthzError test for selecting from a non-existent system 
table.


-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 17
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#17).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
35 files changed, 980 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/17
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 17
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 16:

This patch add a SystemTableScanner and its subclass MetricScanner to "factor 
out the metrics-specific code into a separate class".

-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 16
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#16).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
35 files changed, 960 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/16
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 16
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#16).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
35 files changed, 960 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/16
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 16
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#16).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
35 files changed, 955 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/16
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 16
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#16).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
A be/src/exec/system-table-scanner.cc
A be/src/exec/system-table-scanner.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
35 files changed, 954 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/16
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 16
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has posted comments on this change.

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..


Patch Set 15:

This two patch 
1. add authz test and conjunct test.
2. make the scan range involve all backend node
   (I add a JNI method to transfer information about all backends
from be to fe)
3. fix some bugs

I think it's ready for a review

-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 15
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#15).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
33 files changed, 851 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/15
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 15
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#15).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
33 files changed, 851 insertions(+), 23 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/15
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 15
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#15).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
33 files changed, 863 insertions(+), 35 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/15
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 15
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#15).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/scheduling/simple-scheduler.h
M be/src/service/fe-support.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M common/thrift/StatestoreService.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/FeSupport.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/analysis/AuthorizationTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
A testdata/workloads/functional-query/queries/QueryTest/system-database.test
M tests/query_test/test_scanners.py
33 files changed, 875 insertions(+), 47 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/15
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 15
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#14).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
26 files changed, 748 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/14
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#14).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
M fe/src/test/resources/authz-policy.ini.template
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
26 files changed, 748 insertions(+), 14 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/14
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong 

[Impala-ASF-CR] IMPALA-4024: Add "system" database and expose Impala metrics as a table

[Kathy Sun (Code Review)]

Kathy Sun has uploaded a new patch set (#14).

Change subject: IMPALA-4024: Add "system" database and expose Impala metrics as 
a table
..

IMPALA-4024: Add "system" database and expose Impala metrics as a table

This is to expose metadata (current status of impala, e.g impala-metrics)
into system table, so that users could query them via sql.
Currently, only metrics table is included.
We could add other table later, e.g. queries table

You can run impala-shell.sh and type query like:
select * from system.metrics;
select name, value, description from system.metrics;

Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
---
M be/src/exec/CMakeLists.txt
M be/src/exec/exec-node.cc
A be/src/exec/system-table-scan-node.cc
A be/src/exec/system-table-scan-node.h
M be/src/runtime/coordinator.cc
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/simple-scheduler.cc
M be/src/util/metrics.h
M common/thrift/CatalogObjects.thrift
M common/thrift/Descriptors.thrift
M common/thrift/PlanNodes.thrift
M fe/src/main/java/com/cloudera/impala/analysis/AnalysisContext.java
M fe/src/main/java/com/cloudera/impala/analysis/Analyzer.java
M fe/src/main/java/com/cloudera/impala/analysis/DescriptorTable.java
M fe/src/main/java/com/cloudera/impala/analysis/TupleDescriptor.java
M fe/src/main/java/com/cloudera/impala/catalog/Catalog.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemDb.java
A fe/src/main/java/com/cloudera/impala/catalog/SystemTable.java
M fe/src/main/java/com/cloudera/impala/planner/SingleNodePlanner.java
A fe/src/main/java/com/cloudera/impala/planner/SystemTableScanNode.java
M fe/src/main/java/com/cloudera/impala/service/Frontend.java
M fe/src/test/java/com/cloudera/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTest.java
A testdata/workloads/functional-planner/queries/PlannerTest/system-db.test
25 files changed, 745 insertions(+), 13 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/63/3863/14
-- 
To view, visit http://gerrit.cloudera.org:8080/3863
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I7adbeb45220c468e43b424d70c30b952f6cec2cd
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Kathy Sun 
Gerrit-Reviewer: Kathy Sun 
Gerrit-Reviewer: Tim Armstrong