[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-18 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 6: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5237/


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 6
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Mon, 18 Nov 2019 23:44:15 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-18 Thread Thomas Tauber-Marshall (Code Review)
Thomas Tauber-Marshall has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 5:

(20 comments)

http://gerrit.cloudera.org:8080/#/c/14720/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/14720/5//COMMIT_MSG@7
PS5, Line 7: Add support for retrieval of primary keys
   : and foriegn keys through impala-hs2-server.
nit: keep the title of the commit message to one line


http://gerrit.cloudera.org:8080/#/c/14720/5//COMMIT_MSG@12
PS5, Line 12: GetPrimaryKeys and GetCrossReference
Might be worth explicitly mentioning that the thrift definitions for these were 
taken directly from Hive, and  ways that our behavior differs from Hive's (if 
any)


http://gerrit.cloudera.org:8080/#/c/14720/5//COMMIT_MSG@31
PS5, Line 31: - Authorization information is not available for 
IncompleteTables. In
Could you expand on this a little more, in particular why is not feasible, is 
there any plan to ever fix this, is this consistent with Hive's behavior, etc.


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/analysis/TableDef.java
File fe/src/main/java/org/apache/impala/analysis/TableDef.java:

http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/analysis/TableDef.java@545
PS5, Line 545:
unnecessary blank line


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/analysis/TableDef.java@547
PS5, Line 547: // TODO: Remove this check when we implement PK/FK feature for 
LocalCatalog
 :   //  (IMPALA-9158).
Is this no longer required to deal with the Hive bug described in the comment 
above?


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/catalog/Table.java
File fe/src/main/java/org/apache/impala/catalog/Table.java:

http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/catalog/Table.java@114
PS5, Line 114:   protected final List primaryKeys_ = new 
ArrayList<>();
brief comment


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/catalog/Table.java@116
PS5, Line 116:   protected final List foreignKeys_ = new 
ArrayList<>();
brief comment


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/catalog/Table.java@608
PS5, Line 608:   @Override
// FeTable

Might as well match the rest of this file


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java:

http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java@530
PS5, Line 530: return new ArrayList<>();
Would it be better, here and elsewhere, to do something like throw a 
NotImplementedException or similar to make sure we're not silently returning 
incorrect results?


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/service/MetadataOp.java
File fe/src/main/java/org/apache/impala/service/MetadataOp.java:

http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/service/MetadataOp.java@254
PS5, Line 254: STRING
As far as I can tell, looks like the types you have here don't quite match what 
is used in Hive, in particular I think Hive uses INT for UPDATE_RULE, 
DELETE_RULE, and DEFERRABILITY. see: 
https://github.com/apache/hive/blob/master/service/src/java/org/apache/hive/service/cli/operation/GetCrossReferenceOperation.java#L91

Was it intentional to not match Hive on this?


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/service/MetadataOp.java@266
PS5, Line 266:* Contains lists of databases, lists of table belonging to 
the dbs, list of columns
This comment is stale now


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/service/MetadataOp.java@623
PS5, Line 623: , user
formatting


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/service/MetadataOp.java@649
PS5, Line 649: if (LOG.isTraceEnabled()) LOG.trace("Returning " + 
result.rows.size() + " primary "
 : + "keys for table " + tableName);
Since it doesn't fit on one line, you should use {}, here and below


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/service/MetadataOp.java@662
PS5, Line 662:
nit: formatting


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/service/MetadataOp.java@701
PS5, Line 701:   row.colVals.add(EMPTY_COL_VAL);
Might be nice to note what each of the EMPTY_COL_VALs are for, eg. at the end 
of this line add // PKTABLE_CAT


http://gerrit.cloudera.org:8080/#/c/14720/5/fe/src/main/java/org/apache/impala/service/MetadataOp.java@736
PS5, 

[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-18 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 6:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5049/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 6
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Mon, 18 Nov 2019 19:44:05 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-18 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 6:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5237/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 6
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Mon, 18 Nov 2019 19:00:27 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-18 Thread Anurag Mantripragada (Code Review)
Anurag Mantripragada has uploaded a new patch set (#6). ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..

IMPALA-9104: Add support for retrieval of primary keys
and foriegn keys through impala-hs2-server.

The goal is to let JDBC clients get constraint information
from Impala tables. We implement two new metadata operations in
impala-hs2-server, GetPrimaryKeys and GetCrossReference. For these to work,
new TCLIService methods were added. In FE, these two operations are
implemented to get the information from tables in the catalog.

Much like GetColumns(), tables need to be loaded in order to be able to get
PK/FK information. We wait for the PK table/FK table to load.
In the implementation, PK/FK information is returned
ONLY if the user has access to ALL the columns involved in the PK/FK
relationship.

Testing:
- Added three test tables to our test datasets since most of our FE tests
  relied on dummy tables or testdata. It was difficult to test PK/FK with
  these methods. Also, we can build on this testdata in future when we make
  optimizer improvements.
- Added unit tests in AuthorizationTest and JDBCtest.
- Added e2e test in test_hs2.py

Caveats:
- Authorization information is not available for IncompleteTables. In
  case of foreign key tables, it is not feasible to load all the
  parent tables involved to check for column privileges. Therefore, there
  maybe missing PK/FK relationships for parent tables that are not loaded.
- Retrieval of constraints is not yet supported in LocalCatalog mode. See
  IMPALA-9158.

Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
---
M be/src/rpc/kerberos-test.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-server.h
M bin/rat_exclude_files.txt
M common/thrift/Frontend.thrift
M common/thrift/hive-1-api/TCLIService.thrift
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
A testdata/data/child_table.txt
A testdata/data/parent_table.txt
A testdata/data/parent_table_2.txt
M testdata/datasets/functional/functional_schema_template.sql
M tests/hs2/test_hs2.py
22 files changed, 789 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/14720/6
--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 6
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-17 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 5: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5232/


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Mon, 18 Nov 2019 02:43:30 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-17 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 5:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5232/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Mon, 18 Nov 2019 01:23:14 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-17 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 5:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5041/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sun, 17 Nov 2019 18:59:24 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-17 Thread Anurag Mantripragada (Code Review)
Anurag Mantripragada has uploaded a new patch set (#5). ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..

IMPALA-9104: Add support for retrieval of primary keys
and foriegn keys through impala-hs2-server.

The goal is to let JDBC clients get constraint information
from Impala tables. We implement two new metadata operations in
impala-hs2-server, GetPrimaryKeys and GetCrossReference. For these to work,
new TCLIService methods were added. In FE, these two operations are
implemented to get the information from tables in the catalog.

Much like GetColumns(), tables need to be loaded in order to be able to get
PK/FK information. We wait for the PK table/FK table to load.
In the implementation, PK/FK information is returned
ONLY if the user has access to ALL the columns involved in the PK/FK
relationship.

Testing:
- Added three test tables to our test datasets since most of our FE tests
  relied on dummy tables or testdata. It was difficult to test PK/FK with
  these methods. Also, we can build on this testdata in future when we make
  optimizer improvements.
- Added unit tests in AuthorizationTest and JDBCtest.
- Added e2e test in test_hs2.py

Caveats:
- Authorization information is not available for IncompleteTables. In
  case of foreign key tables, it is not feasible to load all the
  parent tables involved to check for column privileges. Therefore, there
  maybe missing PK/FK relationships for parent tables that are not loaded.
- Retrieval of constraints is not yet supported in LocalCatalog mode. See
  IMPALA-9158.

Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
---
M be/src/rpc/kerberos-test.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-server.h
M bin/rat_exclude_files.txt
M common/thrift/Frontend.thrift
M common/thrift/hive-1-api/TCLIService.thrift
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
A testdata/data/child_table.txt
A testdata/data/parent_table.txt
A testdata/data/parent_table_2.txt
M testdata/datasets/functional/functional_schema_template.sql
M tests/hs2/test_hs2.py
22 files changed, 788 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/14720/5
--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 5
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-16 Thread Anurag Mantripragada (Code Review)
Anurag Mantripragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 4:

Rat check is failing because I added new testdata file. How should I go about 
adding new testdata?


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 16 Nov 2019 22:27:12 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-16 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 4:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/5040/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 16 Nov 2019 22:06:15 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-16 Thread Anurag Mantripragada (Code Review)
Anurag Mantripragada has uploaded a new patch set (#4). ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..

IMPALA-9104: Add support for retrieval of primary keys
and foriegn keys through impala-hs2-server.

The goal is to let JDBC clients get constraint information
from Impala tables. We implement two new metadata operations in
impala-hs2-server, GetPrimaryKeys and GetCrossReference. For these to work,
new TCLIService methods were added. In FE, these two operations are
implemented to get the information from tables in the catalog.

Much like GetColumns(), tables need to be loaded in order to be able to get
PK/FK information. We wait for the PK table/FK table to load.
In the implementation, PK/FK information is returned
ONLY if the user has access to ALL the columns involved in the PK/FK
relationship.

Testing:
- Added three test tables to our test datasets since most of our FE tests
  relied on dummy tables or testdata. It was difficult to test PK/FK with
  these methods. Also, we can build on this testdata in future when we make
  optimizer improvements.
- Added unit tests in AuthorizationTest and JDBCtest.
- Added e2e test in test_hs2.py

Caveats:
- Authorization information is not available for IncompleteTables. In
  case of foreign key tables, it is not feasible to load all the
  parent tables involved to check for column privileges. Therefore, there
  maybe missing PK/FK relationships for parent tables that are not loaded.
- Retrieval of constraints is not yet supported in LocalCatalog mode. See
  IMPALA-9158.

Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
---
M be/src/rpc/kerberos-test.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-server.h
M common/thrift/Frontend.thrift
M common/thrift/hive-1-api/TCLIService.thrift
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
A testdata/data/child_table.txt
A testdata/data/parent_table.txt
A testdata/data/parent_table_2.txt
M testdata/datasets/functional/functional_schema_template.sql
M tests/hs2/test_hs2.py
21 files changed, 785 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/14720/4
--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 4
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-16 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5227/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 16 Nov 2019 20:45:09 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-16 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 3: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5224/


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 16 Nov 2019 12:58:54 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-16 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5220/


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 16 Nov 2019 09:17:31 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-15 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5224/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 16 Nov 2019 02:58:52 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-15 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 3:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/5038/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 16 Nov 2019 02:45:42 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-15 Thread Anurag Mantripragada (Code Review)
Anurag Mantripragada has uploaded a new patch set (#3). ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..

IMPALA-9104: Add support for retrieval of primary keys
and foriegn keys through impala-hs2-server.

The goal is to let JDBC clients get constraint information
from Impala tables. We implement two new metadata operations in
impala-hs2-server, GetPrimaryKeys and GetCrossReference. For these to work,
new TCLIService methods were added. In FE, these two operations are
implemented to get the information from tables in the catalog.

Much like GetColumns(), tables need to be loaded in order to be able to get
PK/FK information. We wait for the PK table/FK table to load.
In the implementation, PK/FK information is returned
ONLY if the user has access to ALL the columns involved in the PK/FK
relationship.

Testing:
- Added three test tables to our test datasets since most of our FE tests
  relied on dummy tables or testdata. It was difficult to test PK/FK with
  these methods. Also, we can build on this testdata in future when we make
  optimizer improvements.
- Added unit tests in AuthorizationTest and JDBCtest.
- Added e2e test in test_hs2.py

Caveats:
- Authorization information is not available for IncompleteTables. In
  case of foreign key tables, it is not feasible to load all the
  parent tables involved to check for column privileges. Therefore, there
  maybe missing PK/FK relationships for parent tables that are not loaded.
- Retrieval of constraints is not yet supported in LocalCatalog mode. See
  IMPALA-9158.

Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
---
M be/src/rpc/kerberos-test.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-server.h
M common/thrift/Frontend.thrift
M common/thrift/hive-1-api/TCLIService.thrift
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M testdata/datasets/functional/functional_schema_template.sql
M tests/hs2/test_hs2.py
18 files changed, 767 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/14720/3
--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 3
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-15 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 2:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/5032/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 16 Nov 2019 00:45:29 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-15 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 1:

Build Failed

https://jenkins.impala.io/job/gerrit-code-review-checks/5031/ : Initial code 
review checks failed. See linked job for details on the failure.


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Sat, 16 Nov 2019 00:38:05 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-15 Thread Anurag Mantripragada (Code Review)
Anurag Mantripragada has uploaded a new patch set (#2). ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..

IMPALA-9104: Add support for retrieval of primary keys
and foriegn keys through impala-hs2-server.

The goal is to let JDBC clients get constraint information
from Impala tables. We implement two new metadata operations in
impala-hs2-server, GetPrimaryKeys and GetCrossReference. For these to work,
new TCLIService methods were added. In FE, these two operations are
implemented to get the information from tables in the catalog.

Much like GetColumns(), tables need to be loaded in order to be able to get
PK/FK information. We wait for the PK table/FK table to load.
In the implementation, PK/FK information is returned
ONLY if the user has access to ALL the columns involved in the PK/FK
relationship.

Testing:
- Added three test tables to our test datasets since most of our FE tests
  relied on dummy tables or testdata. It was difficult to test PK/FK with
  these methods. Also, we can build on this testdata in future when we make
  optimizer improvements.
- Added unit tests in AuthorizationTest and JDBCtest.
- Added e2e test in test_hs2.py

Caveats:
- Authorization information is not available for IncompleteTables. In
  case of foreign key tables, it is not feasible to load all the
  parent tables involved to check for column privileges. Therefore, there
  maybe missing PK/FK relationships for parent tables that are not loaded.
- Retrieval of constraints is not yet supported in LocalCatalog mode. See
  IMPALA-9158.

Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
---
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-server.h
M common/thrift/Frontend.thrift
M common/thrift/hive-1-api/TCLIService.thrift
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M testdata/datasets/functional/functional_schema_template.sql
M tests/hs2/test_hs2.py
17 files changed, 763 insertions(+), 6 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/14720/2
--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 2
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-15 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 1:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/14720/1/fe/src/main/java/org/apache/impala/service/MetadataOp.java
File fe/src/main/java/org/apache/impala/service/MetadataOp.java:

http://gerrit.cloudera.org:8080/#/c/14720/1/fe/src/main/java/org/apache/impala/service/MetadataOp.java@670
PS1, Line 670: PatternMatcher schemaMatcher = 
PatternMatcher.createJdbcPatternMatcher(foreignSchemaName);
line too long (94 > 90)



--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 15 Nov 2019 23:56:25 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-15 Thread Anurag Mantripragada (Code Review)
Anurag Mantripragada has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/14720 )

Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..


Patch Set 1:

Looking for suggestions on overall approach and authorization piece. Thank you!


--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Thomas Tauber-Marshall 
Gerrit-Reviewer: Vihang Karajgaonkar 
Gerrit-Comment-Date: Fri, 15 Nov 2019 23:56:54 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-9104: Add support for retrieval of primary keys and foriegn keys through impala-hs2-server.

2019-11-15 Thread Anurag Mantripragada (Code Review)
Anurag Mantripragada has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/14720


Change subject: IMPALA-9104: Add support for retrieval of primary keys and 
foriegn keys through impala-hs2-server.
..

IMPALA-9104: Add support for retrieval of primary keys
and foriegn keys through impala-hs2-server.

The goal is to let JDBC clients get constraint information
from Impala tables. We implement two new metadata operations in
impala-hs2-server, GetPrimaryKeys and GetCrossReference. For these to work,
new TCLIService methods were added. In FE, these two operations are
implemented to get the information from tables in the catalog.

Much like GetColumns(), tables need to be loaded in order to be able to get
PK/FK information. We wait for the PK table/FK table to load.
In the implementation, PK/FK information is returned
ONLY if the user has access to ALL the columns involved in the PK/FK
relationship.

Testing:
- Added three test tables to our test datasets since most of our FE tests
  relied on dummy tables or testdata. It was difficult to test PK/FK with
  these methods. Also, we can build on this testdata in future when we make
  optimizer improvements.
- Added unit tests in AuthorizationTest and JDBCtest.
- Added e2e test in test_hs2.py

Caveats:
- Authorization information is not available for IncompleteTables. In
  case of foreign key tables, it is not feasible to load all the
  parent tables involved to check for column privileges. Therefore, there
  maybe missing PK/FK relationships for parent tables that are not loaded.
- Retrieval of constraints is not yet supported in LocalCatalog mode. See
  IMPALA-9158.

Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
---
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-server.h
M common/thrift/Frontend.thrift
M common/thrift/hive-1-api/TCLIService.thrift
M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/FeTable.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/MetadataOp.java
M fe/src/test/java/org/apache/impala/authorization/AuthorizationTest.java
M fe/src/test/java/org/apache/impala/service/JdbcTest.java
M testdata/datasets/functional/functional_schema_template.sql
M tests/hs2/test_hs2.py
17 files changed, 762 insertions(+), 6 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/14720/1
--
To view, visit http://gerrit.cloudera.org:8080/14720
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I8942dfbbd4a3be244eed1c61ac2ce17069960477
Gerrit-Change-Number: 14720
Gerrit-PatchSet: 1
Gerrit-Owner: Anurag Mantripragada 
Gerrit-Reviewer: Impala Public Jenkins