[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-12 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has abandoned this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Abandoned

Per conversation on user@, folks are interested in keeping this around.
--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 7
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-07 Thread Dan Hecht (Code Review) 
Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 7: Code-Review+2


--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 7
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Wed, 07 Feb 2018 16:59:37 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-07 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 7:

Rebased. Running 
https://jenkins.impala.io/job/gerrit-verify-dryrun-external/76/ now, but I 
think this is re-reviewable.


--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 7
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Wed, 07 Feb 2018 16:16:35 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-07 Thread Philip Zeyliger (Code Review) 
Hello Jim Apple, Dimitris Tsirogiannis, Alex Behm, Zach Amsden, Dan Hecht, 

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9192

to look at the new patch set (#7).

Change subject: IMPALA-6204: Remove external DataSource
..

IMPALA-6204: Remove external DataSource

Removes DataSourceScanNode, external data sources, and all affiliated
code, tests, and documentation.

When a data source table is encountered, we now throw an exception. To
the user, this looks like:

  [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1');
  Query: create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1')
  Fetched 0 row(s) in 0.11s
  [pannier.ca.cloudera.com:21000] > select * from t;
  Query: select * from t
  Query submitted at: 2018-02-01 17:16:26 (Coordinator: 
http://pannier.ca.cloudera.com:25000)
  ERROR: AnalysisException: Failed to load metadata for table: 't'
  CAUSED BY: TableLoadingException: Failed to load metadata for table: 
default.t. Running 'invalidate metadata default.t' may resolve this problem.
  CAUSED BY: UnsupportedOperationException: Eternal Data source table not 
supported.

A test has been added to capture this behavior.

For the most part, I deleted the unused code. In a few places, a renamed
the Thrift enums and threw errors if they're encountered. For Thrift
structs, I left a comment about the now-skipped id that used to
represent a data-source related entry.

Cherry-picks: not for 2.x

Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
---
M CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/CMakeLists.txt
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
D be/src/exec/data-source-scan-node.cc
D be/src/exec/data-source-scan-node.h
M be/src/exec/exec-node.cc
D be/src/exec/external-data-source-executor.cc
D be/src/exec/external-data-source-executor.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M bin/clean-cmake.sh
M bin/clean.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/Data.thrift
M common/thrift/Descriptors.thrift
D common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/PlanNodes.thrift
M docs/impala.ditamap
M docs/impala_keydefs.ditamap
D docs/topics/impala_create_data_source.xml
D docs/topics/impala_data_sources.xml
D docs/topics/impala_drop_data_source.xml
M docs/topics/impala_new_features.xml
M docs/topics/impala_show.xml
D ext-data-source/.gitignore
D ext-data-source/CMakeLists.txt
D ext-data-source/api/pom.xml
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java
D ext-data-source/pom.xml
D ext-data-source/sample/pom.xml
D 
ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java
D ext-data-source/test/pom.xml
D 
ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableRef.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
D fe/src/main/java/org/apache/impala/catalog/DataSource.java
D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java
D 
fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java
D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-06 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 5:

(2 comments)

Addressed the comments. Removed the remaining enums and renumbered the Thrifts.

There are some conflicts with Tianyi's metadata compression patch, so I'll post 
another patch when I work through the rebase.

http://gerrit.cloudera.org:8080/#/c/9192/5/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:

http://gerrit.cloudera.org:8080/#/c/9192/5/be/src/service/client-request-state.cc@297
PS5, Line 297:   return Status("Data Source not supported.");
> why can we even get here? doesn't the frontend reject this statement?
Done


http://gerrit.cloudera.org:8080/#/c/9192/5/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:

http://gerrit.cloudera.org:8080/#/c/9192/5/common/thrift/CatalogObjects.thrift@401
PS5, Line 401:   //  12: optional TDataSourceTable data_source_table
> I'd also prefer removing everything. I see no point in keeping stuff like t
In a previous life, this was a big no-no. The reasons are for maintaining 
compatibility between different versions of servers and servers and servers and 
clients (such as what would happen in rolling upgrade). We don't really have 
that problem.

A second reason is that if I recorded the network, say, and wanted to make 
sense of things, fields coming and going would drive me mad. In practice, we 
don't do that either.

Anyway--if our practice is to renumber, I've gone ahead and done that.



--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Wed, 07 Feb 2018 04:31:31 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-06 Thread Philip Zeyliger (Code Review) 
Hello Jim Apple, Dimitris Tsirogiannis, Alex Behm, Zach Amsden, Dan Hecht, 

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9192

to look at the new patch set (#6).

Change subject: IMPALA-6204: Remove external DataSource
..

IMPALA-6204: Remove external DataSource

Removes DataSourceScanNode, external data sources, and all affiliated
code, tests, and documentation.

When a data source table is encountered, we now throw an exception. To
the user, this looks like:

  [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1');
  Query: create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1')
  Fetched 0 row(s) in 0.11s
  [pannier.ca.cloudera.com:21000] > select * from t;
  Query: select * from t
  Query submitted at: 2018-02-01 17:16:26 (Coordinator: 
http://pannier.ca.cloudera.com:25000)
  ERROR: AnalysisException: Failed to load metadata for table: 't'
  CAUSED BY: TableLoadingException: Failed to load metadata for table: 
default.t. Running 'invalidate metadata default.t' may resolve this problem.
  CAUSED BY: UnsupportedOperationException: Eternal Data source table not 
supported.

A test has been added to capture this behavior.

For the most part, I deleted the unused code. In a few places, a renamed
the Thrift enums and threw errors if they're encountered. For Thrift
structs, I left a comment about the now-skipped id that used to
represent a data-source related entry.

Cherry-picks: not for 2.x

Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
---
M CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/CMakeLists.txt
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
D be/src/exec/data-source-scan-node.cc
D be/src/exec/data-source-scan-node.h
M be/src/exec/exec-node.cc
D be/src/exec/external-data-source-executor.cc
D be/src/exec/external-data-source-executor.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M bin/clean-cmake.sh
M bin/clean.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/Data.thrift
M common/thrift/Descriptors.thrift
D common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/PlanNodes.thrift
M docs/impala.ditamap
M docs/impala_keydefs.ditamap
D docs/topics/impala_create_data_source.xml
D docs/topics/impala_data_sources.xml
D docs/topics/impala_drop_data_source.xml
M docs/topics/impala_new_features.xml
M docs/topics/impala_show.xml
D ext-data-source/.gitignore
D ext-data-source/CMakeLists.txt
D ext-data-source/api/pom.xml
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java
D ext-data-source/pom.xml
D ext-data-source/sample/pom.xml
D 
ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java
D ext-data-source/test/pom.xml
D 
ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableRef.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
D fe/src/main/java/org/apache/impala/catalog/DataSource.java
D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java
D 
fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java
D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-06 Thread Jim Apple (Code Review) 
Jim Apple has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 5:

> > Was there ever an announcement that this was deprecated?
 >
 > Is https://issues.apache.org/jira/browse/IMPALA-6204 sufficient?

TBH, I'm not thrilled about that. For instance, there is no justification on 
that ticket about why, and no discussion of pros and cons.

While this is in review, Maybe Philip could start a discussion on user@ 
explaining why and asking for feedback. Maybe it will come back unanimous after 
a short amount of time, but I think it's worth a discussion, given how badly 
this could break workloads of existing users. (Obviously they could ETL into 
Parquet or stay on the 2.x branch.)


--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Tue, 06 Feb 2018 19:19:52 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-06 Thread Alex Behm (Code Review) 
Alex Behm has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 5:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/9192/5/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:

http://gerrit.cloudera.org:8080/#/c/9192/5/common/thrift/CatalogObjects.thrift@401
PS5, Line 401:   //  12: optional TDataSourceTable data_source_table
> what's the reasoning for not renumbering for these internal datastructures?
I'd also prefer removing everything. I see no point in keeping stuff like this 
around.



--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Tue, 06 Feb 2018 18:53:43 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-06 Thread Dan Hecht (Code Review) 
Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 5:

(2 comments)

Maybe also manually test that the various SQL related to datasource gives a 
sensible parsing error with this change?

http://gerrit.cloudera.org:8080/#/c/9192/5/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:

http://gerrit.cloudera.org:8080/#/c/9192/5/be/src/service/client-request-state.cc@297
PS5, Line 297:   return Status("Data Source not supported.");
why can we even get here? doesn't the frontend reject this statement?


http://gerrit.cloudera.org:8080/#/c/9192/5/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:

http://gerrit.cloudera.org:8080/#/c/9192/5/common/thrift/CatalogObjects.thrift@401
PS5, Line 401:   //  12: optional TDataSourceTable data_source_table
what's the reasoning for not renumbering for these internal datastructures? I 
guess I don't feel too strongly, but seems like we can accumulate cruft. (Once 
we implement rolling upgrade, then we'll probably need to be stricter.)



--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Tue, 06 Feb 2018 17:45:44 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-06 Thread Jim Apple (Code Review) 
Jim Apple has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 5:

Was there ever an announcement that this was deprecated?


--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Jim Apple 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Tue, 06 Feb 2018 17:07:46 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-05 Thread Alex Behm (Code Review) 
Alex Behm has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 5: Code-Review+1

(1 comment)

lgtm, will give others a chance to look

http://gerrit.cloudera.org:8080/#/c/9192/4/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:

http://gerrit.cloudera.org:8080/#/c/9192/4/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@50
PS4, Line 50: EQ("=", "eq"),
> extdatasources would get a serialized version of operator stuff handed over
nice



--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 5
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Mon, 05 Feb 2018 19:45:35 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-05 Thread Philip Zeyliger (Code Review) 
Hello Dimitris Tsirogiannis, Alex Behm, Zach Amsden, Dan Hecht,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9192

to look at the new patch set (#5).

Change subject: IMPALA-6204: Remove external DataSource
..

IMPALA-6204: Remove external DataSource

Removes DataSourceScanNode, external data sources, and all affiliated
code, tests, and documentation.

When a data source table is encountered, we now throw an exception. To
the user, this looks like:

  [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1');
  Query: create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1')
  Fetched 0 row(s) in 0.11s
  [pannier.ca.cloudera.com:21000] > select * from t;
  Query: select * from t
  Query submitted at: 2018-02-01 17:16:26 (Coordinator: 
http://pannier.ca.cloudera.com:25000)
  ERROR: AnalysisException: Failed to load metadata for table: 't'
  CAUSED BY: TableLoadingException: Failed to load metadata for table: 
default.t. Running 'invalidate metadata default.t' may resolve this problem.
  CAUSED BY: UnsupportedOperationException: Eternal Data source table not 
supported.

A test has been added to capture this behavior.

For the most part, I deleted the unused code. In a few places, a renamed
the Thrift enums and threw errors if they're encountered. For Thrift
structs, I left a comment about the now-skipped id that used to
represent a data-source related entry.

Cherry-picks: not for 2.x

Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
---
M CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/CMakeLists.txt
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
D be/src/exec/data-source-scan-node.cc
D be/src/exec/data-source-scan-node.h
M be/src/exec/exec-node.cc
D be/src/exec/external-data-source-executor.cc
D be/src/exec/external-data-source-executor.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M bin/clean-cmake.sh
M bin/clean.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/Data.thrift
M common/thrift/Descriptors.thrift
D common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/PlanNodes.thrift
M docs/impala.ditamap
M docs/impala_keydefs.ditamap
D docs/topics/impala_create_data_source.xml
D docs/topics/impala_data_sources.xml
D docs/topics/impala_drop_data_source.xml
M docs/topics/impala_new_features.xml
M docs/topics/impala_show.xml
D ext-data-source/.gitignore
D ext-data-source/CMakeLists.txt
D ext-data-source/api/pom.xml
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java
D ext-data-source/pom.xml
D ext-data-source/sample/pom.xml
D 
ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java
D ext-data-source/test/pom.xml
D 
ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableRef.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
D fe/src/main/java/org/apache/impala/catalog/DataSource.java
D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java
D 
fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java
D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-05 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 4:

(1 comment)

I've been generally hesitant of removing anything from Thrift, from old 
protocol buffer days, but I think you all are right that nothing is serialized, 
so it's safe to remove.

>> How can users identify and purge their HMS after this change?

The table property __IMPALA_DATA_SOURCE_NAME will cause a table to be 
unqueryable. That code is new, simple, and I think has to be in there.

Do you think that's sufficient?

http://gerrit.cloudera.org:8080/#/c/9192/4/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:

http://gerrit.cloudera.org:8080/#/c/9192/4/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@50
PS4, Line 50: EQ("=", "eq"),
> what happened here?
extdatasources would get a serialized version of operator stuff handed over to 
them for push downs (extdatasource.thrift.TComparisonOp). Since extdatasources 
no longer exist, the mapping between operators and TComparisonOp is no longer 
necessary.



--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Mon, 05 Feb 2018 19:08:05 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-05 Thread Zach Amsden (Code Review) 
Zach Amsden has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 4:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/9192/4//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/9192/4//COMMIT_MSG@23
PS4, Line 23:   CAUSED BY: UnsupportedOperationException: Eternal Data source 
table not supported.
Eternal Data source?


http://gerrit.cloudera.org:8080/#/c/9192/4//COMMIT_MSG@27
PS4, Line 27: For the most part, I deleted the unused code. In a few places, a 
renamed
I renamed


http://gerrit.cloudera.org:8080/#/c/9192/4/common/thrift/CatalogObjects.thrift
File common/thrift/CatalogObjects.thrift:

http://gerrit.cloudera.org:8080/#/c/9192/4/common/thrift/CatalogObjects.thrift@38
PS4, Line 38:   DEPRECATED_DATA_SOURCE, // removed in Impala 3.0
I'm not sure I see the point of keeping this around instead of just totally 
deleting it.


http://gerrit.cloudera.org:8080/#/c/9192/4/common/thrift/CatalogObjects.thrift@48
PS4, Line 48:   DEPRECATED_DATA_SOURCE_TABLE, // removed in Impala 3.0
Same comment.


http://gerrit.cloudera.org:8080/#/c/9192/4/common/thrift/PlanNodes.thrift
File common/thrift/PlanNodes.thrift:

http://gerrit.cloudera.org:8080/#/c/9192/4/common/thrift/PlanNodes.thrift@43
PS4, Line 43:   DEPRECATED_DATA_SOURCE_NODE, // removed in Impala 3.0
Again, why do we need this?



--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Reviewer: Zach Amsden 
Gerrit-Comment-Date: Mon, 05 Feb 2018 19:01:33 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-05 Thread Alex Behm (Code Review) 
Alex Behm has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 4:

(2 comments)

High-level questions:
- Why are we still keeping datasource related enums and code around?
- How can users identify and purge their HMS after this change?

http://gerrit.cloudera.org:8080/#/c/9192/4/be/src/exec/catalog-op-executor.cc
File be/src/exec/catalog-op-executor.cc:

http://gerrit.cloudera.org:8080/#/c/9192/4/be/src/exec/catalog-op-executor.cc@71
PS4, Line 71: } else if (request.ddl_params.ddl_type == 
TDdlType::DEPRECATED_DROP_DATA_SOURCE) {
Why would we keep this around? SQL that might go through this path doesn't even 
parse anymore.

There are a few other places like this where I don't understand why we'd want 
to keep them.


http://gerrit.cloudera.org:8080/#/c/9192/4/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
File fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java:

http://gerrit.cloudera.org:8080/#/c/9192/4/fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java@50
PS4, Line 50: EQ("=", "eq"),
what happened here?



--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Alex Behm 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Mon, 05 Feb 2018 18:53:07 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-05 Thread Dan Hecht (Code Review) 
Dan Hecht has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 4:

Given that data sources weren't persisted to the HMS, why do we still need the 
various internal enums?  Isn't returning an error on the various DATA SOURCE 
related SQL statements (like CREATE DATA SOURCE) enough, and then everything 
else can be deleted?


--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Dan Hecht 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Mon, 05 Feb 2018 18:44:14 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-05 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 4:

https://jenkins.impala.io/job/gerrit-verify-dryrun-external/74/ running. Last 
commit was missing a 1-line "git amend"; whoops.


--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 4
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Mon, 05 Feb 2018 18:01:14 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-05 Thread Philip Zeyliger (Code Review) 
Hello Dimitris Tsirogiannis,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9192

to look at the new patch set (#4).

Change subject: IMPALA-6204: Remove external DataSource
..

IMPALA-6204: Remove external DataSource

Removes DataSourceScanNode, external data sources, and all affiliated
code, tests, and documentation.

When a data source table is encountered, we now throw an exception. To
the user, this looks like:

  [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1');
  Query: create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1')
  Fetched 0 row(s) in 0.11s
  [pannier.ca.cloudera.com:21000] > select * from t;
  Query: select * from t
  Query submitted at: 2018-02-01 17:16:26 (Coordinator: 
http://pannier.ca.cloudera.com:25000)
  ERROR: AnalysisException: Failed to load metadata for table: 't'
  CAUSED BY: TableLoadingException: Failed to load metadata for table: 
default.t. Running 'invalidate metadata default.t' may resolve this problem.
  CAUSED BY: UnsupportedOperationException: Eternal Data source table not 
supported.

A test has been added to capture this behavior.

For the most part, I deleted the unused code. In a few places, a renamed
the Thrift enums and threw errors if they're encountered. For Thrift
structs, I left a comment about the now-skipped id that used to
represent a data-source related entry.

Cherry-picks: not for 2.x

Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
---
M CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/CMakeLists.txt
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
D be/src/exec/data-source-scan-node.cc
D be/src/exec/data-source-scan-node.h
M be/src/exec/exec-node.cc
D be/src/exec/external-data-source-executor.cc
D be/src/exec/external-data-source-executor.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M bin/clean-cmake.sh
M bin/clean.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/Data.thrift
M common/thrift/Descriptors.thrift
D common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/PlanNodes.thrift
M docs/impala.ditamap
D docs/topics/impala_create_data_source.xml
D docs/topics/impala_data_sources.xml
D docs/topics/impala_drop_data_source.xml
D ext-data-source/.gitignore
D ext-data-source/CMakeLists.txt
D ext-data-source/api/pom.xml
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java
D ext-data-source/pom.xml
D ext-data-source/sample/pom.xml
D 
ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java
D ext-data-source/test/pom.xml
D 
ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableRef.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
D fe/src/main/java/org/apache/impala/catalog/DataSource.java
D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java
D 
fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java
D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/jflex/sql-scanner.flex
M

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-02 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 3:

I think this is now reviewable. I ran core tests successfully with this before 
rebasing. I'm running them again after rebase 
(https://jenkins.impala.io/job/gerrit-verify-dryrun-external/71/). There were 
some conflicts with the keyword change, but nothing interesting.

BTW: +95, -4667!!!


--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 3
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Fri, 02 Feb 2018 22:24:01 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-02 Thread Philip Zeyliger (Code Review) 
Hello Dimitris Tsirogiannis,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9192

to look at the new patch set (#3).

Change subject: IMPALA-6204: Remove external DataSource
..

IMPALA-6204: Remove external DataSource

Removes DataSourceScanNode, external data sources, and all affiliated
code, tests, and documentation.

When a data source table is encountered, we now throw an exception. To
the user, this looks like:

  [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1');
  Query: create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1')
  Fetched 0 row(s) in 0.11s
  [pannier.ca.cloudera.com:21000] > select * from t;
  Query: select * from t
  Query submitted at: 2018-02-01 17:16:26 (Coordinator: 
http://pannier.ca.cloudera.com:25000)
  ERROR: AnalysisException: Failed to load metadata for table: 't'
  CAUSED BY: TableLoadingException: Failed to load metadata for table: 
default.t. Running 'invalidate metadata default.t' may resolve this problem.
  CAUSED BY: UnsupportedOperationException: Eternal Data source table not 
supported.

A test has been added to capture this behavior.

For the most part, I deleted the unused code. In a few places, a renamed
the Thrift enums and threw errors if they're encountered. For Thrift
structs, I left a comment about the now-skipped id that used to
represent a data-source related entry.

Cherry-picks: not for 2.x

Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
---
M CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/CMakeLists.txt
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
D be/src/exec/data-source-scan-node.cc
D be/src/exec/data-source-scan-node.h
M be/src/exec/exec-node.cc
D be/src/exec/external-data-source-executor.cc
D be/src/exec/external-data-source-executor.h
M be/src/runtime/descriptors.cc
M be/src/runtime/descriptors.h
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M bin/clean-cmake.sh
M bin/clean.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/Data.thrift
M common/thrift/Descriptors.thrift
D common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/PlanNodes.thrift
M docs/impala.ditamap
D docs/topics/impala_create_data_source.xml
D docs/topics/impala_data_sources.xml
D docs/topics/impala_drop_data_source.xml
D ext-data-source/.gitignore
D ext-data-source/CMakeLists.txt
D ext-data-source/api/pom.xml
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java
D ext-data-source/pom.xml
D ext-data-source/sample/pom.xml
D 
ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java
D ext-data-source/test/pom.xml
D 
ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableRef.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
D fe/src/main/java/org/apache/impala/catalog/DataSource.java
D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java
D 
fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java
D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/jflex/sql-scanner.flex
M

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-02 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/9192 )

Change subject: IMPALA-6204: Remove external DataSource
..


Patch Set 2: Code-Review-1

Sorry, hold off on this please. I rebased and then did some more grepping and 
found some more tentacles!


--
To view, visit http://gerrit.cloudera.org:8080/9192
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
Gerrit-Change-Number: 9192
Gerrit-PatchSet: 2
Gerrit-Owner: Philip Zeyliger 
Gerrit-Reviewer: Dimitris Tsirogiannis 
Gerrit-Reviewer: Philip Zeyliger 
Gerrit-Comment-Date: Fri, 02 Feb 2018 18:33:27 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-02 Thread Philip Zeyliger (Code Review) 
Hello Dimitris Tsirogiannis,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/9192

to look at the new patch set (#2).

Change subject: IMPALA-6204: Remove external DataSource
..

IMPALA-6204: Remove external DataSource

Removes DataSourceScanNode, external data sources, and all affiliated
code, tests, and documentation.

When a data source table is encountered, we now throw an exception. To
the user, this looks like:

  [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1');
  Query: create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1')
  Fetched 0 row(s) in 0.11s
  [pannier.ca.cloudera.com:21000] > select * from t;
  Query: select * from t
  Query submitted at: 2018-02-01 17:16:26 (Coordinator: 
http://pannier.ca.cloudera.com:25000)
  ERROR: AnalysisException: Failed to load metadata for table: 't'
  CAUSED BY: TableLoadingException: Failed to load metadata for table: 
default.t. Running 'invalidate metadata default.t' may resolve this problem.
  CAUSED BY: UnsupportedOperationException: Eternal Data source table not 
supported.

A test has been added to capture this behavior.

For the most part, I deleted the unused code. In a few places, a renamed
the Thrift enums and threw errors if they're encountered. For Thrift
structs, I left a comment about the now-skipped id that used to
represent a data-source related entry.

Cherry-picks: not for 2.x

Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
---
M CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/CMakeLists.txt
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
D be/src/exec/data-source-scan-node.cc
D be/src/exec/data-source-scan-node.h
M be/src/exec/exec-node.cc
D be/src/exec/external-data-source-executor.cc
D be/src/exec/external-data-source-executor.h
M be/src/runtime/descriptors.cc
M be/src/scheduling/scheduler.cc
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M bin/clean-cmake.sh
M bin/clean.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/Data.thrift
M common/thrift/Descriptors.thrift
D common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/PlanNodes.thrift
M docs/impala.ditamap
D docs/topics/impala_create_data_source.xml
D docs/topics/impala_data_sources.xml
D docs/topics/impala_drop_data_source.xml
D ext-data-source/.gitignore
D ext-data-source/CMakeLists.txt
D ext-data-source/api/pom.xml
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java
D ext-data-source/pom.xml
D ext-data-source/sample/pom.xml
D 
ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java
D ext-data-source/test/pom.xml
D 
ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
D fe/src/main/java/org/apache/impala/catalog/DataSource.java
D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java
D 
fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java
D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M

[Impala-ASF-CR] IMPALA-6204: Remove external DataSource

2018-02-02 Thread Philip Zeyliger (Code Review) 
Philip Zeyliger has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/9192


Change subject: IMPALA-6204: Remove external DataSource
..

IMPALA-6204: Remove external DataSource

Removes DataSourceScanNode, external data sources, and all affiliated
code, tests, and documentation.

When a data source table is encountered, we now throw an exception. To
the user, this looks like:

  [pannier.ca.cloudera.com:21000] > create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1');
  Query: create table t (x int) stored as textfile 
tblproperties('__IMPALA_DATA_SOURCE_NAME'='V1')
  Fetched 0 row(s) in 0.11s
  [pannier.ca.cloudera.com:21000] > select * from t;
  Query: select * from t
  Query submitted at: 2018-02-01 17:16:26 (Coordinator: 
http://pannier.ca.cloudera.com:25000)
  ERROR: AnalysisException: Failed to load metadata for table: 't'
  CAUSED BY: TableLoadingException: Failed to load metadata for table: 
default.t. Running 'invalidate metadata default.t' may resolve this problem.
  CAUSED BY: UnsupportedOperationException: Eternal Data source table not 
supported.

A test has been added to capture this behavior.

For the most part, I deleted the unused code. In a few places, a renamed
the Thrift enums and threw errors if they're encountered. For Thrift
structs, I left a comment about the now-skipped id that used to
represent a data-source related entry.

Cherry-picks: not for 2.x

Change-Id: I02a3a6740466ed7372b71d948c705b30886dcfb6
---
M CMakeLists.txt
M be/generated-sources/gen-cpp/CMakeLists.txt
M be/src/catalog/catalog-util.cc
M be/src/exec/CMakeLists.txt
M be/src/exec/catalog-op-executor.cc
M be/src/exec/catalog-op-executor.h
D be/src/exec/data-source-scan-node.cc
D be/src/exec/data-source-scan-node.h
M be/src/exec/exec-node.cc
D be/src/exec/external-data-source-executor.cc
D be/src/exec/external-data-source-executor.h
M be/src/service/client-request-state.cc
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M bin/clean-cmake.sh
M bin/clean.sh
M buildall.sh
M common/thrift/CMakeLists.txt
M common/thrift/CatalogObjects.thrift
M common/thrift/CatalogService.thrift
M common/thrift/Data.thrift
M common/thrift/Descriptors.thrift
D common/thrift/ExternalDataSource.thrift
M common/thrift/Frontend.thrift
M common/thrift/JniCatalog.thrift
M common/thrift/PlanNodes.thrift
M docs/impala.ditamap
D docs/topics/impala_create_data_source.xml
D docs/topics/impala_data_sources.xml
D docs/topics/impala_drop_data_source.xml
D ext-data-source/.gitignore
D ext-data-source/CMakeLists.txt
D ext-data-source/api/pom.xml
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/util/SerializationUtils.java
D 
ext-data-source/api/src/main/java/org/apache/impala/extdatasource/v1/ExternalDataSource.java
D ext-data-source/pom.xml
D ext-data-source/sample/pom.xml
D 
ext-data-source/sample/src/main/java/org/apache/impala/extdatasource/sample/EchoDataSource.java
D ext-data-source/test/pom.xml
D 
ext-data-source/test/src/main/java/org/apache/impala/extdatasource/AllTypesDataSource.java
M fe/CMakeLists.txt
M fe/pom.xml
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
M fe/src/main/java/org/apache/impala/analysis/Analyzer.java
M fe/src/main/java/org/apache/impala/analysis/BinaryPredicate.java
D fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java
D fe/src/main/java/org/apache/impala/analysis/DropDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java
D fe/src/main/java/org/apache/impala/analysis/ShowDataSrcsStmt.java
M fe/src/main/java/org/apache/impala/catalog/Catalog.java
M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java
D fe/src/main/java/org/apache/impala/catalog/DataSource.java
D fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java
M fe/src/main/java/org/apache/impala/catalog/Table.java
D fe/src/main/java/org/apache/impala/extdatasource/ApiVersion.java
D 
fe/src/main/java/org/apache/impala/extdatasource/ExternalDataSourceExecutor.java
D fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/AnalyzeStmtsTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java