[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up

2024-02-25 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21071 )

Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to 
aborted write id is cleaned up
..


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10304/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/21071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Gerrit-Change-Number: 21071
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Anonymous Coward 
Gerrit-Reviewer: Csaba Ringhofer 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Reviewer: Sai Hemanth Gantasala 
Gerrit-Comment-Date: Mon, 26 Feb 2024 04:16:09 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-12771: Impala catalogd events-skipped may mark the wrong number

2024-02-25 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21045 )

Change subject: IMPALA-12771: Impala catalogd events-skipped may mark the wrong 
number
..


Patch Set 1: Verified-1

Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/10303/


--
To view, visit http://gerrit.cloudera.org:8080/21045
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7aeb04e999b82187eb138c0b643ead259da22f1a
Gerrit-Change-Number: 21045
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Mon, 26 Feb 2024 03:56:00 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up

2024-02-25 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21071 )

Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to 
aborted write id is cleaned up
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/15312/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Gerrit-Change-Number: 21071
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Mon, 26 Feb 2024 03:19:53 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up

2024-02-25 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21071 )

Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to 
aborted write id is cleaned up
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/15311/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Gerrit-Change-Number: 21071
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Mon, 26 Feb 2024 03:04:21 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up

2024-02-25 Thread Quanlong Huang (Code Review)
Hello Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/21071

to look at the new patch set (#2).

Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to 
aborted write id is cleaned up
..

IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id 
is cleaned up

HdfsTable tracks the ValidWriteIdList from HMS. When the table is
reloaded, the ValidWriteIdList is updated to the latest state. An
ABORT_TXN event that is lagging behind could match to aborted write ids
that have already been cleaned up by the HMS housekeeping thread. Such
write ids can't be found in the cached ValidWriteIdList as opened or
aboorted write ids. This hits a Precondition check and fails the event
processing.

This patch fixes the check to allow this case. Also adds more logs for
dealing with write ids.

Tests
 - Add custom-cluster test to start Hive with the housekeeping thread
   turned on and verify such ABORT_TXN event is processed correctly.

Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
---
M bin/create-test-configuration.sh
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
M fe/src/test/resources/hive-site.xml.py
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_events_custom_configs.py
8 files changed, 80 insertions(+), 10 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21071/2
--
To view, visit http://gerrit.cloudera.org:8080/21071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Gerrit-Change-Number: 21071
Gerrit-PatchSet: 2
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 


[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up

2024-02-25 Thread Quanlong Huang (Code Review)
Quanlong Huang has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/21071


Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to 
aborted write id is cleaned up
..

IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id 
is cleaned up

HdfsTable tracks the ValidWriteIdList from HMS. When the table is
reloaded, the ValidWriteIdList is updated to the latest state. An
ABORT_TXN event that is lagging behind could match to aborted write ids
that have already been cleaned up by the HMS housekeeping thread. Such
write ids can't be found in the cached ValidWriteIdList as opened or
aboorted write ids. This hits a Precondition check and fails the event
processing.

This patch fixes the check to allow this case. Also adds more logs for
dealing with write ids.

Tests
 - Add custom-cluster test to start Hive with the housekeeping thread
   turned on and verify such ABORT_TXN event is processed correctly.

Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
---
M bin/create-test-configuration.sh
M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java
M fe/src/main/java/org/apache/impala/catalog/events/MetastoreEvents.java
M 
fe/src/main/java/org/apache/impala/hive/common/MutableValidReaderWriteIdList.java
M fe/src/test/resources/hive-site.xml.py
M tests/common/impala_test_suite.py
M tests/custom_cluster/test_events_custom_configs.py
8 files changed, 78 insertions(+), 10 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/21071/1
--
To view, visit http://gerrit.cloudera.org:8080/21071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Gerrit-Change-Number: 21071
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang 


[Impala-ASF-CR] IMPALA-12827: Fix failures in processing AbortTxnEvent due to aborted write id is cleaned up

2024-02-25 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21071 )

Change subject: IMPALA-12827: Fix failures in processing AbortTxnEvent due to 
aborted write id is cleaned up
..


Patch Set 1:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/21071/1/bin/create-test-configuration.sh
File bin/create-test-configuration.sh:

http://gerrit.cloudera.org:8080/#/c/21071/1/bin/create-test-configuration.sh@158
PS1, Line 158: ln -s "${CONFIG_DIR}/hive-site_housekeeping_on.xml" 
hive-site-housekeeping-on/hive-site.xml
line too long (91 > 90)


http://gerrit.cloudera.org:8080/#/c/21071/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java
File fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java:

http://gerrit.cloudera.org:8080/#/c/21071/1/fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java@842
PS1, Line 842:   LOG.info("EventId: {} EventType: COMMIT_TXN transaction 
id: {}", getEventId(), txnId_);
line too long (93 > 90)


http://gerrit.cloudera.org:8080/#/c/21071/1/tests/custom_cluster/test_events_custom_configs.py
File tests/custom_cluster/test_events_custom_configs.py:

http://gerrit.cloudera.org:8080/#/c/21071/1/tests/custom_cluster/test_events_custom_configs.py@1188
PS1, Line 1188: \
flake8: E502 the backslash is redundant between brackets



--
To view, visit http://gerrit.cloudera.org:8080/21071
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I93b6f684d6e4b94961d804a0c022029249873681
Gerrit-Change-Number: 21071
Gerrit-PatchSet: 1
Gerrit-Owner: Quanlong Huang 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Comment-Date: Mon, 26 Feb 2024 02:39:41 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-12818: Intermediate Result Caching plan node framework

2024-02-25 Thread Yida Wu (Code Review)
Yida Wu has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21035 )

Change subject: IMPALA-12818: Intermediate Result Caching plan node framework
..


Patch Set 3:

(10 comments)

http://gerrit.cloudera.org:8080/#/c/21035/3/be/src/exec/tuple-cache-node.cc
File be/src/exec/tuple-cache-node.cc:

http://gerrit.cloudera.org:8080/#/c/21035/3/be/src/exec/tuple-cache-node.cc@28
PS3, Line 28: allow_tuple_caching
It seems the flag is not used in the class, or do I miss something?


http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:

http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/main/java/org/apache/impala/planner/PlanNode.java@530
PS3, Line 530: tupleCacheInfo == null
Could you please add a comment to describe when or why the tupleCacheInfo is 
null


http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/main/java/org/apache/impala/planner/PlanNode.java@1237
PS3, Line 1237: recurse to children
For clarity, maybe it is better to have a full sentence like, "Recursively 
compute tuple cache info for children nodes"


http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/main/java/org/apache/impala/planner/PlanNode.java@1251
PS3, Line 1251: TODO
Would be good if we have a Jira to follow the TODOs


http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java
File fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java:

http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java@59
PS3, Line 59: String
May consider StringBuilder for hashTrace_ due to string concatenations in 
mergeChild(), StringBuilder could be more efficient


http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/main/java/org/apache/impala/planner/TupleCacheInfo.java@94
PS3, Line 94: TODO
Would be good if we have a Jira to follow the TODOs


http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java
File fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java:

http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/main/java/org/apache/impala/planner/TupleCachePlanner.java@75
PS3, Line 75: TODO
Would be good if we have a Jira to follow the TODOs


http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java
File fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java:

http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java@59
PS3, Line 59: // TODO: incorporate column information into the key
: // verifyDifferentCacheKeys("select id from 
functional.alltypes",
: // "sele
May simply remove the case from the testcase for the commit if we don't support 
it yet


http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java@69
PS3, Line 69: TODO
same as above


http://gerrit.cloudera.org:8080/#/c/21035/3/fe/src/test/java/org/apache/impala/planner/TupleCacheTest.java@79
PS3, Line 79: // TODO: Random functions should make a location ineligible
same as above



--
To view, visit http://gerrit.cloudera.org:8080/21035
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia1f36a87dcce6efd5d1e1f0bc04009bf009b1961
Gerrit-Change-Number: 21035
Gerrit-PatchSet: 3
Gerrit-Owner: Joe McDonnell 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Kurt Deschler 
Gerrit-Reviewer: Yida Wu 
Gerrit-Comment-Date: Mon, 26 Feb 2024 02:35:12 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-12771: Impala catalogd events-skipped may mark the wrong number

2024-02-25 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21045 )

Change subject: IMPALA-12771: Impala catalogd events-skipped may mark the wrong 
number
..


Patch Set 1:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10303/ 
DRY_RUN=true


--
To view, visit http://gerrit.cloudera.org:8080/21045
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7aeb04e999b82187eb138c0b643ead259da22f1a
Gerrit-Change-Number: 21045
Gerrit-PatchSet: 1
Gerrit-Owner: Anonymous Coward 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Quanlong Huang 
Gerrit-Comment-Date: Sun, 25 Feb 2024 23:25:32 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-12842: Make node.js installation aware of CPU architecture

2024-02-25 Thread Laszlo Gaal (Code Review)
Laszlo Gaal has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21070 )

Change subject: IMPALA-12842: Make node.js installation aware of CPU 
architecture
..


Patch Set 1: Code-Review+1

JS_TESTs passed on an ARM-based instance on private infrastructure.


--
To view, visit http://gerrit.cloudera.org:8080/21070
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie22da6237ed6d19cf8af721df471033fdfe6b23a
Gerrit-Change-Number: 21070
Gerrit-PatchSet: 1
Gerrit-Owner: Laszlo Gaal 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Laszlo Gaal 
Gerrit-Reviewer: Michael Smith 
Gerrit-Reviewer: Surya Hebbar 
Gerrit-Comment-Date: Sun, 25 Feb 2024 21:01:15 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source

2024-02-25 Thread Impala Public Jenkins (Code Review)
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21016 )

Change subject: IMPALA-12793: Create JDBC table without data source
..


Patch Set 6:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/15310/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/21016
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Gerrit-Change-Number: 21016
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Sun, 25 Feb 2024 18:42:23 +
Gerrit-HasComments: No


[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source

2024-02-25 Thread Wenzhe Zhou (Code Review)
Wenzhe Zhou has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/21016 )

Change subject: IMPALA-12793: Create JDBC table without data source
..


Patch Set 6:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/21016/5/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
File fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java:

http://gerrit.cloudera.org:8080/#/c/21016/5/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@180
PS5, Line 180: // DataSourceTable is still represented as HDFS tables in 
the metastore but has a
 : // special table property __IMPALA_DATA_SOURCE_NAME to 
indicate that Impala should
 : // use an external data source so we need to add the table 
property with the name
 : //
> I'm not sure we want to restrict the table properties to only the ones that
Changed to check required JDBC parameters.


http://gerrit.cloudera.org:8080/#/c/21016/5/fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java@285
PS5, Line 285: ed by JDBC. Missing required metadata
> Nit: For the JDBC table case, can we pass in dataSourceName as IMPALA_BUILT
Done


http://gerrit.cloudera.org:8080/#/c/21016/5/fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java:

http://gerrit.cloudera.org:8080/#/c/21016/5/fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java@123
PS5, Line 123:  TableLoadingException(String.format(
> See comment from DataSourceTable.java
Done


http://gerrit.cloudera.org:8080/#/c/21016/5/fe/src/main/java/org/apache/impala/util/JsonUtil.java
File fe/src/main/java/org/apache/impala/util/JsonUtil.java:

http://gerrit.cloudera.org:8080/#/c/21016/5/fe/src/main/java/org/apache/impala/util/JsonUtil.java@43
PS5, Line 43: JSON
> Nit: I would prefer either "Json" (lowercase s) or "JSON" (all caps). Betwe
Done


http://gerrit.cloudera.org:8080/#/c/21016/5/tests/custom_cluster/test_ext_data_sources.py
File tests/custom_cluster/test_ext_data_sources.py:

http://gerrit.cloudera.org:8080/#/c/21016/5/tests/custom_cluster/test_ext_data_sources.py@a66
PS5, Line 66:
:
:
:
:
:
:
:
:
:
:
> Do we need to remove this?
Put back this unit-test



--
To view, visit http://gerrit.cloudera.org:8080/21016
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Gerrit-Change-Number: 21016
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh 
Gerrit-Comment-Date: Sun, 25 Feb 2024 18:17:16 +
Gerrit-HasComments: Yes


[Impala-ASF-CR] IMPALA-12793: Create JDBC table without data source

2024-02-25 Thread Wenzhe Zhou (Code Review)
Wenzhe Zhou has uploaded a new patch set (#6). ( 
http://gerrit.cloudera.org:8080/21016 )

Change subject: IMPALA-12793: Create JDBC table without data source
..

IMPALA-12793: Create JDBC table without data source

This patch changes syntax of creating JDBC table statement as
  CREATE TABLE [IF NOT EXISTS] [db_name.]table_name
  (col_name data_type
[constraint_specification]
[COMMENT 'col_comment']
[, ...]
  )
  [COMMENT 'table_comment']
  STORED BY JDBC
  TBLPROPERTIES ('key1'='value1', 'key2'='value2', ...)

Both "STORED BY JDBC" and "STORED AS JDBC" are acceptable. A table
property '__IMPALA_DATA_SOURCE_NAME' is added to the JDBC table with
value 'impalajdbcdatasource', which is shown in the output of command
'show create table'.
Following required JDBC parameters must be specified as table
properties: database.type, jdbc.url, jdbc.driver, driver.url, and table.
Otherwise, AnalysisException will be thrown.

Testing:
 - Added frontend unit tests for new syntax of creating JDBC table.
 - Updated end-to-end unit tests to create JDBC tables without data
   source.
 - Passed core tests

Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
---
M be/src/service/query-options-test.cc
M common/thrift/CatalogObjects.thrift
M fe/src/main/cup/sql-parser.cup
M fe/src/main/java/org/apache/impala/analysis/CreateDataSrcStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeFileStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java
M fe/src/main/java/org/apache/impala/analysis/CreateTableStmt.java
M fe/src/main/java/org/apache/impala/analysis/TableDef.java
M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java
M fe/src/main/java/org/apache/impala/catalog/HdfsFileFormat.java
M fe/src/main/java/org/apache/impala/catalog/local/LocalDataSourceTable.java
A fe/src/main/java/org/apache/impala/util/JsonUtil.java
M fe/src/main/jflex/sql-scanner.flex
M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java
M fe/src/test/java/org/apache/impala/analysis/ParserTest.java
M fe/src/test/java/org/apache/impala/customcluster/LdapHS2Test.java
M testdata/bin/create-ext-data-source-table.sql
M 
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables-predicates.test
M 
testdata/workloads/functional-query/queries/QueryTest/impala-ext-jdbc-tables.test
M 
testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source-with-keystore.test
M testdata/workloads/functional-query/queries/QueryTest/jdbc-data-source.test
M 
testdata/workloads/functional-query/queries/QueryTest/mysql-ext-jdbc-tables.test
M testdata/workloads/functional-query/queries/QueryTest/set.test
M tests/custom_cluster/test_ext_data_sources.py
M tests/query_test/test_ext_data_sources.py
26 files changed, 476 insertions(+), 370 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/16/21016/6
--
To view, visit http://gerrit.cloudera.org:8080/21016
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I765aa86b430246786ad85ab6857cefaf4332c920
Gerrit-Change-Number: 21016
Gerrit-PatchSet: 6
Gerrit-Owner: Wenzhe Zhou 
Gerrit-Reviewer: Abhishek Rawat 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Joe McDonnell 
Gerrit-Reviewer: Wenzhe Zhou 
Gerrit-Reviewer: gaurav singh