[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.05.patch, 
> HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: (was: HIVE-22566.04.patch)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.05.patch, 
> HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: (was: HIVE-22566.03.patch)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.05.patch, 
> HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Status: Open  (was: Patch Available)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.03.patch, 
> HIVE-22566.04.patch, HIVE-22566.05.patch, HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: HIVE-22566.05.patch
Status: Patch Available  (was: Open)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.03.patch, 
> HIVE-22566.04.patch, HIVE-22566.05.patch, HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: HIVE-22566.04.patch
Status: Patch Available  (was: Open)

Resubmitting

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.03.patch, 
> HIVE-22566.04.patch, HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Status: Open  (was: Patch Available)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.03.patch, 
> HIVE-22566.04.patch, HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: (was: HIVE-22566.04.patch)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.03.patch, 
> HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: HIVE-22566.04.patch
Status: Patch Available  (was: Open)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.03.patch, 
> HIVE-22566.04.patch, HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Status: Open  (was: Patch Available)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.03.patch, 
> HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: HIVE-22566.03.patch
Status: Patch Available  (was: Open)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.03.patch, 
> HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Status: Open  (was: Patch Available)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 2019-12-02T20:00:25,545 INFO  

[jira] [Commented] (HIVE-22956) Fix checking if a table is used by a materialized view before dropping

[Pablo Junge (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-22956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17050219#comment-17050219
 ] 

Pablo Junge commented on HIVE-22956:


Thanks [~mgergely], will do.

> Fix checking if a table is used by a materialized view before dropping
> --
>
> Key: HIVE-22956
> URL: https://issues.apache.org/jira/browse/HIVE-22956
> Project: Hive
>  Issue Type: Bug
>  Components: Hive
>Reporter: Miklos Gergely
>Assignee: Miklos Gergely
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22956.01.patch, HIVE-22956.02.patch
>
>
> Currently when a table is dropped we are deriving if it was used by any 
> materialized views from the error message got from the Metastore. Instead of 
> this we should check it in advance, from the HiveMaterializedViewsRegistry.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: HIVE-22566.02.patch
Status: Patch Available  (was: Open)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.02.patch, HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Status: Open  (was: Patch Available)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
   Attachment: HIVE-22566.patch
Fix Version/s: 4.0.0
   Status: Patch Available  (was: Open)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: (was: HIVE-22566.patch)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: metastore.HiveMetaStore 
> (HiveMetaStore.java:logInfo(907)) - 196: 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Status: Open  (was: Patch Available)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: metastore.HiveMetaStore 
> (HiveMetaStore.java:logInfo(907)) - 196: 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: HIVE-22566.patch
Status: Patch Available  (was: Open)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Attachments: HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 2019-12-02T20:00:25,545 INFO  

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: (was: HIVE-22566.patch)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: metastore.HiveMetaStore 
> (HiveMetaStore.java:logInfo(907)) - 196: 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Status: Open  (was: Patch Available)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Attachments: HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: metastore.HiveMetaStore 
> 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Attachment: HIVE-22566.patch
Status: Patch Available  (was: Open)

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Attachments: HIVE-22566.patch
>
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 2019-12-02T20:00:25,545 INFO  

[jira] [Assigned] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge reassigned HIVE-22566:
--

Assignee: Pablo Junge

> Drop table involved in materialized view leaves the table in inconsistent 
> state
> ---
>
> Key: HIVE-22566
> URL: https://issues.apache.org/jira/browse/HIVE-22566
> Project: Hive
>  Issue Type: Bug
>  Components: HiveServer2
>Affects Versions: 3.1.0
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
>
> If you try dropping a table which is part of the definition of a created 
> materialized view, the table is not dropped, which is the desired state as it 
> is part of the materialized view.
> However, there was a "drop" call to the table, so it tried to drop it but did 
> not succeed, leaving it in an inconsistent state.
>  
> Repro:
> ---
> 1) Create tables:
>  
> {code:java}
> CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary 
> FLOAT,  hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES 
> ('transactional'='true'); 
> CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
> INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
> {code}
>  
> 2) Create the VM:
>  
> {code:java}
> CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
> JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
> {code}
>  
> 3) Following is in backend database at this point:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |    81 | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
>  
> 4) Let's drop the 'emps' table:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
> INFO  : Compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 0.05 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): 
> drop table emps
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); 
> Time taken: 10.281 seconds
> INFO  : OK
> No rows affected (16.949 seconds)
> {code}
> No issue displayed
>  
> 5) List tables:
>  
> {code:java}
> 0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
> INFO  : Compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Semantic Analysis Completed (retrial = false)
> INFO  : Returning Hive schema: 
> Schema(fieldSchemas:[FieldSchema(name:tab_name, type:string, comment:from 
> deserializer)], properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.041 seconds
> INFO  : Executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): 
> show tables
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); 
> Time taken: 0.016 seconds
> INFO  : OK
> +---+
> | tab_name  |
> +---+
> | depts |
> | emps  |
> +---+
> 2 rows selected (0.08 seconds)
> {code}
>  
> 6) Now, from the backend-db point of view:
>  
> {code:java}
> mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where 
> DB_ID=16;
> ++---+---+--+---+
> | TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
> ++---+---+--+---+
> |     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
> |     83 |    16 |    83 | depts    | MANAGED_TABLE     |
> |     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
> ++---+---+--+---+
> 3 rows in set (0.00 sec)
> {code}
> The table is left with NULL in SD_ID, making it not available.
>  
> 7) From Metastore.log
>  
> {code:java}
> 2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: metastore.HiveMetaStore 
> (HiveMetaStore.java:logInfo(907)) - 196: 

[jira] [Commented] (HIVE-22587) hive.stats.ndv.error parameter documentation issue in HiveConf.java

[Pablo Junge (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993032#comment-16993032
 ] 

Pablo Junge commented on HIVE-22587:


Thanks to you [~ashutoshc] !

> hive.stats.ndv.error parameter documentation issue in HiveConf.java
> ---
>
> Key: HIVE-22587
> URL: https://issues.apache.org/jira/browse/HIVE-22587
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22587.patch
>
>
> hive.stats.ndv.error parameter documentation should specify that it only 
> affects the FM-Sketch algorithm



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Description: 
If you try dropping a table which is part of the definition of a created 
materialized view, the table is not dropped, which is the desired state as it 
is part of the materialized view.

However, there was a "drop" call to the table, so it tried to drop it but did 
not succeed, leaving it in an inconsistent state.

 

Repro:

---

1) Create tables:

 
{code:java}
CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary FLOAT, 
 hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES ('transactional'='true'); 

CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
{code}
 

2) Create the VM:

 
{code:java}
CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_date FROM emps 
JOIN depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
{code}
 

3) Following is in backend database at this point:

 
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
++---+---+--+---+
|     81 |    16 |    81 | emps     | MANAGED_TABLE     |
|     83 |    16 |    83 | depts    | MANAGED_TABLE     |
|     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
 

4) Let's drop the 'emps' table:

 
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
INFO  : Compiling 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop 
table emps
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time 
taken: 0.05 seconds
INFO  : Executing 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop 
table emps
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time 
taken: 10.281 seconds
INFO  : OK
No rows affected (16.949 seconds)
{code}
No issue displayed

 

5) List tables:

 
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
INFO  : Compiling 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show 
tables
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, 
type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time 
taken: 0.041 seconds
INFO  : Executing 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show 
tables
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time 
taken: 0.016 seconds
INFO  : OK
+---+
| tab_name  |
+---+
| depts |
| emps  |
+---+
2 rows selected (0.08 seconds)
{code}
 

6) Now, from the backend-db point of view:

 
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
++---+---+--+---+
|     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
|     83 |    16 |    83 | depts    | MANAGED_TABLE     |
|     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
The table is left with NULL in SD_ID, making it not available.

 

7) From Metastore.log

 
{code:java}
2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: metastore.HiveMetaStore 
(HiveMetaStore.java:logInfo(907)) - 196: source:172.25.34.150 drop_table : 
tbl=hive.mvs.emps
2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: HiveMetaStore.audit 
(HiveMetaStore.java:logAuditEvent(349)) - ugi=hive   ip=172.25.34.150   
 cmd=source:172.25.34.150 drop_table : tbl=hive.mvs.emps 
2019-12-02T20:00:25,580 INFO  [pool-6-thread-195]: 
metastore.ObjectStore$RetryingExecutor (ObjectStore.java:run(9966)) - 
Attempting to acquire the DB log notification lock: 0 out of 10 retries
javax.jdo.JDODataStoreException: Error executing SQL query "select 
"NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
 ~[datanucleus-api-jdo-4.2.4.jar:?]
 

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Description: 
If you try dropping a table which is part of the definition of a created 
materialized view, the table is not dropped, which is the desired state as it 
is part of the materialized view.

However, there was a "drop" call to the table, so it tried to drop it but did 
not succeed, leaving it in an inconsistent state.

 

Repro:

---

1) Create tables:

 
{code:java}
CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary FLOAT, 
 hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES ('transactional'='true'); 

CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
{code}
 

2) Create the VM:

 
{code:java}
CREATE MATERIALIZED VIEW mv1 AS SELECT empid, deptname, hire_dateFROM emps JOIN 
depts  ON (emps.deptno = depts.deptno) WHERE hire_date >= '2016-01-01';
{code}
 

3) Following is in backend database at this point:

 
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
++---+---+--+---+
|     81 |    16 |    81 | emps     | MANAGED_TABLE     |
|     83 |    16 |    83 | depts    | MANAGED_TABLE     |
|     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
 

4) Let's drop the 'emps' table:

 
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
INFO  : Compiling 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop 
table emps
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time 
taken: 0.05 seconds
INFO  : Executing 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop 
table emps
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time 
taken: 10.281 seconds
INFO  : OK
No rows affected (16.949 seconds)
{code}
No issue displayed

 

5) List tables:

 
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
INFO  : Compiling 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show 
tables
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, 
type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time 
taken: 0.041 seconds
INFO  : Executing 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show 
tables
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time 
taken: 0.016 seconds
INFO  : OK
+---+
| tab_name  |
+---+
| depts |
| emps  |
+---+
2 rows selected (0.08 seconds)
{code}
 

6) Now, from the backend-db point of view:

 
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
++---+---+--+---+
|     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
|     83 |    16 |    83 | depts    | MANAGED_TABLE     |
|     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
The table is left with NULL in SD_ID, making it not available.

 

7) From Metastore.log

 
{code:java}
2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: metastore.HiveMetaStore 
(HiveMetaStore.java:logInfo(907)) - 196: source:172.25.34.150 drop_table : 
tbl=hive.mvs.emps
2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: HiveMetaStore.audit 
(HiveMetaStore.java:logAuditEvent(349)) - ugi=hive   ip=172.25.34.150   
 cmd=source:172.25.34.150 drop_table : tbl=hive.mvs.emps 
2019-12-02T20:00:25,580 INFO  [pool-6-thread-195]: 
metastore.ObjectStore$RetryingExecutor (ObjectStore.java:run(9966)) - 
Attempting to acquire the DB log notification lock: 0 out of 10 retries
javax.jdo.JDODataStoreException: Error executing SQL query "select 
"NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
 ~[datanucleus-api-jdo-4.2.4.jar:?]
  

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Description: 
If you try dropping a table which is part of the definition of a created 
materialized view, the table is not dropped, which is the desired state as it 
is part of the materialized view.

However, there was a "drop" call to the table, so it tried to drop it but did 
not succeed, leaving it in an inconsistent state.

 

Repro:

---

1) Create tables:

 
{code:java}
CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary FLOAT, 
 hire_date TIMESTAMP)STORED AS ORC TBLPROPERTIES ('transactional'='true'); 

CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
INT)STORED AS ORC TBLPROPERTIES ('transactional'='true');
{code}
 

2) Create the VM:

 
{code:java}
CREATE MATERIALIZED VIEW mv1ASSELECT empid, deptname, hire_dateFROM emps JOIN 
depts  ON (emps.deptno = depts.deptno)WHERE hire_date >= '2016-01-01';
{code}
 

3) Following is in backend database at this point:

 
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
++---+---+--+---+
|     81 |    16 |    81 | emps     | MANAGED_TABLE     |
|     83 |    16 |    83 | depts    | MANAGED_TABLE     |
|     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
 

4) Let's drop the 'emps' table:

 
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
INFO  : Compiling 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop 
table emps
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time 
taken: 0.05 seconds
INFO  : Executing 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop 
table emps
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time 
taken: 10.281 seconds
INFO  : OK
No rows affected (16.949 seconds)
{code}
No issue displayed

 

5) List tables:

 
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
INFO  : Compiling 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show 
tables
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, 
type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time 
taken: 0.041 seconds
INFO  : Executing 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show 
tables
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time 
taken: 0.016 seconds
INFO  : OK
+---+
| tab_name  |
+---+
| depts |
| emps  |
+---+
2 rows selected (0.08 seconds)
{code}
 

6) Now, from the backend-db point of view:

 
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
++---+---+--+---+
|     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
|     83 |    16 |    83 | depts    | MANAGED_TABLE     |
|     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
The table is left with NULL in SD_ID, making it not available.

 

7) From Metastore.log

 
{code:java}
2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: metastore.HiveMetaStore 
(HiveMetaStore.java:logInfo(907)) - 196: source:172.25.34.150 drop_table : 
tbl=hive.mvs.emps
2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: HiveMetaStore.audit 
(HiveMetaStore.java:logAuditEvent(349)) - ugi=hive   ip=172.25.34.150   
 cmd=source:172.25.34.150 drop_table : tbl=hive.mvs.emps 
2019-12-02T20:00:25,580 INFO  [pool-6-thread-195]: 
metastore.ObjectStore$RetryingExecutor (ObjectStore.java:run(9966)) - 
Attempting to acquire the DB log notification lock: 0 out of 10 retries
javax.jdo.JDODataStoreException: Error executing SQL query "select 
"NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
 ~[datanucleus-api-jdo-4.2.4.jar:?]

[jira] [Updated] (HIVE-22587) hive.stats.ndv.error parameter documentation issue in HiveConf.java

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22587:
---
Attachment: HIVE-22587.patch
Status: Patch Available  (was: Open)

Changed definition of HIVE_STATS_NDV_ERROR 

 

HIVE_STATS_NDV_ERROR("hive.stats.ndv.error", (float)20.0,
 "The standard error allowed for NDV estimates, expressed in percentage. This 
provides a tradeoff \n" + 
 "between accuracy and compute cost. A lower value for the error indicates 
higher accuracy and a \n" +
 "higher compute cost. (NDV means the number of distinct values.). It only 
affects the FM-Sketch \n" +
 "(not the HLL algorithm which is the default), where it computes the number of 
necessary\n" +
 " bitvectors to achieve the accuracy.")

> hive.stats.ndv.error parameter documentation issue in HiveConf.java
> ---
>
> Key: HIVE-22587
> URL: https://issues.apache.org/jira/browse/HIVE-22587
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
> Attachments: HIVE-22587.patch
>
>
> hive.stats.ndv.error parameter documentation should specify that it only 
> affects the FM-Sketch algorithm



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-22587) hive.stats.ndv.error parameter documentation issue in HiveConf.java

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge reassigned HIVE-22587:
--


> hive.stats.ndv.error parameter documentation issue in HiveConf.java
> ---
>
> Key: HIVE-22587
> URL: https://issues.apache.org/jira/browse/HIVE-22587
> Project: Hive
>  Issue Type: Improvement
>  Components: Hive
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Minor
>
> hive.stats.ndv.error parameter documentation should specify that it only 
> affects the FM-Sketch algorithm



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HIVE-21174) hive.stats.ndv.error parameter documentation issue

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge resolved HIVE-21174.

Resolution: Fixed

> hive.stats.ndv.error parameter documentation issue
> --
>
> Key: HIVE-21174
> URL: https://issues.apache.org/jira/browse/HIVE-21174
> Project: Hive
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, 
> 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, 
> 2.3.4, 3.1.2
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Major
> Fix For: 2.0.2, 2.1.2, 2.4.0, 2.2.1, 2.3.3, 3.0.1, 3.10, 3.2.0, 
> 2.3.4, 3.1.1, 3.1.0, 2.3.2, 2.3.1, 3.0.0, 2.3.0, 2.2.0, 2.1.1, 2.1.0, 2.0.1, 
> 2.0.0
>
>
> Hive documentation for hive.stats.ndv.error does not specify that 
> hive.stats.ndv.error will only affect FM Sketch and not HLL.
>  
> https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (HIVE-21174) hive.stats.ndv.error parameter documentation issue

[Pablo Junge (Jira)]

[ 
https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16989213#comment-16989213
 ] 

Pablo Junge commented on HIVE-21174:


Wiki updated from 

Current documentation shows:

Standard error allowed for NDV estimates, expressed in percentage. This 
provides a tradeoff between accuracy and compute cost. A lower value for the 
error indicates higher accuracy and a higher compute cost. (NDV means number of 
distinct values.)

 

To 



The standard error allowed for NDV estimates, expressed in percentage. This 
provides a tradeoff between accuracy and compute cost. A lower value for the 
error indicates higher accuracy and a higher compute cost. (NDV means the 
number of distinct values.). It only affects the FM-Sketch (not the HLL 
algorithm which is the default), where it computes the number of necessary 
bitvectors to achieve the accuracy.

 

> hive.stats.ndv.error parameter documentation issue
> --
>
> Key: HIVE-21174
> URL: https://issues.apache.org/jira/browse/HIVE-21174
> Project: Hive
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, 
> 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, 
> 2.3.4, 3.1.2
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Major
> Fix For: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, 
> 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, 
> 2.3.4
>
>
> Hive documentation for hive.stats.ndv.error does not specify that 
> hive.stats.ndv.error will only affect FM Sketch and not HLL.
>  
> https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-21174) hive.stats.ndv.error parameter documentation issue

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge reassigned HIVE-21174:
--

Assignee: (was: Pablo Junge)

> hive.stats.ndv.error parameter documentation issue
> --
>
> Key: HIVE-21174
> URL: https://issues.apache.org/jira/browse/HIVE-21174
> Project: Hive
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, 
> 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, 
> 2.3.4, 3.1.2
>Reporter: Pablo Junge
>Priority: Major
> Fix For: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, 
> 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, 
> 2.3.4
>
>
> Hive documentation for hive.stats.ndv.error does not specify that 
> hive.stats.ndv.error will only affect FM Sketch and not HLL.
>  
> https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-21174) hive.stats.ndv.error parameter documentation issue

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge reassigned HIVE-21174:
--

Assignee: Pablo Junge

> hive.stats.ndv.error parameter documentation issue
> --
>
> Key: HIVE-21174
> URL: https://issues.apache.org/jira/browse/HIVE-21174
> Project: Hive
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, 
> 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, 
> 2.3.4, 3.1.2
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Major
> Fix For: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, 
> 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, 
> 2.3.4
>
>
> Hive documentation for hive.stats.ndv.error does not specify that 
> hive.stats.ndv.error will only affect FM Sketch and not HLL.
>  
> https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Assigned] (HIVE-21174) hive.stats.ndv.error parameter documentation issue

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge reassigned HIVE-21174:
--

Assignee: Pablo Junge  (was: Nita Dembla)

> hive.stats.ndv.error parameter documentation issue
> --
>
> Key: HIVE-21174
> URL: https://issues.apache.org/jira/browse/HIVE-21174
> Project: Hive
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, 
> 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, 
> 2.3.4, 3.1.2
>Reporter: Pablo Junge
>Assignee: Pablo Junge
>Priority: Major
> Fix For: 2.0.0, 2.0.1, 2.0.2, 2.1.0, 2.1.1, 2.1.2, 2.2.0, 2.3.0, 
> 3.0.0, 2.4.0, 2.2.1, 2.3.1, 2.3.2, 2.3.3, 3.1.0, 3.0.1, 3.10, 3.2.0, 3.1.1, 
> 2.3.4
>
>
> Hive documentation for hive.stats.ndv.error does not specify that 
> hive.stats.ndv.error will only affect FM Sketch and not HLL.
>  
> https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (HIVE-22566) Drop table involved in materialized view leaves the table in inconsistent state

[Pablo Junge (Jira)]

 [ 
https://issues.apache.org/jira/browse/HIVE-22566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge updated HIVE-22566:
---
Description: 
If you try dropping a table which is part of the definition of a created 
materialized view, the table is not dropped, which is the desired state as it 
is part of the materialized view.

However, there was a "drop" call to the table, so it tried to drop it but did 
not succeed, leaving it in an inconsistent state.

 

Repro:

---

1) Create tables:

 
{code:java}
CREATE TABLE emps (  empid INT,  deptno INT,  name VARCHAR(256),  salary FLOAT, 
 hire_date TIMESTAMP)STORED AS ORCTBLPROPERTIES ('transactional'='true'); 

CREATE TABLE depts (  deptno INT,  deptname VARCHAR(256),  locationid 
INT)STORED AS ORCTBLPROPERTIES ('transactional'='true');
{code}
 

2) Create the VM:

 
{code:java}
CREATE MATERIALIZED VIEW mv1ASSELECT empid, deptname, hire_dateFROM emps JOIN 
depts  ON (emps.deptno = depts.deptno)WHERE hire_date >= '2016-01-01';
{code}
 

3) Following is in backend database at this point:

 
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
++---+---+--+---+
|     81 |    16 |    81 | emps     | MANAGED_TABLE     |
|     83 |    16 |    83 | depts    | MANAGED_TABLE     |
|     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
 

4) Let's drop the 'emps' table:

 
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> drop table emps;
INFO  : Compiling 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop 
table emps
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
INFO  : Completed compiling 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time 
taken: 0.05 seconds
INFO  : Executing 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b): drop 
table emps
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20191202200025_c13079d0-8695-4485-8a18-14804b8b014b); Time 
taken: 10.281 seconds
INFO  : OK
No rows affected (16.949 seconds)
{code}
No issue displayed

 

5) List tables:

 
{code:java}
0: jdbc:hive2://c1122-node2.squadron.support.> show tables;
INFO  : Compiling 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show 
tables
INFO  : Semantic Analysis Completed (retrial = false)
INFO  : Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:tab_name, 
type:string, comment:from deserializer)], properties:null)
INFO  : Completed compiling 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time 
taken: 0.041 seconds
INFO  : Executing 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413): show 
tables
INFO  : Starting task [Stage-0:DDL] in serial mode
INFO  : Completed executing 
command(queryId=hive_20191202200125_ca12565b-1d4d-4433-a602-ecf685863413); Time 
taken: 0.016 seconds
INFO  : OK
+---+
| tab_name  |
+---+
| depts |
| emps  |
+---+
2 rows selected (0.08 seconds)
{code}
 

6) Now, from the backend-db point of view:

 
{code:java}
mysql> select TBL_ID, DB_ID, SD_ID, TBL_NAME, TBL_TYPE from TBLS where DB_ID=16;
++---+---+--+---+
| TBL_ID | DB_ID | SD_ID | TBL_NAME | TBL_TYPE          |
++---+---+--+---+
|     81 |    16 |  NULL | emps     | MANAGED_TABLE     |
|     83 |    16 |    83 | depts    | MANAGED_TABLE     |
|     84 |    16 |    84 | mv1      | MATERIALIZED_VIEW |
++---+---+--+---+
3 rows in set (0.00 sec)
{code}
The table is left with NULL in SD_ID, making it not available.

 

7) From Metastore.log

 
{code:java}
2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: metastore.HiveMetaStore 
(HiveMetaStore.java:logInfo(907)) - 196: source:172.25.34.150 drop_table : 
tbl=hive.mvs.emps
2019-12-02T20:00:25,545 INFO  [pool-6-thread-195]: HiveMetaStore.audit 
(HiveMetaStore.java:logAuditEvent(349)) - ugi=hive   ip=172.25.34.150   
 cmd=source:172.25.34.150 drop_table : tbl=hive.mvs.emps 
2019-12-02T20:00:25,580 INFO  [pool-6-thread-195]: 
metastore.ObjectStore$RetryingExecutor (ObjectStore.java:run(9966)) - 
Attempting to acquire the DB log notification lock: 0 out of 10 retries
javax.jdo.JDODataStoreException: Error executing SQL query "select 
"NEXT_EVENT_ID" from "NOTIFICATION_SEQUENCE" for update".
at 
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:543)
 ~[datanucleus-api-jdo-4.2.4.jar:?]
at 

[jira] [Assigned] (HIVE-21174) hive.stats.ndv.error parameter documentation issue

[Pablo Junge (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HIVE-21174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pablo Junge reassigned HIVE-21174:
--


> hive.stats.ndv.error parameter documentation issue
> --
>
> Key: HIVE-21174
> URL: https://issues.apache.org/jira/browse/HIVE-21174
> Project: Hive
>  Issue Type: Improvement
>  Components: Documentation
>Affects Versions: 2.3.4, 3.1.1, 3.1.0, 2.3.2, 2.3.1, 3.0.0, 2.3.0, 2.2.0, 
> 2.1.1, 2.1.0, 2.0.1, 2.0.0, 2.0.2, 2.1.2, 2.4.0, 2.2.1, 2.3.3, 3.0.1, 3.10, 
> 3.2.0, 3.1.2
>Reporter: Pablo Junge
>Assignee: Nita Dembla
>Priority: Major
> Fix For: 2.0.2, 2.1.2, 2.4.0, 2.2.1, 2.3.3, 3.0.1, 3.10, 3.2.0, 
> 3.1.2, 2.3.4, 3.1.1, 3.1.0, 2.3.2, 2.3.1, 3.0.0, 2.3.0, 2.2.0, 2.1.1, 2.1.0, 
> 2.0.1, 2.0.0
>
>
> Hive documentation for hive.stats.ndv.error does not specify that 
> hive.stats.ndv.error will only affect FM Sketch and not HLL.
>  
> https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)