[jira] [Updated] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties

2018-08-14 Thread Sergey Shelukhin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-20343:

   Resolution: Fixed
Fix Version/s: 4.0.0
   Status: Resolved  (was: Patch Available)

Committed to master, thanks for the review!
[~Rajkumar Singh] I couldn't repro the location issue, but it's a separate one.
Can you see if you can repro it when specifying txn + properties fully? This 
patch makes sure that ambiguous partial specification like that is not allowed.

If the issue still happens, can you file a separate JIRA, and also see if it 
reproes when full path w/schema is specified for the location (i.e. 
hdfs://). 
We might need more information about the repro, since so far I couldn't repro 
it locally.


> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-20343.patch
>
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc  TBLPROPERTIES 
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected 
> "insert_only"
> {code}
>  describe formatted ctasexampleinsertonly
>  
> +---++---+
> |   col_name| data_type   
>|comment|
> +---++---+
> | # col_name| data_type   
>| comment   |
> | name  | varchar(8)  
>|   |
> | time  | double  
>|   |
> |   | NULL
>| NULL  |
> | # Detailed Table Information  | NULL
>| NULL  |
> | Database: | default 
>| NULL  |
> | OwnerType:| USER
>| NULL  |
> | Owner:| hive
>| NULL  |
> | CreateTime:   | Wed Aug 08 21:35:15 UTC 2018
>| NULL  |
> | LastAccessTime:   | UNKNOWN 
>| NULL  |
> | Retention:| 0   
>| NULL  |
> | Location: | 
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
>  | NULL  |
> | Table Type:   | MANAGED_TABLE   
>| NULL  |
> | Table Parameters: | NULL
>| NULL  |
> |   | COLUMN_STATS_ACCURATE   
>| {}|
> |   | bucketing_version   
>| 2 |
> |   | numFiles
>| 1 |
> |   | numRows 
>| 1 |
> |   | rawDataSize 
>| 0 |
> |   | totalSize   
>| 754   |
> |   | transactional   
>| true  |
> |   | transactional_properties
>| default   |
> |   | transient_lastDdlTime   
>| 1533764115|
> |   | NULL
>| NULL  |
> | # Storage Information | NULL
>| NULL

[jira] [Updated] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties

2018-08-13 Thread Sergey Shelukhin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-20343:

Status: Patch Available  (was: Open)

> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20343.patch
>
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc  TBLPROPERTIES 
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected 
> "insert_only"
> {code}
>  describe formatted ctasexampleinsertonly
>  
> +---++---+
> |   col_name| data_type   
>|comment|
> +---++---+
> | # col_name| data_type   
>| comment   |
> | name  | varchar(8)  
>|   |
> | time  | double  
>|   |
> |   | NULL
>| NULL  |
> | # Detailed Table Information  | NULL
>| NULL  |
> | Database: | default 
>| NULL  |
> | OwnerType:| USER
>| NULL  |
> | Owner:| hive
>| NULL  |
> | CreateTime:   | Wed Aug 08 21:35:15 UTC 2018
>| NULL  |
> | LastAccessTime:   | UNKNOWN 
>| NULL  |
> | Retention:| 0   
>| NULL  |
> | Location: | 
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
>  | NULL  |
> | Table Type:   | MANAGED_TABLE   
>| NULL  |
> | Table Parameters: | NULL
>| NULL  |
> |   | COLUMN_STATS_ACCURATE   
>| {}|
> |   | bucketing_version   
>| 2 |
> |   | numFiles
>| 1 |
> |   | numRows 
>| 1 |
> |   | rawDataSize 
>| 0 |
> |   | totalSize   
>| 754   |
> |   | transactional   
>| true  |
> |   | transactional_properties
>| default   |
> |   | transient_lastDdlTime   
>| 1533764115|
> |   | NULL
>| NULL  |
> | # Storage Information | NULL
>| NULL  |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde   
>| NULL  |
> | InputFormat:  | 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL  |
> | OutputFormat: | 
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat   | NULL  |
> | Compressed:   | No  
>| NULL  |
> | Num Buckets:  | -1  
>| NULL  |
> | Bucket Columns:   | []

[jira] [Updated] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties

2018-08-13 Thread Sergey Shelukhin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sergey Shelukhin updated HIVE-20343:

Attachment: HIVE-20343.patch

> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-20343.patch
>
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc  TBLPROPERTIES 
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected 
> "insert_only"
> {code}
>  describe formatted ctasexampleinsertonly
>  
> +---++---+
> |   col_name| data_type   
>|comment|
> +---++---+
> | # col_name| data_type   
>| comment   |
> | name  | varchar(8)  
>|   |
> | time  | double  
>|   |
> |   | NULL
>| NULL  |
> | # Detailed Table Information  | NULL
>| NULL  |
> | Database: | default 
>| NULL  |
> | OwnerType:| USER
>| NULL  |
> | Owner:| hive
>| NULL  |
> | CreateTime:   | Wed Aug 08 21:35:15 UTC 2018
>| NULL  |
> | LastAccessTime:   | UNKNOWN 
>| NULL  |
> | Retention:| 0   
>| NULL  |
> | Location: | 
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
>  | NULL  |
> | Table Type:   | MANAGED_TABLE   
>| NULL  |
> | Table Parameters: | NULL
>| NULL  |
> |   | COLUMN_STATS_ACCURATE   
>| {}|
> |   | bucketing_version   
>| 2 |
> |   | numFiles
>| 1 |
> |   | numRows 
>| 1 |
> |   | rawDataSize 
>| 0 |
> |   | totalSize   
>| 754   |
> |   | transactional   
>| true  |
> |   | transactional_properties
>| default   |
> |   | transient_lastDdlTime   
>| 1533764115|
> |   | NULL
>| NULL  |
> | # Storage Information | NULL
>| NULL  |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde   
>| NULL  |
> | InputFormat:  | 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL  |
> | OutputFormat: | 
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat   | NULL  |
> | Compressed:   | No  
>| NULL  |
> | Num Buckets:  | -1  
>| NULL  |
> | Bucket Columns:   | []

[jira] [Updated] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties

2018-08-09 Thread Eugene Koifman (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Koifman updated HIVE-20343:
--
Component/s: (was: Hive)
 Transactions

> Hive 3: CTAS does not respect transactional_properties
> --
>
> Key: HIVE-20343
> URL: https://issues.apache.org/jira/browse/HIVE-20343
> Project: Hive
>  Issue Type: Bug
>  Components: Transactions
>Affects Versions: 3.1.0
> Environment: hive-3
>Reporter: Rajkumar Singh
>Priority: Major
>
> Steps to reproduce:
> {code}
> create table ctasexampleinsertonly stored as orc  TBLPROPERTIES 
> ("transactional_properties"="insert_only") as select * from testtable limit 1;
> {code}
> look for transactional_properties which is 'default' not the expected 
> "insert_only"
> {code}
>  describe formatted ctasexampleinsertonly
>  
> +---++---+
> |   col_name| data_type   
>|comment|
> +---++---+
> | # col_name| data_type   
>| comment   |
> | name  | varchar(8)  
>|   |
> | time  | double  
>|   |
> |   | NULL
>| NULL  |
> | # Detailed Table Information  | NULL
>| NULL  |
> | Database: | default 
>| NULL  |
> | OwnerType:| USER
>| NULL  |
> | Owner:| hive
>| NULL  |
> | CreateTime:   | Wed Aug 08 21:35:15 UTC 2018
>| NULL  |
> | LastAccessTime:   | UNKNOWN 
>| NULL  |
> | Retention:| 0   
>| NULL  |
> | Location: | 
> hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly
>  | NULL  |
> | Table Type:   | MANAGED_TABLE   
>| NULL  |
> | Table Parameters: | NULL
>| NULL  |
> |   | COLUMN_STATS_ACCURATE   
>| {}|
> |   | bucketing_version   
>| 2 |
> |   | numFiles
>| 1 |
> |   | numRows 
>| 1 |
> |   | rawDataSize 
>| 0 |
> |   | totalSize   
>| 754   |
> |   | transactional   
>| true  |
> |   | transactional_properties
>| default   |
> |   | transient_lastDdlTime   
>| 1533764115|
> |   | NULL
>| NULL  |
> | # Storage Information | NULL
>| NULL  |
> | SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde   
>| NULL  |
> | InputFormat:  | 
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL  |
> | OutputFormat: | 
> org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat   | NULL  |
> | Compressed:   | No  
>| NULL  |
> | Num Buckets:  | -1  
>| NULL  |
> | Bucket Columns:   | []  
>| NULL

[jira] [Updated] (HIVE-20343) Hive 3: CTAS does not respect transactional_properties

2018-08-08 Thread Rajkumar Singh (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HIVE-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajkumar Singh updated HIVE-20343:
--
Description: 
Steps to reproduce:
{code}
create table ctasexampleinsertonly stored as orc  TBLPROPERTIES 
("transactional_properties"="insert_only") as select * from testtable limit 1;
{code}
look for transactional_properties which is 'default' not the expected 
"insert_only"
{code}
 describe formatted ctasexampleinsertonly

 
+---++---+
|   col_name| data_type 
 |comment|
+---++---+
| # col_name| data_type 
 | comment   |
| name  | varchar(8)
 |   |
| time  | double
 |   |
|   | NULL  
 | NULL  |
| # Detailed Table Information  | NULL  
 | NULL  |
| Database: | default   
 | NULL  |
| OwnerType:| USER  
 | NULL  |
| Owner:| hive  
 | NULL  |
| CreateTime:   | Wed Aug 08 21:35:15 UTC 2018  
 | NULL  |
| LastAccessTime:   | UNKNOWN   
 | NULL  |
| Retention:| 0 
 | NULL  |
| Location: | 
hdfs://xx:8020/warehouse/tablespace/managed/hive/ctasexampleinsertonly 
| NULL  |
| Table Type:   | MANAGED_TABLE 
 | NULL  |
| Table Parameters: | NULL  
 | NULL  |
|   | COLUMN_STATS_ACCURATE 
 | {}|
|   | bucketing_version 
 | 2 |
|   | numFiles  
 | 1 |
|   | numRows   
 | 1 |
|   | rawDataSize   
 | 0 |
|   | totalSize 
 | 754   |
|   | transactional 
 | true  |
|   | transactional_properties  
 | default   |
|   | transient_lastDdlTime 
 | 1533764115|
|   | NULL  
 | NULL  |
| # Storage Information | NULL  
 | NULL  |
| SerDe Library:| org.apache.hadoop.hive.ql.io.orc.OrcSerde 
 | NULL  |
| InputFormat:  | 
org.apache.hadoop.hive.ql.io.orc.OrcInputFormat| NULL  |
| OutputFormat: | 
org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat   | NULL  |
| Compressed:   | No
 | NULL  |
| Num Buckets:  | -1
 | NULL  |
| Bucket Columns:   | []
 | NULL  |
| Sort Columns: | []
 | NULL  |
| Storage Desc Params:  | NULL  
 | NULL  |
|   | serialization.format  
 | 1 |
+---++---+
{code}

not sure whether its a cosmatic issue but it does creates a problem with insert 
{code}
CREATE TABLE TABLE42 ROW FORMAT SERDE