[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-04-04 Thread Peter Vary (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Vary updated HIVE-15995:
--
   Resolution: Fixed
Fix Version/s: 3.0.0
   Status: Resolved  (was: Patch Available)

Pushed to master.

Thanks for the patch [~szita]!

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, 
> cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-03-22 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: Patch Available  (was: In Progress)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, 
> cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-03-22 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Attachment: HIVE-15995.5.patch

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, 
> cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-03-22 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Attachment: (was: HIVE-15995.5.patch)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-03-22 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Attachment: HIVE-15995.5.patch

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, 
> cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-03-22 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: In Progress  (was: Patch Available)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.5.patch, HIVE-15995.patch, 
> cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-03-06 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: Patch Available  (was: In Progress)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-03-06 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Attachment: HIVE-15995.4.patch

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.4.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-03-06 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: In Progress  (was: Patch Available)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-22 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: In Progress  (was: Patch Available)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-22 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Attachment: HIVE-15995.3.patch

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-22 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: Patch Available  (was: In Progress)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.3.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-12 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: Patch Available  (was: In Progress)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-12 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: In Progress  (was: Patch Available)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-12 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Attachment: HIVE-15995.2.patch

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-12 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: In Progress  (was: Patch Available)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-12 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: Patch Available  (was: In Progress)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.2.patch, 
> HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-07 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Status: In Progress  (was: Patch Available)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HIVE-15995.1.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-07 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Attachment: HIVE-15995.1.patch

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Fix For: 2.1.0
>
> Attachments: HIVE-15995.1.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2018-02-07 Thread Adam Szita (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adam Szita updated HIVE-15995:
--
Fix Version/s: (was: 2.1.0)
Affects Version/s: 3.0.0
 Target Version/s: 3.0.0  (was: 2.1.2, 3.0.0)
   Status: Patch Available  (was: In Progress)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 2.1.0, 1.2.1, 3.0.0
>Reporter: Michal Ferlinski
>Assignee: Adam Szita
>Priority: Major
> Attachments: HIVE-15995.1.patch, HIVE-15995.patch, cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2017-03-25 Thread Pengcheng Xiong (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pengcheng Xiong updated HIVE-15995:
---
Target Version/s: 2.1.2, 3.0.0  (was: 2.2.0, 2.1.2)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Michal Ferlinski
>Assignee: Michal Ferlinski
> Fix For: 2.1.0
>
> Attachments: cx1.avsc, cx2.avsc, HIVE-15995.patch
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2017-02-24 Thread Michal Ferlinski (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michal Ferlinski updated HIVE-15995:

Target Version/s: 2.2.0, 2.1.2

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Michal Ferlinski
>Assignee: Michal Ferlinski
> Fix For: 2.1.0
>
> Attachments: cx1.avsc, cx2.avsc, HIVE-15995.patch
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2017-02-24 Thread Michal Ferlinski (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michal Ferlinski updated HIVE-15995:

Affects Version/s: 1.2.1
Fix Version/s: 2.1.0
  Component/s: Metastore

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>  Components: Metastore
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Michal Ferlinski
>Assignee: Michal Ferlinski
> Fix For: 2.1.0
>
> Attachments: cx1.avsc, cx2.avsc, HIVE-15995.patch
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2017-02-24 Thread Michal Ferlinski (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michal Ferlinski updated HIVE-15995:

Description: 
Hive enables table schema evolution via properties. For avro e.g. we can alter 
the 'avro.schema.url' property to update table schema to the next version. 
Updating properties however doesn't affect column list stored in metastore DB 
so the table is not in the newest version when returned from metastore API. 
This is problem for tools working with metastore (e.g. Presto).

To solve this issue I suggest to introduce new DDL statement syncing metastore 
columns with those from serde:
{code}
ALTER TABLE user_test1 UPDATE COLUMNS
{code}

Note that this is format independent solution. 

To reproduce, follow the instructions below:
- Create table based on avro schema version 1 (cxv1.avsc)
{code}
CREATE EXTERNAL TABLE user_test1
  PARTITIONED BY (dt string)
  ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
  STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
  OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
  LOCATION
  '/tmp/schema-evolution/user_test1'
  TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
{code}
- Update schema to version 2 (cx2.avsc)
{code}
ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
'/tmp/schema-evolution/cx2.avsc');
{code}
- Print serde columns (top info) and metastore columns (Detailed Table 
Information):
{code}
DESCRIBE EXTENDED user_test1
{code}

  was:
Hive enables table schema evolution via properties. For avro e.g. we could 
alter the 'avro.schema.url' property to update table schema to the next 
version. Updating properties however doesn't affect column list stored in 
metastore DB so the table is not in the newest version when returned from 
metastore API. This is problem for tools working with metastore (e.g. Presto).

To solve this issue I suggest to introduce new DDL statement syncing metastore 
columns with those from serde:
{code}
ALTER TABLE user_test1 UPDATE COLUMNS
{code}

This will be serde format independent solution. 
Let me know what you think. Patch is in progress.

To reproduce, follow the instructions below:
- Create table based on avro schema version 1 (cxv1.avsc)
{code}
CREATE EXTERNAL TABLE user_test1
  PARTITIONED BY (dt string)
  ROW FORMAT SERDE
  'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
  STORED AS INPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
  OUTPUTFORMAT
  'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
  LOCATION
  '/tmp/schema-evolution/user_test1'
  TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
{code}
- Update schema to version 2 (cx2.avsc)
{code}
ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
'/tmp/schema-evolution/cx2.avsc');
{code}
- Print serde columns (top info) and metastore columns (Detailed Table 
Information):
{code}
DESCRIBE EXTENDED user_test1
{code}


> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Michal Ferlinski
>Assignee: Michal Ferlinski
> Attachments: cx1.avsc, cx2.avsc, HIVE-15995.patch
>
>
> Hive enables table schema evolution via properties. For avro e.g. we can 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> Note that this is format independent solution. 
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2017-02-24 Thread Michal Ferlinski (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michal Ferlinski updated HIVE-15995:

Affects Version/s: 2.1.0

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Michal Ferlinski
>Assignee: Michal Ferlinski
> Attachments: cx1.avsc, cx2.avsc, HIVE-15995.patch
>
>
> Hive enables table schema evolution via properties. For avro e.g. we could 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> This will be serde format independent solution. 
> Let me know what you think. Patch is in progress.
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2017-02-24 Thread Michal Ferlinski (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michal Ferlinski updated HIVE-15995:

Assignee: Michal Ferlinski
  Status: Patch Available  (was: Open)

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>Reporter: Michal Ferlinski
>Assignee: Michal Ferlinski
> Attachments: cx1.avsc, cx2.avsc, HIVE-15995.patch
>
>
> Hive enables table schema evolution via properties. For avro e.g. we could 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> This will be serde format independent solution. 
> Let me know what you think. Patch is in progress.
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2017-02-24 Thread Michal Ferlinski (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michal Ferlinski updated HIVE-15995:

Attachment: HIVE-15995.patch

This patch adds new DDL statement to sync metastore columns with serde fields: 
ALTER TABLE tab_name UPDATE COLUMNS;
Please review the code.

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>Reporter: Michal Ferlinski
> Attachments: cx1.avsc, cx2.avsc, HIVE-15995.patch
>
>
> Hive enables table schema evolution via properties. For avro e.g. we could 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> This will be serde format independent solution. 
> Let me know what you think. Patch is in progress.
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (HIVE-15995) Syncing metastore table with serde schema

2017-02-21 Thread Michal Ferlinski (JIRA) 

 [ 
https://issues.apache.org/jira/browse/HIVE-15995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michal Ferlinski updated HIVE-15995:

Attachment: cx2.avsc
cx1.avsc

> Syncing metastore table with serde schema
> -
>
> Key: HIVE-15995
> URL: https://issues.apache.org/jira/browse/HIVE-15995
> Project: Hive
>  Issue Type: Bug
>Reporter: Michal Ferlinski
> Attachments: cx1.avsc, cx2.avsc
>
>
> Hive enables table schema evolution via properties. For avro e.g. we could 
> alter the 'avro.schema.url' property to update table schema to the next 
> version. Updating properties however doesn't affect column list stored in 
> metastore DB so the table is not in the newest version when returned from 
> metastore API. This is problem for tools working with metastore (e.g. Presto).
> To solve this issue I suggest to introduce new DDL statement syncing 
> metastore columns with those from serde:
> {code}
> ALTER TABLE user_test1 UPDATE COLUMNS
> {code}
> This will be serde format independent solution. 
> Let me know what you think. Patch is in progress.
> To reproduce, follow the instructions below:
> - Create table based on avro schema version 1 (cxv1.avsc)
> {code}
> CREATE EXTERNAL TABLE user_test1
>   PARTITIONED BY (dt string)
>   ROW FORMAT SERDE
>   'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
>   STORED AS INPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
>   OUTPUTFORMAT
>   'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
>   LOCATION
>   '/tmp/schema-evolution/user_test1'
>   TBLPROPERTIES ('avro.schema.url'='/tmp/schema-evolution/cx1.avsc');
> {code}
> - Update schema to version 2 (cx2.avsc)
> {code}
> ALTER TABLE user_test1 SET TBLPROPERTIES ('avro.schema.url' = 
> '/tmp/schema-evolution/cx2.avsc');
> {code}
> - Print serde columns (top info) and metastore columns (Detailed Table 
> Information):
> {code}
> DESCRIBE EXTENDED user_test1
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)