subject:"\[jira\] \[Commented\] \(HIVE\-9445\) Revert HIVE\-5700 \- enforce single date format for partition column storage"

[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

2015-02-12 Thread Matt Foley (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318761#comment-14318761
 ] 

Matt Foley commented on HIVE-9445:
--

[~brocknoland], HIVE-5700 is stated to be fixed in version 0.13.  Above, this 
HIVE-9445 is stated to be fixed in version 0.15.  Was the problem also reverted 
in 0.13 and 0.14 branches?  Thanks.

 Revert HIVE-5700 - enforce single date format for partition column storage
 --

 Key: HIVE-9445
 URL: https://issues.apache.org/jira/browse/HIVE-9445
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
 Fix For: 0.15.0

 Attachments: HIVE-9445.1.patch, HIVE-9445.1.patch


 HIVE-5700 has the following issues:
 * HIVE-8730 - fails mysql upgrades
 * Does not upgrade all metadata, e.g. {{PARTITIONS.PART_NAME}} See comments 
 in HIVE-5700.
 * Completely corrupts postgres, see below.
 With a postgres metastore on 0.12, I executed the following:
 {noformat}
 CREATE TABLE HIVE5700_DATE_PARTED (line string) PARTITIONED BY (ddate date);
 CREATE TABLE HIVE5700_STRING_PARTED (line string) PARTITIONED BY (ddate 
 string);
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='NOT_DATE');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150121');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150122');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='2015-01-23');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='NOT_DATE');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150121');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150122');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='2015-01-23');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='NOT_DATE');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='20150121');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='20150122');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='2015-01-23');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='NOT_DATE');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='20150121');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='20150122');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='2015-01-23');
 hive show partitions HIVE5700_DATE_PARTED;  
 OK
 ddate=20150121
 ddate=20150122
 ddate=2015-01-23
 ddate=NOT_DATE
 Time taken: 0.052 seconds, Fetched: 4 row(s)
 hive show partitions HIVE5700_STRING_PARTED;
 OK
 ddate=20150121
 ddate=20150122
 ddate=2015-01-23
 ddate=NOT_DATE
 Time taken: 0.051 seconds, Fetched: 4 row(s)
 {noformat}
 I then took a dump of the database named {{postgres-pre-upgrade.sql}} and the 
 data in the dump looks good:
 {noformat}
 [root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY 
 PARTITION_KEY_VALS' postgres-pre-upgrade.sql 
 COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME, 
 SD_ID, TBL_ID) FROM stdin;
 3 1421943647  0   ddate=NOT_DATE  6   2
 4 1421943647  0   ddate=20150121  7   2
 5 1421943648  0   ddate=20150122  8   2
 6 1421943664  0   ddate=NOT_DATE  9   3
 7 1421943664  0   ddate=20150121  10  3
 8 1421943665  0   ddate=20150122  11  3
 9 1421943694  0   ddate=2015-01-2312  2
 101421943695  0   ddate=2015-01-2313  3
 \.
 --
 COPY PARTITION_KEY_VALS (PART_ID, PART_KEY_VAL, INTEGER_IDX) FROM 
 stdin;
 3 NOT_DATE0
 4 201501210
 5 201501220
 6 NOT_DATE0
 7 201501210
 8 201501220
 9 2015-01-23  0
 102015-01-23  0
 \.
 {noformat}
 I then upgraded to 0.13 and subsequently upgraded the MS with the following 
 command: {{schematool -dbType postgres -upgradeSchema -verbose}}
 The file {{postgres-post-upgrade.sql}} is the post-upgrade db dump. As you 
 can see the data is completely corrupt.
 {noformat}
 [root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY 
 PARTITION_KEY_VALS' postgres-post-upgrade.sql 
 COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME, 
 SD_ID, TBL_ID) FROM stdin;
 3 1421943647  0   ddate=NOT_DATE  6   2
 4 1421943647  0   ddate=20150121  7   2
 5

[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

2015-02-12 Thread Brock Noland (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14319345#comment-14319345
 ] 

Brock Noland commented on HIVE-9445:


[~thejas] [~vikram.dixit] - I believe you guys RM those two branches. Would you 
apply this change to them?

 Revert HIVE-5700 - enforce single date format for partition column storage
 --

 Key: HIVE-9445
 URL: https://issues.apache.org/jira/browse/HIVE-9445
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
 Fix For: 1.1.0

 Attachments: HIVE-9445.1.patch, HIVE-9445.1.patch


 HIVE-5700 has the following issues:
 * HIVE-8730 - fails mysql upgrades
 * Does not upgrade all metadata, e.g. {{PARTITIONS.PART_NAME}} See comments 
 in HIVE-5700.
 * Completely corrupts postgres, see below.
 With a postgres metastore on 0.12, I executed the following:
 {noformat}
 CREATE TABLE HIVE5700_DATE_PARTED (line string) PARTITIONED BY (ddate date);
 CREATE TABLE HIVE5700_STRING_PARTED (line string) PARTITIONED BY (ddate 
 string);
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='NOT_DATE');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150121');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150122');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='2015-01-23');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='NOT_DATE');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150121');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150122');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='2015-01-23');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='NOT_DATE');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='20150121');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='20150122');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='2015-01-23');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='NOT_DATE');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='20150121');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='20150122');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='2015-01-23');
 hive show partitions HIVE5700_DATE_PARTED;  
 OK
 ddate=20150121
 ddate=20150122
 ddate=2015-01-23
 ddate=NOT_DATE
 Time taken: 0.052 seconds, Fetched: 4 row(s)
 hive show partitions HIVE5700_STRING_PARTED;
 OK
 ddate=20150121
 ddate=20150122
 ddate=2015-01-23
 ddate=NOT_DATE
 Time taken: 0.051 seconds, Fetched: 4 row(s)
 {noformat}
 I then took a dump of the database named {{postgres-pre-upgrade.sql}} and the 
 data in the dump looks good:
 {noformat}
 [root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY 
 PARTITION_KEY_VALS' postgres-pre-upgrade.sql 
 COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME, 
 SD_ID, TBL_ID) FROM stdin;
 3 1421943647  0   ddate=NOT_DATE  6   2
 4 1421943647  0   ddate=20150121  7   2
 5 1421943648  0   ddate=20150122  8   2
 6 1421943664  0   ddate=NOT_DATE  9   3
 7 1421943664  0   ddate=20150121  10  3
 8 1421943665  0   ddate=20150122  11  3
 9 1421943694  0   ddate=2015-01-2312  2
 101421943695  0   ddate=2015-01-2313  3
 \.
 --
 COPY PARTITION_KEY_VALS (PART_ID, PART_KEY_VAL, INTEGER_IDX) FROM 
 stdin;
 3 NOT_DATE0
 4 201501210
 5 201501220
 6 NOT_DATE0
 7 201501210
 8 201501220
 9 2015-01-23  0
 102015-01-23  0
 \.
 {noformat}
 I then upgraded to 0.13 and subsequently upgraded the MS with the following 
 command: {{schematool -dbType postgres -upgradeSchema -verbose}}
 The file {{postgres-post-upgrade.sql}} is the post-upgrade db dump. As you 
 can see the data is completely corrupt.
 {noformat}
 [root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY 
 PARTITION_KEY_VALS' postgres-post-upgrade.sql 
 COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME, 
 SD_ID, TBL_ID) FROM stdin;
 3 1421943647  0   ddate=NOT_DATE  6   2
 4 1421943647  0   ddate=20150121  7   2
 5 1421943648  0   ddate=20150122  8   2
 6 1421943664  0

[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

2015-01-29 Thread Sergey Shelukhin (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14297384#comment-14297384
 ] 

Sergey Shelukhin commented on HIVE-9445:


Hmm, looks like I missed the java part of the change that was not merely code 
move. Partition spec validation should not have been thrown out... Let me file 
a JIRA to add it back

 Revert HIVE-5700 - enforce single date format for partition column storage
 --

 Key: HIVE-9445
 URL: https://issues.apache.org/jira/browse/HIVE-9445
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
 Fix For: 0.15.0

 Attachments: HIVE-9445.1.patch, HIVE-9445.1.patch


 HIVE-5700 has the following issues:
 * HIVE-8730 - fails mysql upgrades
 * Does not upgrade all metadata, e.g. {{PARTITIONS.PART_NAME}} See comments 
 in HIVE-5700.
 * Completely corrupts postgres, see below.
 With a postgres metastore on 0.12, I executed the following:
 {noformat}
 CREATE TABLE HIVE5700_DATE_PARTED (line string) PARTITIONED BY (ddate date);
 CREATE TABLE HIVE5700_STRING_PARTED (line string) PARTITIONED BY (ddate 
 string);
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='NOT_DATE');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150121');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150122');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='2015-01-23');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='NOT_DATE');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150121');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150122');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='2015-01-23');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='NOT_DATE');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='20150121');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='20150122');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='2015-01-23');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='NOT_DATE');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='20150121');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='20150122');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='2015-01-23');
 hive show partitions HIVE5700_DATE_PARTED;  
 OK
 ddate=20150121
 ddate=20150122
 ddate=2015-01-23
 ddate=NOT_DATE
 Time taken: 0.052 seconds, Fetched: 4 row(s)
 hive show partitions HIVE5700_STRING_PARTED;
 OK
 ddate=20150121
 ddate=20150122
 ddate=2015-01-23
 ddate=NOT_DATE
 Time taken: 0.051 seconds, Fetched: 4 row(s)
 {noformat}
 I then took a dump of the database named {{postgres-pre-upgrade.sql}} and the 
 data in the dump looks good:
 {noformat}
 [root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY 
 PARTITION_KEY_VALS' postgres-pre-upgrade.sql 
 COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME, 
 SD_ID, TBL_ID) FROM stdin;
 3 1421943647  0   ddate=NOT_DATE  6   2
 4 1421943647  0   ddate=20150121  7   2
 5 1421943648  0   ddate=20150122  8   2
 6 1421943664  0   ddate=NOT_DATE  9   3
 7 1421943664  0   ddate=20150121  10  3
 8 1421943665  0   ddate=20150122  11  3
 9 1421943694  0   ddate=2015-01-2312  2
 101421943695  0   ddate=2015-01-2313  3
 \.
 --
 COPY PARTITION_KEY_VALS (PART_ID, PART_KEY_VAL, INTEGER_IDX) FROM 
 stdin;
 3 NOT_DATE0
 4 201501210
 5 201501220
 6 NOT_DATE0
 7 201501210
 8 201501220
 9 2015-01-23  0
 102015-01-23  0
 \.
 {noformat}
 I then upgraded to 0.13 and subsequently upgraded the MS with the following 
 command: {{schematool -dbType postgres -upgradeSchema -verbose}}
 The file {{postgres-post-upgrade.sql}} is the post-upgrade db dump. As you 
 can see the data is completely corrupt.
 {noformat}
 [root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY 
 PARTITION_KEY_VALS' postgres-post-upgrade.sql 
 COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME, 
 SD_ID, TBL_ID) FROM stdin;
 3 1421943647  0   ddate=NOT_DATE  6   2
 4 1421943647  0   ddate=20150121  7   2
 5

[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

2015-01-25 Thread Xuefu Zhang (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291147#comment-14291147
 ] 

Xuefu Zhang commented on HIVE-9445:
---

+1

 Revert HIVE-5700 - enforce single date format for partition column storage
 --

 Key: HIVE-9445
 URL: https://issues.apache.org/jira/browse/HIVE-9445
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
 Attachments: HIVE-9445.1.patch, HIVE-9445.1.patch


 HIVE-5700 has the following issues:
 * HIVE-8730 - fails mysql upgrades
 * Does not upgrade all metadata, e.g. {{PARTITIONS.PART_NAME}} See comments 
 in HIVE-5700.
 * Completely corrupts postgres, see below.
 With a postgres metastore on 0.12, I executed the following:
 {noformat}
 CREATE TABLE HIVE5700_DATE_PARTED (line string) PARTITIONED BY (ddate date);
 CREATE TABLE HIVE5700_STRING_PARTED (line string) PARTITIONED BY (ddate 
 string);
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='NOT_DATE');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150121');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150122');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='2015-01-23');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='NOT_DATE');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150121');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150122');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='2015-01-23');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='NOT_DATE');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='20150121');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='20150122');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='2015-01-23');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='NOT_DATE');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='20150121');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='20150122');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='2015-01-23');
 hive show partitions HIVE5700_DATE_PARTED;  
 OK
 ddate=20150121
 ddate=20150122
 ddate=2015-01-23
 ddate=NOT_DATE
 Time taken: 0.052 seconds, Fetched: 4 row(s)
 hive show partitions HIVE5700_STRING_PARTED;
 OK
 ddate=20150121
 ddate=20150122
 ddate=2015-01-23
 ddate=NOT_DATE
 Time taken: 0.051 seconds, Fetched: 4 row(s)
 {noformat}
 I then took a dump of the database named {{postgres-pre-upgrade.sql}} and the 
 data in the dump looks good:
 {noformat}
 [root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY 
 PARTITION_KEY_VALS' postgres-pre-upgrade.sql 
 COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME, 
 SD_ID, TBL_ID) FROM stdin;
 3 1421943647  0   ddate=NOT_DATE  6   2
 4 1421943647  0   ddate=20150121  7   2
 5 1421943648  0   ddate=20150122  8   2
 6 1421943664  0   ddate=NOT_DATE  9   3
 7 1421943664  0   ddate=20150121  10  3
 8 1421943665  0   ddate=20150122  11  3
 9 1421943694  0   ddate=2015-01-2312  2
 101421943695  0   ddate=2015-01-2313  3
 \.
 --
 COPY PARTITION_KEY_VALS (PART_ID, PART_KEY_VAL, INTEGER_IDX) FROM 
 stdin;
 3 NOT_DATE0
 4 201501210
 5 201501220
 6 NOT_DATE0
 7 201501210
 8 201501220
 9 2015-01-23  0
 102015-01-23  0
 \.
 {noformat}
 I then upgraded to 0.13 and subsequently upgraded the MS with the following 
 command: {{schematool -dbType postgres -upgradeSchema -verbose}}
 The file {{postgres-post-upgrade.sql}} is the post-upgrade db dump. As you 
 can see the data is completely corrupt.
 {noformat}
 [root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY 
 PARTITION_KEY_VALS' postgres-post-upgrade.sql 
 COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME, 
 SD_ID, TBL_ID) FROM stdin;
 3 1421943647  0   ddate=NOT_DATE  6   2
 4 1421943647  0   ddate=20150121  7   2
 5 1421943648  0   ddate=20150122  8   2
 6 1421943664  0   ddate=NOT_DATE  9   3
 7 1421943664  0   ddate=20150121  10  3
 8 1421943665  0   ddate=20150122  11

[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

2015-01-24 Thread Hive QA (JIRA)


[ 
https://issues.apache.org/jira/browse/HIVE-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291014#comment-14291014
 ] 

Hive QA commented on HIVE-9445:
---



{color:green}Overall{color}: +1 all checks pass

Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12694410/HIVE-9445.1.patch

{color:green}SUCCESS:{color} +1 7365 tests passed

Test results: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2515/testReport
Console output: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2515/console
Test logs: 
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2515/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12694410 - PreCommit-HIVE-TRUNK-Build

 Revert HIVE-5700 - enforce single date format for partition column storage
 --

 Key: HIVE-9445
 URL: https://issues.apache.org/jira/browse/HIVE-9445
 Project: Hive
  Issue Type: Bug
  Components: Metastore
Affects Versions: 0.13.0, 0.14.0, 0.13.1, 0.15.0, 0.14.1
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Blocker
 Attachments: HIVE-9445.1.patch, HIVE-9445.1.patch


 HIVE-5700 has the following issues:
 * HIVE-8730 - fails mysql upgrades
 * Does not upgrade all metadata, e.g. {{PARTITIONS.PART_NAME}} See comments 
 in HIVE-5700.
 * Completely corrupts postgres, see below.
 With a postgres metastore on 0.12, I executed the following:
 {noformat}
 CREATE TABLE HIVE5700_DATE_PARTED (line string) PARTITIONED BY (ddate date);
 CREATE TABLE HIVE5700_STRING_PARTED (line string) PARTITIONED BY (ddate 
 string);
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='NOT_DATE');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150121');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='20150122');
 ALTER TABLE HIVE5700_DATE_PARTED ADD PARTITION (ddate='2015-01-23');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='NOT_DATE');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150121');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='20150122');
 ALTER TABLE HIVE5700_STRING_PARTED ADD PARTITION (ddate='2015-01-23');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='NOT_DATE');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='20150121');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='20150122');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_DATE_PARTED PARTITION (ddate='2015-01-23');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='NOT_DATE');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='20150121');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='20150122');
 LOAD DATA LOCAL INPATH '/tmp/single-line-of-data' INTO TABLE 
 HIVE5700_STRING_PARTED PARTITION (ddate='2015-01-23');
 hive show partitions HIVE5700_DATE_PARTED;  
 OK
 ddate=20150121
 ddate=20150122
 ddate=2015-01-23
 ddate=NOT_DATE
 Time taken: 0.052 seconds, Fetched: 4 row(s)
 hive show partitions HIVE5700_STRING_PARTED;
 OK
 ddate=20150121
 ddate=20150122
 ddate=2015-01-23
 ddate=NOT_DATE
 Time taken: 0.051 seconds, Fetched: 4 row(s)
 {noformat}
 I then took a dump of the database named {{postgres-pre-upgrade.sql}} and the 
 data in the dump looks good:
 {noformat}
 [root@hive5700-1-1 ~]# egrep -A9 '^COPY PARTITIONS|^COPY 
 PARTITION_KEY_VALS' postgres-pre-upgrade.sql 
 COPY PARTITIONS (PART_ID, CREATE_TIME, LAST_ACCESS_TIME, PART_NAME, 
 SD_ID, TBL_ID) FROM stdin;
 3 1421943647  0   ddate=NOT_DATE  6   2
 4 1421943647  0   ddate=20150121  7   2
 5 1421943648  0   ddate=20150122  8   2
 6 1421943664  0   ddate=NOT_DATE  9   3
 7 1421943664  0   ddate=20150121  10  3
 8 1421943665  0   ddate=20150122  11  3
 9 1421943694  0   ddate=2015-01-2312  2
 101421943695  0   ddate=2015-01-2313  3
 \.
 --
 COPY PARTITION_KEY_VALS (PART_ID, PART_KEY_VAL, INTEGER_IDX) FROM 
 stdin;
 3 NOT_DATE0
 4 201501210
 5 201501220
 6 NOT_DATE0
 7 201501210
 8 201501220
 9 2015-01-23  0

[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

[jira] [Commented] (HIVE-9445) Revert HIVE-5700 - enforce single date format for partition column storage

5 matches

Site Navigation

Mail list logo

Footer information