Danny Teok created HIVE-5774: -------------------------------- Summary: INSERT OVERWRITE DYNAMIC PARTITION on LARGE DATA Key: HIVE-5774 URL: https://issues.apache.org/jira/browse/HIVE-5774 Project: Hive Issue Type: Bug Components: Database/Schema Environment: debian 6.0.7 Reporter: Danny Teok Priority: Critical
After several forensic analysis, we are convinced that there is a bug when rebuilding using dynamic partition over more than 30 days. Row counts do not match. In details: Part A -- original_table 2013-01-01; 394,755 rows 2013-01-02; 424,448 2013-01-03; 427,201 ... 2013-10-30; 3,234,472 Part B -- copy_of_original_table_new 2013-01-01; 372,628 rows 2013-01-02; 400,553 2013-01-03; 403,495 ... 2013-10-30; 2,865,877 The query that is used to populate the original table is the same for populating the "copy_of_original_table_new" table. When we rebuilt for 1 day, e.g. 2013-01-01, the number of row counts of the copy_of_original_table_new matched up exactly with orignal_table. When we rebuilt for 7 days, the number of row counts matched up exactly. When we rebuilt for 15 days, the number of row counts matched up exactly. When we rebuilt for 303 days (10 months), everything fxxked up. No matches. When we rebuilt for 35 days, 80% matched up exactly. The other 20% are out from hundreds to tens of thousands of rows (a variance of up to 3%) In other words, the more days that are specified in the WHERE dt BETWEEN dateStart AND dateEnd, the dates will be out, i.e. no matching row count with original_table. However, of those 20% that are out, we rebuilt each of them statically with the corresponding date. The result is astonishingly surprising -- they matched the original_table row count! Apologize in advance if this is not technical enough, but I hope the message is clear. We believe there is a bug. Not sure how to check our Hive version, but our Hadoop's version is "Hadoop 2.0.0-cdh4.1.1" For a glimpse of the INSERT OVERWRITE sql, it's here -- http://pastebin.com/g1qxsUm2 -- This message was sent by Atlassian JIRA (v6.1#6144)