> On Aug. 28, 2014, 6:05 a.m., Szehon Ho wrote: > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 1040 > > <https://reviews.apache.org/r/24688/diff/3/?file=669965#file669965line1040> > > > > Yep, thats what I meant.
I think this option seemed not useful. Any bigger number than one reducer, which is default for order-by, will make better performance, then why don't we try with that? - Navis ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/24688/#review51747 ----------------------------------------------------------- On Aug. 27, 2014, 2:18 a.m., Navis Ryu wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/24688/ > ----------------------------------------------------------- > > (Updated Aug. 27, 2014, 2:18 a.m.) > > > Review request for hive. > > > Bugs: HIVE-7669 > https://issues.apache.org/jira/browse/HIVE-7669 > > > Repository: hive-git > > > Description > ------- > > The source table has 600 Million rows and it has a String column > "l_shipinstruct" which has 4 unique values. (Ie. these 4 values are repeated > across the 600 million rows) > > We are sorting it based on this string column "l_shipinstruct" as shown in > the below HiveQL with the following parameters. > {code:sql} > set hive.optimize.sampling.orderby=true; > set hive.optimize.sampling.orderby.number=10000000; > set hive.optimize.sampling.orderby.percent=0.1f; > > insert overwrite table lineitem_temp_report > select > l_orderkey, l_partkey, l_suppkey, l_linenumber, l_quantity, > l_extendedprice, l_discount, l_tax, l_returnflag, l_linestatus, l_shipdate, > l_commitdate, l_receiptdate, l_shipinstruct, l_shipmode, l_comment > from > lineitem > order by l_shipinstruct; > {code} > Stack Trace > Diagnostic Messages for this Task: > {noformat} > Error: java.lang.RuntimeException: Error in configuring object > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109) > at > org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75) > at > org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133) > at > org.apache.hadoop.mapred.MapTask$OldOutputCollector.<init>(MapTask.java:569) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:430) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:342) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) > Caused by: java.lang.reflect.InvocationTargetException > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:601) > at > org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106) > ... 10 more > Caused by: java.lang.IllegalArgumentException: Can't read partitions file > at > org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116) > at > org.apache.hadoop.mapred.lib.TotalOrderPartitioner.configure(TotalOrderPartitioner.java:42) > at > org.apache.hadoop.hive.ql.exec.HiveTotalOrderPartitioner.configure(HiveTotalOrderPartitioner.java:37) > ... 15 more > Caused by: java.io.IOException: Split points are out of order > at > org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:96) > ... 17 more > {noformat} > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 7f4afd9 > common/src/java/org/apache/hadoop/hive/conf/Validator.java cea9c41 > ql/src/java/org/apache/hadoop/hive/ql/exec/HiveTotalOrderPartitioner.java > 6c22362 > ql/src/java/org/apache/hadoop/hive/ql/exec/PartitionKeySampler.java 166461a > ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecDriver.java ef72039 > ql/src/test/org/apache/hadoop/hive/ql/exec/TestPartitionKeySampler.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/24688/diff/ > > > Testing > ------- > > > Thanks, > > Navis Ryu > >
