> On July 22, 2016, 8:17 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 1730 > > <https://reviews.apache.org/r/49881/diff/4/?file=1449676#file1449676line1730> > > > > Lets use different config name here. Overloading this one at too many > > places.
Added "hive.load.dynamic.partitions.thread" in latst patch. > On July 22, 2016, 8:17 p.m., Ashutosh Chauhan wrote: > > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java, line 1763 > > <https://reviews.apache.org/r/49881/diff/4/?file=1449676#file1449676line1763> > > > > This is going to retrieve every partitions in Table. This may be > > problematic for tables which already had large number of partitions. Added getPartitionsForPath which fetches the partitions on need basis (instead of fetching all partitions) - Rajesh ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/49881/#review143246 ----------------------------------------------------------- On July 25, 2016, 4:43 a.m., Rajesh Balamohan wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/49881/ > ----------------------------------------------------------- > > (Updated July 25, 2016, 4:43 a.m.) > > > Review request for hive and Ashutosh Chauhan. > > > Bugs: HIVE-14204 > https://issues.apache.org/jira/browse/HIVE-14204 > > > Repository: hive-git > > > Description > ------- > > Lots of time is spent in sequential fashion to load dynamic partitioned > dataset in driver side. > > E.g simple dynamic partitioned load as follows takes 300+ seconds > > INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from > tpcds_bin_partitioned_orc_200.web_sales; > > Time taken to load dynamic partitions: 309.22 seconds > > > Diffs > ----- > > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java e92466f > metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java > 5adfa02 > metastore/src/java/org/apache/hadoop/hive/metastore/Warehouse.java d624d1b > > ql/src/java/org/apache/hadoop/hive/metastore/SynchronizedMetaStoreClient.java > PRE-CREATION > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbLockManager.java b4ae1d1 > ql/src/java/org/apache/hadoop/hive/ql/lockmgr/DbTxnManager.java 7b0369d > ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 9d927bd > > Diff: https://reviews.apache.org/r/49881/diff/ > > > Testing > ------- > > > Thanks, > > Rajesh Balamohan > >
