----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71707/#review218505 -----------------------------------------------------------
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java Lines 328 (patched) <https://reviews.apache.org/r/71707/#comment306253> Hey Attila, the solution looks good however, as other fileSystems might face similar issues in the future using this recursive method (i.e. Azure Blob storage) wouldn't it make sense to have hdfs a the base case and others separately? and maybe throw a warn message here when the filesystem is not supported? - Panos Garefalakis On Nov. 5, 2019, 3:32 p.m., Attila Magyar wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71707/ > ----------------------------------------------------------- > > (Updated Nov. 5, 2019, 3:32 p.m.) > > > Review request for hive, Ashutosh Chauhan, Peter Vary, and Slim Bouguerra. > > > Bugs: HIVE-22411 > https://issues.apache.org/jira/browse/HIVE-22411 > > > Repository: hive-git > > > Description > ------- > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > > Therefore insertion time goes up linearly. > > > Diffs > ----- > > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java > 38e843aeacf > > standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java > bf206fffc26 > > > Diff: https://reviews.apache.org/r/71707/diff/2/ > > > Testing > ------- > > measured and plotted insertation time > > > Thanks, > > Attila Magyar > >
