[ 
https://issues.apache.org/jira/browse/HIVE-23791?focusedWorklogId=453565&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-453565
 ]

ASF GitHub Bot logged work on HIVE-23791:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 01/Jul/20 17:56
            Start Date: 01/Jul/20 17:56
    Worklog Time Spent: 10m 
      Work Description: pvary commented on a change in pull request #1196:
URL: https://github.com/apache/hive/pull/1196#discussion_r448527274



##########
File path: ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java
##########
@@ -2614,28 +2633,25 @@ public static Path getVersionFilePath(Path deltaOrBase) 
{
           + " from " + jc.get(ValidTxnWriteIdList.VALID_TABLES_WRITEIDS_KEY));
       return null;
     }
-    Directory acidInfo = AcidUtils.getAcidState(fs, dir, jc, idList, null, 
false);
+    if (fs == null) {
+      fs = dir.getFileSystem(jc);
+    }
+    // Collect the all of the files/dirs
+    Map<Path, HdfsDirSnapshot> hdfsDirSnapshots = 
AcidUtils.getHdfsDirSnapshots(fs, dir);

Review comment:
       Ohh.. I think I get it now.
   * You are right that this will do stuff which is not really needed in this 
case - namely creating objects which are not needed at here 
(dirSnapshot.metaDataFile/dirSnapshot.acidFormatFile), also we might list and 
create objects which are not needed in this snapshot. On the other hand the 
costly part on S3 (and on HDFS as well) is the number of remote calls, which is 
reduced to a single listing instead of doing the listing for every directory 
1-by-1.
   * It is not possible that it does not scan some location which needed. If 
this happens then this is a bug in AcidUtils.getAcidState, as it has to return 
every directory which is readable
   
   What I do not understand in your comment is "this method would return a 
something (it could still be a map) which could fill in stuff from hdfs if its 
not cached already" - the main thing we would like to avoid here is the need of 
reading the HDFS again and again. The only way to realize that something is 
missing is reading the directory again... or I miss something :)
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 453565)
    Time Spent: 40m  (was: 0.5h)

> Optimize ACID stats generation
> ------------------------------
>
>                 Key: HIVE-23791
>                 URL: https://issues.apache.org/jira/browse/HIVE-23791
>             Project: Hive
>          Issue Type: Improvement
>          Components: Statistics, Transactions
>            Reporter: Peter Vary
>            Assignee: Peter Vary
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 40m
>  Remaining Estimate: 0h
>
> Currently basic stats generation uses file listing for getting statistics, 
> and also uses a file listing for getting the acid state. We should optimize 
> this.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to