[jira] [Work logged] (HIVE-27158) Store hive columns stats in puffin files for iceberg tables

[ASF GitHub Bot (Jira)]

     [ 
https://issues.apache.org/jira/browse/HIVE-27158?focusedWorklogId=855896&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-855896
 ]

ASF GitHub Bot logged work on HIVE-27158:
-----------------------------------------

                Author: ASF GitHub Bot
            Created on: 10/Apr/23 16:08
            Start Date: 10/Apr/23 16:08
    Worklog Time Spent: 10m 
      Work Description: simhadri-g commented on code in PR #4131:
URL: https://github.com/apache/hive/pull/4131#discussion_r1161856243


##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -361,6 +378,83 @@ private Table 
getTable(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
     return table;
   }
 
+
+  @Override
+  public boolean canSetColStatistics() {
+    return getStatsSource().equals(ICEBERG);
+  }
+
+  @Override
+  public boolean 
canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+    Table table = Catalogs.loadTable(conf, 
Utilities.getTableDesc(hmsTable).getProperties());
+    if (table.currentSnapshot() != null) {
+      Path statsPath = getStatsPath(table);
+      if (getStatsSource().equals(ICEBERG)) {
+        try (FileSystem fs = statsPath.getFileSystem(conf)) {
+          if (fs.exists(statsPath)) {
+            return true;
+          }
+        } catch (IOException e) {
+          LOG.warn(e.getMessage());

Review Comment:
   Done.



##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -361,6 +378,83 @@ private Table 
getTable(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
     return table;
   }
 
+
+  @Override
+  public boolean canSetColStatistics() {
+    return getStatsSource().equals(ICEBERG);
+  }
+
+  @Override
+  public boolean 
canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+    Table table = Catalogs.loadTable(conf, 
Utilities.getTableDesc(hmsTable).getProperties());
+    if (table.currentSnapshot() != null) {
+      Path statsPath = getStatsPath(table);
+      if (getStatsSource().equals(ICEBERG)) {
+        try (FileSystem fs = statsPath.getFileSystem(conf)) {
+          if (fs.exists(statsPath)) {
+            return true;
+          }
+        } catch (IOException e) {
+          LOG.warn(e.getMessage());
+        }
+      }
+    }
+    return false;
+  }
+
+  @Override
+  public List<ColumnStatisticsObj> 
getColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+    Table table = Catalogs.loadTable(conf, 
Utilities.getTableDesc(hmsTable).getProperties());

Review Comment:
   Done



##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -361,6 +378,83 @@ private Table 
getTable(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
     return table;
   }
 
+
+  @Override
+  public boolean canSetColStatistics() {
+    return getStatsSource().equals(ICEBERG);
+  }
+
+  @Override
+  public boolean 
canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+    Table table = Catalogs.loadTable(conf, 
Utilities.getTableDesc(hmsTable).getProperties());
+    if (table.currentSnapshot() != null) {
+      Path statsPath = getStatsPath(table);
+      if (getStatsSource().equals(ICEBERG)) {
+        try (FileSystem fs = statsPath.getFileSystem(conf)) {
+          if (fs.exists(statsPath)) {
+            return true;
+          }
+        } catch (IOException e) {
+          LOG.warn(e.getMessage());
+        }
+      }
+    }
+    return false;
+  }
+
+  @Override
+  public List<ColumnStatisticsObj> 
getColStatistics(org.apache.hadoop.hive.ql.metadata.Table hmsTable) {
+    Table table = Catalogs.loadTable(conf, 
Utilities.getTableDesc(hmsTable).getProperties());
+    String statsPath = getStatsPath(table).toString();
+    LOG.info("Using stats from puffin file at:" + statsPath);

Review Comment:
   Done





Issue Time Tracking
-------------------

    Worklog Id:     (was: 855896)
    Time Spent: 8h  (was: 7h 50m)

> Store hive columns stats in puffin files for iceberg tables
> -----------------------------------------------------------
>
>                 Key: HIVE-27158
>                 URL: https://issues.apache.org/jira/browse/HIVE-27158
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Simhadri Govindappa
>            Assignee: Simhadri Govindappa
>            Priority: Major
>              Labels: pull-request-available
>          Time Spent: 8h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)