[
https://issues.apache.org/jira/browse/HIVE-27158?focusedWorklogId=851927&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-851927
]
ASF GitHub Bot logged work on HIVE-27158:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 21/Mar/23 05:04
Start Date: 21/Mar/23 05:04
Worklog Time Spent: 10m
Work Description: rbalamohan commented on code in PR #4131:
URL: https://github.com/apache/hive/pull/4131#discussion_r1142898325
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -349,6 +364,98 @@ public Map<String, String> getBasicStatistics(Partish
partish) {
return stats;
}
+
+ @Override
+ public boolean canSetColStatistics() {
+ String statsSource = HiveConf.getVar(conf,
HiveConf.ConfVars.HIVE_USE_STATS_FROM).toLowerCase();
+ return statsSource.equals(PUFFIN) ? true : false;
+ }
+
+ @Override
+ public boolean
canProvideColStatistics(org.apache.hadoop.hive.ql.metadata.Table tbl) {
+
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable = tbl;
+ TableDesc tableDesc = Utilities.getTableDesc(hmsTable);
+ Table table = Catalogs.loadTable(conf, tableDesc.getProperties());
+ if (table.currentSnapshot() != null) {
+ String statsSource = HiveConf.getVar(conf,
HiveConf.ConfVars.HIVE_USE_STATS_FROM).toLowerCase();
+ String statsPath = table.location() + "/stats/" + table.name() +
table.currentSnapshot().snapshotId();
+ if (statsSource.equals(PUFFIN)) {
+ try (FileSystem fs = new Path(table.location()).getFileSystem(conf)) {
+ if (fs.exists(new Path(statsPath))) {
+ return true;
+ }
+ } catch (IOException e) {
+ LOG.warn(e.getMessage());
+ }
+ }
+ }
+ return false;
+ }
+
+ @Override
+ public List<ColumnStatisticsObj>
getColStatistics(org.apache.hadoop.hive.ql.metadata.Table tbl) {
+
+ org.apache.hadoop.hive.ql.metadata.Table hmsTable = tbl;
+ TableDesc tableDesc = Utilities.getTableDesc(hmsTable);
+ Table table = Catalogs.loadTable(conf, tableDesc.getProperties());
+ String statsSource = HiveConf.getVar(conf,
HiveConf.ConfVars.HIVE_USE_STATS_FROM).toLowerCase();
+ switch (statsSource) {
+ case ICEBERG:
+ // Place holder for iceberg stats
+ break;
+ case PUFFIN:
+ String snapshotId = table.name() +
table.currentSnapshot().snapshotId();
Review Comment:
Is this mainly for col stats? As in, do we plan to standardize this for
iceberg?
As of today, it seems to be querying HMS for basic stats and with this, it
may have a mix of HMS + FileReading.
Issue Time Tracking
-------------------
Worklog Id: (was: 851927)
Time Spent: 0.5h (was: 20m)
> Store hive columns stats in puffin files for iceberg tables
> -----------------------------------------------------------
>
> Key: HIVE-27158
> URL: https://issues.apache.org/jira/browse/HIVE-27158
> Project: Hive
> Issue Type: Improvement
> Reporter: Simhadri Govindappa
> Assignee: Simhadri Govindappa
> Priority: Major
> Labels: pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
