[
https://issues.apache.org/jira/browse/HIVE-25373?focusedWorklogId=628861&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-628861
]
ASF GitHub Bot logged work on HIVE-25373:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Jul/21 04:18
Start Date: 28/Jul/21 04:18
Worklog Time Spent: 10m
Work Description: rbalamohan commented on a change in pull request #2520:
URL: https://github.com/apache/hive/pull/2520#discussion_r677960313
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/ColStatsProcessor.java
##########
@@ -177,25 +182,36 @@ private ColumnStatisticsDesc buildColumnStatsDesc(Table
table, String partName,
public int persistColumnStats(Hive db, Table tbl) throws HiveException,
MetaException, IOException {
// Construct a column statistics object from the result
- List<ColumnStatistics> colStats = constructColumnStatsFromPackedRows(tbl);
- // Persist the column statistics object to the metastore
- // Note, this function is shared for both table and partition column stats.
- if (colStats.isEmpty()) {
- return 0;
- }
- SetPartitionsStatsRequest request = new
SetPartitionsStatsRequest(colStats, Constants.HIVE_ENGINE);
- request.setNeedMerge(colStatDesc.isNeedMerge());
- HiveTxnManager txnMgr = AcidUtils.isTransactionalTable(tbl)
- ? SessionState.get().getTxnMgr() : null;
+ long writeId = -1;
+ ValidWriteIdList validWriteIdList = null;
+ HiveTxnManager txnMgr = AcidUtils.isTransactionalTable(tbl) ?
SessionState.get().getTxnMgr() : null;
if (txnMgr != null) {
- request.setWriteId(txnMgr.getAllocatedTableWriteId(tbl.getDbName(),
tbl.getTableName()));
- ValidWriteIdList validWriteIdList =
- AcidUtils.getTableValidWriteIdList(conf,
AcidUtils.getFullTableName(tbl.getDbName(), tbl.getTableName()));
- if (validWriteIdList != null) {
- request.setValidWriteIdList(validWriteIdList.toString());
+ writeId = txnMgr.getAllocatedTableWriteId(tbl.getDbName(),
tbl.getTableName());
+ validWriteIdList =
+ AcidUtils.getTableValidWriteIdList(conf,
AcidUtils.getFullTableName(tbl.getDbName(), tbl.getTableName()));
+ }
+
+ boolean done = false;
+ long maxNumStats =
conf.getLongVar(HiveConf.ConfVars.HIVE_STATS_MAX_NUM_STATS);
+ while (!done) {
+ List<ColumnStatistics> colStats = new ArrayList<>();
+ done = constructColumnStatsFromPackedRows(tbl, colStats, maxNumStats);
+
+ // Persist the column statistics object to the metastore
+ // Note, this function is shared for both table and partition column
stats.
+ if (colStats.isEmpty()) {
+ continue;
+ }
+ SetPartitionsStatsRequest request = new
SetPartitionsStatsRequest(colStats, Constants.HIVE_ENGINE);
+ request.setNeedMerge(colStatDesc.isNeedMerge());
+ if (txnMgr != null) {
+ request.setWriteId(writeId);
+ if (validWriteIdList != null) {
+ request.setValidWriteIdList(validWriteIdList.toString());
+ }
}
+ db.setPartitionColumnStatistics(request);
Review comment:
Info level logging would be good to have here. Since it will be printed
for every 10K updates, it should be fine and will be helpful for debugging as
well. And if possible provide time taken for overall buildColumnStatsDesc.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 628861)
Time Spent: 20m (was: 10m)
> Modify buildColumnStatsDesc to send configured number of stats for updation
> ---------------------------------------------------------------------------
>
> Key: HIVE-25373
> URL: https://issues.apache.org/jira/browse/HIVE-25373
> Project: Hive
> Issue Type: Sub-task
> Reporter: mahesh kumar behera
> Assignee: mahesh kumar behera
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The number of stats sent for updation should be controlled to avoid thrift
> error in case the size exceeds the limit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
