kasakrisz commented on a change in pull request #2991:
URL: https://github.com/apache/hive/pull/2991#discussion_r800706374
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java
##########
@@ -512,6 +513,8 @@ private String toString(Map<String, String> parameters) {
if (dpPartSpecs != null) {
// load the list of DP partitions and return the list of partition
specs
list.addAll(dpPartSpecs);
+ // Reload partition metadata because another BasicStatsTask instance
may have updated the stats.
+ list = db.getPartitionsByNames(table,
list.stream().map(Partition::getName).collect(Collectors.toList()));
Review comment:
I think these are thrift API calls from HS2 to HMS and the number of
calls depend on the number of partitions and
`hive.metastore.batch.retrieve.max`.
Also there is some communication from HMS to backend db which is also done
in batches.
I explored another way:
* In the StatsTask instead of applying the stats on the Partition objects
collect it to a delta objects.
* Implement a HMS call and backend DB call to update stats based on the
collected delta objects.
However this would be a bigger change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
