hemanthumashankar0511 commented on code in PR #6317:
URL: https://github.com/apache/hive/pull/6317#discussion_r2804553772
##########
ql/src/java/org/apache/hadoop/hive/ql/plan/MapWork.java:
##########
@@ -658,8 +658,18 @@ public String getSamplingTypeString() {
@Override
public void configureJobConf(JobConf job) {
super.configureJobConf(job);
+ // Configure each table only once, even if we read thousands of its
partitions.
+ // This avoids repeating expensive work (like loading storage drivers) for
every single partition.
+ Set<TableDesc> processedTables = new HashSet<>();
+
for (PartitionDesc partition : aliasToPartnInfo.values()) {
- PlanUtils.configureJobConf(partition.getTableDesc(), job);
+ TableDesc tableDesc = partition.getTableDesc();
+
+ // If we haven't seen this table before, configure it and remember it.
+ // If we have seen it, skip it.
+ if (tableDesc != null && processedTables.add(tableDesc)) {
Review Comment:
Good point! I agree readability is important.
I'm happy to change it to `contains()` + `add()` if you think that's
clearer. Let me know your preference and I'll update the PR accordingly.
```java
if (tableDesc != null && !processedTables.contains(tableDesc)) {
processedTables.add(tableDesc);
PlanUtils.configureJobConf(tableDesc, job);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
