rangareddy commented on issue #17363:
URL: https://github.com/apache/hudi/issues/17363#issuecomment-5351239284
This issue was reviewed as part of the JIRA-migrated backlog triage
(HUDI-8867).
**Findings: the code pattern you flagged is unchanged on `master`. Routing
to the performance validation pass for the impact numbers.**
`hudi-client/hudi-client-common/src/main/java/org/apache/hudi/index/HoodieIndexUtils.java:190`:
```java
public static List<Pair<String, HoodieBaseFile>>
getLatestBaseFilesForAllPartitions(final List<String> partitions,
final HoodieEngineContext context,
final HoodieTable hoodieTable) {
context.setJobStatus(HoodieIndexUtils.class.getSimpleName(), "Load latest
base files from all partitions: " + hoodieTable.getConfig().getTableName());
return context.flatMap(partitions, partitionPath -> {
List<Pair<String, HoodieBaseFile>> filteredFiles =
getLatestBaseFilesForPartition(partitionPath, hoodieTable).stream()
.map(baseFile -> Pair.of(partitionPath, baseFile))
.collect(toList());
return filteredFiles.stream();
}, Math.max(partitions.size(), 1));
}
```
That is still one single-partition lookup per partition, with parallelism
set to the partition count - the pattern your thread dump points at. The
callers are unchanged too: `HoodieGlobalSimpleIndex.java:106` and
`HoodieSimpleIndex.java:137`.
So the refactor this ticket asks for has not happened, and the direction (a
bulk/`listPartitions`-style call, especially for the metadata table) still
looks right.
The reason this is going to the performance queue rather than straight to a
fix: the severity claim - timeline server saturation and a stuck application -
depends on partition count and deployment shape, and needs measuring on a
realistic table before the change is scoped. The code change itself is not in
doubt; the priority and the target API shape benefit from numbers.
Keeping this open.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
