aidar-stripe commented on PR #3100:
URL: https://github.com/apache/celeborn/pull/3100#issuecomment-2666352745
@FMX thanks for the link! I think you are absolutely right here, we were
running a version of Celeborn client (it's been 0.5.1 with some of our commits
for integrity checks, which were disabled).
I could confirm that PbGetReducerFileGroupResponse conversion code only
takes primaries there:
```
val fileGroup =
pbGetReducerFileGroupResponse.getFileGroupsMap.asScala.map {
case (partitionId, fileGroup) =>
(
partitionId,
PbSerDeUtils.fromPbPackedPartitionLocationsPair(
fileGroup.getPartitionLocationsPair)._1.asScala.toSet.asJava)
}.asJava
```
This explains consistency of the failures that we've seen much better than
the potential concurrency issue with the HashSet. I would still like to merge
in the PR though, I think usage of ConcurrentHashSet still more appropriate
there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
