yew1eb commented on code in PR #3825:
URL: https://github.com/apache/celeborn/pull/3825#discussion_r3843710129
##########
common/src/main/java/org/apache/celeborn/common/protocol/PartitionLocation.java:
##########
@@ -243,32 +229,93 @@ public String getUniqueId() {
return id + "-" + epoch;
}
+ public long getUniqueIdLong() {
+ return toUniqueIdLong(id, epoch);
+ }
+
+ public static long toUniqueIdLong(int id, int epoch) {
+ return ((long) id << 32) | (epoch & 0xffffffffL);
+ }
+
+ public static long toUniqueIdLong(String uniqueId) {
+ Long packedUniqueId = tryToUniqueIdLong(uniqueId);
+ if (packedUniqueId == null) {
+ throw new IllegalArgumentException("Invalid partition location unique
id: " + uniqueId);
+ }
+ return packedUniqueId;
+ }
+
+ public static Long tryToUniqueIdLong(String uniqueId) {
+ if (uniqueId == null || uniqueId.length() < 3) {
+ return null;
+ }
+ int idStart = uniqueId.charAt(0) == '-' ? 1 : 0;
+ int separatorIndex = uniqueId.indexOf('-', idStart);
+ if (separatorIndex < 0) {
+ return null;
+ }
+ long id = tryParseInt(uniqueId, 0, separatorIndex);
+ long epoch = tryParseInt(uniqueId, separatorIndex + 1, uniqueId.length());
+ if (id == INVALID_PARSED_INT || epoch == INVALID_PARSED_INT) {
+ return null;
+ }
+ return toUniqueIdLong((int) id, (int) epoch);
+ }
Review Comment:
`getUniqueIdLong` / `toUniqueIdLong` / `tryToUniqueIdLong` (and
`tryParseInt` below) have no production call site in this PR —
`WorkerPartitionLocationInfo` keeps `String` keys. Shall we drop them to keep
the protocol class minimal, or are they intended for a follow-up?
##########
common/pom.xml:
##########
@@ -158,6 +158,12 @@
<artifactId>mockito-core</artifactId>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.openjdk.jol</groupId>
+ <artifactId>jol-core</artifactId>
+ <version>0.17</version>
+ <scope>test</scope>
+ </dependency>
<dependency>
Review Comment:
Nit: please move the version to `<dependencyManagement>` in the root
`pom.xml` (e.g. a `jol.version` property), per project convention. Also,
`jol-core` is not declared in `project/CelebornBuild.scala`, so the benchmark
can't run under SBT — either add it there or note that it's Maven-only.
##########
client/src/main/java/org/apache/celeborn/client/read/CelebornInputStream.java:
##########
@@ -172,6 +172,27 @@ public int partitionsRead() {
public abstract int partitionsRead();
+ static boolean shouldSkipLocation(
+ boolean rangeReadFilter, int startMapIndex, int endMapIndex,
PartitionLocation location) {
+ if (!rangeReadFilter || endMapIndex == Integer.MAX_VALUE) {
+ return false;
+ }
+ RoaringBitmap bitmap = location.getMapIdBitMapIfPresent();
+ if (bitmap == null && location.hasPeer()) {
+ bitmap = location.getPeer().getMapIdBitMapIfPresent();
+ }
+ if (bitmap == null) {
+ // Missing filter metadata cannot prove that this location is irrelevant.
+ return false;
+ }
+ for (int i = startMapIndex; i < endMapIndex; i++) {
+ if (bitmap.contains(i)) {
+ return false;
+ }
Review Comment:
Nit (can be a follow-up): this per-index scan can be a single range query,
O(containers) instead of O(range):
```java
return !bitmap.intersects(startMapIndex, (long) endMapIndex);
```
`intersects(long, long)` is available in RoaringBitmap 1.0.6.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
