FANNG1 commented on PR #10671:
URL: https://github.com/apache/gravitino/pull/10671#issuecomment-4533227967
Thanks for the detailed explanation on in-memory pagination being the only
practical option given the current `Catalog` interface constraints. That makes
sense.
However, I still think the offset-based token is a concern worth addressing,
since it can silently produce incorrect results (missing or duplicate entries)
without any indication to the caller — not just as an edge case, but as a
predictable failure mode whenever the catalog is active.
**Proposed alternative: name-based cursor token**
Since the full list is already materialized in memory on every request, we
can make pagination significantly more robust at no extra cost:
1. **Sort the list deterministically** (e.g., by name alphabetically) before
slicing.
2. **Use the last returned item's name as the `pageToken`** (cursor),
instead of a numeric offset.
3. On the next request, binary-search or scan the sorted list for the cursor
name and return items after it.
```java
// Example sketch
List<Namespace> sorted = all.stream()
.sorted(Comparator.comparing(ns -> ns.toString()))
.collect(toList());
int startIdx = 0;
if (pageToken != null && !pageToken.isEmpty()) {
// Find the first item strictly after the cursor
startIdx = IntStream.range(0, sorted.size())
.filter(i -> sorted.get(i).toString().compareTo(pageToken) > 0)
.findFirst()
.orElse(sorted.size());
}
List<Namespace> page = sorted.subList(startIdx, Math.min(startIdx +
pageSize, sorted.size()));
String nextToken = page.isEmpty() ? null : page.get(page.size() -
1).toString();
```
**Why this is more resilient:**
| Scenario | Offset-based (current) | Name-cursor (proposed) |
|---|---|---|
| Item deleted before cursor | Subsequent page **skips** an item | Cursor
still valid, no data lost |
| Item created before cursor | Subsequent page **duplicates** vicinity |
Item already passed, correctly skipped |
| Item created after cursor | Correctly included | Correctly included |
| Cursor item itself deleted | No issue (offset just moves) | First item
alphabetically after cursor name is returned — graceful |
The only remaining edge case is items created *after* the cursor but
*before* the current page's last item — these will be missed, which is the same
behavior as most keyset-pagination implementations and generally acceptable.
This is a much smaller window than the current approach.
Since the list is already sorted by most underlying catalog implementations
(Hive, JDBC), the sort step may even be a no-op in practice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
