SubhamSinghal commented on code in PR #23869:
URL: https://github.com/apache/datafusion/pull/23869#discussion_r3683382117
##########
datafusion/physical-plan/src/topk/mod.rs:
##########
@@ -1811,6 +1811,435 @@ impl PartitionedTopKRank {
}
}
+/// A run of rows from a single source [`RecordBatch`] sharing one
+/// distinct ORDER BY value. Materialized at emit time via
+/// [`take_record_batch`].
+///
+/// The batch is referenced by `batch_id` rather than held directly, so the
+/// operator-scoped [`RecordBatchStore`] can charge each distinct source
+/// batch once however many entries reference it. Holding a batch per entry
+/// and charging its bytes per entry would inflate the reservation by a
+/// factor of (partitions × K), since a single batch contributes an entry to
+/// every partition and ob group it touches.
+#[derive(Debug)]
+struct GroupEntry {
+ /// Indices into the batch identified by `batch_id`. Always non-empty
+ /// by construction.
+ row_indices: Vec<u32>,
+ /// Key into `PartitionedTopKDenseRank::store`.
+ batch_id: u32,
+}
+
+/// Per-partition state for `DENSE_RANK()` semantics.
+///
+/// A `HashMap<Vec<u8>, Vec<GroupEntry>>` keyed by the row-encoded ORDER
+/// BY bytes, capped at `k` distinct keys. Each key's `Vec<GroupEntry>`
+/// holds every row seen at that ob value, one entry per contributing
+/// source `RecordBatch`.
+struct DenseRankPartitionState {
+ groups: HashMap<Vec<u8>, Vec<GroupEntry>>,
+ /// The same keys as `groups`, in a max-heap: the admission boundary
+ /// (the largest tracked ob value) is an O(1) `peek()` check, and
+ /// admission / removal are O(log K).
+ ///
Review Comment:
@kosiew are you referring to RANK code? at line 2100 I can see:
```
state.groups.insert(
ob_key,
vec![GroupEntry {
row_indices: run_indices,
batch_id,
}],
);
```
which is referring to batch_id. Looks like stale comment but please confirm.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
