LakshSingla commented on code in PR #16911:
URL: https://github.com/apache/druid/pull/16911#discussion_r1723270853
##########
extensions-core/multi-stage-query/src/main/java/org/apache/druid/msq/kernel/ShuffleSpec.java:
##########
@@ -68,4 +70,17 @@ public interface ShuffleSpec
* @throws IllegalStateException if kind is {@link ShuffleKind#GLOBAL_SORT}
with more than one target partition
*/
int partitionCount();
+
+ /**
+ * Limit that can be applied during shuffling. This is provided to enable
performance optimizations.
+ *
+ * Implementations may apply this limit to each partition individually, or
may apply it to the entire resultset
+ * (across all partitions). Either approach is valid, so downstream logic
must handle either one.
+ *
+ * Implementations may also ignore this hint completely, or may apply a
limit that is somewhat higher than this hint.
+ */
+ default long limitHint()
Review Comment:
I was wondering if there's any merit in separating the limit hint into a
POJO, and having both limit and offset baked into it. Something like
`DefaultLimitSpec` without the columns. It's up to the reader to calculate the
combined limit and use that, instead of baking that knowledge into a long.
I was thinking of use cases when we want to pushdown this limit into other
portions of MSQ's stack and want to distinguish between rows [0..offset)
(thrown away) and rows [offset, offset + limit) (kept)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
