jon-wei commented on a change in pull request #7133: 6088 - Time Ordering On
Scans
URL: https://github.com/apache/incubator-druid/pull/7133#discussion_r263617590
##########
File path: docs/content/querying/scan-query.md
##########
@@ -168,18 +148,38 @@ The format of the result when resultFormat equals to
`compactedList`:
"events" : [
["2013-01-01T00:00:00.000Z", "1", "article", "0", "0",
"11._korpus_(NOVJ)", "sl", "0", "EmausBot", 1.0, 39.0, 39.0, 39.0, 0.0],
["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "112_U.S._580",
"en", "1", "MZMcBride", 1.0, 70.0, 70.0, 70.0, 0.0],
- ["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "113_U.S._243",
"en", "1", "MZMcBride", 1.0, 77.0, 77.0, 77.0, 0.0],
- ["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "113_U.S._73",
"en", "1", "MZMcBride", 1.0, 70.0, 70.0, 70.0, 0.0],
- ["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "113_U.S._756",
"en", "1", "MZMcBride", 1.0, 68.0, 68.0, 68.0, 0.0]
+ ["2013-01-01T00:00:00.000Z", "0", "article", "0", "0", "113_U.S._243",
"en", "1", "MZMcBride", 1.0, 77.0, 77.0, 77.0, 0.0]
]
} ]
```
-The biggest difference between select query and scan query is that, scan query
doesn't retain all rows in memory before rows can be returned to client.
-It will cause memory pressure if too many rows required by select query.
-Scan query doesn't have this issue.
-Scan query can return all rows without issuing another pagination query, which
is extremely useful when query against Historical or realtime process directly.
-
+## Time Ordering
+
+The Scan query currently supports ordering based on timestamp for non-legacy
queries. Note that using time ordering
+will yield results that do not indicate which segment rows are from.
Furthermore, time ordering is only supported
+where the result set limit is less than
`druid.query.scan.maxRowsQueuedForTimeOrdering` rows and less than
+`druid.query.scan.maxSegmentsTimeOrderedInMemory` segments are scanned per
Historical. The reasoning behind these
+limitations is that the implementation of time ordering uses two strategies
that can consume too much heap memory
+if left unbounded. These strategies (listed below) are chosen on a
per-Historical basis depending on query
+result set limit and the number of segments being scanned.
+
+1. Priority Queue: Each segment on a Historical is opened sequentially. Every
row is added to a bounded priority
+queue which is ordered by timestamp. For every row above the result set
limit, the row with the earliest (if descending)
+or latest (if ascending) timestamp will be dequeued. After every row has been
processed, the sorted contents of the
+priority queue are streamed back to the Broker(s) in batches. Attempting to
load too many rows into memory runs the
+risk of Historical nodes running out of memory. The
`druid.query.scan.maxRowsQueuedForTimeOrdering` property protects
+from this by limiting the number of rows in the query result set when time
ordering is used.
+
+2. K-Way/N-Way Merge: Each segment on a Historical is opened in parallel.
Since each segment's rows are already
Review comment:
nit: suggest picking "K" or "N" but not both
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
