HeartSaVioR commented on a change in pull request #33038:
URL: https://github.com/apache/spark/pull/33038#discussion_r667611207
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
##########
@@ -60,21 +60,17 @@ trait ReadStateStore {
def get(key: UnsafeRow): UnsafeRow
/**
- * Get key value pairs with optional approximate `start` and `end` extents.
- * If the State Store implementation maintains indices for the data based on
the optional
- * `keyIndexOrdinal` over fields `keySchema` (see
`StateStoreProvider.init()`), then it can use
- * `start` and `end` to make a best-effort scan over the data. Default
implementation returns
- * the full data scan iterator, which is correct but inefficient. Custom
implementations must
- * ensure that updates (puts, removes) can be made while iterating over this
iterator.
+ * Return an iterator containing all the key-value pairs which are matched
with
+ * the given prefix key.
+ *
+ * Spark will provide numColsPrefixKey greater than 0 in
StateStoreProvider.init method if
+ * the state store is responsible to handle the request for prefix scan. The
schema of the
+ * prefix key should be same with the leftmost `numColsPrefixKey` columns of
the key schema.
*
- * @param start UnsafeRow having the `keyIndexOrdinal` column set with
appropriate starting value.
- * @param end UnsafeRow having the `keyIndexOrdinal` column set with
appropriate ending value.
- * @return An iterator of key-value pairs that is guaranteed not miss any
key between start and
- * end, both inclusive.
+ * It is expected to throw exception if Spark calls this method without
setting numColsPrefixKey
+ * to the greater than 0.
*/
- def getRange(start: Option[UnsafeRow], end: Option[UnsafeRow]):
Iterator[UnsafeRowPair] = {
- iterator()
- }
+ def prefixScan(prefixKey: UnsafeRow): Iterator[UnsafeRowPair]
Review comment:
I just commented on "partial implementation" of state store.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
