Smityz opened a new issue #723: URL: https://github.com/apache/incubator-pegasus/issues/723
## Proposal Redesign of Pegasus Scanner ### Background Pegasus provides three interfaces `on_get_scanner` `on_scan` and `on_clear_scanner` , for clients to execute scanning tasks. If we want to full scan the whole table, at first, the client will call `on_get_scanner` on each partition, and then partitions return a `context_id` which is a random number generated by the server to record some parameters such as `hash_key_filter_type`, `batch_size` and the context of this scanning task. Secondly, the client uses this `context_id` to call `on_scan` and completes scanning in the corresponding partition in turn. Servers will scan the whole data of the table on the disk, and return compliant value to the client in batches. If the tasking end or any error happened, the client will call `on_clear_scanner` to clear its context_id on the server. ### Problem Statement In actual use, such a design will cause some problems. 1. **prefix scan is too slow** If we execute this scanning task: ```shell full_scan --hash_key_filter_type prefix --hash_key_filter_pattern 2021-04-21 ``` Server will scan all the data in the table, then returns the prefix match key of the pattern. But we can speed it up by using prefix seeking futures of RocksDB. 2. **scanning task is easily failed** Although we have a batch size to limit the scan time, it does not work if the data is sparse. In the case above, we need to scan almost the whole partition but it is possible that there is no row which matches the prefix,then it will be easy to timeout. ### Proposal **For problem 1** 1. Pegasus store key schema in RocksDB is like `[hashkey_len(2bytes)][hashkey][sortkey]`, so we can't directly use prefix seeking. But we can prefix seek `[01][prefix_pattern]`,`[02][prefix_pattern]`,`[03][prefix_pattern]`...`[65535][prefix_pattern]` in RocksDB. 2. Client can parallelly scan all the partitions instead of one by one. **For problem 2** 1. We can set a `HeartbeatCheck` during scanning like [Hbase StoreScanner](https://github.com/apache/hbase/blob/048ca4e43fdf8b341c9ade5a9d455f627fc76041/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java#L591), pegasus sever sends heartbeat packets periodically to avoid timeout, which performed like a stream. 2. We can change the way to count batch size: compliant value number -> already scan value number -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@pegasus.apache.org For additional commands, e-mail: dev-h...@pegasus.apache.org
