ninsmiracle opened a new pull request, #1216:
URL: https://github.com/apache/incubator-pegasus/pull/1216
### What problem does this PR solve? <!--add issue link with summary if
exists-->
#1215
To solve bulk load cu problem.
### What is changed and how does it work?
## Date Structure
`hashkey: data in format of "2022-10-12 00:33:25"
sortkey: "bulkload_cu@appId:partitionID"
value: JSON in structure of {"appId:[bulkloadCU]"}`
Design reason:
-Keep the same format with CU, which is convenient for other billing system
collection.
## How replica get BulkLoad CU date?
As we know, when RocksDB doing BulkLoad, enegine should download a file
named meta_date first.For pegasus ,each partition has a meta_data file that
contains the size and md5 information of each sst.This file has some date like
this format:
`{"files":[{"name":"1.sst","size":14273079,"md5":"9ca4d0f21912eab77cf387ef9bb95eb8"},{"name":"10.sst","size":8908447,"md5":"534b8a64b3d4659833599ebdf0719988"}}`
The replica will first download the file and then start download sst file.
During this process, the replica may cache the file locally. If the file name
is detected, the file size will be checked. If the verification is successful,
the download will be skipped, otherwise, the download will be re-downloaded.
After the download is successful, the bulkload-related counter in replica_stub
will increase the file size. This piece of logic was originally intended to
support update BulkLoad download progress to update download process.
Look at those code about update download progress.We can see that download
progress is calculated by current downloaded size. THE **current downloaded
size** will be reset when BulkLoad roll back or the whole BulkLoad process
failed. We can store that variable in another way to implement BulkLoad CU
feature.
If the file already exists and the verification passes, the filesize will
be passed directly to cur_download_size. In this way, each replica gets a
cur_download_size that is periodically updated (one update per meta request).
(The logic follows the existing process report)
Then, the secondaries converts the cur_download_size to bulkload_cu size,
wraps it in the response, and sends it to the primary. The primary is
aggregated into the download amount of the partition, packaged in the response
and sent back to the meta.
##How meta deal with BulkLoad CU which replica report?
The meta maintains two new maps, which store the table ip and the total
download amount, as well as the shard gpid and shard download amount.
When the APP_STATUS is DOWNLOADING and DOWNLOADED, we assign the download
amount in the response of each primary to the meta
_partitions_total_downloaded_file_size map(mapping partition info gpid to
BulkLoad CU). Since it resets after an exception, we don't need to worry about
data errors.
When the APP_STATUS turn to FAILED state, we reset the
app_bulk_load_file_size directly, and this bulkload import completely failed.
When all app shards have completed the ingestion, we need to update
related app status on ZooKeeper nodes. After receiving the reply that the ZK
modification is successful, we will write
_partitions_total_downloaded_file_size to the stat table.
The write method is similar with the method we record readCU,writerCU,it's a
async write method.
##### Tests <!-- At least one of them must be included. -->
- Manual test :
I test it in a pegasus cluster. It run as expected when we do a BulkLoad or
we do BulkLoad on two map in the same cluster at the same time.
Even we cancel BulkLoad and recover it, the program will run successfully.
FYI: BulkLoad CU will record ervery replica real downloads size.In other
words,if you have to download a 100MB SST file.And 3 servers for one partition
in your cluster, every partition should download 300MD total.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
