Greetings everyone, HBASE-24749 has been proposed almost a year ago, introducing a new StoreFile tracker as a way to allow for any hbase hfile modifications to be safely completed without needing a file system rename. This seems pretty relevant for deployments over S3 file systems, where rename operations are not atomic and can have a performance degradation when multiple requests get concurrently submitted to the same bucket. We had done superficial tests and ycsb runs, where individual renames of files larger than 5GB can take a few hundreds of seconds to complete. We also observed impacts in write loads throughput, the bottleneck potentially being the renames.
With S3 being an important piece of my employer cloud solution, we would like to help it move forward. We plan to contribute new patches per the original design/Jira, but we’d also be happy to review changes from the original authors, too. Please let us know if anyone has any concerns, otherwise we’ll start to self-assign issues on HBASE-24749 Wellington
