I do not believe that introducing new dependency on external systems is the right approach in the long term. Yes, we can try memcached with extstore storage on SSD (open source Redis/Valkey lack SSD support), but several issues immediately arise:
1. Operational and maintenance burden. Running a separate memcached cluster in HA mode requires additional expertise and operational overhead. Achieving HA would require either a client-side support via modified memcached client lib or using proxies (mcrouter or own memcached proxy). 2. Performance and latency requirements. HBase is a demanding client of an external cache system - it requires high throughput and low latency (because this is what local cache is supposed to provide), Since all traffic will flow through proxies for HA, we risk introducing additional network hop and creating bottlenecks exactly where we can least afford them. 3. Uncharted territory and risk. To my best knowledge no one has successfully deployed this architecture at scale. We would be entering untested territory with unclear operational characteristics, failure semantics and long - term stability implications. This could be considered only as last resort option only, imo. best -Vlad On Wed, Feb 25, 2026 at 8:18 PM 张铎(Duo Zhang) <[email protected]> wrote: > > I mean an external SSD based storage, not inside the hbase cluster. > We have a memcached block cache implementation in our code base, and I > think it is also easy to implement a redis based block cache, then > there will be bunch of open source projects which can be used, for > example, like this one > > https://github.com/OpenAtomFoundation/pikiwidb > > Thanks. > > Wellington Chevreuil <[email protected]> 于2026年2月25日周三 23:50写道： > > > > Our main reason to pick hdfs storage types/policies was to leverage > > redundancy and distribution. The current architecture has limitations with > > OS upgrades and node scaling, which are common use cases. > > > > For OS upgrades, the updated instance must be restarted, the SSD disk is > > reset and the whole cache is lost, requiring reads to be served back from > > S3, however, if we now use the SSD disks for HDFS SSD policy, the reads > > would still be served by nearby DNs, provided nodes restarted are less than > > the replication factor. > > > > For node scaling, regions need to be moved around different servers. That > > too is problematic, currently, as it causes newly moved regions to be > > uncached and read from S3. > > > > We originally thought about wrapping hdfs ssd storage on a bucketcache file > > engine or a brand new block cache implementation, but bucket cache requires > > random file access to store blocks on different segments of few large local > > files, whilst implementing a new block cache type that saves each block > > directly as a single file in hdfs could hammer namenode with too many small > > files. > > > > But of course, any distributed cache solution that could be mounted over > > these disks and wrapped by a block cache implementation can solve this. The > > tradeoff would be yet another service to deploy and manage within the hbase > > cluster itself, whilst HDFS is already present here because we need it for > > WALs. > > > > Em qua., 25 de fev. de 2026 às 02:19, 张铎(Duo Zhang) <[email protected]> > > escreveu: > > > > > What about implementing a SSD based external block cache storage? > > > > > > Wellington Chevreuil <[email protected]> 于2026年2月25日周三 > > > 06:12写道： > > > > > > > > Dear HBase Dev Community, > > > > > > > > In our quest to pursue optimal performance for hbase when configuring > > > root > > > > dir over cloud storage, we have been relying heavily on local cache > > > > using > > > > ephemeral SSD disks normally available on given cloud provider instance > > > > types. > > > > > > > > Currently, our reference architecture for such use cases leverages file > > > > based BucketCache, but that has certain limitations, and we are now at > > > > early stages of testing an alternative approach using hdfs storage types > > > > and policies to mirror root dir content from S3 to hdfs dirs mounted on > > > > these ephemeral SSD disks. Additional technical details are documented > > > here > > > > < > > > https://docs.google.com/document/d/1XcUSBrsVTQLq_j_Tq6FrC_NRPiVNxdIU8gu2gDPT-eM/edit?tab=t.0 > > > > > > > > . > > > > > > > > So far in our PoC, we have implemented and tested the read/write from/to > > > > the different file system locations, we would like to continue this > > > feature > > > > implementation in collaboration with the community, so we plan to open > > > > an > > > > umbrella jira ticket, together with individual subtasks for each > > > > functionality described in the document mentioned above. Comments and > > > > feedback are very welcome. > > > > > > > > Thanks, > > > > Wellington > > >
