For the offheap write path related things, The parent JIRA https://issues.apache.org/jira/browse/HBASE-15179
We have raised subtasks under that. Many of them are closed and there are 3 main sub-jiras now. All the 3 are now patch available. Subtasks include -> Reading the requests in to a ByteBuffer pool of fixed size. -> Making ChunkPool work with direct byte buffers -> Sizing heuristics of onheap and offheap cells (along with memstore flush sizing). -> Async WAL support. There were some POCs that we done before going with the patches against trunk. The POC reports and design docs are available in the following links https://docs.google.com/document/d/1fj5P8JeutQ-Uadb29ChDscMu MaJqaMNRI86C4k5S1rQ/edit https://docs.google.com/document/d/1bYIkwNVVyYk8bgik-C5x_yc_ 2hWHyi2-9KMwEFWLfzY/edit The POC reports suggest that we get around 24% improvement with 100 threads when using the PE tool. In these POCs we have also used the Y! compacting memstore and its variants called Pipeline Memstore(which is under discussion) to prove how far their work can help offheap because it could reduce the onheap overhead of the cells in memstore. They can explain their current state of affairs when they do write up here. We are now testing with AsyncWAL to see if that could be made the default WAL because currently in HDFS DFSOutputSTream we don't have support for bytebuffers. We need to copy the cell contents to onheap byte[]. Currenly as part of HBASE-15786 there is a workaround we are doing to over come that. Sorry. I thought I had sent this mail but I just saw that it is in my Drafts. Regards Ram On Mon, Nov 14, 2016 at 5:27 AM, Stack <[email protected]> wrote: > Thanks for the writeup Stephen. > > See below. > > On Fri, Nov 11, 2016 at 5:15 PM, Stephen Jiang <[email protected]> > wrote: > > > Hello, fellow HBASE developers, > > > > We are making progress towards HBASE 2.0 releases. I am using the > > following queries to search for on-going HBASE 2.0 feature work items > > (project = HBase AND (fixVersion = 2.0.0 OR affectedVersion = 2.0.0) AND > > resolution is EMPTY AND (issuetype != Bug AND issuetype != Test AND > > issuetype != Sub-task) ORDER BY issuetype DESC), at this time, we have > > 247! That is a lot. At some time in near future, we definitely need to > > trim down the list; otherwise, 2.0 will never be released. > > > > > Agree. Suggest we all do our own review but at a certain stage, it is up to > the RMs to make a call on what shape of 2.0 will be. > > > > > For now, Matteo and I are tracking some big projects that are on-going: > > > > (1). HBASE-14350 the stable Assignment Manager (using Procedure V2) > > - This is a blocker to have branch-2 cut. In the past few weeks, we made > > good progress and majority of implementation is done. The patches are > > under review and testing. Matto is drafting a document for review. > > > > (2). HBASE-14414 Backup/Restore Phase 3 > > - Currently it is blocked by HBASE-14123. The giant HBASE-14123 patches > > was discussed and reviewed from the community (see > > http://www.mail-archive.com/[email protected]/msg41090.html for the > > long > > discussion); and all feedback were taken care of in the latest patch. > > Currently I marked HBASE-14123 as 2.0 blocker, as without it, the further > > develpment of backup/restore is blocked - the backup/restore is a key > > enterprise feature for 2.0 release. I think HBASE-14123 is ready for > > another round of vote. > > > > > IMO, not a blocker since it ancillary function but something we should get > in. > > > > > (3). HBASE-15179 Offheap > > - This is another important feature that I think it is MUST for 2.0. > Since > > stack works closely with Intel developers, he has some insight on this > > project: "Intel are betting on this. Alibaba are using the offheap read > > path and interested in write path too. This work is still very much up in > > the air and being worked out as we go (especially the Y! Israel inmemory > > compaction component). It is a little shakey dependent on mslab pooling, > > blockcache being on by default, async wal being default, and then > dependent > > on lots of perf and ITBLL testing." > > > > > The above is from a private, hurried email that does not do the current > state justice. > > Ram and Anoop are the authorities on offheaping and have been keeping up > their state in an associated doc. The lads might be persuaded do a summary > of current state here. > > Ditto for inmemory compaction. Our Anastasia and Eshcar are working hard at > the moment doing laste pieces and perf testing. Maybe we can a status > dumped here in dev. > > > > > (4). HBASE-16952 Protobuf3 > > - Good news, stack got the majority of work done already. This is a MUST > > for 2.0 release. Now we only have a small sub-task HBASE-16967 > (findbugs) > > left. > > > > > To be clear, pb3 is under the offheaping umbrella. Let's add async WAL to > this list, also needed by offheaping. > > Will be back after taking a look at current 2.0 list. > > Agree that hbase2 needs to support hadoop3. > > St.Ack > > > > > > (5). HBASE-14070 Logical Clock > > - This needs to be done before 2.0 release. At this time, seems not > making > > much progress. > > > > (6). HBASE-16833 JAVA Async Client > > - A lot of progress was made by Duo and his associates. We should be in > a > > good shape in JAVA client when 2.0 release. > > > > (7). HBASE-14850 C++ Async Client > > - Another project that is making good progress. I know some customer > wants > > this. This is long overdue project. Hopefully we will make those > customer > > happy in 2.0 release with a good performed C++ client. > > > > Please let me know other on-going projects that needs to be in HBASE 2.0 > > release (stack mentioned "logical file system tier", but I am not sure > > whether it would make enough progress to have a realistic chance making > > 2.0). As I said at the beginning of this email, at some point soon, we > > will be more picky and trim down the unfinished features in 2.0. > > > > Happy Weekend! > > Stephen > > >
