I like the idea of multi node testing, because the mini cluster does an admirable job but cannot truly emulate a production deploy because of various singletons in our code or that of our dependencies. It would also be pretty nice if k8s was the substrate — and ultimately is used to inject chaos too (via hbase-it) — because it would help us detect if we violate required discipline for that common deployment target, like inappropriate caching of DNS resolutions concurrent with pod cycling, to pick a historical example.
Even just periodic execution of ITBLL would be nice. So I guess the next question is what does that require of us, the larger community. Who proposed the work? Who performs it? We should open some JIRAs to kick things off? > On Mar 10, 2022, at 8:32 AM, Sean Busbey <[email protected]> wrote: > > Hi folks! > > Quick background: all of the automated testing for nightly and PR > contributions is now running on a dedicated Jenkins instance ( > ci-hbase.apache.org ). We moved our existing 10 dedicated nodes off of > the ci-hadoop controller and thanks to a new anonymous donor we were > able to add an additional 10 nodes. > > The new donor gave enough of a contribution that we can make some > decisions as a community about expanding these resources further. > > The new 10 nodes run 2 executors each (same as our old nodes), have > this shape, and are considered "medium" by the provider we're getting > them from: > > 64GB DDR4 ECC RAM > Intel® Xeon® E-2176G hexa-core processor with Hyper-Threading Coffee Lake. > 2 x 960 GB NVMe SSD Datacenter Edition (RAID 1) > > To give an idea of what the current testing workload of our project > looks like, we can use the built in jenkins utilization tooling for > our general purpose label 'hbase'[0]. > > If you look at the last 4 days of utilization[1] we have a couple of > periods with a small backlog of ~2 executors worth of work. The > measurements are very rolled up so it's hard to tell specifics. On the > chart of the last day or so[2] we can see two periods of 1-2 hours > where we have a backlog of 2-4 executors worth of work. > > for comparison, the chart for immediately after we had to burn off ~3 > days of backlog because our worker nodes were offline back at the end > of february shows no queue[3]. > > I think we could possibly benefit from adding 1-2 additional medium > worker nodes, but the long periods where we have ~half our executors > idle makes me think some refactoring or timing changes would maybe be > a better way to improve our current steady state workload. > > One thing that we currently lack is robust integration testing of a > cluster deployment. At the moment our nightly jobs spin up a test that > makes a single node version of Hadoop and then a single node hbase on > top of it. It then does a trivial functionality test[4]. > > The host provider we use for jenkins worker nodes has a large node shaped > like: > 160GB RAM > Intel® Xeon® W-2295 18-Core Cascade-Lake W Hyper-Threading > 2 x 960GB NVMe drives as RAID1 > > A pretty short path to improvement would be if we got 1 or 2 of these > nodes and moved our integration test to use the minikube project[5] to > run a local kubernetes environment. We could then deploy a small but > multinode Hadoop and HBase cluster and run e.g. ITBLL against it in > addition to whatever checking of cli commands, shell expectations, > etc. > > What do y'all think? > > [0]: https://ci-hbase.apache.org/label/hbase/load-statistics > [1]: > https://issues.apache.org/jira/secure/attachment/13040941/ci-hbase-long-graph-20220310.png > [2]: > https://issues.apache.org/jira/secure/attachment/13040940/ci-hbase-medium-graph-20220310.png > [3]: > https://issues.apache.org/jira/secure/attachment/13040939/ci-hbase-medium-graph-20220223.png > [4]: > https://github.com/apache/hbase/blob/master/dev-support/hbase_nightly_pseudo-distributed-test.sh > [5]: https://minikube.sigs.k8s.io/docs/
