we use HDP 2.3.4 (HBase 1.1.2, phoenix 4.4 - but with a lot of backports
from later versions)
The key of the data table is <customerid- 11 bytes><userid- 36 bytes- but
it really is a right padded long due to historic data><event type
The two global indexes are <customerid-11 bytes><event type
long><timestamp> <user id> and <customerid-11 bytes><campaign id
Around 100B rows in the main table. The main issues we see are
# Sudden spikes in queueSize - going all the way to 1G limit and staying
there, without any correlated client traffic
# Boatloads of these errors 2016-11-30 11:28:54,907 INFO
org.apache.hadoop.hbase.DoNotRetryIOException: ERROR 2008 (INT10): ERROR
2008 (INT10): Unable to find cached index metadata. key=120521194876100862
region=<region-key>. Index update failed
We have cross datacenter WAL replication enabled.
We saw PHOENIX-1718, and changed all recommended timeouts to 1 hour. Our
HBase version has HBase-11705. We also discovered that the queuesize is
global (across general/replication/priority queues) and if it reaches the
1GB limit, calls to all queues will drop. That was interesting because even
though the replication handlers have a different queue, the size is
counted globally, affecting others. Please correct me on this. I hope I'm
wrong on this one :)
Our challenge has been to understand what's HBase doing under various
scenarios. We monitor call queue lengths, sizes and latencies as the
primary alerting mechanism to tell us something is going on with HBase.
On Wed, Nov 30, 2016 at 1:15 PM, Ted Yu <ted...@yahoo.com.invalid> wrote:
> Neelesh:Can you share more details about the sluggish cluster performance
> (such as version of hbase / phoenix, your schema, region server log
> snippet, stack traces, etc) ?
> As hbase / phoenix evolve, I hope the performance keeps getting better for
> your use case.
> On Wednesday, November 30, 2016 10:07 AM, Neelesh <neele...@gmail.com>
> We use both, in different capacities. Cassandra is an x-DC archive store
> with mostly batch writes and occasional key based reads. Hbase is for
> real-time event ingestion. Our experience so far on hbase + phoenix is that
> when it works, it is fast and scales like crazy. But if you ever hit a snag
> around data patterns, you will have a VERY hard time figuring out what's
> going on. A combination of global phoenix indexes and heavy writes leave an
> entire cluster sluggish, if there is a hint of hotspotting.
> On the other hand, we had a big struggle getting Cassandra when a node
> recovery was in progress. What with twice the amount of disk requirements
> during recovery etc. Other than that, it is quiet.
> But the access patterns are not the same.
> I think the old rule still stays. If you are already on hadoop , or
> interested in using/analysing data in several different ways, go with hbase
> . If you just need a big data store with a few predefined query patterns,
> Cassandra is good
> Of course, I'm biased towards HBase.
> On Nov 30, 2016 7:02 AM, "Mich Talebzadeh" <mich.talebza...@gmail.com>
> > Hi Guys,
> > Used Hbase on HDFS reasonably well. Happy to to stick with it and more
> > Hive/Phoenix views and Phoenix indexes where I can.
> > I have a bunch of users now vocal about the use case for Cassandra and
> > whether it can do a better job than Hbase.
> > Unfortunately I am no expert on Cassandra. However, some use case fit
> > be very valuable.
> > Thanks
> > Dr Mich Talebzadeh
> > LinkedIn * https://www.linkedin.com/profile/view?id=
> > AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw
> > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> > OABUrV8Pw>*
> > http://talebzadehmich.wordpress.com
> > *Disclaimer:* Use it at your own risk. Any and all responsibility for any
> > loss, damage or destruction of data or any other property which may arise
> > from relying on this email's technical content is explicitly disclaimed.
> > The author will in no case be liable for any monetary damages arising
> > such loss, damage or destruction.