> We also discovered that the queuesize is
> global (across general/replication/priority queues) and if it reaches the
> 1GB limit, calls to all queues will drop. That was interesting because
even
> though the replication handlers have  a different queue, the size is
> counted globally, affecting others. Please correct me on this. I hope I'm
> wrong on this one :)

You are right.  The size is checked before the call is dispatched to the
different queues.
It sounds like something that can be improved.


Jerry


On Thu, Dec 1, 2016 at 8:04 AM, Neelesh <neele...@gmail.com> wrote:

> Yes, PHOENIX-1718 has about the same information
>
> On Dec 1, 2016 1:12 AM, "Ted Yu" <ted...@yahoo.com> wrote:
>
> > w.r.t. "Unable to find cached index metadata" error, have you seen this ?
> >
> > http://search-hadoop.com/m/Phoenix/9UY0h2YBSOhgbflB1?
> > subj=Re+Global+Secondary+Index+ERROR+2008+INT10+Unable+
> > to+find+cached+index+metadata+PHOENIX+1718+
> >
> > Cheers
> >
> >
> > On Wednesday, November 30, 2016 11:59 PM, Neelesh <neele...@gmail.com>
> > wrote:
> >
> >
> > Ted,
> > we use HDP 2.3.4 (HBase 1.1.2, phoenix 4.4 - but with a lot of backports
> > from later versions)
> >
> > The key of the data table is  <customerid- 11 bytes><userid- 36 bytes-
> >  but it really is a right padded long due to historic data><event type
> > long><timestamp><random long>
> >
> > The two global indexes are <customerid-11 bytes><event type
> > long><timestamp> <user id> and <customerid-11 bytes><campaign id
> > long><timestamp>
> > Around 100B rows in the main table. The main issues we see are
> >
> > # Sudden spikes in queueSize - going all the way to 1G limit and staying
> > there, without any correlated client traffic
> > # Boatloads of these errors 2016-11-30 11:28:54,907 INFO
> >  [RW.default.writeRpcServer.handler=43,queue=9,port=16020]
> > util.IndexManagementUtil: Rethrowing org.apache.hadoop.hbase.
> DoNotRetryIOException:
> > ERROR 2008 (INT10): ERROR 2008 (INT10): Unable to find cached index
> > metadata.  key=120521194876100862 region=<region-key>. Index update
> failed
> >
> > We have cross datacenter WAL replication enabled.
> > We saw PHOENIX-1718, and changed all recommended timeouts to 1 hour.  Our
> > HBase version has HBase-11705. We also discovered that the queuesize is
> > global (across general/replication/priority queues) and if it reaches the
> > 1GB limit, calls to all queues will drop. That was interesting because
> even
> > though the replication handlers have  a different queue, the size is
> > counted globally, affecting others. Please correct me on this. I hope I'm
> > wrong on this one :)
> >
> > Our challenge has been to understand what's HBase doing under various
> > scenarios. We monitor call queue lengths, sizes and latencies as the
> > primary alerting mechanism to tell us something is going on with HBase.
> >
> > Thanks!
> > -neelesh
> >
> > On Wed, Nov 30, 2016 at 1:15 PM, Ted Yu <ted...@yahoo.com.invalid>
> wrote:
> >
> > Neelesh:Can you share more details about the sluggish cluster performance
> > (such as version of hbase / phoenix, your schema, region server log
> > snippet, stack traces, etc) ?
> > As hbase / phoenix evolve, I hope the performance keeps getting better
> for
> > your use case.
> > Cheers
> >
> >     On Wednesday, November 30, 2016 10:07 AM, Neelesh <
> neele...@gmail.com>
> > wrote:
> >
> >
> >  We use both, in different capacities. Cassandra is an x-DC archive store
> > with mostly batch writes and occasional key based reads. Hbase is for
> > real-time event ingestion. Our experience so far on hbase + phoenix is
> that
> > when it works, it is fast and scales like crazy. But if you ever hit a
> snag
> > around data patterns, you will have a VERY hard time figuring out what's
> > going on. A combination of global phoenix indexes and heavy writes leave
> an
> > entire cluster sluggish, if there is a hint of hotspotting.
> >
> > On the other hand, we had a big struggle getting Cassandra when a node
> > recovery was in progress. What with twice the amount of disk requirements
> > during recovery etc. Other than that, it is quiet.
> > But the access patterns are not the same.
> >
> > I think the old rule still stays. If you are already on hadoop , or
> > interested in using/analysing data in several different ways, go with
> hbase
> > . If you just need a big data store with a few predefined query patterns,
> > Cassandra is good
> >
> > Of course, I'm biased towards HBase.
> >
> > On Nov 30, 2016 7:02 AM, "Mich Talebzadeh" <mich.talebza...@gmail.com>
> > wrote:
> >
> > > Hi Guys,
> > >
> > > Used Hbase on HDFS reasonably well. Happy to to stick with it and more
> > with
> > > Hive/Phoenix views and Phoenix indexes where I can.
> > >
> > > I have a bunch of users now vocal about the use case for Cassandra and
> > > whether it can do a better job than Hbase.
> > >
> > > Unfortunately I am no expert on Cassandra. However, some use case fit
> > would
> > > be very valuable.
> > >
> > > Thanks
> > >
> > > Dr Mich Talebzadeh
> > >
> > >
> > >
> > > LinkedIn * https://www.linkedin.com/ profile/view?id=
> > <https://www.linkedin.com/profile/view?id=>
> > > AAEAAAAWh2gBxianrbJd6zP6AcPCCd OABUrV8Pw
> > > <https://www.linkedin.com/ profile/view?id=
> > AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCd
> >
> > > OABUrV8Pw>*
> > >
> > >
> > >
> > > http://talebzadehmich. wordpress.com
> > <http://talebzadehmich.wordpress.com/>
> > >
> > >
> > > *Disclaimer:* Use it at your own risk. Any and all responsibility for
> any
> > > loss, damage or destruction of data or any other property which may
> arise
> > > from relying on this email's technical content is explicitly
> disclaimed.
> > > The author will in no case be liable for any monetary damages arising
> > from
> > > such loss, damage or destruction.
> > >
> >
> >
> >
> >
> >
> >
> >
> >
>

Reply via email to