Re: ABORTING region server and following HBase cluster "crash"

2018-11-02 Thread Vincent Poon
Indexes in Phoenix should not in theory cause any cluster outage. An index write failure should just disable the index, not cause a crash. In practice, there have been some bugs around race conditions, the most dangerous of which accidentally trigger a KillServerOnFailurePolicy which then

Re: ABORTING region server and following HBase cluster "crash"

2018-11-02 Thread Neelesh
By no means am I judging Phoenix based on this. This is simply a design trade-off (scylladb goes the same route and builds global indexes). I appreciate all the effort that has gone in to Phoenix, and it was indeed a life saver. But the technical point remains that single node failures have

Re: Python phoenixdb adapter and JSON serialization on PQS

2018-11-02 Thread Manoj Ganesan
Thanks Josh for the response! I would definitely like to use protobuf serialization, but I'm observing performance issues trying to run queries with a large number of results. One problem is that I observe PQS runs out of memory, when its trying to (what looks like to me) serialize the results in

Re: ABORTING region server and following HBase cluster "crash"

2018-11-02 Thread Josh Elser
I would strongly disagree with the assertion that this is some unavoidable problem. Yes, an inverted index is a data structure which, by design, creates a hotspot (phrased another way, this is "data locality"). Lots of extremely smart individuals have spent a significant amount of time and

Re: Python phoenixdb adapter and JSON serialization on PQS

2018-11-02 Thread Josh Elser
I would strongly suggest you do not use the JSON serialization. The JSON support is implemented via Jackson which has no means to make backwards compatibility "easy". On the contrast, protobuf makes this extremely easy and we have multiple examples over the past years where we've been able to

Re: ABORTING region server and following HBase cluster "crash"

2018-11-02 Thread Neelesh
I think this is an unavoidable problem in some sense, if global indexes are used. Essentially global indexes create a graph of dependent region servers due to index rpc calls from one RS to another. Any single failure is bound to affect the entire graph, which under reasonable load becomes the

Re: Phoenix Performances & Uses Cases

2018-11-02 Thread Neelesh
Another observation with Phoenix global indexes - at very large volumes of writes, a single region server failure cascades to the entire cluster very quickly On Sat, Oct 27, 2018, 4:50 AM Nicolas Paris wrote: > Hi > > I am benchmarking phoenix to better understand its strength and > weaknesses.