Certainly the EMR support makes it easy to deploy a test cluster to kick around the tires.
On Sat, Sep 6, 2014 at 10:03 AM, James Taylor <[email protected]> wrote: > I don't have experience running Phoenix in AWS. Andrew Purtell is a > good person to ask. I'm curious if our support under their EMR helps > in any way: http://phoenix.apache.org/phoenix_on_emr.html > > Thanks, > James > > On Sat, Sep 6, 2014 at 12:27 AM, Alex Kamil <[email protected]> wrote: >> not sure, that's not my experience with phoenix, but if you have unstable >> network connection to your storage (which is EBS is well known for) it may >> affect the results >> >> >> On Sat, Sep 6, 2014 at 3:14 AM, Vikas Agarwal <[email protected]> wrote: >>> >>> Of course, I can do a lot of optimizations. However, my concern is that >>> what I am missing that is causing Phoenix to perform bad while exactly on >>> same time, Hbase is giving results amazingly fast. >>> >>> >>> On Sat, Sep 6, 2014 at 12:41 PM, Alex Kamil <[email protected]> wrote: >>>> >>>> well it is still network attached, If you allocate enough heap to fit the >>>> whole thing in memory (in hbase/conf/hbase-env.sh) you could probably >>>> eliminate this as a possible reason >>>> >>>> >>>> On Sat, Sep 6, 2014 at 2:43 AM, Vikas Agarwal <[email protected]> >>>> wrote: >>>>> >>>>> EBS but with new generation SSD not magnetic one. >>>>> >>>>> >>>>> On Sat, Sep 6, 2014 at 12:11 PM, Alex Kamil <[email protected]> >>>>> wrote: >>>>>> >>>>>> do you use EBS or ephemeral storage, I found EBS performance to be >>>>>> somewhat unpredictable >>>>>> >>>>>> >>>>>> On Sat, Sep 6, 2014 at 2:37 AM, Vikas Agarwal <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> Hbase is 0.98.0 >>>>>>> Phoenix is 4.0 >>>>>>> >>>>>>> >>>>>>> On Sat, Sep 6, 2014 at 12:04 PM, Vikas Agarwal <[email protected]> >>>>>>> wrote: >>>>>>>> >>>>>>>> Yes, that is why it is a trouble for me. However, on contrary, HBase >>>>>>>> shell is also on the same machine and same environment, so if it is an >>>>>>>> issue >>>>>>>> of resource (CPU or memory) it should have affected the HBase too, but >>>>>>>> HBase >>>>>>>> is able to give me results within 0.0150 seconds. :( >>>>>>>> >>>>>>>> No, I haven't tested it outside AWS. I guess, it should not be the >>>>>>>> case due to much better performance by native HBase query on HBase >>>>>>>> shell. >>>>>>>> >>>>>>>> >>>>>>>> On Sat, Sep 6, 2014 at 11:59 AM, James Taylor >>>>>>>> <[email protected]> wrote: >>>>>>>>> >>>>>>>>> Something is up in your environment. What version of Phoenix and >>>>>>>>> HBase >>>>>>>>> are you using and in what environment? Have you tried this locally, >>>>>>>>> outside of AWS to compare? >>>>>>>>> >>>>>>>>> Take a look at our perf numbers, generated more-or-less daily, and >>>>>>>>> which run over more data that what you're testing against: >>>>>>>>> >>>>>>>>> http://phoenix-bin.github.io/client/performance/phoenix-20140904095313.htm >>>>>>>>> >>>>>>>>> Some of these are point queries and they take in the neighborhood of >>>>>>>>> 0.01 seconds. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> James >>>>>>>>> >>>>>>>>> On Fri, Sep 5, 2014 at 10:48 PM, Vikas Agarwal >>>>>>>>> <[email protected]> wrote: >>>>>>>>> > Missed to mention that count query (posted in my last mail) is >>>>>>>>> > also taking >>>>>>>>> > very long time to return the count. >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > On Sat, Sep 6, 2014 at 11:17 AM, Vikas Agarwal >>>>>>>>> > <[email protected]> >>>>>>>>> > wrote: >>>>>>>>> >> >>>>>>>>> >> As I mentioned, schema is nothing but bunch of fields (some being >>>>>>>>> >> integers, longs and text) along with primary key (row key) and I >>>>>>>>> >> am making >>>>>>>>> >> simple query to get result for a particular primary key, nothing >>>>>>>>> >> more than >>>>>>>>> >> that. >>>>>>>>> >> >>>>>>>>> >> 0: jdbc:phoenix:localhost> SELECT count(1) FROM table_name; >>>>>>>>> >> >>>>>>>>> >> +------------+ >>>>>>>>> >> >>>>>>>>> >> | COUNT(1) | >>>>>>>>> >> >>>>>>>>> >> +------------+ >>>>>>>>> >> >>>>>>>>> >> | 4667515 | >>>>>>>>> >> >>>>>>>>> >> +------------+ >>>>>>>>> >> >>>>>>>>> >> 1 row selected (132.11 seconds) >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> On Sat, Sep 6, 2014 at 11:09 AM, Puneet Kumar Ojha >>>>>>>>> >> <[email protected]> wrote: >>>>>>>>> >>> >>>>>>>>> >>> If you can share the schema,data type,cardinality of each >>>>>>>>> >>> dimension and >>>>>>>>> >>> usual queries, I can help to design a schema with performance of >>>>>>>>> >>> less than 1 >>>>>>>>> >>> sec using Phoenix. >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> Thanks >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> ------ Original message------ >>>>>>>>> >>> >>>>>>>>> >>> From: James Taylor >>>>>>>>> >>> >>>>>>>>> >>> Date: Sat, Sep 6, 2014 10:15 AM >>>>>>>>> >>> >>>>>>>>> >>> To: user; >>>>>>>>> >>> >>>>>>>>> >>> Subject:Re: Phoenix response time >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> >>>>>>>>> >>> Vikas, >>>>>>>>> >>> Please post your schema and query. >>>>>>>>> >>> Thanks, >>>>>>>>> >>> James >>>>>>>>> >>> >>>>>>>>> >>> On Fri, Sep 5, 2014 at 9:18 PM, Vikas Agarwal >>>>>>>>> >>> <[email protected]> >>>>>>>>> >>> wrote: >>>>>>>>> >>> > Ours is also a single node setup right now and as of now there >>>>>>>>> >>> > are less >>>>>>>>> >>> > than >>>>>>>>> >>> > 1 million rows which is expected to grow around 100m at >>>>>>>>> >>> > minimum. >>>>>>>>> >>> > >>>>>>>>> >>> > I am aware of secondary indexes but when I am querying on >>>>>>>>> >>> > primary/row >>>>>>>>> >>> > key, >>>>>>>>> >>> > why would it take so much time? >>>>>>>>> >>> > >>>>>>>>> >>> > I am directly querying using sqlline for Phoenix and hbase >>>>>>>>> >>> > shell for >>>>>>>>> >>> > HBase >>>>>>>>> >>> > query. I am not expecting to do any fine tuning for such small >>>>>>>>> >>> > dataset. >>>>>>>>> >>> > I am >>>>>>>>> >>> > assumimg a minimum performance level out of the box. >>>>>>>>> >>> > >>>>>>>>> >>> > On Friday, September 5, 2014, yeshwanth kumar >>>>>>>>> >>> > <[email protected]> >>>>>>>>> >>> > wrote: >>>>>>>>> >>> >> >>>>>>>>> >>> >> hi vikas, >>>>>>>>> >>> >> >>>>>>>>> >>> >> we used phoenix on a 4 core/23Gb machine, as a single node >>>>>>>>> >>> >> setup. >>>>>>>>> >>> >> used HDP 2.1 >>>>>>>>> >>> >> our table has 50-70M rows, >>>>>>>>> >>> >> select on that table took less than 2 seconds. >>>>>>>>> >>> >> Aggregation queries took less than 8 seconds. >>>>>>>>> >>> >> for achieving good performance we created secondary index on >>>>>>>>> >>> >> the >>>>>>>>> >>> >> table. >>>>>>>>> >>> >> >>>>>>>>> >>> >> make sure you finetuned hbase, >>>>>>>>> >>> >> enabling compression on the data makes a difference in >>>>>>>>> >>> >> response. >>>>>>>>> >>> >> if u distribute the data and load over all regions in hbase, >>>>>>>>> >>> >> look at the performance tips mentioned in phoenix blog >>>>>>>>> >>> >> >>>>>>>>> >>> >> -yeshwanth >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> Cheers, >>>>>>>>> >>> >> Yeshwanth >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> >> On Fri, Sep 5, 2014 at 5:42 PM, Vikas Agarwal >>>>>>>>> >>> >> <[email protected]> >>>>>>>>> >>> >> wrote: >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> Hi, >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> Preface: We are testing phoenix using Hortonworks >>>>>>>>> >>> >>> distribution for >>>>>>>>> >>> >>> HBase >>>>>>>>> >>> >>> on Amazon EC2 instance (r3.large, 2 CPU/15 GB RAM). >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> With contrast to performance benchmarks, I found Phoenix to >>>>>>>>> >>> >>> be very >>>>>>>>> >>> >>> slow >>>>>>>>> >>> >>> in querying even on primary key or row key. So, tried to >>>>>>>>> >>> >>> increase the >>>>>>>>> >>> >>> RAM >>>>>>>>> >>> >>> for HBase and Phoenix and increasing the CPU and RAM by >>>>>>>>> >>> >>> upgrading the >>>>>>>>> >>> >>> EC2 >>>>>>>>> >>> >>> machine type to r3.xlarge (4 CPU, 30 GB RAM). Results were >>>>>>>>> >>> >>> like this: >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> Time takes in returning result of query on row key: >>>>>>>>> >>> >>> With Storm running and very less RAM available: 50 sec >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> With Storm stopped and RAM available to Phoenix and HBase: >>>>>>>>> >>> >>> 18 sec >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> With new machine of next higher category (4 CPU and 30 GB >>>>>>>>> >>> >>> RAM): 8 sec >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> Pure HBase query by row key with Storm stopped and (2 CPU, >>>>>>>>> >>> >>> 15 GB >>>>>>>>> >>> >>> RAM): >>>>>>>>> >>> >>> 0.0150 seconds. :) >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> So, the difference seems to be many fold of what native >>>>>>>>> >>> >>> HBase is >>>>>>>>> >>> >>> providing to us. I am not able to understand how it can be >>>>>>>>> >>> >>> possible? >>>>>>>>> >>> >>> What I >>>>>>>>> >>> >>> am missing here? >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> -- >>>>>>>>> >>> >>> Regards, >>>>>>>>> >>> >>> Vikas Agarwal >>>>>>>>> >>> >>> 91 – 9928301411 >>>>>>>>> >>> >>> >>>>>>>>> >>> >>> InfoObjects, Inc. >>>>>>>>> >>> >>> Execution Matters >>>>>>>>> >>> >>> http://www.infoobjects.com >>>>>>>>> >>> >>> 2041 Mission College Boulevard, #280 >>>>>>>>> >>> >>> Santa Clara, CA 95054 >>>>>>>>> >>> >>> +1 (408) 988-2000 Work >>>>>>>>> >>> >>> +1 (408) 716-2726 Fax >>>>>>>>> >>> >> >>>>>>>>> >>> >> >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > -- >>>>>>>>> >>> > Regards, >>>>>>>>> >>> > Vikas Agarwal >>>>>>>>> >>> > 91 – 9928301411 >>>>>>>>> >>> > >>>>>>>>> >>> > InfoObjects, Inc. >>>>>>>>> >>> > Execution Matters >>>>>>>>> >>> > http://www.infoobjects.com >>>>>>>>> >>> > 2041 Mission College Boulevard, #280 >>>>>>>>> >>> > Santa Clara, CA 95054 >>>>>>>>> >>> > +1 (408) 988-2000 Work >>>>>>>>> >>> > +1 (408) 716-2726 Fax >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> >>>>>>>>> >> -- >>>>>>>>> >> Regards, >>>>>>>>> >> Vikas Agarwal >>>>>>>>> >> 91 – 9928301411 >>>>>>>>> >> >>>>>>>>> >> InfoObjects, Inc. >>>>>>>>> >> Execution Matters >>>>>>>>> >> http://www.infoobjects.com >>>>>>>>> >> 2041 Mission College Boulevard, #280 >>>>>>>>> >> Santa Clara, CA 95054 >>>>>>>>> >> +1 (408) 988-2000 Work >>>>>>>>> >> +1 (408) 716-2726 Fax >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > -- >>>>>>>>> > Regards, >>>>>>>>> > Vikas Agarwal >>>>>>>>> > 91 – 9928301411 >>>>>>>>> > >>>>>>>>> > InfoObjects, Inc. >>>>>>>>> > Execution Matters >>>>>>>>> > http://www.infoobjects.com >>>>>>>>> > 2041 Mission College Boulevard, #280 >>>>>>>>> > Santa Clara, CA 95054 >>>>>>>>> > +1 (408) 988-2000 Work >>>>>>>>> > +1 (408) 716-2726 Fax >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Regards, >>>>>>>> Vikas Agarwal >>>>>>>> 91 – 9928301411 >>>>>>>> >>>>>>>> InfoObjects, Inc. >>>>>>>> Execution Matters >>>>>>>> http://www.infoobjects.com >>>>>>>> 2041 Mission College Boulevard, #280 >>>>>>>> Santa Clara, CA 95054 >>>>>>>> +1 (408) 988-2000 Work >>>>>>>> +1 (408) 716-2726 Fax >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> Vikas Agarwal >>>>>>> 91 – 9928301411 >>>>>>> >>>>>>> InfoObjects, Inc. >>>>>>> Execution Matters >>>>>>> http://www.infoobjects.com >>>>>>> 2041 Mission College Boulevard, #280 >>>>>>> Santa Clara, CA 95054 >>>>>>> +1 (408) 988-2000 Work >>>>>>> +1 (408) 716-2726 Fax >>>>>> >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> Vikas Agarwal >>>>> 91 – 9928301411 >>>>> >>>>> InfoObjects, Inc. >>>>> Execution Matters >>>>> http://www.infoobjects.com >>>>> 2041 Mission College Boulevard, #280 >>>>> Santa Clara, CA 95054 >>>>> +1 (408) 988-2000 Work >>>>> +1 (408) 716-2726 Fax >>>> >>>> >>> >>> >>> >>> -- >>> Regards, >>> Vikas Agarwal >>> 91 – 9928301411 >>> >>> InfoObjects, Inc. >>> Execution Matters >>> http://www.infoobjects.com >>> 2041 Mission College Boulevard, #280 >>> Santa Clara, CA 95054 >>> +1 (408) 988-2000 Work >>> +1 (408) 716-2726 Fax >> >> -- Best regards, - Andy Problems worthy of attack prove their worth by hitting back. - Piet Hein (via Tom White)
