Re: real time Query engine Spark-SQL on Hbase

2015-05-02 Thread Siddharth Ubale
this. Thanks, Siddharth From: ayan guha guha.a...@gmail.com Sent: 01 May 2015 04:38 To: Ted Yu Cc: user@spark.apache.org; Siddharth Ubale; matei.zaha...@gmail.com; Prakash Hosalli; Amit Kumar Subject: Re: real time Query engine Spark-SQL on Hbase And if I may ask

Re: real time Query engine Spark-SQL on Hbase

2015-05-02 Thread Ted Yu
*To:* Ted Yu *Cc:* user@spark.apache.org; Siddharth Ubale; matei.zaha...@gmail.com; Prakash Hosalli; Amit Kumar *Subject:* Re: real time Query engine Spark-SQL on Hbase And if I may ask, how long it takes in hbase CLI? I would not expect spark to improve performance of hbase. At best spark

Re: real time Query engine Spark-SQL on Hbase

2015-04-30 Thread Ted Yu
bq. a single query on one filter criteria Can you tell us more about your filter ? How selective is it ? Which hbase release are you using ? Cheers On Thu, Apr 30, 2015 at 7:23 AM, Siddharth Ubale siddharth.ub...@syncoms.com wrote: Hi, I want to use Spark as Query engine on HBase with

Re: real time Query engine Spark-SQL on Hbase

2015-04-30 Thread ayan guha
And if I may ask, how long it takes in hbase CLI? I would not expect spark to improve performance of hbase. At best spark will push down the filter to hbase. So I would try to optimise any additional overhead like bringing data into spark. On 1 May 2015 00:56, Ted Yu yuzhih...@gmail.com wrote:

Re: real time Query engine Spark-SQL on Hbase

2015-04-30 Thread Corey Nolet
A tad off topic, but could still be relevant. Accumulo's design is a tad different in the realm of being able to shard and perform set intersections/unions server-side (through seeks). I've got an adapter for Spark SQL on top of a document store implementation in Accumulo that accepts the

real time Query engine Spark-SQL on Hbase

2015-04-30 Thread Siddharth Ubale
Hi, I want to use Spark as Query engine on HBase with sub second latency. I am using Spark 1.3 version. And followed the steps below on Hbase table with around 3.5 lac rows : 1. Mapped the Dataframe to Hbase table .RDDCustomers maps to the hbase table which is used to create the