I would start with
https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
and also take a look at
https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Enjoy,
Wei
%83Bytes
.toLong(byte[] bytes)
HTH,
Anil Gupta
On Wed, Oct 3, 2012 at 10:42 PM, Weishung Chung weish...@gmail.com
wrote:
Hi,
I store a long value as byte array in HBase cell using an Increment
object.
When I try to access the value via javascript using rest api, how do I
decode
I was running the hbase rest service and trying to access the data from
javascript client side.
Thank you :)
On Wed, Oct 3, 2012 at 11:44 PM, anil gupta anilgupt...@gmail.com wrote:
Sorry, i don't know java script.
On Wed, Oct 3, 2012 at 11:40 PM, Weishung Chung weish...@gmail.com
wrote
, and the result you get back is incorrect?
Can you reply with more detail for reproduction of the problem?
On Fri, Sep 28, 2012 at 2:16 PM, Weishung Chung weish...@gmail.com
wrote:
Hi users,
I use hbase increment and somehow when I get the value back via hbase
rest
api via javascript
Hi HBase users,
I need to pull data from 2 HBase tables in a mapreduce job. For 1 table
input, I use TableMapReduceUtil.initTableMapperJob. Is there another method
for multitable inputs ?
Thank you,
Wei Shung
Basically a join of two data sets on the same row key.
On Fri, Aug 10, 2012 at 6:12 AM, Amandeep Khurana ama...@gmail.com wrote:
How do you want to use two tables? Can you explain your algo a bit?
On Fri, Aug 10, 2012 at 6:40 PM, Weishung Chung weish...@gmail.com
wrote:
Hi HBase users
working with HBase
(or for that matter most NoSQL stores).
On Fri, Aug 10, 2012 at 6:52 PM, Weishung Chung weish...@gmail.com
wrote:
Basically a join of two data sets on the same row key.
On Fri, Aug 10, 2012 at 6:12 AM, Amandeep Khurana ama...@gmail.com
wrote:
How do you want
but they are in production now
On Fri, Aug 10, 2012 at 6:39 AM, Weishung Chung weish...@gmail.com wrote:
Thank you, I am trying to avoid to fetch by gets and would like to do
something like hadoop MultipleInputs.
Yes, it would be nice if i could denormalize and remodel the schema.
On Fri
, August 10, 2012, Bryan Beaudreault wrote:
Use 3 jobs: 1 to scan each table. The third could do a map-side join.
Make
sure to use the same sort and partitions on the first two.
Sent from iPhone.
On Aug 10, 2012, at 9:41 AM, Weishung Chung weish...@gmail.com
javascript:;
wrote
Thank you all. Practice makes perfect :)
On Tue, Apr 17, 2012 at 5:46 PM, Michael Segel michael_se...@hotmail.comwrote:
In theory, you could go as large as a region size minus the key and
overhead. (rows can't span regions)
Realistically you'd want to go much smaller.
Sent from my iPhone
Thank you !
On Sun, Mar 11, 2012 at 4:07 PM, Doug Meil doug.m...@explorysmedical.comwrote:
Hi there-
Have you seen the examples in here?
http://hbase.apache.org/book.html#mapreduce
On 3/11/12 4:59 PM, Weishung Chung weish...@gmail.com wrote:
Hey users,
I am trying to store
Hey users,
I am trying to store mapreduce output directly to HBase. Basically I have a
regular mapper reading from files and would like to modify the reducer to
store the output to HBase table. It looks like I could extends
TableReducer. My mapper output is of Text key and IntWritable value.
Any
cells of it or
add new cells.
Please let me know if you have more Qs!
Alex Baranau
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
On Wed, May 18, 2011 at 1:19 AM, Weishung Chung weish...@gmail.com
wrote:
I have another question. For overwriting, do I need
, Weishung Chung weish...@gmail.com
wrote:
I have another question. For overwriting, do I need to delete the
existing
one before re-writing it?
On Sat, May 14, 2011 at 10:17 AM, Weishung Chung weish...@gmail.com
wrote:
Yes, it's simple yet useful. I am integrating
I haven't found a fix. But, in the saveVersion.sh, I enter the user name
directly to get it going.
For all the test cases, do I need to have hbase up and running?
On Fri, Apr 29, 2011 at 2:56 PM, Weishung Chung weish...@gmail.com wrote:
It is the path separator, \, I guess it can't handle
Hello my fellow hbase users,
I have a question regarding the test cases in HBase. When I run the mvn
build, do I need to have hbase up and running ?
Thank you,
Wei Shung
Thank you...running the build now :)
On Tue, May 3, 2011 at 11:16 AM, Ted Yu yuzhih...@gmail.com wrote:
Test cases aren't dependent on physical hbase cluster.
On Tue, May 3, 2011 at 8:58 AM, Weishung Chung weish...@gmail.com wrote:
I haven't found a fix. But, in the saveVersion.sh, I enter
Thank you JD...building now :)
On Tue, May 3, 2011 at 1:20 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote:
No, it uses its own MiniHBaseCluster.
J-D
On Tue, May 3, 2011 at 9:04 AM, Weishung Chung weish...@gmail.com wrote:
Hello my fellow hbase users,
I have a question regarding
Hello my fellow HBase users,
I need your expertise here...
I was running
mvn clean site install assembly:single
and got the following compilation errors on package-info.java
[ERROR]
c:\workspace\hbase\target\generated-sources\org\apache\hadoop\hbase\package-info.java:[5,40]
illegal escape
it to an issue.
Yours,
St.Ack
On Fri, Apr 29, 2011 at 8:51 AM, Weishung Chung weish...@gmail.com
wrote:
Hello my fellow HBase users,
I need your expertise here...
I was running
mvn clean site install assembly:single
and got the following compilation errors on package-info.java
Awesome, I need to try it out :) Thank you !
On Thu, Apr 21, 2011 at 9:23 AM, Alex Baranau alex.barano...@gmail.comwrote:
Aha, so you want to count it as single scan (or just differently) when
determining the load?
The current code looks like this:
class DistributedScanner:
public static
Hello,
I am seeing reloading region because regionserver didn't accept updates
message in the middle of populating the table and was wondering what could
be wrong with it?
Thank you so much,
of low end old
machines. Could I increase the region size so that this could be avoided
because I am not very satisfied with the uploading speed ? What are the pros
and cons?
Thanks again :)
On Wed, Apr 20, 2011 at 2:22 PM, Stack st...@duboce.net wrote:
On Wed, Apr 20, 2011 at 12:09 PM, Weishung
Thanks J-D great info !
On Wed, Apr 20, 2011 at 4:45 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote:
Take a look at this section of the book:
http://hbase.apache.org/book/performance.html
J-D
On Wed, Apr 20, 2011 at 12:44 PM, Weishung Chung weish...@gmail.com
wrote:
Hello Stack
Thank you so much for the informative info. It really helps me out.
For secondary index, even without transaction, I would think one could still
build a secondary index on another key especially if we have row level
locking. Correct me if I am wrong.
Also, I have read about clustered B-Tree used
ask that hbase support its needs in some special way.
It is very instructive to see how it was constructed.
Dave
-Original Message-
From: Weishung Chung [mailto:weish...@gmail.com]
Sent: Friday, March 25, 2011 9:27 AM
To: user@hbase.apache.org
Subject: Re: Stargate+hbase
Thank
I was exploring different file formats in hadoop initially but somehow the
search spanned into hbase i guess
Sorry for the confusion Ryan :D
On Wed, Mar 23, 2011 at 7:18 AM, Harsh J qwertyman...@gmail.com wrote:
On Wed, Mar 23, 2011 at 8:36 AM, Ryan Rawson ryano...@gmail.com wrote:
Curious,
Hey my fellow hadoop/hbase developers,
I just came across this google compression/decompression package yesterday,
could we make a good use of this compression scheme in hadoop? It's written
in C++ though.
http://code.google.com/p/snappy/
http://code.google.com/p/snappy/I haven't looked close
cell
value? If so, to read a row, multiple key/value pair reads have to be done?
Thank you :)
On Tue, Mar 22, 2011 at 9:09 AM, Weishung Chung weish...@gmail.com wrote:
Thank you, I will definitely take a look. Also, the TFile spec below helps
me to understand more,
what an exciting work
-format-to.html
On Tue, Mar 22, 2011 at 10:58 AM, Vivek Krishna vivekris...@gmail.comwrote:
http://nosql.mypopescu.com/post/3220921756/hbase-internals-hfile-explained
might help.
Viv
On Tue, Mar 22, 2011 at 11:43 AM, Weishung Chung weish...@gmail.comwrote:
My fellow superb hbase experts
I found this useful article that explains the internal storage of HFile
http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
http://www.larsgeorge.com/2009/10/hbase-architecture-101-storage.html
On Tue, Mar 22, 2011 at 11:31 AM, Weishung Chung weish...@gmail.com wrote:
I also
anywhere in
HBase.
-ryan
On Sat, Mar 19, 2011 at 9:01 AM, Weishung Chung weish...@gmail.com
wrote:
I am browsing through the hadoop.io package and was wondering what other
file formats are available in hadoop other than SequenceFile and TFile?
Is all data written through hadoop including
I am browsing through the hadoop.io package and was wondering what other
file formats are available in hadoop other than SequenceFile and TFile?
Is all data written through hadoop including those from hbase saved in the
above formats? It seems like SequenceFile is in key value pair format.
Thank
Thank you for the info, HFile looks interesting, can't wait to dig into the
code and get a better understanding of HFile !
On Sat, Mar 19, 2011 at 11:28 AM, Harsh J qwertyman...@gmail.com wrote:
Hello,
On Sat, Mar 19, 2011 at 9:31 PM, Weishung Chung weish...@gmail.com
wrote:
Is all data
I am trying to estimate the cost of hosting own HBase cluster vs using EC2.
Could anyone give me some guidance?
Cluster size ~ 6 to 8 nodes
Usage ~ at least 12 hours/day with lot of read/write operations. (I know I
need to have more concrete usage number here)
Thank you so much :)
I have a question about the distribution of row keys in HBase.
If multiple tables have the same row keys, are they stored in the same
region ? I am trying to better understand the key distribution in HBase.
The same keys rows will be accessed at the same time and I am trying to
avoid any hot spot
Thank you :D
On Wed, Mar 9, 2011 at 4:43 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote:
Different tables, different regions, so they won't be together.
J-D
On Wed, Mar 9, 2011 at 2:38 PM, Weishung Chung weish...@gmail.com wrote:
I have a question about the distribution of row keys
/
St.Ack
On Thu, Mar 3, 2011 at 10:48 AM, Weishung Chung weish...@gmail.com
wrote:
Bill, thank you for the clarification.
Ted, good info, i will iterate the map backwards then :)
Another question I have is about unit testing in HBase, any
recommendation
about the best way to simulate
ends
with one. Those in between have non-empy byte[]s to specify their
boundaries.
On Thu, Mar 3, 2011 at 7:18 AM, Weishung Chung weish...@gmail.com
wrote:
Thanks, Stack!
Got a few more questions.
Does every region start with an empty byte[] and end with one too?
Also
Awesome, thanks a lot. I will try them out n let u guys know the result.
On Tue, Mar 1, 2011 at 11:54 PM, 陈加俊 cjjvict...@gmail.com wrote:
Yes I do it like this. But I hava another problem I can't count the rows of
one table fast.
On Wed, Mar 2, 2011 at 12:58 PM, Ted Yu yuzhih...@gmail.com
I tried the method as Stack suggested to find the first row, it works :) I
have yet to learn about Filter and would like to use it too. I was wondering
which method would give a better performance.
As for the max, I will try it out tomorrow.
I thought I could use the getEndKeys() method but it
There is a HTablePool class in org.apache.hadoop.hbase.client package. I was
wondering is there any performance advantages/speedup in using the pool
instead of HTable?
Thank you
only.
Reducing zookeeper connections has performance advantage.
On Tue, Mar 1, 2011 at 8:47 AM, Weishung Chung weish...@gmail.com wrote:
There is a HTablePool class in org.apache.hadoop.hbase.client package. I
was
wondering is there any performance advantages/speedup in using the pool
I checked out 0.90 and now I see the difference :) Thanks.
On Tue, Mar 1, 2011 at 12:30 PM, Weishung Chung weish...@gmail.com wrote:
Stack: thanks
Ted: I am using HBase 0.20, I have yet to check out 0.90.0
On Tue, Mar 1, 2011 at 11:49 AM, Ted Yu yuzhih...@gmail.com wrote:
I assume your
How to get the first or last row in the HBase table? like the min(), max()
in mysql?
Thank you.
-.
*/
Explains it, no?
Lars
On Fri, Feb 4, 2011 at 8:41 AM, Weishung Chung weish...@gmail.com
wrote:
I am looking at the following protected HTableDescriptor's constructor,
but i can't figure out the purpose of the
MapImmutableBytesWritable,ImmutableBytesWritable
values ? What does
I am looking forward to it :)
btw, I found this informative video about Sqoop.
http://www.cloudera.com/blog/2009/12/hadoop-world-sqoop-database-import-for-hadoop/
On Thu, Feb 3, 2011 at 4:44 PM, arv...@cloudera.com arv...@cloudera.comwrote:
On Thu, Feb 3, 2011 at 2:17 PM, Weishung Chung weish
I am looking for tool that allows me to import data from mysql to hbase. Any
suggestion?
Thank you :)
:23 PM, Buttler, David buttl...@llnl.gov
wrote:
Sqoop?
http://archive.cloudera.com/cdh/3/sqoop/SqoopUserGuide.html
-Original Message-
From: Weishung Chung [mailto:weish...@gmail.com]
Sent: Thursday, February 03, 2011 1:18 PM
To: user@hbase.apache.org
Subject
thank you for the clarification :) I am reading about sqoop now...
On Thu, Feb 3, 2011 at 3:41 PM, Mark Kerzner markkerz...@gmail.com wrote:
scribe is a tool for log aggregation, at face value, not mysql
On Thu, Feb 3, 2011 at 3:37 PM, Weishung Chung weish...@gmail.com wrote:
Thank you
About Sqoop, could I import the data specifically into HBase ?
I know I could write a program to read from mysql and use HBase API to write
to HBase.
On Thu, Feb 3, 2011 at 3:49 PM, Weishung Chung weish...@gmail.com wrote:
thank you for the clarification :) I am reading about sqoop now
/FLUME-6
Branch: https://github.com/cloudera/flume/tree/hbase
-Thanks,
Dani.
http://www.cc.gatech.edu/~iar3/
On Thu, Feb 3, 2011 at 5:11 PM, arv...@cloudera.com arv...@cloudera.com
wrote:
On Thu, Feb 3, 2011 at 1:56 PM, Weishung Chung weish...@gmail.com
wrote:
About Sqoop, could I
I am looking at the following protected HTableDescriptor's constructor,
but i can't figure out the purpose of the
MapImmutableBytesWritable,ImmutableBytesWritable
values ? What does it contain?
protected HTableDescriptor(final byte [] name, HColumnDescriptor[]
families,
Is zookeeper responsible for the backup/replication of -ROOT- and .META.
files? It looks like I need multiple HBase masters setup to achieve high
availability. In the multiple masters setup, would there be any data loss in
the switch over after the first master became unavailable.
Is zookeeper responsible for the backup/replication of ROOT and META files?
It looks like I need multiple HBase masters setup to achieve high
availability. In the multiple masters setup, would there be any data loss in
the switch over after the first master became unavailable.
Thank you
? Sorry because I haven't got a
chance to browse deeper in the zookeeper codes yet :(
On Fri, Jan 28, 2011 at 10:35 AM, Stack st...@duboce.net wrote:
On Fri, Jan 28, 2011 at 8:10 AM, Weishung Chung weish...@gmail.com
wrote:
Is zookeeper responsible for the backup/replication of -ROOT- and .META
Great, thank you :D
I guess I need to read up more on zookeeper.
On Fri, Jan 28, 2011 at 10:56 AM, Stack st...@duboce.net wrote:
On Fri, Jan 28, 2011 at 8:52 AM, Weishung Chung weish...@gmail.com
wrote:
Correct me if I am wrong :)
In HConnectionManager, it seems to me that a zookeeper
What's the timeout mechanism in HConnection? Could I keep an instance of
HTable, and letting it idle for a long time like a few hours and reuse it
again for getting data?
Thank you.
I am trying to understand the HBase data insertion steps/procedures. Could
anyone kindly enlighten me about this method?
org.apache.hadoop.hbase.client.HConnectionManager
private HRegionLocation locateRegionInMeta(final byte [] parentTable,
final byte [] tableName, final byte [] row,
Great ! Definitely will check out the package tomorrow ! Thanks :)
On Thu, Jan 27, 2011 at 6:12 PM, tsuna tsuna...@gmail.com wrote:
Note that if you don't want to deal with XML configuration files, you
can try asynchbase (https://github.com/stumbleupon/asynchbase). It's
an alternative fully
ever browse its
source code. Thanks.
On Thu, Jan 27, 2011 at 6:54 PM, Stack st...@duboce.net wrote:
That method does a lookup into the catalog .META. table to find where
a region is located. What else you need to know?
St.Ack
On Thu, Jan 27, 2011 at 3:24 PM, Weishung Chung weish...@gmail.com
I want to group tables in different databases, could I create database and
have tables under it as in mysql?
Thank you
I am abit confused after reading the HBaseConfiguratin source code. When
does Configuration call the loadResource(Properties properties, Object name,
boolean quiet) method and perform the parsing of the xmls? immediately after
the initialization of the HBaseConfiguration instance?
Thank you so
It seems like the properties are only lazily initialized from xmls on the
first call to get a certain property. I guess it was done for performance
purposes. Correct me if i am wrong. Thank you.
On Wed, Jan 26, 2011 at 12:35 PM, Weishung Chung weish...@gmail.com wrote:
I am abit confused after
Correct me if i am wrong. If I want to pass in the path of hbase-site.xml to
the HBaseConfiguration,
Create HBaseConfiguration() will still add the default xmls from the
CLASSPATH and will clear them out when I call addResource(String
pathToResource).
Thanks !
On Wed, Jan 26, 2011 at 4:14 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote:
I happens on this call:
https://github.com/apache/hadoop/blob/trunk/src/core/org/apache/hadoop/conf/Configuration.java#L1002
J-D
On Wed, Jan 26, 2011 at 10:43 AM, Weishung Chung weish...@gmail.com
at 10:58 AM, Weishung Chung weish...@gmail.com
wrote:
Correct me if i am wrong. If I want to pass in the path of hbase-site.xml
to
the HBaseConfiguration,
Create HBaseConfiguration() will still add the default xmls from the
CLASSPATH and will clear them out when I call addResource(String
is HConnectionImplementation
in org.apache.hadoop.hbase.client.HConnectionManager
Are you not using an IDE? Tracking this stuff down without one will be
very
slow.
On Thu, Jan 20, 2011 at 1:57 PM, Weishung Chung weish...@gmail.com
wrote:
I am diving into the HBase source code for to get
Ok, rechecked out the project as java project in Eclipse and now F3 key is
working and I should be able to browse the source with ease, I hope :)
On Thu, Jan 20, 2011 at 4:31 PM, Weishung Chung weish...@gmail.com wrote:
Thank you for the prompt response. I am using Eclipse IDE, Ted, you
)
attributes!
Alex Baranau
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop -
HBase
On Mon, Jan 10, 2011 at 10:45 PM, Weishung Chung weish...@gmail.com
wrote:
Ok, i will test it, thanks again :)
On Mon, Jan 10, 2011 at 1:53 PM, Jean-Daniel Cryans
jdcry
/columns/values will be returned to the client. This does not relate to
the number of rows sent per RPC. See
http://hbase.apache.org/docs/current/api/org/apache/hadoop/hbase/filter/package-summary.htmlfor
more information about filters.
JG
-Original Message-
From: Weishung Chung
What is the good way to randomize the primary key which is a timestamp in
HBase to avoid hotspotting?
Thank you so much :)
on your requirements
On Mon, Jan 10, 2011 at 9:33 AM, Weishung Chung weish...@gmail.com
wrote:
What is the good way to randomize the primary key which is a timestamp
in
HBase to avoid hotspotting?
Thank you so much :)
it is a problem.
On Mon, Jan 10, 2011 at 8:20 AM, Weishung Chung weish...@gmail.com
wrote:
Thank you for the replies.
Most of the queries, (70%) will be for scanning a range of consecutive
times, with some single timestamp query (30%)
But there are multiple tables with the same range
What is the difference between the above put method with the following
capability of the HBaseHUT package ?
https://github.com/sematext/HBaseHUT
On Mon, Jan 10, 2011 at 10:58 AM, Weishung Chung weish...@gmail.com wrote:
Does HTable.put(ListPut puts) method perform a batch insert with a single
the results.
On Mon, Jan 10, 2011 at 11:56 AM, Weishung Chung weish...@gmail.com
wrote:
Thank you for your prompt response. I am a bit confused about the prefix.
If i were to use prefix for the timestamp key, when come to query time,
how
should i specify the row key to search for? How
Does HBase have the capability to partition dataset by range like the MySQL
partitioning eg. partition the datetime, row key by month?
Thank you.
into using the bulk uploader tools such as
http://hbase.apache.org/docs/r0.89.20100924/bulk-loads.html
J-D
On Mon, Jan 10, 2011 at 10:10 AM, Weishung Chung weish...@gmail.com
wrote:
Thank you :)
Could I use org.apache.hadoop.hbase.io.BatchUpdate ? Would it be faster
than
the put
as well as you think you do.
Dave
-Original Message-
From: Weishung Chung [mailto:weish...@gmail.com]
Sent: Monday, January 10, 2011 10:14 AM
To: user@hbase.apache.org
Subject: customize partitioning of regionserver
Does HBase have the capability to partition dataset by range like
Ok, i will test it, thanks again :)
On Mon, Jan 10, 2011 at 1:53 PM, Jean-Daniel Cryans jdcry...@apache.orgwrote:
Depending on the level of super fastness you need, it may or may not
be fast enough. Better to test it, as usual.
J-D
On Mon, Jan 10, 2011 at 11:12 AM, Weishung Chung weish
side. But then you might be constrained by your
subsequent step if you can't parallelize that nicely.
Dave
-Original Message-
From: Weishung Chung [mailto:weish...@gmail.com]
Sent: Monday, January 10, 2011 11:02 AM
To: user@hbase.apache.org
Subject: Re: customize partitioning
81 matches
Mail list logo
