Hi Stack,
Re Lustre use: I'm not a hardware infrastructure type of guy, but I can tell
you that we have a very fast interconnect for access into the global filesystem:
The Olympus Infiniband topology is a combination of 2:1 oversubscribed 36 port
leaf switches and direct links into a 648 port
While generating some load against a library that makes extensive use of
HTablePool in 0.92, I noticed that the largest heap consumer was
java.lang.ref.Finalizer. Digging in, I discovered that HTablePool's
internal PooledHTable extends HTable, which instantiates a
ThreadPoolExecutor and
Hi All,
This issue has been solved by passing Hbase configuration on the mapreduce
codes directly.
*conf* = HBaseConfiguration.*create*();
*conf*.clear();
*conf.set(hbase.zookeeper.quorum, namenode hostname/IP);*
* *
*conf.set(hbase.zookeeper.property.clientPort,client port);*
* *
On Mon, Jul 2, 2012 at 11:16 PM, Taylor, Ronald C
ronald.tay...@pnnl.gov wrote:
...
Thanks for the interesting setup description Ronald. Definitely
interested in how things progress.
Go easy,
St.Ack
You can log a Jira where you attach your patch.
Thanks
On Jul 2, 2012, at 8:13 PM, Ryan Brush rbr...@gmail.com wrote:
While generating some load against a library that makes extensive use of
HTablePool in 0.92, I noticed that the largest heap consumer was
java.lang.ref.Finalizer. Digging
Hello,
I have a simpel table with 1.5 billion rows and one column familiy 'F'.
Each row key is 33 bytes and the cell values are void. By doing the math I
would expect this table to take up (33+1)x1.5*10^9 = 51GB. However if I do
a hadoop dfs -du I get that the table takes up ~82GB. This is after
Hi
I am using HBase 0.90.6 - cdh3u4. I am inserting data into the HBase table
almost every second using my Java code. After running the program for 3-4 days,
I am getting heap space error while running Hbase on certain nodes of my
cluster. I have allocated 2G as the heapspace of Hbase. I
Hi,
For development purpose, I have set up HBase in pseudodistributed mode.
I have following line in hbase-env.sh file
export HBASE_MANAGES_ZK=true
HBase shell works fine. But client API is not working.
My client code is as follows
Configuration config =
Hello Ananda,
Add these two lines in your client and sww if it works for you :
config.set(hbase.zookeeper.property.clientPort,2181);
config.set(hbase.master, localhost:6);
Regards,
Mohammad Tariq
On Tue, Jul 3, 2012 at 3:49 PM, AnandaVelMurugan Chandra Mohan
Hi,
Thanks for the response. Sadly I am still getting same error.
On Tue, Jul 3, 2012 at 3:58 PM, Mohammad Tariq donta...@gmail.com wrote:
Hello Ananda,
Add these two lines in your client and sww if it works for you :
config.set(hbase.zookeeper.property.clientPort,2181);
Can you paste the contents of your /etc/hosts and hbase-site.xml files??
Regards,
Mohammad Tariq
On Tue, Jul 3, 2012 at 4:06 PM, AnandaVelMurugan Chandra Mohan
ananthu2...@gmail.com wrote:
Hi,
Thanks for the response. Sadly I am still getting same error.
On Tue, Jul 3, 2012 at 3:58
Stack/Lars,
Here's an entry for OCLC:
OCLC (www.worldcat.org) uses HBase as the main data store for WorldCat,
a union catalog which aggregates the collections of 72,000 libraries in
112 countries and territories. WorldCat is currently comprised of
nearly 1 billion records with nearly 2 billion
Timestamps on the cells themselves?
# Versions?
On Jul 3, 2012, at 4:54 AM, Sever Fundatureanu wrote:
Hello,
I have a simpel table with 1.5 billion rows and one column familiy 'F'.
Each row key is 33 bytes and the cell values are void. By doing the math I
would expect this table to take
Hi,
You're over thinking this.
Take a step back and remember that you can store anything you want as a byte
stream in a column.
Literally.
So you have a record that could be a text blob. Store it in one column. Use
JSON to define its structure and fields.
The only thing that makes it
Hi,
These are text from the files
/etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
hbase-site.xml
What's the status of Hadoop and IPV6 vs IPV4?
On Jul 3, 2012, at 7:07 AM, AnandaVelMurugan Chandra Mohan wrote:
Hi,
These are text from the files
/etc/hosts
127.0.0.1 localhost
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost
On Tue, Jul 3, 2012 at 1:29 PM, Buckley,Ron buckl...@oclc.org wrote:
Stack/Lars,
Here's an entry for OCLC:
OCLC (www.worldcat.org) uses HBase as the main data store for WorldCat,
a union catalog which aggregates the collections of 72,000 libraries in
112 countries and territories. WorldCat
Right, forgot about the timestamps. These should be a long value each, so 8
bytes. The versioning is set to 1 so it shouldn't count.
Note the column qualifier is also void on each entry.
So now we get (33+1+8)x1.5*10^9 = 63GB, still a 19GB difference...
Thanks,
Sever
On Tue, Jul 3, 2012 at 1:48
On Tue, Jul 3, 2012 at 2:17 PM, Sever Fundatureanu
fundatureanu.se...@gmail.com wrote:
Right, forgot about the timestamps. These should be a long value each, so 8
bytes. The versioning is set to 1 so it shouldn't count.
Note the column qualifier is also void on each entry.
So now we get
Are you sure about the Hbase shell???Are you able to create tables ,
or list the tables through shell??
Regards,
Mohammad Tariq
On Tue, Jul 3, 2012 at 5:41 PM, Michael Segel michael_se...@hotmail.com wrote:
What's the status of Hadoop and IPV6 vs IPV4?
On Jul 3, 2012, at 7:07 AM,
Comparisons are fine.
Try to not think of this in terms of rows and columns, but in terms of records.
Think of each record as being atomic.
Create a list of all of the components that make up that record.
Then combine like components in to structures.
Like the Street Address. Add in a
Thanks for the link.
I followed the link and fixed my hdfs url too.
But when I start hbase, hbase master and zookeeper processes are not
starting
I tried starting Hbase master manually and I got this error.
ERROR master.HMasterCommandLine: Failed to start master
java.io.IOException: CRC check
What do you mean by I tried starting Hbase master manually and I got
this error.??..By manually do you mean through the shell??How were
you trying to do it earlier??And if possible could you please post the
modified core-site.xml and hbase-site.xml files.
Regards,
Mohammad Tariq
On Tue, Jul
For starting Hbase master manually
I did cd to HBASE_HOME\bin
Then did ./hbase master start
Contents of my hbase-site.xml
configuration
property
namehbase.rootdir/name
valuehdfs://localhost:8020/user/eucalyptus/hbase/value
descriptionThe directory shared by RegionServers.
Change the value of fs.default.name to hdfs://localhost:8020 and
restart everything again..It should be a combination of host:port.
Regards,
Mohammad Tariq
On Tue, Jul 3, 2012 at 8:12 PM, AnandaVelMurugan Chandra Mohan
ananthu2...@gmail.com wrote:
For starting Hbase master manually
I did
That did not help. Still I dont see hbase master and zookeeper processes. I
am thinking of starting everything from scratch. Any suggestions?
On Tue, Jul 3, 2012 at 8:17 PM, Mohammad Tariq donta...@gmail.com wrote:
Change the value of fs.default.name to hdfs://localhost:8020 and
restart
I was only du'ing the table dir. The tmp dirs only had a couple of hundred
bytes in my case.
The HFile tool only gives the avgKeyLen=46. This does not include 4 bytes
KeyLength + 4 bytes ValueLength.
Now indeed I get a total of 54 bytes/KV *1.5 billion ~= 81GB. Probably
there are also leftovers
My HMaster and HRegionservers start and run for awhile.
Looking at the messages, there are appear to be some Datanodes with some
issues, HLogSplitter has some block issues, the HMaster appears to drop
off the network (i know bad), then it comes back, and then the cluster
runs for about 10 more
On Tuesday, July 3, 2012 at 10:08 AM, Jay Wilson wrote:
2012-07-03 09:05:00,530 ERROR
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Couldn't close
log at
hdfs://devrackA-00:8020/var/hbase-hadoop/hbase/-ROOT-/70236052/recovered.edits/046.temp
Would Datanode issues impact the HMaster stability?
Yes and no. If you have only a few datanodes down, their should be no
issue. When there are enough missing datanodes to make some blocks not
available at all in the cluster, there are many tasks that can not be
done anymore (to say the least,
Hi,
I just upgraded the hbase to 0.90.6. I am usign a mapr distribution.
When I start up the mster, I get the
following error message:
2012-07-03 13:30:01,214 FATAL org.apache.hadoop.hbase.master.HMaster:
Unhandled exception.
Starting shutdown.
org.apache.hadoop.ipc.RemoteException:
In your case, likely you are hitting the blocking store files
(hbase.hstore.blockingStoreFiles default:7) and/or
hbase.hregion.memstore.block.multiplier - check out
http://hbase.apache.org/book/config.files.html for more details on
this configurations and how they affect your insert performance.
I have reread the sections in the O'Reilly HBase book on cluster
configuration and troubleshooting and I am still getting session
expired after X number of minutes. X being anywhere from 15 to 20 minutes.
There is 0 load on the cluster and it's using a dedicated isolated
network. No jobs running
Hi,
The KV storage will be like
KeyLength (4 bytes) + Value length(4 bytes) + rowkeylength(2bytes) + rowkey(..
bytes) + CF length(1 byte) + CF (...bytes) + Qualifier(..bytes) + timestamp(8
bytes) + type(1 byte) + value (...bytes)
If you are using HFile V2 there will be memstoreTS also added
Jay,
You need to modify the zoo.cfg to reflect the quorum.
server.0=localhost:2888:3888 will change to something like
server.0=zk_host_1:2888:3888
server.1=zk_host_2:2888:3888
server.3=zk_host_3:2888:3888
The same config needs to be on all the zookeeper hosts.
Also, I assume it's a self
35 matches
Mail list logo
