RE: Contributing to hbase but test with less hardware
Hi Mike, Against most people's better judgment, I am running a hadoop/hbase cluster on previously unused desktops that were sitting in my company's basement. Not to be self promoting, but I documented some of the tweaks I did to make it run with very good stability on my website at http://www.travishegner.com/2009/11/hbase-020-low-memory-performance-tuning.html. The document is a little dated, but the basic concept was to cut everything in half. I like to think of it as a mini-cluster. I set it up mostly for learning about parallel computing, but now we use it in pseudo-production to house historical server logs for distributed grep, and (low volume) website analytics. I've been collecting various data from our network for about 6-8 months and have only accumulated 21GB worth... Even a small amount has proven to be quite useful for us. I run my master as a virtual machine to have some sort of redundancy/protection for it. Mini-Cluster Specs: Hadoop: 0.20.3 HBase: 0.20.5 Raw DFS space: ~1 TB Master: ESXi VM 2 Processors 2GB RAM 20 GB Hard disk NameNode, JobTracker, HMaster, Zookeeper Workers (5 total): Gateway E4100 Pentium 4 HT 3.2Ghz 2GB RAM 250 GB Hard disk DataNode, TaskTracker, RegionServer It's not exactly a top dollar cluster, but it suites our needs pretty well, and it's a great learning experience for me. Thanks, Travis Hegner http://www.travishegner.com/ -Original Message- From: hbase...@aol.com [mailto:hbase...@aol.com] Sent: Tuesday, October 26, 2010 6:01 PM To: user@hbase.apache.org Subject: Contributing to hbase but test with less hardware Dear HBase devs, I am reading the HBase sources and have also read the http://wiki.apache.org/hadoop/Hbase/HowToContribute wiki page including the jira noob label suggestion. If I do not have 4 or 5 machines at home with sufficient RAM to test changes on a meaningful HBase cluster, what are my other alternatives? Apart from companies with large deployments (su, cloudera, y!) and where devs can remotely test their changes, is there a free/cheap cluster for the less fortunate others who own laptops with 2GB RAM? Thanks for your help. Mike The information contained in this communication is confidential and is intended only for the use of the named recipient. Unauthorized use, disclosure, or copying is strictly prohibited and may be unlawful. If you have received this communication in error, you should know that you are bound to confidentiality, and should please immediately notify the sender or our IT Department at 866.459.4599.
RE: JAVA CLIENT==10/08/31 20:27:54 INFO ipc.HbaseRPC: Problem connecting to server: /10.0.3.85:60020
Shuja, If you are not running any type of DNS/rDNS service, then make sure the /etc/hosts file on each of your nodes maps each node to the IP address you want it to resolve to. Thanks, Travis Hegner http://www.travishegner.com/ -Original Message- From: Shuja Rehman [mailto:shujamug...@gmail.com] Sent: Wednesday, September 01, 2010 6:02 AM To: user@hbase.apache.org Subject: JAVA CLIENT==10/08/31 20:27:54 INFO ipc.HbaseRPC: Problem connecting to server: /10.0.3.85:60020 Hi All I have used these configuration settings to access hbase server from java client HBaseConfiguration config = new HBaseConfiguration(); config.clear(); config.set(hbase.zookeeper.quorum, myserver.mycompany.com:2181); config.set(hbase.zookeeper.property.clientPort,2181); The problem is that its trying to connect to local ip address instead of global IP. INFO zookeeper.ZooKeeper: Initiating client connection, connectString= myserver.mycompany.com:2181 sessionTimeout=6 watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@fd54d6 10/08/31 20:27:33 INFO zookeeper.ClientCnxn: Opening socket connection to server myserver.mycompany.com/203.14.166.12:2181http://myserver.mycompany.com/20314.166.12:2181 10/08/31 20:27:33 INFO zookeeper.ClientCnxn: Socket connection established to myserver.mycompany.com/203.14.166.12:2181, initiating session 10/08/31 20:27:34 INFO zookeeper.ClientCnxn: Session establishment complete on server myserver.mycompany.com/203.14.166.12:2181, sessionid = 0x12acaa705e30004, negotiated timeout = 4 10/08/31 20:27:54 INFO ipc.HbaseRPC: Problem connecting to server: / 10.0.3.85:60020 10/08/31 20:28:15 INFO ipc.HbaseRPC: Problem connecting to server: / 10.0.3.85:60020 10/08/31 20:28:36 INFO ipc.HbaseRPC: Problem connecting to server: / 10.0.3.85:60020 but it should try to connect to global ip not local ip. so need to resolve this issue.I think we might need to put somewhere global Ip Addresss. I have checked the zk dump and it has the following information HBase tree in ZooKeeper is rooted at /hbase Cluster up? true Master address: 10.0.3.85:6 Region server holding ROOT: 10.0.3.85:60020 Region servers: - 10.0.3.100:60020 - 10.0.3.85:60020 Quorum Server Statistics: - hadoop.zoniversal.com:2181 Zookeeper version: 3.3.1-942149, built on 05/07/2010 17:14 GMT Clients: /10.0.3.85:57636[1](queued=0,recved=1994,sent=1994) /10.0.3.85:57654[1](queued=0,recved=1991,sent=1991) /10.0.3.85:49253[0](queued=0,recved=1,sent=0) /10.0.3.85:46493[1](queued=0,recved=2041,sent=2042) /10.0.3.100:50387[1](queued=0,recved=2016,sent=2017) Latency min/avg/max: 0/0/118 Received: 8056 Sent: 8057 Outstanding: 0 Zxid: 0xda Mode: standalone Node count: 11 Anybody there to help me out?? Thanks -- Regards Shuja-ur-Rehman Baig http://pk.linkedin.com/in/shujamughal Cell: +92 3214207445 The information contained in this communication is confidential and is intended only for the use of the named recipient. Unauthorized use, disclosure, or copying is strictly prohibited and may be unlawful. If you have received this communication in error, you should know that you are bound to confidentiality, and should please immediately notify the sender or our IT Department at 866.459.4599.
RE: Secondary Index versus Full Table Scan
Going out on a limb, I think it will perform MUCH faster with multiple copies, as the data is already sitting in each mappers memory, ready to be accessed locally. The time to process per mapper should be very dramatically reduced. With that in mind, you only have to scale up as disk space requires it, and disk space is cheap. With your current method, adding three more identical data nodes, is only going to cut your time in half. So unless you have the budget to get the number of machines required, it's at least worth a try to have multiple copies, at least that only costs your time. HTH, Travis Hegner http://www.travishegner.com/ -Original Message- From: Luke Forehand [mailto:luke.foreh...@networkedinsights.com] Sent: Tuesday, August 03, 2010 12:37 PM To: user@hbase.apache.org Subject: Re: Secondary Index versus Full Table Scan Edward Capriolo edlinuxg...@... writes: Generally speaking: If you are doing full range scans of a table indexes will not help. Adding indexes will make the performance worse, it will take longer to load your data and now fetching the data will involve two lookups instead of one. If you are doing full range scans adding more nodes should result in linear scale up. Edward, Can you clarify what full range scan means? I am not doing full range scans, but I am doing relatively large range scans (3 million records), so I think what you are saying applies. Thanks for the insight. We initially implemented the secondary index out of a need to have our main data sorted by multiple dimensions for various use cases. Now I'm thinking it may be better to have multiple copies of our main data, sorted in multiple ways, to avoid the two lookups. So I'm faced with two options right now; multiple copies of the data sorted in multiple ways to do range scans, or buy a lot more servers and do full scans. Given these two choices, do people have general recommendations on which makes the most sense? Thanks! -Luke The information contained in this communication is confidential and is intended only for the use of the named recipient. Unauthorized use, disclosure, or copying is strictly prohibited and may be unlawful. If you have received this communication in error, you should know that you are bound to confidentiality, and should please immediately notify the sender or our IT Department at 866.459.4599.
RE: operations, how 'hard' is it to keep the servers humming
The biggest issue you'll likely have is hardware, so if you are running ec2, that is out the window. I run my datanodes on 'old' desktop grade hardware... Single Power Supply, 2GB RAM, Single HT P4 procs, and single 250GB disks. I know, it's bad, but for my current purposes, it works pretty well. Once the cluster is up and running, and I'm not changing configs and constantly restarting, it will run for weeks without intervention. If you run on server grade hardware, built to tighter specs, the chances of failure (therefore intervention for repair or replacement) are lower. If you run on ec2, then someone else is dealing with the hardware, and you can just use the cluster... Travis Hegner http://www.travishegner.com/ -Original Message- From: S Ahmed [mailto:sahmed1...@gmail.com] Sent: Wednesday, July 21, 2010 3:36 PM To: user@hbase.apache.org Subject: Re: operations, how 'hard' is it to keep the servers humming Can you define what you mean by 'complete junk' ? I plan on using ec2. On Wed, Jul 21, 2010 at 3:23 PM, Hegner, Travis theg...@trilliumit.comwrote: That question is completely dependant on the size of the cluster you are looking at setting up, which is then dependant on how much data you want to store and/or process. A one man show should be able to handle 10-20 machines without too much trouble, unless you run complete junk. I run a 6 node cluster on complete junk, and I rarely have had to tinker with it since setting it up. Travis Hegner http://www.travishegner.com/ -Original Message- From: S Ahmed [mailto:sahmed1...@gmail.com] Sent: Wednesday, July 21, 2010 2:59 PM To: user@hbase.apache.org Subject: operations, how 'hard' is it to keep the servers humming From a operations standpoint, is setup a hbase cluster and keeping them running a fairly complex task? i.e. if I am a 1-man show, would it be a smart choice to build on top of hbase or is a crazy idea? The information contained in this communication is confidential and is intended only for the use of the named recipient. Unauthorized use, disclosure, or copying is strictly prohibited and may be unlawful. If you have received this communication in error, you should know that you are bound to confidentiality, and should please immediately notify the sender or our IT Department at 866.459.4599. The information contained in this communication is confidential and is intended only for the use of the named recipient. Unauthorized use, disclosure, or copying is strictly prohibited and may be unlawful. If you have received this communication in error, you should know that you are bound to confidentiality, and should please immediately notify the sender or our IT Department at 866.459.4599.
RE: Client Connection to Zookeeper times out during mapreduce
Thanks St.Ack! I must have lost that config after my last hadoop upgrade. Much appreciated! Travis -Original Message- From: saint@gmail.com [mailto:saint@gmail.com] On Behalf Of Stack Sent: Thursday, July 15, 2010 11:42 AM To: user@hbase.apache.org Subject: Re: Client Connection to Zookeeper times out during mapreduce Is this intentional? Attempting connection to server localhost/127.0.0.1:2181 I'd guess not. Would seem to indicate then that your conf dir is not on the CLASSPATH when the child task runs? You've seen http://hbase.apache.org/docs/r0.20.5/api/org/apache/hadoop/hbase/mapreduce/package-summary.html#classpath? St.Ack On Thu, Jul 15, 2010 at 7:16 AM, Hegner, Travis theg...@trilliumit.com wrote: Afterthought: hadoop: 0.20.2, r911707 hbase: 0.20.5, r950246 Thanks Again! -Original Message- From: Hegner, Travis [mailto:theg...@trilliumit.com] Sent: Thursday, July 15, 2010 10:15 AM To: user@hbase.apache.org Subject: Client Connection to Zookeeper times out during mapreduce Hi All! I apologize in advance for what is probably a rookie mistake, but it's been a while since I've done a map reduce with Hbase as input or output. I'm assuming that this is a cluster config problem because even the built-in rowcounter mapreduce job is failing. The hbase shell is operating normally, and I can even access Hbase from thrift. Cluster: 1 master: NN, JT, ZK, HM 5 slaves: DN, TT, RS When attempting to run a mapreduce job with Hbase as a source, my map tasks are failing with the following logged exceptions: 2010-07-15 09:16:22,383 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201007150924_0003/attempt_201007150924_0003_m_00_0/work 2010-07-15 09:16:22,386 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=6 watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@80fa6f 2010-07-15 09:16:22,389 INFO org.apache.zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false 2010-07-15 09:16:22,402 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server localhost/127.0.0.1:2181 2010-07-15 09:16:22,407 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to sun.nio.ch.selectionkeyi...@1d85f79 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) 2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown input java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) 2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown output java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) 2010-07-15 09:16:22,532 WARN org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase -- check quorum servers, currently=localhost:2181 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:974) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625) at org.apache.hadoop.hbase.client.HConnectionManager
RE: High ingest rate and FIN_WAIT1 problems
Hi Thomas, I ran into a very similar issue when running slony-I on postgresql to replicate 15-20 databases. Adjusting the TCP_FIN_TIMEOUT parameters for the kernel may help to slow (or hopefully stop), the leaking sockets. I found some notes about adjusting TCP parameters here: http://www.hikaro.com/linux/tweaking-tcpip-syctl-conf.html with the specific excerpt regarding the TCP_FIN_TIMEOUT: TCP_FIN_TIMEOUT This setting determines the time that must elapse before TCP/IP can release a closed connection and reuse its resources. During this TIME_WAIT state, reopening the connection to the client costs less than establishing a new connection. By reducing the value of this entry, TCP/IP can release closed connections faster, making more resources available for new connections. Addjust this in the presense of many connections sitting in the TIME_WAIT state: # echo 30 /proc/sys/net/ipv4/tcp_fin_timeout Try setting this lower on your master. You may also consider these on the same link: TCP_TW_RECYCLE It enables fast recycling of TIME_WAIT sockets. The default value is 0 (disabled). The sysctl documentation incorrectly states the default as enabled. It can be changed to 1 (enabled) in many cases. Known to cause some issues with hoststated (load balancing and fail over) if enabled, should be used with caution. echo 1 /proc/sys/net/ipv4/tcp_tw_recycle TCP_TW_REUSE This allows reusing sockets in TIME_WAIT state for new connections when it is safe from protocol viewpoint. Default value is 0 (disabled). It is generally a safer alternative to tcp_tw_recycle echo 1 /proc/sys/net/ipv4/tcp_tw_reuse The above commands will not persist reboots, but the link explains how. The list experts may be able to give more insight on which, if any, of these settings are safe to manipulate, and what risks or issues you may encounter specifically with Hbase while adjusting these settings. Hope This Helps, Travis Hegner -Original Message- From: Thomas Downing [mailto:tdown...@proteus-technologies.com] Sent: Friday, July 16, 2010 10:33 AM To: user@hbase.apache.org Subject: High ingest rate and FIN_WAIT1 problems Hi, I am a complete HBase and HDFS newbie, so I apologize in advance for the inevitable bloopers. We are doing feasibility testing on NoSql data store options, with rather high ingest rate requirements. So far, HBase is looking good, with only one issue identified. Running at an ingest rate of ~30K rows per second on a 4 2.2Mhz CPU 8G RAM machine I am slowly leaking sockets. This is a single node setup - no replication. The CPU load is only about 50%-60%, with the majority of that in userland, system and iowait are averaging less than 3%. There is no swapping going on. The problem is that on the datanode there are a large number of sockets in FIN_WAIT1, with corresponding peers on master in ESTABLISHED. These pairs hang around for quite some time, at at my ingest rate this means that the total sockets held by datanode and master is slowly going up. If my understanding of TCP is correct, then this indicates that the master peer has stopped reading incoming data from the datanode - i.e, it is sending a window of zero; and that the datanode has called close(2) on it's peer. There was a thread some time ago: http://www.mail-archive.com/hbase-u...@hadoop.apache.org/msg03329.html There was no real conclusion. I have played with the config params as suggested on that thread, but no luck yet. Also, in that case the problem seemed to be between datanodes for replication operations - not the case with me. Changing timeouts to avoid the slow increase might not really solve the problem IFF the master peer has in fact ceased to read it's socket. The data outstanding in the TCP stack buffer would be lost. Whether that would imply data loss is beyond me. I am posting this here as although the only logs with errors/exceptions are the datanode logs, netstat and wireshark seem to indicate that the problem is on the master side. The master, namenode, regionserver and zookeeper and logs shows no warning or errors. The datanode log shows this, over and over: 2010-07-16 00:33:09,269 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: DatanodeRegistration(127.0.0.1:50010, storageID=DS-1028643313-10.1.1.200-50010-1279026099917, infoPort=50075, ipcPort=50020):Got exception while serving blk_3684861726145519813_22386 to /127.0.0.1: java.net.SocketTimeoutException: 48 millis timeout while waiting for channel to be ready for write. ch : java.nio.channels.SocketChannel[connected local=/127.0.0.1:50010 remote=/127.0.0.1:54774] at org.apache.hadoop.net.SocketIOWithTimeout.waitForIO(SocketIOWithTimeout.java:246) at org.apache.hadoop.net.SocketOutputStream.waitForWritable(SocketOutputStream.java:159) at org.apache.hadoop.net.SocketOutputStream.transferToFully(SocketOutputStream.java:198) at
Client Connection to Zookeeper times out during mapreduce
Hi All! I apologize in advance for what is probably a rookie mistake, but it's been a while since I've done a map reduce with Hbase as input or output. I'm assuming that this is a cluster config problem because even the built-in rowcounter mapreduce job is failing. The hbase shell is operating normally, and I can even access Hbase from thrift. Cluster: 1 master: NN, JT, ZK, HM 5 slaves: DN, TT, RS When attempting to run a mapreduce job with Hbase as a source, my map tasks are failing with the following logged exceptions: 2010-07-15 09:16:22,383 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201007150924_0003/attempt_201007150924_0003_m_00_0/work 2010-07-15 09:16:22,386 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=6 watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@80fa6f 2010-07-15 09:16:22,389 INFO org.apache.zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false 2010-07-15 09:16:22,402 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server localhost/127.0.0.1:2181 2010-07-15 09:16:22,407 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to sun.nio.ch.selectionkeyi...@1d85f79 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) 2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown input java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) 2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown output java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) 2010-07-15 09:16:22,532 WARN org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase -- check quorum servers, currently=localhost:2181 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:974) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:134) at org.apache.hadoop.hbase.client.HTable.init(HTable.java:112) at org.apache.hadoop.hbase.mapreduce.TableInputFormat.setConf(TableInputFormat.java:87) at
RE: Client Connection to Zookeeper times out during mapreduce
Afterthought: hadoop: 0.20.2, r911707 hbase: 0.20.5, r950246 Thanks Again! -Original Message- From: Hegner, Travis [mailto:theg...@trilliumit.com] Sent: Thursday, July 15, 2010 10:15 AM To: user@hbase.apache.org Subject: Client Connection to Zookeeper times out during mapreduce Hi All! I apologize in advance for what is probably a rookie mistake, but it's been a while since I've done a map reduce with Hbase as input or output. I'm assuming that this is a cluster config problem because even the built-in rowcounter mapreduce job is failing. The hbase shell is operating normally, and I can even access Hbase from thrift. Cluster: 1 master: NN, JT, ZK, HM 5 slaves: DN, TT, RS When attempting to run a mapreduce job with Hbase as a source, my map tasks are failing with the following logged exceptions: 2010-07-15 09:16:22,383 INFO org.apache.zookeeper.ZooKeeper: Client environment:user.dir=/tmp/hadoop-hadoop/mapred/local/taskTracker/jobcache/job_201007150924_0003/attempt_201007150924_0003_m_00_0/work 2010-07-15 09:16:22,386 INFO org.apache.zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181 sessionTimeout=6 watcher=org.apache.hadoop.hbase.client.hconnectionmanager$clientzkwatc...@80fa6f 2010-07-15 09:16:22,389 INFO org.apache.zookeeper.ClientCnxn: zookeeper.disableAutoWatchReset is false 2010-07-15 09:16:22,402 INFO org.apache.zookeeper.ClientCnxn: Attempting connection to server localhost/127.0.0.1:2181 2010-07-15 09:16:22,407 WARN org.apache.zookeeper.ClientCnxn: Exception closing session 0x0 to sun.nio.ch.selectionkeyi...@1d85f79 java.net.ConnectException: Connection refused at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:933) 2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown input java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:638) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:999) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) 2010-07-15 09:16:22,416 WARN org.apache.zookeeper.ClientCnxn: Ignoring exception during shutdown output java.nio.channels.ClosedChannelException at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:649) at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1004) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:970) 2010-07-15 09:16:22,532 WARN org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper: Failed to create /hbase -- check quorum servers, currently=localhost:2181 org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:780) at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:808) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:405) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:432) at org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:545) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:974) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:625) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:630) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:601) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:670) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:634) at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java
RE: How to search and make indexes in ColumnFamilies with unknown columns ?
As I said before, I'm not very familiar with the API for scans/filters/etc.
If you are not worried about realtime access to that query, then you could run
a map reduce job that takes in all rows, you could validate whether
Courses:Maths exists in each row or not. If it exists context.write(Maths,
1); and then reduce it overall to accumulate a total.
Even better since you'd be running a mapreduce, for each course:
context.write(course, 1); and reduce the overall output with the course name
as the key, this will give you the total number in each course as a sorted list.
If you want realtime access, then potentially explore a secondary table as an
index, which is kept up to date by the inserting application, or by a scheduled
map/reduce.
Thanks,
Travis Hegner
http://www.travishegner.com/
-Original Message-
From: SyedShoaib [mailto:shoaib_ta...@hotmail.com]
Sent: Friday, June 25, 2010 5:44 AM
To: hbase-u...@hadoop.apache.org
Subject: RE: How to search and make indexes in ColumnFamilies with unknown
columns ?
Thank you very much for your help. If we keep courses as columns, the problem
remains the same. Actually, the number of columns are unknown. There can be
1000 subjects in one row. There may be only two subjects in another row.
These subjects are unknown to us while we are programming through client
API. The user will insert them on runtime. Now how a Filter in Client API
will search a particular course in all columns of a ColumnFamily? All the
filters I have explored search only in a single column of a ColumnFamily at
one time. Thats the real problem.
Many thanks for the help again.
regards,
Hegner, Travis wrote:
I'm not an expert by any means, but I wonder if you were to store the
course name/type as the column name, and some arbitrary but useful value
as the value, for example:
Student_Courses // Table Name
{
Student: // Column Family
{
ID = 12345678
Name = John Smith
}
Courses: // Column Family with any number of columns:
{
Maths = 2010_Fall
Computer = 2011_Spring
.
.
Science = 2011_Spring
}
}
The API may be better suited to handle filtering by column name, rather
than value, but as I said, I'm no expert, and I have very little
experience filtering via the API.
Assuming the filter works correctly, you could simply ignore the value
retrieved if it wasn't needed. Be careful about putting too large of a
value in though, as that could affect performance. This is one of the
beauties of a column oriented schema, you can store useful, valuable
information as a column name.
I do know that with this type of schema, the columns would be accessed
like:
get(row_id, Courses:Maths[, version]);
or something to that effect anyway...
Hope This Helps, Good Luck!
Travis Hegner
http://www.travishegner.com/
-Original Message-
From: SyedShoaib [mailto:shoaib_ta...@hotmail.com]
Sent: Thursday, June 24, 2010 8:26 AM
To: hbase-u...@hadoop.apache.org
Subject: How to search and make indexes in ColumnFamilies with unknown
columns ?
Hi,
I am new to HBase and have just worked on it for few days. I have two
questions. Any kind of help is fully appreciated and many thanks in
advance.
1) Suppose I have a columnFamily with unknown number of columns. I want to
search a value in this columnFamily. That value can be present in any
column
of this columnFamily. How will I search a value in whole columnFamily? For
further elaboration please consider a simple scenario:
For example: A student can have any number of courses. Schema in HBase
could
be:
Student_Courses // Table Name
{
Student: // Column Family
{
ID:
Name:
}
Courses: // Column Family with any number of columns:
{
Course_1: Maths
Course_2: Computer
.
.
Course_n: Science
}
}
If I want to search all rows with a value “Maths” in any of the column
inside columnFamily “Course:” what will I do ? I can search for any value
through SingleColumnValueFilter by mentioning ColumnFamily and Prefix
e.g.
Student:Name. But how will I search a value in Course: columnFamily
keeping the fact in mind that I dont know how many columns I have in it.
2) How will I make an index on this columnFamily (“Course:”) ? I know
indexes are made on columns but the columns are unknown in number! I can
make an index on Student:Name. But what to do if I want to make a single
index on complete “Courses:” ColumnFamily? Is it possible? It will help me
a
lot during a search like SHOW ME ALL THE STUDENTS REGISTERED IN MATHS.
Regards,
--
View this message in context:
http://old.nabble.com/How-to-search-and-make-indexes-in-ColumnFamilies-with-unknown-columns---tp28981932p28981932.html
Sent from the HBase User mailing list archive at Nabble.com.
The information contained
