Is there a setting to cap row size?

2011-04-07 Thread Bryan Keller 
I have a wide table schema for an HBase table, where I model a one-to-many 
relationship of purchase orders and line items. Each row is a purchase order, 
and I add columns for each line item. Under normal circumstances I don't expect 
more than a few thousand columns per row, totalling less than 1mb per row in 
general.

In one of my stress tests, I was inserting many line items into the same row. 
Eventually, the row's region server shut down. In the log there was an 
IOException with an error about the wite-ahead log failing to close, with the 
subsequent regionserver shutdown.

Once in this state, the only way I could manage to get my system functional 
again was to wipe the /hbase directory in HDFS and start from scratch.

To avoid having my system susceptible to total data loss due to some bad import 
data or the like, I'd like to limit the size of the row so it will simply throw 
an exception if it reaches a certain size (either in bytes or in number of 
columns). Does such a setting exist?

Re: Is there a setting to cap row size?

2011-04-07 Thread Ryan Rawson 
Sounds like you are having a HDFS related problem.  Check those
datanode logs for errors.

As for a setting for max row size, this might not be so easy to do,
since during the Put time we don't actually know anything about the
existing row data. To find that out we'd have to go and read the row
first then make a decision.

There are some sources for how HBase stores on disk, we are also very
similar to the bigtable paper.

-ryan

On Thu, Apr 7, 2011 at 7:24 AM, Bryan Keller brya...@gmail.com wrote:
 I have a wide table schema for an HBase table, where I model a one-to-many 
 relationship of purchase orders and line items. Each row is a purchase order, 
 and I add columns for each line item. Under normal circumstances I don't 
 expect more than a few thousand columns per row, totalling less than 1mb per 
 row in general.

 In one of my stress tests, I was inserting many line items into the same row. 
 Eventually, the row's region server shut down. In the log there was an 
 IOException with an error about the wite-ahead log failing to close, with the 
 subsequent regionserver shutdown.

 Once in this state, the only way I could manage to get my system functional 
 again was to wipe the /hbase directory in HDFS and start from scratch.

 To avoid having my system susceptible to total data loss due to some bad 
 import data or the like, I'd like to limit the size of the row so it will 
 simply throw an exception if it reaches a certain size (either in bytes or in 
 number of columns). Does such a setting exist?

timing out for hdfs errors faster

2011-04-07 Thread Jack Levin 
Hello, I get those errors sometimes:

2011-04-07 07:49:41,527 WARN org.apache.hadoop.hdfs.DFSClient: Failed
to connect to /10.103.7.5:50010 for file
/hbase/media_data/1c95bfcf0dd19800b1f44278627259ae/att/7725092577730365184
for block 802538788372768807:java.net.SocketTimeoutException: 6
millis timeout while waiting for channel to be ready for read. ch :
java.nio.channels.SocketChannel[connected local=/10.101.6.8:40801
remote=/10.103.7.5:50010

What would be configuration setting to shorten the timeout say to 5
seconds?  What about retries (if any).

-Jack

org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLo

2011-04-07 Thread Shuja Rehman 
Hi

I am trying to read from hbase the following code.

http://pastebin.com/wvVVUT3p

it reads for first 4-5 times but after that it start throwing this exception

SEVERE: null
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
at
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
at org.apache.hadoop.hbase.client.HConnectionManager$HCo


The full details of exception are here.

http://pastebin.com/Yj4VgvHG

I am using CDH3b4 in pseudo enviornment.

Any clue???
thnx

-- 
Regards
Shuja-ur-Rehman Baig
http://pk.linkedin.com/in/shujamughal

file is already being created by NN_Recovery

2011-04-07 Thread Daniel Iancu 

Hello everybody
We've run into this, now popular, error on our cluster

2011-04-07 16:28:00,654 WARN IPC Server handler 0 on 8020 
org.apache.hadoop.hdfs.StateChange - DIR* NameSystem.startFile: failed 
to create file 
/hbase/.logs/search-hadoop-eu001.v300.gmx.net,60020,1302075782687/search-hadoop-eu001.v300.gmx.net%3A60020.1302075783467 
for 
DFSClient_hb_m_search-namenode-eu002.v300.gmx.net:6_1302186078300 on 
client 10.1.100.32, because this file is already being created by 
NN_Recovery on 10.1.100.61


I've read a couple of threads around it, still it seems that nobody 
pinpointed the cause of it? The only solution here remains to delete the 
log file and lose data ?


I've seen  this error on almost any cluster we've installed so far, 
deleting logs was not concerning since all were test clusters. Now we 
got this on the production cluster, and strange, this cluster was just 
installed, there is no table and no data, no activity there. So what 
logs is master trying to create?


We are running the latest CDH3B4 from Cloudera.

Thanks for any hints,
Daniel

Re: file is already being created by NN_Recovery

2011-04-07 Thread Jack Levin 
If you have socket.dfs.timeout set to 0, consider removing it, most of
our issues like that went away after that.  This problem occurs when
you have datanode crash, and there is a conflict with the lease on the
file (which should expire in one hour, this is unconfigurable hard
timeout).   If you do end up in situation like that, the only way we
could resolve it is like this:

# stop the master
# hadoop fs -cp file new_file
# hadoop fs -rm file
# hadoop fs -cp new_file file
# start master, and watch it replay the log.

This appears to break the lease as new .log file does not have this issue.

-Jack

On Thu, Apr 7, 2011 at 9:35 AM, Daniel Iancu daniel.ia...@1and1.ro wrote:
 Hello everybody
 We've run into this, now popular, error on our cluster

 2011-04-07 16:28:00,654 WARN IPC Server handler 0 on 8020
 org.apache.hadoop.hdfs.StateChange - DIR* NameSystem.startFile: failed to
 create file
 /hbase/.logs/search-hadoop-eu001.v300.gmx.net,60020,1302075782687/search-hadoop-eu001.v300.gmx.net%3A60020.1302075783467
 for DFSClient_hb_m_search-namenode-eu002.v300.gmx.net:6_1302186078300 on
 client 10.1.100.32, because this file is already being created by
 NN_Recovery on 10.1.100.61

 I've read a couple of threads around it, still it seems that nobody
 pinpointed the cause of it? The only solution here remains to delete the log
 file and lose data ?

 I've seen  this error on almost any cluster we've installed so far, deleting
 logs was not concerning since all were test clusters. Now we got this on the
 production cluster, and strange, this cluster was just installed, there is
 no table and no data, no activity there. So what logs is master trying to
 create?

 We are running the latest CDH3B4 from Cloudera.

 Thanks for any hints,
 Daniel

Task process exit with nonzero status of 255.

2011-04-07 Thread Shahnawaz Saifi 
Hi,

While executing MR with 472G data, I am running into following error:

java.io.IOException: Task process exit with nonzero status of 255.
at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)




2011-04-07 07:54:15,442 INFO org.apache.zookeeper.ClientCnxn: Opening socket
connection to server 10.40.99.22:2181
2011-04-07 07:54:15,442 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to 10.40.99.22:2181, initiating session
2011-04-07 07:54:15,443 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
for server 10.40.99.22:2181, unexpected error, closing socket connection and
attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
at sun.nio.ch.IOUtil.read(IOUtil.java:204)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
2011-04-07 07:54:15,545 FATAL
com.example.dataengine.aggregation.aggregationMR_HDFS:
*
2011-04-07 07:54:15,545 FATAL
com.example.dataengine.aggregation.aggregationMR_HDFS:
org.apache.hadoop.hbase.ZooKeeperConnectionException:
org.apache.zookeeper.KeeperException$ConnectionLossException:
KeeperErrorCode = ConnectionLoss for /hbase
2011-04-07 07:54:15,546 FATAL
com.example.dataengine.aggregation.aggregationMR_HDFS:
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.init(HConnectionManager.java:292)
org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
org.apache.hadoop.hbase.client.HTable.init(HTable.java:167)
org.apache.hadoop.hbase.client.HTable.init(HTable.java:145)
com.example.dataengine.aggregation.aggregationMR_HDFS$aggregationMapper.setup(aggregationMR_HDFS.java:324)
org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
org.apache.hadoop.mapred.Child.main(Child.java:170)

2011-04-07 07:54:15,546 FATAL
com.example.dataengine.aggregation.aggregationMR_HDFS:
*


-- 
Thanks,
Shah

Re: Task process exit with nonzero status of 255.

2011-04-07 Thread Jean-Daniel Cryans 
Check the log of the zookeeper at the adress that's printed, it may be
a problem of too many connections (in which case you need to make sure
you reuse the configuration objects).

J-D

On Thu, Apr 7, 2011 at 9:49 AM, Shahnawaz Saifi shahsa...@gmail.com wrote:
 Hi,

 While executing MR with 472G data, I am running into following error:

 java.io.IOException: Task process exit with nonzero status of 255.
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:418)




 2011-04-07 07:54:15,442 INFO org.apache.zookeeper.ClientCnxn: Opening socket
 connection to server 10.40.99.22:2181
 2011-04-07 07:54:15,442 INFO org.apache.zookeeper.ClientCnxn: Socket
 connection established to 10.40.99.22:2181, initiating session
 2011-04-07 07:54:15,443 WARN org.apache.zookeeper.ClientCnxn: Session 0x0
 for server 10.40.99.22:2181, unexpected error, closing socket connection and
 attempting reconnect
 java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcher.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:237)
    at sun.nio.ch.IOUtil.read(IOUtil.java:204)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
    at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)
 2011-04-07 07:54:15,545 FATAL
 com.example.dataengine.aggregation.aggregationMR_HDFS:
 *
 2011-04-07 07:54:15,545 FATAL
 com.example.dataengine.aggregation.aggregationMR_HDFS:
 org.apache.hadoop.hbase.ZooKeeperConnectionException:
 org.apache.zookeeper.KeeperException$ConnectionLossException:
 KeeperErrorCode = ConnectionLoss for /hbase
 2011-04-07 07:54:15,546 FATAL
 com.example.dataengine.aggregation.aggregationMR_HDFS:
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.init(HConnectionManager.java:292)
 org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:155)
 org.apache.hadoop.hbase.client.HTable.init(HTable.java:167)
 org.apache.hadoop.hbase.client.HTable.init(HTable.java:145)
 com.example.dataengine.aggregation.aggregationMR_HDFS$aggregationMapper.setup(aggregationMR_HDFS.java:324)
 org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)
 org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:621)
 org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
 org.apache.hadoop.mapred.Child.main(Child.java:170)

 2011-04-07 07:54:15,546 FATAL
 com.example.dataengine.aggregation.aggregationMR_HDFS:
 *


 --
 Thanks,
 Shah

Re: file is already being created by NN_Recovery

2011-04-07 Thread Stack 
The RegionServer is down for sure?  Else it sounds like an issue that
was addressed by the addition of a new short-circuit API call added to
HDFS on the hadoop-0.20-append branch.  The patches that added this
new call went into the branch quite a while ago.   They are:

 HDFS-1554. New semantics for recoverLease. (hairong)

 HDFS-1555. Disallow pipelien recovery if a file is already being
lease recovered. (hairong)

These patches are not in CDH3b*.  They are in the CDH3 release which
is due any day now.

HBase 0.90.2 makes use of the new API: See
https://issues.apache.org/jira/browse/HBASE-3285.  Attached to that
issue is a patch for CDH3b2, a patch we are running here at SU.  Shout
if you need a version of this patch for CDH3b3/4.

St.Ack


On Thu, Apr 7, 2011 at 9:35 AM, Daniel Iancu daniel.ia...@1and1.ro wrote:
 Hello everybody
 We've run into this, now popular, error on our cluster

 2011-04-07 16:28:00,654 WARN IPC Server handler 0 on 8020
 org.apache.hadoop.hdfs.StateChange - DIR* NameSystem.startFile: failed to
 create file
 /hbase/.logs/search-hadoop-eu001.v300.gmx.net,60020,1302075782687/search-hadoop-eu001.v300.gmx.net%3A60020.1302075783467
 for DFSClient_hb_m_search-namenode-eu002.v300.gmx.net:6_1302186078300 on
 client 10.1.100.32, because this file is already being created by
 NN_Recovery on 10.1.100.61

 I've read a couple of threads around it, still it seems that nobody
 pinpointed the cause of it? The only solution here remains to delete the log
 file and lose data ?

 I've seen  this error on almost any cluster we've installed so far, deleting
 logs was not concerning since all were test clusters. Now we got this on the
 production cluster, and strange, this cluster was just installed, there is
 no table and no data, no activity there. So what logs is master trying to
 create?

 We are running the latest CDH3B4 from Cloudera.

 Thanks for any hints,
 Daniel

Re: timing out for hdfs errors faster

2011-04-07 Thread Jean-Daniel Cryans 
 Another question, why would dfsclient setting for sockettimeout (for
 data reading) would be set so high by default if HBASE is expected to
 be real time?  Shouldn't it be few seconds (5?).

Not all clusters are used for real time applications, also usually
users first try to cram as much data as they can and see if it holds,
disregard their hardware, if they are swapping, or anything that might
make things slow. A lot of configurations are set to high values for
those reasons.

 2011-04-07 07:49:41,527 WARN org.apache.hadoop.hdfs.DFSClient: Failed
 to connect to /10.103.7.5:50010 for file
 /hbase/media_data/1c95bfcf0dd19800b1f44278627259ae/att/7725092577730365184
 for block 802538788372768807:java.net.SocketTimeoutException: 6
 millis timeout while waiting for channel to be ready for read. ch :
 java.nio.channels.SocketChannel[connected local=/10.101.6.8:40801
 remote=/10.103.7.5:50010

 What would be configuration setting to shorten the timeout say to 5
 seconds?  What about retries (if any).

Something is up with that Datanode as the region server isn't even
able to establish a channel to it. The retries are done with other
replicas (no need to hit the same faulty datanode twice). Looking at
the code, the timeout for reads is set with dfs.socket.timeout

J-D

Re: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = Connecti

2011-04-07 Thread Jean-Daniel Cryans 
You should be seeing more log lines related to ZooKeeper before that.

Also make sure your client connects to the zk server.

J-D

On Thu, Apr 7, 2011 at 9:11 AM, Shuja Rehman shujamug...@gmail.com wrote:
 Hi

 I am trying to read from hbase the following code.

 http://pastebin.com/wvVVUT3p

 it reads for first 4-5 times but after that it start throwing this exception

 SEVERE: null
 org.apache.hadoop.hbase.ZooKeeperConnectionException:
 org.apache.hadoop.hbase.ZooKeeperConnectionException:
 org.apache.zookeeper.KeeperException$ConnectionLossException:
 KeeperErrorCode = ConnectionLoss for /hbase
        at
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:988)
        at
 org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:301)
        at org.apache.hadoop.hbase.client.HConnectionManager$HCo


 The full details of exception are here.

 http://pastebin.com/Yj4VgvHG

 I am using CDH3b4 in pseudo enviornment.

 Any clue???
 thnx

 --
 Regards
 Shuja-ur-Rehman Baig
 http://pk.linkedin.com/in/shujamughal

Re: timing out for hdfs errors faster

2011-04-07 Thread Stack 
On Thu, Apr 7, 2011 at 7:58 AM, Jack Levin magn...@gmail.com wrote:
 Hello, I get those errors sometimes:

 2011-04-07 07:49:41,527 WARN org.apache.hadoop.hdfs.DFSClient: Failed
 to connect to /10.103.7.5:50010 for file
 /hbase/media_data/1c95bfcf0dd19800b1f44278627259ae/att/7725092577730365184
 for block 802538788372768807:java.net.SocketTimeoutException: 6
 millis timeout while waiting for channel to be ready for read. ch :
 java.nio.channels.SocketChannel[connected local=/10.101.6.8:40801
 remote=/10.103.7.5:50010

 What would be configuration setting to shorten the timeout say to 5
 seconds?  What about retries (if any).


0.90.0 added a timeout to the RPC (See HBASE-3154  'HBase RPC should
support timeout').  The default is 60 seconds.  To change the config.,
set hbase.rpc.timeout.  Retries should be going on in the upper
layers.  As to why 60 seconds, my guess is that the author and
reviewer were being conservative.  Previous there was no timeout.

St.Ack

Re: HTable.put hangs on bulk loading

2011-04-07 Thread Jean-Daniel Cryans 
There's nothing of use in the pasted logs unfortunately, and the log
didn't get attached to your mail (happens often). Consider putting on
a web server or pastebin.

Also I see you are on an older version, upgrading isn't going to fix
your issue (which is probably related to your environment or
configuration) but at least it's gonna be easier for us to support
you.

J-D

On Wed, Apr 6, 2011 at 7:10 PM, ajay.gov agovindara...@yahoo.com wrote:

 I am doing a load test for which I need to load a table with many rows.  I
 have a small java program that has a for loop and calls HTable.put.  I am
 inserting a map of 2 items into a table that has one column family. The
 limit of the for loop is currently 2. However after 15876 rows the call
 to Put hangs. I am using autoFlush on the HTable. Any ideas why this may
 happen?

 The table configuration:
 DESCRIPTION                                          ENABLED
  {NAME = 'TABLE2', FAMILIES = [{NAME = 'TABLE2_CF true
  1', BLOOMFILTER = 'NONE', REPLICATION_SCOPE = '0'
  , COMPRESSION = 'NONE', VERSIONS = '3', TTL = '2
  147483647', BLOCKSIZE = '65536', IN_MEMORY = 'fal
  se', BLOCKCACHE = 'true'}]}

 The HBase config on the client is the one in the hbase-default.xml. Some
 values:
 hbase.client.write.buffer=2097152
 hbase.client.pause=1000
 hbase.client.retries.number=10

 If i use another client I am able to put items to the table. I am also able
 to scan items from the table using the hbase shell.

 I have attached the server configuratio
 I don't see anything in the region server or master logs. I have them here.

 The master server log:
 2011-04-06 19:02:40,149 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.rootScanner scanning meta region {server:
 184.106.69.238:60020, regionname: -ROOT-,,0.70236052, startKey: }
 2011-04-06 19:02:40,152 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.rootScanner scan of 1 row(s) of meta region {server:
 184.106.69.238:60020, regionname: -ROOT-,,0.70236052, startKey: } complete
 2011-04-06 19:02:40,157 INFO org.apache.hadoop.hbase.master.ServerManager: 1
 region servers, 0 dead, average load 42.0
 2011-04-06 19:03:15,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.metaScanner scanning meta region {server:
 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey: }
 2011-04-06 19:03:15,265 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.metaScanner scan of 40 row(s) of meta region {server:
 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey: }
 complete
 2011-04-06 19:03:15,266 INFO org.apache.hadoop.hbase.master.BaseScanner: All
 1 .META. region(s) scanned


 The region server logs:
 2011-04-06 19:02:21,294 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
 Creating region TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
 2011-04-06 19:02:21,295 INFO org.apache.hadoop.hbase.regionserver.HRegion:
 Onlined TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
 sequenceid=1

 --
 View this message in context: 
 http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31338874.html
 Sent from the HBase User mailing list archive at Nabble.com.

Re: timing out for hdfs errors faster

2011-04-07 Thread Stack 
Jack:  Pardon me.  What J-D said.  You were asking about DN timeout.
Below I write about RS timeout.
St.Ack

On Thu, Apr 7, 2011 at 10:28 AM, Stack st...@duboce.net wrote:
 On Thu, Apr 7, 2011 at 7:58 AM, Jack Levin magn...@gmail.com wrote:
 Hello, I get those errors sometimes:

 2011-04-07 07:49:41,527 WARN org.apache.hadoop.hdfs.DFSClient: Failed
 to connect to /10.103.7.5:50010 for file
 /hbase/media_data/1c95bfcf0dd19800b1f44278627259ae/att/7725092577730365184
 for block 802538788372768807:java.net.SocketTimeoutException: 6
 millis timeout while waiting for channel to be ready for read. ch :
 java.nio.channels.SocketChannel[connected local=/10.101.6.8:40801
 remote=/10.103.7.5:50010

 What would be configuration setting to shorten the timeout say to 5
 seconds?  What about retries (if any).


 0.90.0 added a timeout to the RPC (See HBASE-3154  'HBase RPC should
 support timeout').  The default is 60 seconds.  To change the config.,
 set hbase.rpc.timeout.  Retries should be going on in the upper
 layers.  As to why 60 seconds, my guess is that the author and
 reviewer were being conservative.  Previous there was no timeout.

 St.Ack

Re: timing out for hdfs errors faster

2011-04-07 Thread Jack Levin 
Thanks, How about setting hbase-site.xml with

dfs.datanode.socket.write.timeout
dfs.datanode.socket.read.write.timeout

If tcp connection is established, but harddrive fails right after
that, I do not want to wait 60 seconds to read, I want to quicky
timeout and move to next datanode.

-Jack



On Thu, Apr 7, 2011 at 10:14 AM, Jean-Daniel Cryans jdcry...@apache.org wrote:
 Another question, why would dfsclient setting for sockettimeout (for
 data reading) would be set so high by default if HBASE is expected to
 be real time?  Shouldn't it be few seconds (5?).

 Not all clusters are used for real time applications, also usually
 users first try to cram as much data as they can and see if it holds,
 disregard their hardware, if they are swapping, or anything that might
 make things slow. A lot of configurations are set to high values for
 those reasons.

 2011-04-07 07:49:41,527 WARN org.apache.hadoop.hdfs.DFSClient: Failed
 to connect to /10.103.7.5:50010 for file
 /hbase/media_data/1c95bfcf0dd19800b1f44278627259ae/att/7725092577730365184
 for block 802538788372768807:java.net.SocketTimeoutException: 6
 millis timeout while waiting for channel to be ready for read. ch :
 java.nio.channels.SocketChannel[connected local=/10.101.6.8:40801
 remote=/10.103.7.5:50010

 What would be configuration setting to shorten the timeout say to 5
 seconds?  What about retries (if any).

 Something is up with that Datanode as the region server isn't even
 able to establish a channel to it. The retries are done with other
 replicas (no need to hit the same faulty datanode twice). Looking at
 the code, the timeout for reads is set with dfs.socket.timeout

 J-D

Re: timing out for hdfs errors faster

2011-04-07 Thread Jack Levin 
I meant to say dfs.datanode.socket.read.timeout

-Jack

On Thu, Apr 7, 2011 at 10:54 AM, Jack Levin magn...@gmail.com wrote:
 Thanks, How about setting hbase-site.xml with

 dfs.datanode.socket.write.timeout
 dfs.datanode.socket.read.write.timeout

 If tcp connection is established, but harddrive fails right after
 that, I do not want to wait 60 seconds to read, I want to quicky
 timeout and move to next datanode.

 -Jack



 On Thu, Apr 7, 2011 at 10:14 AM, Jean-Daniel Cryans jdcry...@apache.org 
 wrote:
 Another question, why would dfsclient setting for sockettimeout (for
 data reading) would be set so high by default if HBASE is expected to
 be real time?  Shouldn't it be few seconds (5?).

 Not all clusters are used for real time applications, also usually
 users first try to cram as much data as they can and see if it holds,
 disregard their hardware, if they are swapping, or anything that might
 make things slow. A lot of configurations are set to high values for
 those reasons.

 2011-04-07 07:49:41,527 WARN org.apache.hadoop.hdfs.DFSClient: Failed
 to connect to /10.103.7.5:50010 for file
 /hbase/media_data/1c95bfcf0dd19800b1f44278627259ae/att/7725092577730365184
 for block 802538788372768807:java.net.SocketTimeoutException: 6
 millis timeout while waiting for channel to be ready for read. ch :
 java.nio.channels.SocketChannel[connected local=/10.101.6.8:40801
 remote=/10.103.7.5:50010

 What would be configuration setting to shorten the timeout say to 5
 seconds?  What about retries (if any).

 Something is up with that Datanode as the region server isn't even
 able to establish a channel to it. The retries are done with other
 replicas (no need to hit the same faulty datanode twice). Looking at
 the code, the timeout for reads is set with dfs.socket.timeout

 J-D

Hadoop Append Github

2011-04-07 Thread Jason Rutherglen 
Is https://github.com/facebook/hadoop-20-append the Github branch for
Hadoop Append 0.20?

Re: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = Connecti

2011-04-07 Thread Shuja Rehman 
here is more log. now it is not connecting at all.

11/04/07 23:02:55 WARN hbase.HBaseConfiguration: instantiating
HBaseConfiguration() is deprecated. Please use HBaseConfiguration#create()
to construct a plain Configuration
11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.3.2-CDH3B4--1, built on 02/21/2011 22:45 GMT
11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client environment:host.name=DV
11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
environment:java.version=1.6.0_21
11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
environment:java.vendor=Sun Microsystems Inc.
11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
environment:java.home=C:\Program Files\Java\jdk1.6.0_21\jre
11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
environment:java.class.path=D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\build\web\WEB-INF\classes;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\commons-logging-1.1.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\log4j-1.2.15.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\activation.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\webservices-api.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\webservices-extra-api.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\webservices-extra.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\webservices-rt.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\webservices-tools.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restapi\jsr311-api-1.1.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\asm-3.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jackson-core-asl-1.1.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jersey-client-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jersey-core-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jersey-json-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jersey-server-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jersey-spring-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jettison-1.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\oauth-client-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\oauth-signature-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\activation.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jaxb-api.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jsr173_api.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\hadoop-0.20.2-CDH3B4-core.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\hbase-0.90.1-CDH3B4.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\mysql-connector-java-5.1.7-bin.jar;D:\FreeLauncing\DataWareHouse\UIDataTransporter\dist\UIDataTransporter.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\zookeeper-3.3.2-CDH3B4.jar;C:\Program
Files\Apache Software Foundation\Apache Tomcat
6.0.26\lib\annotations-api.jar;C:\Program Files\Apache Software
Foundation\Apache Tomcat 6.0.26\lib\catalina-ant.jar;C:\Program Files\Apache
Software Foundation\Apache Tomcat 6.0.26\lib\catalina-ha.jar;C:\Program
Files\Apache Software Foundation\Apache Tomcat
6.0.26\lib\catalina-tribes.jar;C:\Program Files\Apache Software
Foundation\Apache Tomcat 6.0.26\lib\catalina.jar;C:\Program Files\Apache
Software Foundation\Apache Tomcat 6.0.26\lib\el-api.jar;C:\Program
Files\Apache Software Foundation\Apache Tomcat
6.0.26\lib\jasper-el.jar;C:\Program Files\Apache Software Foundation\Apache
Tomcat 6.0.26\lib\jasper.jar;C:\Program Files\Apache Software
Foundation\Apache Tomcat 6.0.26\lib\jsp-api.jar;C:\Program Files\Apache
Software Foundation\Apache Tomcat 6.0.26\lib\servlet-api.jar;C:\Program
Files\Apache Software Foundation\Apache Tomcat
6.0.26\lib\tomcat-coyote.jar;C:\Program Files\Apache Software
Foundation\Apache Tomcat 6.0.26\lib\tomcat-dbcp.jar;C:\Program Files\Apache
Software Foundation\Apache Tomcat 6.0.26\lib\tomcat-i18n-es.jar;C:\Program
Files\Apache Software Foundation\Apache Tomcat
6.0.26\lib\tomcat-i18n-fr.jar;C:\Program Files\Apache Software
Foundation\Apache Tomcat 6.0.26\lib\tomcat-i18n-ja.jar;C:\Program
Files\Apache Software Foundation\Apache Tomcat 6.0.26\bin\tomcat-juli.jar
11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
environment:java.library.path=C:\Program
Files\Java\jdk1.6.0_21\jre\bin;.;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\Program
Files\Common Files\Microsoft Shared\Windows Live;C:\Program Files\PC
Connectivity

Re: Hadoop Append Github

2011-04-07 Thread Jean-Daniel Cryans 
That's the one published by Facebook, the one maintained by Apache is
https://github.com/apache/hadoop-common/tree/branch-0.20-append

J-D

On Thu, Apr 7, 2011 at 11:04 AM, Jason Rutherglen
jason.rutherg...@gmail.com wrote:
 Is https://github.com/facebook/hadoop-20-append the Github branch for
 Hadoop Append 0.20?

Re: Hadoop Append Github

2011-04-07 Thread Stack 
That one looks dead Jason.  There was a bulk upload in December and
nought since.
St.Ack


On Thu, Apr 7, 2011 at 11:04 AM, Jason Rutherglen
jason.rutherg...@gmail.com wrote:
 Is https://github.com/facebook/hadoop-20-append the Github branch for
 Hadoop Append 0.20?

Re: Hadoop Append Github

2011-04-07 Thread Jason Rutherglen 
Ah ok, Google turned up the one I posted, I wonder why this one was
harder to find?

Thanks!

On Thu, Apr 7, 2011 at 11:07 AM, Jean-Daniel Cryans jdcry...@apache.org wrote:
 That's the one published by Facebook, the one maintained by Apache is
 https://github.com/apache/hadoop-common/tree/branch-0.20-append

 J-D

 On Thu, Apr 7, 2011 at 11:04 AM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is https://github.com/facebook/hadoop-20-append the Github branch for
 Hadoop Append 0.20?

Re: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = Connecti

2011-04-07 Thread Jean-Daniel Cryans 
If you look at 204.13.166.85's zookeeper log, do you see anything that
looks bad around the time you ran this?

J-D

On Thu, Apr 7, 2011 at 11:04 AM, Shuja Rehman shujamug...@gmail.com wrote:
 here is more log. now it is not connecting at all.

 11/04/07 23:02:55 WARN hbase.HBaseConfiguration: instantiating
 HBaseConfiguration() is deprecated. Please use HBaseConfiguration#create()
 to construct a plain Configuration
 11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
 environment:zookeeper.version=3.3.2-CDH3B4--1, built on 02/21/2011 22:45 GMT
 11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client environment:host.name=DV
 11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
 environment:java.version=1.6.0_21
 11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
 environment:java.vendor=Sun Microsystems Inc.
 11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
 environment:java.home=C:\Program Files\Java\jdk1.6.0_21\jre
 11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
 environment:java.class.path=D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\build\web\WEB-INF\classes;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\commons-logging-1.1.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\log4j-1.2.15.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\activation.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\webservices-api.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\webservices-extra-api.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\webservices-extra.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\webservices-rt.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\metro\webservices-tools.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restapi\jsr311-api-1.1.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\asm-3.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jackson-core-asl-1.1.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jersey-client-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jersey-core-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jersey-json-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jersey-server-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jersey-spring-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jettison-1.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\oauth-client-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\oauth-signature-1.1.5.1.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\activation.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jaxb-api.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\restlib\jsr173_api.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\hadoop-0.20.2-CDH3B4-core.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\hbase-0.90.1-CDH3B4.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\mysql-connector-java-5.1.7-bin.jar;D:\FreeLauncing\DataWareHouse\UIDataTransporter\dist\UIDataTransporter.jar;D:\FreeLauncing\DataWareHouse\WebServices\WebServiceSvn\lib\CopyLibs\zookeeper-3.3.2-CDH3B4.jar;C:\Program
 Files\Apache Software Foundation\Apache Tomcat
 6.0.26\lib\annotations-api.jar;C:\Program Files\Apache Software
 Foundation\Apache Tomcat 6.0.26\lib\catalina-ant.jar;C:\Program Files\Apache
 Software Foundation\Apache Tomcat 6.0.26\lib\catalina-ha.jar;C:\Program
 Files\Apache Software Foundation\Apache Tomcat
 6.0.26\lib\catalina-tribes.jar;C:\Program Files\Apache Software
 Foundation\Apache Tomcat 6.0.26\lib\catalina.jar;C:\Program Files\Apache
 Software Foundation\Apache Tomcat 6.0.26\lib\el-api.jar;C:\Program
 Files\Apache Software Foundation\Apache Tomcat
 6.0.26\lib\jasper-el.jar;C:\Program Files\Apache Software Foundation\Apache
 Tomcat 6.0.26\lib\jasper.jar;C:\Program Files\Apache Software
 Foundation\Apache Tomcat 6.0.26\lib\jsp-api.jar;C:\Program Files\Apache
 Software Foundation\Apache Tomcat 6.0.26\lib\servlet-api.jar;C:\Program
 Files\Apache Software Foundation\Apache Tomcat
 6.0.26\lib\tomcat-coyote.jar;C:\Program Files\Apache Software
 Foundation\Apache Tomcat 6.0.26\lib\tomcat-dbcp.jar;C:\Program Files\Apache
 Software Foundation\Apache Tomcat 6.0.26\lib\tomcat-i18n-es.jar;C:\Program
 Files\Apache Software Foundation\Apache Tomcat
 6.0.26\lib\tomcat-i18n-fr.jar;C:\Program Files\Apache Software
 Foundation\Apache Tomcat 6.0.26\lib\tomcat-i18n-ja.jar;C:\Program
 Files\Apache Software Foundation\Apache Tomcat 6.0.26\bin\tomcat-juli.jar
 11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
 environment:java.library.path=C:\Program

Re: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = Connecti

2011-04-07 Thread Shuja Rehman 
i got the log file and it says that

2011-04-07 11:17:41,864 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
/182.178.254.222 - max is 10
2011-04-07 11:17:45,453 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
/182.178.254.222 - max is 10
2011-04-07 11:17:47,470 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
/182.178.254.222 - max is 10
2011-04-07 11:17:49,964 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
/182.178.254.222 - max is 10
2011-04-07 11:17:52,101 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
/182.178.254.222 - max is 10
2011-04-07 11:17:54,211 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
/182.178.254.222 - max is 10
2011-04-07 11:17:54,251 - WARN  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639] - Exception causing close of session
0x12f30d63bc90013 due to java.io.IOException: Connection reset by peer
2011-04-07 11:17:54,251 - INFO  [NIOServerCxn.Factory:
0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for
client /182.178.254.222:27546 which had sessionid 0x12f30d63bc90013
2011-04-07 11:18:24,004 - INFO  [SessionTracker:ZooKeeperServer@314] -
Expiring session 0x12f30d63bc90013, timeout of 4ms exceeded
2011-04-07 11:18:24,005 - INFO  [ProcessThread:-1:PrepRequestProcessor@387]
- Processed session termination for sessionid: 0x12f30d63bc90013


so how to close the connection? i am also using now htable.close() in the
end of funtion.

On Thu, Apr 7, 2011 at 11:21 PM, Shuja Rehman shujamug...@gmail.com wrote:

 where zookeeper keeps its logs? i check the /usr/lib/zookeeper and there is
 no log file inside. also not in bin folder.




 On Thu, Apr 7, 2011 at 11:17 PM, Jean-Daniel Cryans 
 jdcry...@apache.orgwrote:

 If you look at 204.13.166.85's zookeeper log, do you see anything that
 looks bad around the time you ran this?

 J-D

 On Thu, Apr 7, 2011 at 11:04 AM, Shuja Rehman shujamug...@gmail.com
 wrote:
  here is more log. now it is not connecting at all.
 
  11/04/07 23:02:55 WARN hbase.HBaseConfiguration: instantiating
  HBaseConfiguration() is deprecated. Please use
 HBaseConfiguration#create()
  to construct a plain Configuration
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:zookeeper.version=3.3.2-CDH3B4--1, built on 02/21/2011 22:45
 GMT
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client environment:
 host.name=DV
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:java.version=1.6.0_21
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:java.vendor=Sun Microsystems Inc.
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:java.home=C:\Program Files\Java\jdk1.6.0_21\jre
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client

Re: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = Connecti

2011-04-07 Thread Jean-Daniel Cryans 
So regarding finding your logs and other stuff related to that, since
you are using CDH you should always check their documentation.

In ZooKeeper there's a configurable limit of 30 connections per IP.
HTable.close won't close the connection since you can have multiple
HTables using the same cluster. Always make sure you reuse the same
configuration object and always reuse the HTables, don't create new
ones per session or something like that.

This is all described in HTable's class javadoc:
http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html

J-D

On Thu, Apr 7, 2011 at 11:35 AM, Shuja Rehman shujamug...@gmail.com wrote:
 i got the log file and it says that

 2011-04-07 11:17:41,864 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:45,453 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:47,470 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:49,964 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:52,101 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:54,211 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:54,251 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639] - Exception causing close of session
 0x12f30d63bc90013 due to java.io.IOException: Connection reset by peer
 2011-04-07 11:17:54,251 - INFO  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for
 client /182.178.254.222:27546 which had sessionid 0x12f30d63bc90013
 2011-04-07 11:18:24,004 - INFO  [SessionTracker:ZooKeeperServer@314] -
 Expiring session 0x12f30d63bc90013, timeout of 4ms exceeded
 2011-04-07 11:18:24,005 - INFO  [ProcessThread:-1:PrepRequestProcessor@387]
 - Processed session termination for sessionid: 0x12f30d63bc90013


 so how to close the connection? i am also using now htable.close() in the
 end of funtion.

 On Thu, Apr 7, 2011 at 11:21 PM, Shuja Rehman shujamug...@gmail.com wrote:

 where zookeeper keeps its logs? i check the /usr/lib/zookeeper and there is
 no log file inside. also not in bin folder.




 On Thu, Apr 7, 2011 at 11:17 PM, Jean-Daniel Cryans 
 jdcry...@apache.orgwrote:

 If you look at 204.13.166.85's zookeeper log, do you see anything that
 looks bad around the time you ran this?

 J-D

 On Thu, Apr 7, 2011 at 11:04 AM, Shuja Rehman shujamug...@gmail.com
 wrote:
  here is more log. now it is not connecting at all.
 
  11/04/07 23:02:55 WARN hbase.HBaseConfiguration: instantiating
  HBaseConfiguration() is deprecated. Please use
 HBaseConfiguration#create()
  to construct a plain Configuration
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:zookeeper.version=3.3.2-CDH3B4--1, built on 02/21/2011 22:45
 GMT
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client environment:
 host.name=DV
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:java.version=1.6.0_21
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:java.vendor=Sun Microsystems Inc.
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:java.home=C:\Program Files\Java\jdk1.6.0_21\jre
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client

Re: HTable.put hangs on bulk loading

2011-04-07 Thread ajay.gov 

Sorry, my server config was not attached. Its here:
http://pastebin.com/U41QZGiq

thanks
-ajay



ajay.gov wrote:
 
 I am doing a load test for which I need to load a table with many rows.  I
 have a small java program that has a for loop and calls HTable.put.  I am
 inserting a map of 2 items into a table that has one column family. The
 limit of the for loop is currently 2. However after 15876 rows the
 call to Put hangs. I am using autoFlush on the HTable. Any ideas why this
 may happen? 
 
 The table configuration:
 DESCRIPTION  ENABLED  
   
  {NAME = 'TABLE2', FAMILIES = [{NAME = 'TABLE2_CF true 
   
  1', BLOOMFILTER = 'NONE', REPLICATION_SCOPE = '0'  
   
  , COMPRESSION = 'NONE', VERSIONS = '3', TTL = '2  
   
  147483647', BLOCKSIZE = '65536', IN_MEMORY = 'fal  
   
  se', BLOCKCACHE = 'true'}]}
 
 The HBase config on the client is the one in the hbase-default.xml. Some
 values:
 hbase.client.write.buffer=2097152
 hbase.client.pause=1000
 hbase.client.retries.number=10
 
 If i use another client I am able to put items to the table. I am also
 able to scan items from the table using the hbase shell.
 
 I have attached the server configuratio
 I don't see anything in the region server or master logs. I have them
 here.
 
 The master server log:
 2011-04-06 19:02:40,149 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.rootScanner scanning meta region {server:
 184.106.69.238:60020, regionname: -ROOT-,,0.70236052, startKey: }
 2011-04-06 19:02:40,152 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.rootScanner scan of 1 row(s) of meta region {server:
 184.106.69.238:60020, regionname: -ROOT-,,0.70236052, startKey: }
 complete
 2011-04-06 19:02:40,157 INFO org.apache.hadoop.hbase.master.ServerManager:
 1 region servers, 0 dead, average load 42.0
 2011-04-06 19:03:15,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.metaScanner scanning meta region {server:
 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey: }
 2011-04-06 19:03:15,265 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.metaScanner scan of 40 row(s) of meta region {server:
 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey: }
 complete
 2011-04-06 19:03:15,266 INFO org.apache.hadoop.hbase.master.BaseScanner:
 All 1 .META. region(s) scanned
 
 
 The region server logs:
 2011-04-06 19:02:21,294 DEBUG
 org.apache.hadoop.hbase.regionserver.HRegion: Creating region
 TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
 2011-04-06 19:02:21,295 INFO org.apache.hadoop.hbase.regionserver.HRegion:
 Onlined TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
 sequenceid=1
 
 

-- 
View this message in context: 
http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31345580.html
Sent from the HBase User mailing list archive at Nabble.com.

Re: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.hadoop.hbase.ZooKeeperConnectionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = Connecti

2011-04-07 Thread Jean-Daniel Cryans 
To help usability, I created https://issues.apache.org/jira/browse/HBASE-3755

J-D

On Thu, Apr 7, 2011 at 11:39 AM, Jean-Daniel Cryans jdcry...@apache.org wrote:
 So regarding finding your logs and other stuff related to that, since
 you are using CDH you should always check their documentation.

 In ZooKeeper there's a configurable limit of 30 connections per IP.
 HTable.close won't close the connection since you can have multiple
 HTables using the same cluster. Always make sure you reuse the same
 configuration object and always reuse the HTables, don't create new
 ones per session or something like that.

 This is all described in HTable's class javadoc:
 http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTable.html

 J-D

 On Thu, Apr 7, 2011 at 11:35 AM, Shuja Rehman shujamug...@gmail.com wrote:
 i got the log file and it says that

 2011-04-07 11:17:41,864 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:45,453 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:47,470 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:49,964 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:52,101 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:54,211 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn$Factory@247] - Too many connections from
 /182.178.254.222 - max is 10
 2011-04-07 11:17:54,251 - WARN  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@639] - Exception causing close of session
 0x12f30d63bc90013 due to java.io.IOException: Connection reset by peer
 2011-04-07 11:17:54,251 - INFO  [NIOServerCxn.Factory:
 0.0.0.0/0.0.0.0:2181:NIOServerCnxn@1435] - Closed socket connection for
 client /182.178.254.222:27546 which had sessionid 0x12f30d63bc90013
 2011-04-07 11:18:24,004 - INFO  [SessionTracker:ZooKeeperServer@314] -
 Expiring session 0x12f30d63bc90013, timeout of 4ms exceeded
 2011-04-07 11:18:24,005 - INFO  [ProcessThread:-1:PrepRequestProcessor@387]
 - Processed session termination for sessionid: 0x12f30d63bc90013


 so how to close the connection? i am also using now htable.close() in the
 end of funtion.

 On Thu, Apr 7, 2011 at 11:21 PM, Shuja Rehman shujamug...@gmail.com wrote:

 where zookeeper keeps its logs? i check the /usr/lib/zookeeper and there is
 no log file inside. also not in bin folder.




 On Thu, Apr 7, 2011 at 11:17 PM, Jean-Daniel Cryans 
 jdcry...@apache.orgwrote:

 If you look at 204.13.166.85's zookeeper log, do you see anything that
 looks bad around the time you ran this?

 J-D

 On Thu, Apr 7, 2011 at 11:04 AM, Shuja Rehman shujamug...@gmail.com
 wrote:
  here is more log. now it is not connecting at all.
 
  11/04/07 23:02:55 WARN hbase.HBaseConfiguration: instantiating
  HBaseConfiguration() is deprecated. Please use
 HBaseConfiguration#create()
  to construct a plain Configuration
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:zookeeper.version=3.3.2-CDH3B4--1, built on 02/21/2011 22:45
 GMT
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client environment:
 host.name=DV
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:java.version=1.6.0_21
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:java.vendor=Sun Microsystems Inc.
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client
  environment:java.home=C:\Program Files\Java\jdk1.6.0_21\jre
  11/04/07 23:02:55 INFO zookeeper.ZooKeeper: Client

Re: Hadoop Append Github

2011-04-07 Thread Jason Rutherglen 
Is http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
different than the Github one at
https://github.com/apache/hadoop-common/tree/branch-0.20-append ?

As I can apply the HDFS-347 patch successfully to the SVN version,
however the Github one has a number of rejects.  Are they different or
is there some uniqueness involved with Github, or something simple I'm
missing.

On Thu, Apr 7, 2011 at 11:10 AM, Stack st...@duboce.net wrote:
 That one looks dead Jason.  There was a bulk upload in December and
 nought since.
 St.Ack


 On Thu, Apr 7, 2011 at 11:04 AM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is https://github.com/facebook/hadoop-20-append the Github branch for
 Hadoop Append 0.20?

Re: Hadoop Append Github

2011-04-07 Thread Jean-Daniel Cryans 
As far as I can tell, they are at the same revision.

J-D

On Thu, Apr 7, 2011 at 1:19 PM, Jason Rutherglen
jason.rutherg...@gmail.com wrote:
 Is http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
 different than the Github one at
 https://github.com/apache/hadoop-common/tree/branch-0.20-append ?

 As I can apply the HDFS-347 patch successfully to the SVN version,
 however the Github one has a number of rejects.  Are they different or
 is there some uniqueness involved with Github, or something simple I'm
 missing.

 On Thu, Apr 7, 2011 at 11:10 AM, Stack st...@duboce.net wrote:
 That one looks dead Jason.  There was a bulk upload in December and
 nought since.
 St.Ack


 On Thu, Apr 7, 2011 at 11:04 AM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is https://github.com/facebook/hadoop-20-append the Github branch for
 Hadoop Append 0.20?

Re: Hadoop Append Github

2011-04-07 Thread Jason Rutherglen 
How did you compare?

On Thu, Apr 7, 2011 at 1:37 PM, Jean-Daniel Cryans jdcry...@apache.org wrote:
 As far as I can tell, they are at the same revision.

 J-D

 On Thu, Apr 7, 2011 at 1:19 PM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
 different than the Github one at
 https://github.com/apache/hadoop-common/tree/branch-0.20-append ?

 As I can apply the HDFS-347 patch successfully to the SVN version,
 however the Github one has a number of rejects.  Are they different or
 is there some uniqueness involved with Github, or something simple I'm
 missing.

 On Thu, Apr 7, 2011 at 11:10 AM, Stack st...@duboce.net wrote:
 That one looks dead Jason.  There was a bulk upload in December and
 nought since.
 St.Ack


 On Thu, Apr 7, 2011 at 11:04 AM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is https://github.com/facebook/hadoop-20-append the Github branch for
 Hadoop Append 0.20?

Re: Hadoop Append Github

2011-04-07 Thread Jason Rutherglen 
It looks like they may [somehow] be different?  The latest change to
SVN happened 2011-01-10 whereas the Github one was changed Wed Apr 6
19:03:37 2011?

Here's from Github:

commit 53d6ff79e8c4ee850cf4e592ddd20b8e116a8513
Author: Konstantin Shvachko s...@apache.org
Date:   Wed Apr 6 19:03:37 2011 +
Committing HADOOP-6949 to branch 0.22.
git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1089562
13f79535-47bb-0310-9956-ffa450edef68

from http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append

Path: .
URL: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
Repository Root: http://svn.apache.org/repos/asf
Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
Revision: 1089967
Node Kind: directory
Schedule: normal
Last Changed Author: hairong
Last Changed Rev: 1057313
Last Changed Date: 2011-01-10 11:01:36 -0800 (Mon, 10 Jan 2011)



On Thu, Apr 7, 2011 at 2:05 PM, Jason Rutherglen
jason.rutherg...@gmail.com wrote:
 How did you compare?

 On Thu, Apr 7, 2011 at 1:37 PM, Jean-Daniel Cryans jdcry...@apache.org 
 wrote:
 As far as I can tell, they are at the same revision.

 J-D

 On Thu, Apr 7, 2011 at 1:19 PM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
 different than the Github one at
 https://github.com/apache/hadoop-common/tree/branch-0.20-append ?

 As I can apply the HDFS-347 patch successfully to the SVN version,
 however the Github one has a number of rejects.  Are they different or
 is there some uniqueness involved with Github, or something simple I'm
 missing.

 On Thu, Apr 7, 2011 at 11:10 AM, Stack st...@duboce.net wrote:
 That one looks dead Jason.  There was a bulk upload in December and
 nought since.
 St.Ack


 On Thu, Apr 7, 2011 at 11:04 AM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is https://github.com/facebook/hadoop-20-append the Github branch for
 Hadoop Append 0.20?

Re: Hadoop Append Github

2011-04-07 Thread Jean-Daniel Cryans 
That last change on github was for trunk, not the append branch. The
last one I see in that branch is:

HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.
Hairong Kuang (author)
January 10, 2011

Same as in SVN.

J-D

On Thu, Apr 7, 2011 at 2:09 PM, Jason Rutherglen
jason.rutherg...@gmail.com wrote:
 It looks like they may [somehow] be different?  The latest change to
 SVN happened 2011-01-10 whereas the Github one was changed Wed Apr 6
 19:03:37 2011?

 Here's from Github:

 commit 53d6ff79e8c4ee850cf4e592ddd20b8e116a8513
 Author: Konstantin Shvachko s...@apache.org
 Date:   Wed Apr 6 19:03:37 2011 +
 Committing HADOOP-6949 to branch 0.22.
 git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1089562
 13f79535-47bb-0310-9956-ffa450edef68

 from http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append

 Path: .
 URL: http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
 Repository Root: http://svn.apache.org/repos/asf
 Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
 Revision: 1089967
 Node Kind: directory
 Schedule: normal
 Last Changed Author: hairong
 Last Changed Rev: 1057313
 Last Changed Date: 2011-01-10 11:01:36 -0800 (Mon, 10 Jan 2011)



 On Thu, Apr 7, 2011 at 2:05 PM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 How did you compare?

 On Thu, Apr 7, 2011 at 1:37 PM, Jean-Daniel Cryans jdcry...@apache.org 
 wrote:
 As far as I can tell, they are at the same revision.

 J-D

 On Thu, Apr 7, 2011 at 1:19 PM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is 
 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
 different than the Github one at
 https://github.com/apache/hadoop-common/tree/branch-0.20-append ?

 As I can apply the HDFS-347 patch successfully to the SVN version,
 however the Github one has a number of rejects.  Are they different or
 is there some uniqueness involved with Github, or something simple I'm
 missing.

 On Thu, Apr 7, 2011 at 11:10 AM, Stack st...@duboce.net wrote:
 That one looks dead Jason.  There was a bulk upload in December and
 nought since.
 St.Ack


 On Thu, Apr 7, 2011 at 11:04 AM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is https://github.com/facebook/hadoop-20-append the Github branch for
 Hadoop Append 0.20?

Re: Hadoop Append Github

2011-04-07 Thread Jason Rutherglen 
How using Github were you able to see only the log for the given branch/URL?

I'm not sure why the patch won't apply.  I ran diff, and there are
differences, though they're mostly in the scripts and other non-source
code files.

On Thu, Apr 7, 2011 at 2:20 PM, Jean-Daniel Cryans jdcry...@apache.org wrote:
 That last change on github was for trunk, not the append branch. The
 last one I see in that branch is:

 HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.
 Hairong Kuang (author)
 January 10, 2011

 Same as in SVN.

 J-D

 On Thu, Apr 7, 2011 at 2:09 PM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 It looks like they may [somehow] be different?  The latest change to
 SVN happened 2011-01-10 whereas the Github one was changed Wed Apr 6
 19:03:37 2011?

 Here's from Github:

 commit 53d6ff79e8c4ee850cf4e592ddd20b8e116a8513
 Author: Konstantin Shvachko s...@apache.org
 Date:   Wed Apr 6 19:03:37 2011 +
 Committing HADOOP-6949 to branch 0.22.
 git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1089562
 13f79535-47bb-0310-9956-ffa450edef68

 from 
 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append

 Path: .
 URL: 
 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
 Repository Root: http://svn.apache.org/repos/asf
 Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
 Revision: 1089967
 Node Kind: directory
 Schedule: normal
 Last Changed Author: hairong
 Last Changed Rev: 1057313
 Last Changed Date: 2011-01-10 11:01:36 -0800 (Mon, 10 Jan 2011)



 On Thu, Apr 7, 2011 at 2:05 PM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 How did you compare?

 On Thu, Apr 7, 2011 at 1:37 PM, Jean-Daniel Cryans jdcry...@apache.org 
 wrote:
 As far as I can tell, they are at the same revision.

 J-D

 On Thu, Apr 7, 2011 at 1:19 PM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is 
 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
 different than the Github one at
 https://github.com/apache/hadoop-common/tree/branch-0.20-append ?

 As I can apply the HDFS-347 patch successfully to the SVN version,
 however the Github one has a number of rejects.  Are they different or
 is there some uniqueness involved with Github, or something simple I'm
 missing.

 On Thu, Apr 7, 2011 at 11:10 AM, Stack st...@duboce.net wrote:
 That one looks dead Jason.  There was a bulk upload in December and
 nought since.
 St.Ack


 On Thu, Apr 7, 2011 at 11:04 AM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is https://github.com/facebook/hadoop-20-append the Github branch for
 Hadoop Append 0.20?

Re: Hadoop Append Github

2011-04-07 Thread Jean-Daniel Cryans 
So from that page:

https://github.com/apache/hadoop-common

Switch to the append branch then click on the right on the history button.

Make sure you switched to the append branch in your cloned git repo
too, by default you are on trunk.

J-D

On Thu, Apr 7, 2011 at 4:08 PM, Jason Rutherglen
jason.rutherg...@gmail.com wrote:
 How using Github were you able to see only the log for the given branch/URL?

 I'm not sure why the patch won't apply.  I ran diff, and there are
 differences, though they're mostly in the scripts and other non-source
 code files.

 On Thu, Apr 7, 2011 at 2:20 PM, Jean-Daniel Cryans jdcry...@apache.org 
 wrote:
 That last change on github was for trunk, not the append branch. The
 last one I see in that branch is:

 HDFS-1554. New semantics for recoverLease. Contributed by Hairong Kuang.
 Hairong Kuang (author)
 January 10, 2011

 Same as in SVN.

 J-D

 On Thu, Apr 7, 2011 at 2:09 PM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 It looks like they may [somehow] be different?  The latest change to
 SVN happened 2011-01-10 whereas the Github one was changed Wed Apr 6
 19:03:37 2011?

 Here's from Github:

 commit 53d6ff79e8c4ee850cf4e592ddd20b8e116a8513
 Author: Konstantin Shvachko s...@apache.org
 Date:   Wed Apr 6 19:03:37 2011 +
 Committing HADOOP-6949 to branch 0.22.
 git-svn-id: https://svn.apache.org/repos/asf/hadoop/common/trunk@1089562
 13f79535-47bb-0310-9956-ffa450edef68

 from 
 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append

 Path: .
 URL: 
 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
 Repository Root: http://svn.apache.org/repos/asf
 Repository UUID: 13f79535-47bb-0310-9956-ffa450edef68
 Revision: 1089967
 Node Kind: directory
 Schedule: normal
 Last Changed Author: hairong
 Last Changed Rev: 1057313
 Last Changed Date: 2011-01-10 11:01:36 -0800 (Mon, 10 Jan 2011)



 On Thu, Apr 7, 2011 at 2:05 PM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 How did you compare?

 On Thu, Apr 7, 2011 at 1:37 PM, Jean-Daniel Cryans jdcry...@apache.org 
 wrote:
 As far as I can tell, they are at the same revision.

 J-D

 On Thu, Apr 7, 2011 at 1:19 PM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is 
 http://svn.apache.org/repos/asf/hadoop/common/branches/branch-0.20-append
 different than the Github one at
 https://github.com/apache/hadoop-common/tree/branch-0.20-append ?

 As I can apply the HDFS-347 patch successfully to the SVN version,
 however the Github one has a number of rejects.  Are they different or
 is there some uniqueness involved with Github, or something simple I'm
 missing.

 On Thu, Apr 7, 2011 at 11:10 AM, Stack st...@duboce.net wrote:
 That one looks dead Jason.  There was a bulk upload in December and
 nought since.
 St.Ack


 On Thu, Apr 7, 2011 at 11:04 AM, Jason Rutherglen
 jason.rutherg...@gmail.com wrote:
 Is https://github.com/facebook/hadoop-20-append the Github branch for
 Hadoop Append 0.20?

Re: HTable.put hangs on bulk loading

2011-04-07 Thread Ajay Govindarajan 
Thanks for pointing this out. I have uploaded the server config at:
http://pastebin.com/U41QZGiq

thanks
-ajay








From: Jean-Daniel Cryans jdcry...@apache.org
To: user@hbase.apache.org
Sent: Thursday, April 7, 2011 10:29 AM
Subject: Re: HTable.put hangs on bulk loading

There's nothing of use in the pasted logs unfortunately, and the log
didn't get attached to your mail (happens often). Consider putting on
a web server or pastebin.

Also I see you are on an older version, upgrading isn't going to fix
your issue (which is probably related to your environment or
configuration) but at least it's gonna be easier for us to support
you.

J-D

On Wed, Apr 6, 2011 at 7:10 PM, ajay.gov agovindara...@yahoo.com wrote:

 I am doing a load test for which I need to load a table with many rows.  I
 have a small java program that has a for loop and calls HTable.put.  I am
 inserting a map of 2 items into a table that has one column family. The
 limit of the for loop is currently 2. However after 15876 rows the call
 to Put hangs. I am using autoFlush on the HTable. Any ideas why this may
 happen?

 The table configuration:
 DESCRIPTION                                          ENABLED
  {NAME = 'TABLE2', FAMILIES = [{NAME = 'TABLE2_CF true
  1', BLOOMFILTER = 'NONE', REPLICATION_SCOPE = '0'
  , COMPRESSION = 'NONE', VERSIONS = '3', TTL = '2
  147483647', BLOCKSIZE = '65536', IN_MEMORY = 'fal
  se', BLOCKCACHE = 'true'}]}

 The HBase config on the client is the one in the hbase-default.xml. Some
 values:
 hbase.client.write.buffer=2097152
 hbase.client.pause=1000
 hbase.client.retries.number=10

 If i use another client I am able to put items to the table. I am also able
 to scan items from the table using the hbase shell.

 I have attached the server configuratio
 I don't see anything in the region server or master logs. I have them here.

 The master server log:
 2011-04-06 19:02:40,149 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.rootScanner scanning meta region {server:
 184.106.69.238:60020, regionname: -ROOT-,,0.70236052, startKey: }
 2011-04-06 19:02:40,152 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.rootScanner scan of 1 row(s) of meta region {server:
 184.106.69.238:60020, regionname: -ROOT-,,0.70236052, startKey: } complete
 2011-04-06 19:02:40,157 INFO org.apache.hadoop.hbase.master.ServerManager: 1
 region servers, 0 dead, average load 42.0
 2011-04-06 19:03:15,252 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.metaScanner scanning meta region {server:
 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey: }
 2011-04-06 19:03:15,265 INFO org.apache.hadoop.hbase.master.BaseScanner:
 RegionManager.metaScanner scan of 40 row(s) of meta region {server:
 184.106.69.238:60020, regionname: .META.,,1.1028785192, startKey: }
 complete
 2011-04-06 19:03:15,266 INFO org.apache.hadoop.hbase.master.BaseScanner: All
 1 .META. region(s) scanned


 The region server logs:
 2011-04-06 19:02:21,294 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
 Creating region TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.
 2011-04-06 19:02:21,295 INFO org.apache.hadoop.hbase.regionserver.HRegion:
 Onlined TABLE2,,1302141740486.010a5ae704ed53f656cbddb8e489162a.; next
 sequenceid=1

 --
 View this message in context: 
 http://old.nabble.com/HTable.put-hangs-on-bulk-loading-tp31338874p31338874.html
 Sent from the HBase User mailing list archive at Nabble.com.

zookeper warning with 0.90.1 hbase

2011-04-07 Thread Venkatesh 

 I see lot of these warnings..everything seems to be working otherwise..Is this 
something that can be ignored?


 2011-04-07 21:29:15,032 WARN Timer-0-SendThread(..:2181) 
org.apache.zookeeper.ClientCnxn - Session 0x0 for server :2181, unexpected 
error, closing socket connection and attempting reconnect
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcher.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
at sun.nio.ch.IOUtil.read(IOUtil.java:200)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)


2011-04-07 21:29:15,032 DEBUG Timer-0-SendThread(..:2181) 
org.apache.zookeeper.ClientCnxn - Ignoring exception during shutdown input
java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at 
sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
at 
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1205)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:11

Re: zookeper warning with 0.90.1 hbase

2011-04-07 Thread Stack 
They happen on the end of a map task or on shutdown?  If so, yes,
ignore (or, if you want to have nice clean shutdown, figure how
Session 0x0 was set up -- was it you -- and call appropriate close in
time).

St.Ack

On Thu, Apr 7, 2011 at 6:33 PM, Venkatesh vramanatha...@aol.com wrote:

  I see lot of these warnings..everything seems to be working otherwise..Is 
 this something that can be ignored?


  2011-04-07 21:29:15,032 WARN Timer-0-SendThread(..:2181) 
 org.apache.zookeeper.ClientCnxn - Session 0x0 for server :2181, 
 unexpected error, closing socket connection and attempting reconnect
 java.io.IOException: Connection reset by peer
        at sun.nio.ch.FileDispatcher.read0(Native Method)
        at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:21)
        at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:233)
        at sun.nio.ch.IOUtil.read(IOUtil.java:200)
        at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:236)
        at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:858)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1130)


 2011-04-07 21:29:15,032 DEBUG Timer-0-SendThread(..:2181) 
 org.apache.zookeeper.ClientCnxn - Ignoring exception during shutdown input
 java.net.SocketException: Transport endpoint is not connected
        at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
        at 
 sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640)
        at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360)
        at 
 org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1205)
        at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:11