Re: Snapshot failure

2014-10-29 Thread Ian Brooks
Hi, I applied the suggested timeout changes and it lasted longer before it started failing, but its now just throwing the same errors on procedure.Subprocedure: Subprocedure pool is full! Looking through the logs, it looks like the nodes may not correctly updating they completion state after

Re: Snapshot failure

2014-10-17 Thread Ian Brooks
Hi, I got a change finally to apply the timeout changes and it lasted longer before it started failing, but its now just throwing the same errors on procedure.Subprocedure: Subprocedure pool is full! Looking through the logs, it looks like the nodes may not correctly updating they completion

Re: Snapshot failure

2014-10-16 Thread Ian Brooks
Hi, I got a change finally to apply the timeout changes and it lasted longer before it started failing, but its now just throwing the same errors on procedure.Subprocedure: Subprocedure pool is full! Looking through the logs, it looks like the nodes may not correctly updating they completion

Re: Snapshot failure

2014-09-23 Thread Ted Yu
Subprocedure pool is full. How many snapshot requests did you submit before seeing the following ? Cheers On Sep 23, 2014, at 2:28 AM, Ian Brooks i.bro...@sensewhere.com wrote: Hi, I'm seeing an issue on our hbase cluster which is preventing snapshots from working. So far the only way i

Re: Snapshot failure

2014-09-23 Thread Ian Brooks
Hi, I'm running snapshots on 7 tables every hour, the last run managed 15 rounds of snapshots before starting to fail. I did notice the following while getting the success count 2014-09-10 11:01:55,453 DEBUG [rs(##,16020,1410126483043)-snapshot-pool151-thread-1]

Re: Snapshot failure

2014-09-23 Thread Ted Yu
Here're the config parameters related to controlling snapshot timeout: property namehbase.snapshot.master.timeoutMillis/name !-- Change from default of 60s to 600s to allow for slow flushing of tables -- value60/value description This is the time HBase master waits for