Hi Ted,
I tired doing this multiple time and this is what I see. Looks like some
property settings for my timeout are not good and shell gets a ERROR with
timeout.
Am I missing some config change as mentioned earlier have these two
property defined.
hbase-site.xml
<property>
<name>hbase.snapshot.master.timeoutMillis</name>
<value>1800000</value>
</property>
<property>
<name>hbase.snapshot.region.timeout</name>
<value>1800000</value>
</property>
<property>
<name>hbase.snapshot.enabled</name>
<value>true</value>
</property>
tail -f master log:
2015-05-08 19:47:26,983 WARN org.apache.hadoop.hbase.HBaseConfiguration:
Config option "hbase.regionserver.lease.period" is deprecated. Instead, use
"hbase.client.scanner.timeout.period"
hbase shell:
hbase(main):004:0> snapshot 'table','snapshot_test_3'
ERROR: Snapshot 'snapshot_test_3' wasn't completed in expectedTime:60000 ms
Here is some help for this command:
Take a snapshot of specified table. Examples:
hbase> snapshot 'sourceTable', 'snapshotName'
hbase> snapshot 'namespace:sourceTable', 'snapshotName', {SKIP_FLUSH =>
true}
hbase(main):005:0> snapshot 'device_table','snapshot_test_3'
ERROR: org.apache.hadoop.hbase.snapshot.SnapshotExistsException: Snapshot
'snapshot_test_3' already stored on the filesystem.
at
org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:526)
at org.apache.hadoop.hbase.master.HMaster.snapshot(HMaster.java:2842)
at
org.apache.hadoop.hbase.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java:42291)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2031)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108)
at
org.apache.hadoop.hbase.ipc.FifoRpcScheduler$1.run(FifoRpcScheduler.java:74)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Here is some help for this command:
Take a snapshot of specified table. Examples:
hbase> snapshot 'sourceTable', 'snapshotName'
hbase> snapshot 'namespace:sourceTable', 'snapshotName', {SKIP_FLUSH =>
true}
On Fri, May 8, 2015 at 11:58 AM, Ted Yu <[email protected]> wrote:
> bq. ERROR: Snapshot 'snapshot_test' wasn't completed in expectedTime:60000
> ms
>
> Rahul:
> Please check your master log around 15:15 (when the timeout happened). You
> should find some clue there.
>
> Cheers
>
> On Fri, May 8, 2015 at 11:40 AM, rahul malviya <[email protected]
> >
> wrote:
>
> > We have 16000 regions with approx 70 regions per regionserver.
> >
> > Thanks,
> > Rahul
> >
> > On Fri, May 8, 2015 at 11:25 AM, Jean-Marc Spaggiari <
> > [email protected]> wrote:
> >
> > > How many regions do you have on this table, how many RS and how many
> > > regions per RS?
> > >
> > > JM
> > >
> > > 2015-05-08 14:21 GMT-04:00 Rahul Malviya <[email protected]>:
> > >
> > > > Hi,
> > > >
> > > > I want to run MR job on snapshot of large HBase table but I am
> getting
> > > > timeout errors while doing so. Can you tell me what I am doing wrong
> ?
> > > >
> > > > I am using hbase-0.98.6-cdh5.2.0.
> > > >
> > > > hbase-site.xml
> > > > <property>
> > > > <name>hbase.snapshot.master.timeoutMillis</name>
> > > > <value>1800000</value>
> > > > </property>
> > > > <property>
> > > > <name>hbase.snapshot.region.timeout</name>
> > > > <value>1800000</value>
> > > > </property>
> > > > <property>
> > > > <name>hbase.snapshot.enabled</name>
> > > > <value>true</value>
> > > > </property>
> > > >
> > > > Error :
> > > > $ hbase shell
> > > > HBase Shell; enter 'help<RETURN>' for list of supported commands.
> > > > Type "exit<RETURN>" to leave the HBase Shell
> > > > Version 0.98.6-cdh5.2.0, rUnknown, Sat Oct 11 15:14:25 PDT 2014
> > > >
> > > > hbase(main):001:0> snapshot ’table','snapshot_test'
> > > > SLF4J: Class path contains multiple SLF4J bindings.
> > > > SLF4J: Found binding in
> > > >
> > >
> >
> [jar:file:/ngs/app/awdp/RELEASE-5.2/hbase-0.98.6-cdh5.2.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > SLF4J: Found binding in
> > > >
> > >
> >
> [jar:file:/ngs/app/awdp/RELEASE-5.2/hadoop-2.5.0-cdh5.2.0/share/hadoop/mapreduce1/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > SLF4J: Found binding in
> > > >
> > >
> >
> [jar:file:/ngs/app/awdp/RELEASE-5.2/hadoop-2.5.0-cdh5.2.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> > > > SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings <
> > > > http://www.slf4j.org/codes.html#multiple_bindings> for an
> explanation.
> > > > SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> > > >
> > > > ERROR: Snapshot 'snapshot_test' wasn't completed in
> expectedTime:60000
> > ms
> > > >
> > > > Here is some help for this command:
> > > > Take a snapshot of specified table. Examples:
> > > >
> > > > hbase> snapshot 'sourceTable', 'snapshotName'
> > > > hbase> snapshot 'namespace:sourceTable', 'snapshotName', {SKIP_FLUSH
> > =>
> > > > true}
> > > >
> > > >
> > > > hbase(main):002:0>
> > > >
> > > > Thanks,
> > > > Rahul
> > >
> >
>
