Re: Hbase on docker container with persistent storage

Hi Udbhav, Great work on hbase docker deployment was done in https://issues.apache.org/jira/browse/HBASE-12721 you may start your journey from there. As for rest of your questions maybe there are some folks here that were doing similar testing and may give you more info. Regards Samir On Thu,

Re: Very High CPU usage in RegionServer

Hi Sandeep, What sort of load is on cluster when this is happening ? What logs say in moment when cluster is this state ? It is strange that whole cluster is unresponsive i can remember few cases when this is possible: - "hbase:meta" table is in transition or unavailable - zookeeper is

Re: Splitting causes HBase to crash

In most cases like can try following: 1. stop hbase 2. clean hbase zookeeper data (hbase zkcli --> rmr /hbase) 3. start hbase Regards Samir On Fri, May 13, 2016 at 9:17 AM, Gunnar Tapper wrote: > Hi, > > I'm doing some development testing with Apache Trafodion running

Re: Region Server Crash On Upsert Query Execution

Hi Amit, Check regionserver logs, usual suspects, are log running GC and HDFS client related issues. Check for compaction queue. Regrads Samir On Thu, Mar 31, 2016 at 10:48 AM, Amit Shah wrote: > Hi, > > We have been experimenting hbase (version 1.0) and phoenix (version

Re: Hbase data migration from version 0.94 to 0.98

Thanks for sharing this Rahul, this is quite useful approach for folks who have replication already configured and can afford downtime during migration. Regards Samir On Tue, Mar 1, 2016 at 7:26 PM, rahul gidwani wrote: > Hi Samir, > > We did this migration without any

Re: Hbase data migration from version 0.94 to 0.98

We have recently tested similar migration path source: hadoop1/hbase0.94 --> destination: hadoop2:/hbase0.98. Here is steps that we have performed : 1. stop source hbase cluster 2. use distcp to copy /hbase folder to destination cluster 3. run hbase migrate script on destination cluster (on

Re: HBase Cluster not responding after shutting down one of slave nodes

ecked that blog,helped me to understand what is going on when datanode > dies. > But still not sure why it took 15 minutes. > I will check the server logs again, carefully. > > Regards, > Hironori Ogibayashi > > 2016-02-18 22:56 GMT+09:00 Samir Ahmic <ahmic.sa...@gmai

Re: HBase Cluster not responding after shutting down one of slave nodes

apps/hbase/data/WALs/host7516.mydomain,16020,1455618299902-splitting/host7516.mydomain%2C16020%2C1455618299902..meta.1455690873222.meta, > length=83, corrupted=false, progress failed=false > 2016-02-17 15:41:20,482 INFO [RS_LOG_REPLAY_OPS-HOST7518:16020-0] > coordination.ZkSplitLogWorkerCoord

Re: HBase Cluster not responding after shutting down one of slave nodes

Hi Hironori, Based on your logs you have shutdown regionserver that was hosting system "hbase:meta" table. This is special case since user regions can not be reassigned to other servers until "hbase:meta" table region is reassigned. This can partially explain delay in recovery time you have

Re: Regionserver IP in Meta

Hi Anna What version of hbase you are using ? In most cases regoinserver will be bind on address which is resolved value of HOSTNAME environment variable. Try "ping $HOSTNAME" to determine hostname of server then you can add this line to /etc/hosts: eth0_ip_address

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Hi, This looks like some WALs splitting operations are not able to finish causing master to abort. First thing i would try is to stop cluster, clear zk data and then try to start it again. If that does not solve issue you probably have case of corrupted WALs files. In that case you will have to

Re: Table copies in hbase

to get all column families for source > table and create the destination table with them and then set pre-splits. > Right?But is there any other way to do the same? > > Thanks again, > Sumit > > > From: Samir Ahmic <ahmic.sa...@gmail.com> > To: user@hbase.apache.org

Re: Table copies in hbase

Hi Sumit, IMHO snapshots are easiest way to copy tables you need just to steps: 1. create snapshot 2. clone snapshot to another table What version of hbase you are using? Regarding you question what will happen in case operation fails in case of cloning snapshot i have never experience such

Re: Table copies in hbase

want then run CopyTable tool. Regards Samir On Wed, Jan 20, 2016 at 12:24 PM, Ted Yu <yuzhih...@gmail.com> wrote: > There is also backup / restore (work in progress): > > https://issues.apache.org/jira/browse/HBASE-7912 > > FYI > > On Wed, Jan 20, 2016 at 2:1

Re: constantly adding snapshot info

Based on what i have found in code there are two processes related to snapshots that are triggered every 5 minutes by default SnapshotHFileCleaner and SnapshotLogCleaner but none of them can explain why do you see new snapshots created. Have you been restoring snapshots, that can explain new

Re: constantly adding snapshot info

Hi Brian, Did you have created snapshots on cluster while you were upgrading to new hbase version ? Based on your logs there are some snapshots operations running on your cluster. Can you dig your logs to find what is triggering SnapshotManifestLoader there should some more info just above this

Re: High get/scan rates on HBase table even if no readers are on

HI Mukesh, Did you try to change logging levels in in $HBASE_CONF_DIR/log4j.properties? You can enable this lines to get debug info in lour logs: # Enable this to get detailed connection error/retry logging. #

Re: How to optimize the GC in HBase

Hi, What sort of load did you have on regionserver when server died? I see exception happens around compaction and flush operations. Did you check server's cpu, disks, memory, network... If pause was not caused by GC it may be that root cause is somewhere else. There is also hdfs exception in logs

Re: Phantom region server and PENDING_OPEN regions

Hi Kristoffer, It looks like you have some issue with name resolution. Try to remove incorrect value from reslove.conf (node.dc1.consul) and then restart hbase cluster. Regarding issue with region in transition check master log for "hbase:meta,,1.1588230740" there should be exception explaining

Re: Phantom region server and PENDING_OPEN regions

ehow it feels as if amb2.node.dc1.consul shadows > > the real host amb2.service.consul. > > > > I was looking into the source code and found the configuration > > 'hbase.regionserver.hostname' - could that be of help here to remove > > the node.dc1 host? > > >

Re: ZooKeeper still running in a decommissioned regionserver

Hi Oussama, What is the value of "hbase.zookeeper.quorum" in your config ? How many zookeeper instances you have in your cluster? You can stop zookeeper process on server with "hbase-daemon.sh stop zookeeper" command but be careful you mast have majority of zk ensemble up in order to zk cluster

Re: Phantom region server and PENDING_OPEN regions

rvice.consul > > PING amb1.service.consul (172.17.0.89) 56(84) bytes of data. > 64 bytes from amb1.node.dc1.consul (172.17.0.89): icmp_seq=1 ttl=64 > time=0.070 ms > > ping amb2 from amb2.service.consul > > PING amb2.service.consul (172.17.0.90) 56(84) bytes of data. > 6

Re: Export Snapshot to remote cluster and then Clone_Snapshot from exported data

Hi, Can you see snapshot on remote cluster? If you can see snapshot you can use clone snapshot command from hbase shell to create table. Regards Samir On Oct 14, 2015 6:38 PM, "anil gupta" wrote: > Hi, > > I exported snapshot of a table to remote cluster. Now, i want to

Re: Export Snapshot to remote cluster and then Clone_Snapshot from exported data

and? > > On Wed, Oct 14, 2015 at 11:19 AM, Samir Ahmic <ahmic.sa...@gmail.com> > wrote: > > > Hi, > > Can you see snapshot on remote cluster? If you can see snapshot you can > use > > clone snapshot command from hbase shell to create table. > > Regards &

Re: Export Snapshot to remote cluster and then Clone_Snapshot from exported data

structions to export the snapshot. What > do you mean by not exporting it to correct directory? > I am using HDP. Do you mean to that i just need to copy this exported in > same directory structure as other snapshots? > > > On Wed, Oct 14, 2015 at 11:36 AM, Samir Ahmic <a

Re: reginoserver shutdown

Hi, Based on you logs: 2015-09-01 15:35:58,047 INFO [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine *(eg GC): pause of approximately 4954ms* you had long running GC which cause timeout in communication between regionserver and zookeeper: 2015-09-01 15:36:04,970 INFO

Re: How to detect whether hbase cluster is up and ready for accepting requests.

Hi, Depending on your hbase version you can get ClusterStatus over api. https://hbase.apache.org/apidocs/org/apache/hadoop/hbase/ClusterStatus.html Regards Samir On Tue, Sep 1, 2015 at 2:03 PM, James Teng wrote: > Just as mentioned in the subject, is there any api to

Re: RegionServer Shutdown

Hi, It look like you are missing htrace jar in your hadoop classpath. You can check it with: $ hadoop classpath | tr : \n | grep htrace If it is not in classpath you will need to include it in hadop classpth. HTrace jar is located in $HBASE_HOME/lib. Regards Samir On Tue, Jul 7, 2015 at 1:15

Re: RegionServer Shutdown

-daemon.sh start regionserver command. Thanks! On Tue, Jul 7, 2015 at 5:19 PM, Samir Ahmic ahmic.sa...@gmail.com wrote: Hi, It look like you are missing htrace jar in your hadoop classpath. You can check it with: $ hadoop classpath | tr : \n | grep htrace If it is not in classpath you

Re: Regionserver won't start

Hello, In RegioServer log file you should have exception above this one you paste that explains why rs process is exiting. You will probably find root cause of issue there. Regards Samir On Thu, Mar 26, 2015 at 8:43 AM, ocassano ocass...@staff.voo.be wrote: Hello everyone, I’m facing a

Re: Regionserver won't start

Well there should be some more info in logs. You can try to start RegionServer process with hbase regionserver start commanf that will log everything on terminal so you can track where is issue. Regards Samir On Thu, Mar 26, 2015 at 10:58 AM, ocassano ocass...@staff.voo.be wrote: Hi Samir,

Re: Regionserver won't start

This cloud be hdfs related. Did you check your hdfs is running correctly ? On Thu, Mar 26, 2015 at 2:34 PM, ocassano ocass...@staff.voo.be wrote: That's what I did to get more logs. But I can't see anything going wrong. regionserver_start_failed.txt

Re: Regionserver won't start

In this case regionserver process is unable to initialize ShutdownHook thread. To be honest this something that i have rarely seen, if you are unable to resolve it i suggest new clean install because sometimes wrong jars can end up in classpath and is difficult to find root cause especially if

Re: Regionserver won't start

And this line in your log looks strange: 2015-03-23 16:24:31,327 INFO [regionserver60020] zookeeper.ZooKeeper: Client environment:java.class.path= ... did you remove classpath elements or they dont show in log ? On Thu, Mar 26, 2015 at 3:53 PM, Samir Ahmic ahmic.sa...@gmail.com wrote: Can

Re: Regionserver won't start

Can you check hbase classpath command maybe you have some strange hadoop version in your hbase classpath ? On Thu, Mar 26, 2015 at 3:37 PM, ocassano ocass...@staff.voo.be wrote: HDFS is up and running fine. It's HEALTHY and not in SAFE MODE. I watched at HDFS logs while I'm trying to start

Re: TableInputFormatBase Cannot resolve the host name

Do you use DNS server for name resolution ? Did you setup reverse DNS zone in your cluster ? I have seen this errors before when there is no reverse DNS setup. I believe that TableInputFormatBase class requires revers DNS name resolution. Regards Samir On Wed, Jun 25, 2014 at 10:57 AM, Li Li

Re: TableInputFormatBase Cannot resolve the host name

? On Wed, Jun 25, 2014 at 5:08 PM, Samir Ahmic ahmic.sa...@gmail.com wrote: Do you use DNS server for name resolution ? Did you setup reverse DNS zone in your cluster ? I have seen this errors before when there is no reverse DNS setup. I believe that TableInputFormatBase class requires revers

Re: Regionserver crash

Hi Ian, What hadoop fsck / says ? Maybe you have some corrupted data on your cluster. Also try using hbase hbck do investigate issue. If you have disk space issues try adding more data nodes to your cluster. Regarding errors you have send they are thrown because ProtobufLogWriter is unable to

Re: Regionserver crash

was clean at the time of the crash and after. hadoop version 2.4 hbase version 0.98.3 -Ian Brooks On Tuesday 17 Jun 2014 13:43:45 Samir Ahmic wrote: Hi Ian, What hadoop fsck / says ? Maybe you have some corrupted data on your cluster. Also try using hbase hbck do investigate issue. If you

Re: Timeouts during major compaction

Hi, You are definitely experiencing performance degradation cussed by major compaction. How big is your write load ? Maybe you should consider to decrease time interval between major compactions if you are writing a lot data to cluster. Also here is part of hbase book which explains how to

Re: Adding a new column family in existing table

Hi Vimal, Check 'alter' command in hbase shell. Run help 'alter' in hbase shell there is very good explanation for command usage. Regarding impact alter command will change table schema on all regions belonging to table you are altering. Regards Samir On Fri, May 30, 2014 at 9:07 AM, Vimal

Re: meta server hungs ?

Hi, This exception: exception=java.net.SocketTimeoutException: Call to hadoop77/192.168.1.87:60020 failed because java.net.SocketTimeoutException: 6 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.1.123:33117

Re: 答复: meta server hungs ?

-sweep generation total 16252928K, used 11162086K [0x00041ae0, 0x0007fae0, 0x0007fae0) concurrent-mark-sweep perm gen total 81072K, used 48660K [0x0007fae0, 0x0007ffd2c000, 0x0008) -邮件原件- 发件人: Samir Ahmic [mailto:ahmic.sa...@gmail.com] 发送

Re: rowcount in a specified timerange

Hi, Hansi Take look at https://github.com/apache/hbase/blob/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/RowCounter.java . You can modified it and and scan.setTimeRange(long minStamp, long maxStamp) that will give you option to count rows in specific time range. Anther

Re: Getting all rows from .META. for a specific table.

Hi David, You may try this in hbase shell: scan '.META.', {FILTER=PrefixFilter('table_name')} this will scan all regions in .META. table with table_name prefix. Cheers:) Samir On Fri, Apr 25, 2014 at 11:49 AM, David Koch ogd...@googlemail.com wrote: Hi, How do I get all rows from

Re: Getting all rows from .META. for a specific table.

Hi David, Here is quick bash script that will extract encoded region name. #!/bin/bash table=$1 echo scan '.META.', {FILTER=\PrefixFilter('$table')\} cmd_file hbase shell cmd_file | grep $table |awk '{print $1}' | cut -d , -f3 | cut -d . -f2 |tail -n+2 | uniq Regards On Fri, Apr 25, 2014

Re: Copying data from one Hbase cluster to Another Hbase cluster

Hi Vimal, I have few options how to move data from one hbase cluster to another: 1. You can use org.apache.hadoop.hbase.mapreduce.Export tool to export tables to HDFS and then you can use hadoop distcp to move data to another cluster. When data is place on second cluster you can use

Re: Copying data from one Hbase cluster to Another Hbase cluster

cluster. Is there any other solution which does not require map reduce at all ? On Fri, Feb 14, 2014 at 2:41 PM, Samir Ahmic ahmic.sa...@gmail.com wrote: Hi Vimal, I have few options how to move data from one hbase cluster to another: 1. You can use

Re: WUI - Regionserver shows duplicated entries (IP-name)

Hi Federico, You can restart master that should clean rs list. Regards 2014-02-05 18:15 GMT+01:00 Federico Gaule fga...@despegar.com: Dear all, I've introduced a new regionserver to my cluster. First, there was no rDNS solving names, so it was named with it IP address. Then i stop the

Re: max HStoreFile size

Hi Amit, Yes. You can set split policy per table. Here is relevant part of hbase book: http://hbase.apache.org/book/regions.arch.html The policy can set globally through the HBaseConfiguration used or on a per table basis: HTableDescriptor myHtd = ...;

Re: Inconsistency in hbase tables

Hi, Which version of hbase you are using ? Did you try fixing with hbase hbck -fixMeta ?. Also if hbck don't fix this issue for you you can use the OfflineMetaRepar tool to create META and ROOT regions and tables. You can find very useful instruction here:

Re: Unable to find region for hello_world,,99999999999999 after 10 tries.

Hi Fernando, Check logs on RegionServers there should be explanation why you got NotServingRegionException. You said that you got error after 10 min of running script ? It could be that you have some resources issue, memory disk, cpu. Cheers On Thu, Jan 16, 2014 at 2:28 PM, Fernando Iwamoto -

Re: HBase connection hangs

Hi Mark, Base on this exception: 1. 1862 [pool-2-thread-1] WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper - Possibly transient ZooKeeper, quorum=mark-7:2181, exception=org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for

Re: What cause region server to timeout other than long gc?

Hi Henry, Are you using automatic major compaction or you turned it off ? From your logs i can see that issue appeared after compaction is triggered. If few compactions are run at same time it can severely affect IO performance leading to errors on RS.(which may be your case). Here is relevant

Re: Will hbase automatically distribute the data across region servers or NOT..??

Hi Vamshi, What version of hbase you are using ? Regarding *hbase.hregion.max.filesize *property from hbase-site.xml you should check this from hbase-default.xml : property namehbase.hregion.max.filesize/name value*1073741824*0/value value is in bytes* description Maximum

HBase Snapshots as backup solution

Hi all, We are using hbase-0.94.6.1 and at moment i'm evaluation Snapshots as backup solution for moving data between clusters. I'm wondering if someone have similar experience and what are pros and cons ? Also is Snapshot future stable enough for this sort of operation ? Thanks, Samir

Re: 答复: Getting list of Region In Transition via command line

Hi, Asaf You can run* status 'detailed' | hbase shell* from command line that will give you detailed status including regions in transition and also you can use *hbase hbc*k command , and third options is to run *hbase zkcli* and then *ls /hbase/unassign* that will display encoded names of

hbase-0.94.6.1 balancer issue

Hi, all I'm evaluating hbase-0.94.6.1 and i have 48 regions on 2 node cluster. I was restarting on of RSs and after that tried to balance cluster by running balancer from shell. After running command regions were not distributed to second RS and i found this line i master log: 2013-04-12

Re: hbase-0.94.6.1 balancer issue

. This is normal behavior for 0.94. I don't know for versions before that. Also, are you sure you need 48 tables? And not less tables with more CFs? JM 2013/4/12 Samir Ahmic ahmic.sa...@gmail.com Hi, JM I have 48 tables and as you said it is 1 region per table since i did not reach

Re: hbase-0.94.6.1 balancer issue

distributed... unless there's an edge case where when you have only tables with 1 region it puts them all on the same server :) Thx, J-D On Fri, Apr 12, 2013 at 12:37 PM, Samir Ahmic ahmic.sa...@gmail.com wrote: Thanks for explaining Jean-Marc, We are using 0.90.4 for very long time

Re: hbase-0.94.6.1 balancer issue

HBASE-7060 explains my case, i'm using 0.94.6.1 and looks like issue is still present. Thanks for replaying guys Cheers:)

Re: Cannot connect to local hbase database

Hi, Paul Look like to me that you did not point your java program to correct HBase configuration. Look at HTablePool constructor at: http://hbase.apache.org/apidocs/org/apache/hadoop/hbase/client/HTablePool.html#HTablePool() Maybe you should try it like this: Configuration conf =

Re: Regions in transition

Hi, Kiran Welcome to beautiful world of HBase transition states :) . When i face RIT issue this are steps that i use to resolve it: 1. hbase hbck -fixAssignments (this depends on your version of hbase it can be also just -fix) If you don't have luck with 1. then you will need manual

Re: HBASE-5603 fix

Sure thing. I will fill JIRA Cheers On Wed, Feb 6, 2013 at 6:20 PM, Ted Yu yuzhih...@gmail.com wrote: HBASE-5603 has been closed. Mind filing a new JIRA ? On Wed, Feb 6, 2013 at 2:47 AM, Samir Ahmic ahmic.sa...@gmail.com wrote: Hi all, I have tried rolling-restart.sh and it still hangs

Re: number of region servers is wrong

Hi Lu, I remember that i had similar issue with wrong number of region servers reported to master. In my case it was issue with reverse name resolution so i think you should check DNS settings and /etc/hosts. Try ping -c 2 $HOSTNAME on regionserver that is reported twice

Region is in transition, in CLOSING state

Hi, We are using hbase-0.90.4, and after one of our region servers went down master reported that one region is in transition, in CLOSING state: INFO org.apache.hadoop.hbase.master.AssignmentManager: Regions in transition timed out: access_tokens,,1312452518317.1878726708 state=CLOSING,