Re: several doubts about region split?

Inline. J-D On Wed, Jul 17, 2013 at 7:10 AM, yonghu wrote: > Thanks for your quick response! > > For the question one, what will be the latency? How long we need to wait > until the daughter regions are again online? Usually a matter of 1-2 seconds. > > regards! > > Yong > > > > On Wed, Jul 17

Re: Memory leak in HBase replication ?

Yean WARN won't give us anything, and please try to get us a fat log. Post it on pastebin or such. Thx, J-D On Wed, Jul 17, 2013 at 11:03 AM, Anusauskas, Laimonas < lanusaus...@corp.untd.com> wrote: > J-D, > > I have log level org.apache=WARN and there is only following in the logs > before GC

Re: Memory leak in HBase replication ?

1GB is a pretty small heap and it could be that the default size for logs to replicate is set to high. The default for replication.source.size.capacity is 64MB. Can you set it much lower on your master cluster (on each RS), like 2MB, and see if it makes a difference? The logs and the jstack seem t

Re: Memory leak in HBase replication ?

Yes... your master cluster must have helluva backup to replicate :) Seems to make a good argument to lower the default setting. What do you think? J-D On Wed, Jul 17, 2013 at 3:37 PM, Anusauskas, Laimonas < lanusaus...@corp.untd.com> wrote: > Thanks, setting replication.source.size.capacity to

Re: Bulk Load on HBase 0.95.1-hadoop1

0.95.1 is a developer preview release, if you are just starting with HBase please grab the stable release from 0.94, for example http://mirrors.sonic.net/apache/hbase/stable/ J-D On Thu, Jul 18, 2013 at 1:51 PM, Jonathan Cardoso wrote: > I was trying to follow the instructions from > this

Re: Altering table column family attributes without disabling the table

You could always set hbase.online.schema.update.enable to true on your master, restart it (but not the cluster), and you could do what you are describing... but it's a risky feature to use before 0.96.0. Did you also set hbase.replication to true? If not, you'll have to do it on the region servers

Re: Excessive .META scans

Can you tell who's doing it? You could enable IPC debug for a few secs to see who's coming in with scans. You could also try to disable pre-fetching, set hbase.client.prefetch.limit to 0 Also, is it even causing a problem or you're just worried it might since it doesn't look "normal"? J-D On Mo

Re: Can't solve the "Unable to load realm info from SCDynamicStore" error

"Unable to load realm info from SCDynamicStore" is only a warning and a red herring. What seems to be happening is that your shell can't reach zookeeper. Are Zookeeper and HBase running? What other health checks have you done? J-D On Tue, Jul 30, 2013 at 10:28 PM, Seth Edwards wrote: > I am som

Re: HDFS Restart with Replication

I can't think of a way how your missing blocks would be related to HBase replication, there's something else going on. Are all the datanodes checking back in? J-D On Thu, Aug 1, 2013 at 2:17 PM, Patrick Schless wrote: > I'm running: > CDH4.1.2 > HBase 0.92.1 > Hadoop 2.0.0 > > Is there an issue

Re: HDFS Restart with Replication

> nothing special about data05, and it seems to be in the cluster, the same > as anyone else. > > > On Thu, Aug 1, 2013 at 5:04 PM, Jean-Daniel Cryans wrote: > >> I can't think of a way how your missing blocks would be related to >> HBase replication, there's so

Re: HDFS Restart with Replication

ster, but not the Region > Servers, then restarting HDFS. What's the correct order of operations for > bouncing everything? > > > On Thu, Aug 1, 2013 at 5:21 PM, Jean-Daniel Cryans wrote: > >> Can you follow the life of one of those blocks though the Namenode and >>

Re: HDFS Restart with Replication

J-D >> >> On Fri, Aug 2, 2013 at 8:27 AM, Patrick Schless >> wrote: >> > Ah, I bet the issue is that I'm stopped the HMaster, but not the Region >> > Servers, then restarting HDFS. What's the correct order of operations for >> > bouncing everything

Re: Major Compaction in 0.90.6

Inline. J-D On Mon, Aug 19, 2013 at 2:48 AM, Monish r wrote: > Hi guys, > I have the following questions in HBASE 0.90.6 > > 1. Does hbase use only one compaction thread to handle both major and minor > compaction? > Yes, look at CompactSplitThread > > 2. If hbase uses multiple compaction t

Re: Replication queue?

You can find a lot here: http://hbase.apache.org/replication.html And how many logs you can queue is how much disk space you have :) On Tue, Aug 20, 2013 at 7:23 AM, Jean-Marc Spaggiari < jean-m...@spaggiari.org> wrote: > Hi, > > If I have a master -> slave replication, and master went down, re

Re: Major Compaction in 0.90.6

pache.hadoop.hbase.regionserver.Store: > Completed major compaction of 1 file(s), new > > file=hdfs://x.x.x.x:9000/hbase/NOTIFICATION_HISTORY/b00086bca62ee55796a960002291aca4/n/4754838096619480671 > > i find a new file is created for every major compaction triggger. > > Regards

Re: Is downgrade from 0.96.0 to 0.94.6 possible?

FYI you'll be in the same situation with 0.95.2, actually worse since it's really just a developer preview release. But if you meant "try" in its strict sense, ie use it on a test cluster, then yes please do. The more people we get to try it out the better 0.96.0 will be. J-D On Thu, Aug 22, 20

Re: [Question: replication] why only one regionserver is used during replication? 0.94.9

Region servers replicate data written to them, so look at how your regions are distributed. J-D On Tue, Aug 27, 2013 at 11:29 AM, Demai Ni wrote: > hi, guys, > > I am using hbase 0.94.9. And setup replication from a 4-nodes master(3 > regserver) to a 3-nodes slave(2 regserver). > > I can tell

Re: counter Increment gives DonotRetryException

You probably put a string in there that was a number, and increment expects a 8 bytes long. For example, if you did: put 't1', '9row27', 'columnar:column1', '1' Then did an increment on that, it would fail. J-D On Thu, Aug 29, 2013 at 4:42 AM, yeshwanth kumar wrote: > i am newbie to Hbase, >

Re: HBase distributed mode issue

What's your /etc/hosts on the master like? HBase does a simple lookup to get the machine's hostname and it "seems" your need reports itself as being localhost. On Tue, Sep 3, 2013 at 6:23 AM, Omkar Joshi wrote: > I'm trying to set up a 2-node HBase cluster in distributed mode. > > Somehow, my re

Re: Getting column values in batches for a single row

Scan.setBatch does what you are looking for, since with a Get there's no way to iterate over mutliple calls: https://github.com/apache/hbase/blob/0.94.2/src/main/java/org/apache/hadoop/hbase/client/Scan.java#L306 Just make sure to make the Scan start at the row you want and stop right after it. J

Re: Performance analysis in Hbase

Yeah there isn't a whole lot of documentation about metrics. Could it be that you are still running on a default 1GB heap and you are pounding it with multiple clients? Try raising the heap size? FWIW I gave a presentation at HBaseCon with Kevin O'dell about HBase operations which could shed some

Re: High cpu usage on a region server

Or roll back to CDH 4.2's HBase. They are fully compatible. J-D On Thu, Sep 12, 2013 at 10:25 AM, lars hofhansl wrote: > Not that I am aware of. Reduce the HFile block size will lessen this > problem (but then cause other issues). > > It's just a fix to the RegexStringFilter. You can just reco

Re: Information about hbase 0.96

Release date is: when it gets released. We are currently going through release candidates and as soon as one gets accepted we release it. I'd like to say it's gonna happen this month but who knows. There's probably one or two presentations online that explain what's in 0.96.0, but the source of tr

Re: Command to delete based on column Family + rowkey

HBASE-8753 doesn't seem related. Right now there's nothing in the shell that does the equivalent of this: Delete.deleteFamily(byte [] family) But it's possible to run java code in the jruby shell so in the end you can still do it, just takes more lines. J-D On Mon, Sep 16, 2013 at 1:45 AM, Te

Re: user_permission ERROR: Unknown table

What are you trying to do bg? If you want to setup user permissions you also need to have a "secure" HBase (the link that Ted posted) which involves Kerberos. J-D On Mon, Sep 16, 2013 at 1:33 PM, Ted Yu wrote: > See http://hbase.apache.org/book.html#d0e5135 > > > On Mon, Sep 16, 2013 at 1:06 P

Re: HBase Negation or NOT operator

You can always remove the NOT clause by changing the statement, but I'm wondering what your use case really is. HBase doesn't have secondary indexes so, unless you are doing a short-ish scan (let's say a million rows), it means you want to do a full table scan and that doesn't scale. J-D On Tue,

Re: user_permission ERROR: Unknown table

Ah I see, well unless you setup "Secure HBase" there won't be any perms enforcement. So in which way is your application failing to use "Selector"? Do you have an error message or stack trace handy? J-D On Tue, Sep 17, 2013 at 5:43 AM, BG wrote: > Well we are trying to find out why our applic

Re: show processlist equivalent in Hbase

(putting cdh user in BCC, please don't cross-post) The web UIs for both the master and the region server have a section called Tasks and has a bunch of links like this: Tasks Show All Monitored Tasks Show non-RPC Tasks Show All RPC Handler Tasks Show Active RPC Calls Show Client Operations View

Re: openTSDB lose large amount of data when the client are writing

Could happen if a region moves since locks aren't persisted, but if I were you I'd ask on the opentsdb mailing list first. J-D On Thu, Sep 19, 2013 at 10:09 AM, Tianying Chang wrote: > Hi, > > I have a customer who use openTSDB. Recently we found that only less than > 10% data are written, res

Re: Bulkload into empty table with configureIncrementalLoad()

You need to create the table with pre-splits, see http://hbase.apache.org/book.html#perf.writing J-D On Thu, Sep 19, 2013 at 9:52 AM, Dolan Antenucci wrote: > I have about 1 billion values I am trying to load into a new HBase table > (with just one column and column family), but am running into

Re: Hbase ports

On Mon, Sep 23, 2013 at 9:14 AM, John Foxinhead wrote: > Hi all. I'm doing a project for my university so that i have to know > perfectly how all the Hbase ports work. Studing the documentation i found > that Zookeeper accept connection on port 2181, Hbase master on port 6 > and Hbase regionse

Re: Hbase Compression

On flushing we do some cleanup, like removing deleted data that was already in the MemStore or extra versions. Could it be that you are overwriting recently written data? 48MB is the size of the Memstore that accumulated while the flushing happened. J-D On Tue, Sep 24, 2013 at 3:50 AM, aiyoh79

Re: Export API using start and stop row key !

You'd need to use 0.94 (or CDH4.2+ since you are mentioning being on CDH) to have access to TableInputFormat.SCAN_ROW_START and SCAN_ROW_STOP then all you need to do is to copy Export's code and add what you're missing. J-D On Tue, Sep 24, 2013 at 5:42 PM, karunakar wrote: > Hi Experts, > > I

Re: Replication

That means that the master cluster isn't able to see any region servers in the slave cluster... is cluster b up? Can you create tables? J-D On Fri, Sep 27, 2013 at 3:23 AM, Arnaud Lamy wrote: > Hi, > > I tried to configure a replication with 2 boxes (a&b). A hosts hbase & zk > and b only hbase

Re: What is causing my mappers to execute so damn slow?

Your details are missing important bits like you configurations, Hadoop/HBase versions, etc. Doing those random reads inside your MR job, especially if they are reading cold data, will indeed make it slower. Just to get an idea, if you skip doing the Gets, how fast does it became? J-D On Fri, S

Re: What is causing my mappers to execute so damn slow?

00 rows.. > > > On Fri, Sep 27, 2013 at 11:12 PM, Jean-Daniel Cryans >wrote: > > > Your details are missing important bits like you configurations, > > Hadoop/HBase versions, etc. > > > > Doing those random reads inside your MR job, especially if they are > r

Re: HBase stucked because HDFS fails to replicate blocks

I like the way you were able to dig down into multiple logs and present us the information, but it looks more like GC than an HDFS failure. In your region server log, go back to the first FATAL and see if it got a session expired from ZK and other messages like a client not being able to talk to a

Re: hbase.master parameter?

hbase.master was removed when we added zookeeper, so now a client will do a lookup in ZK instead of talking to a pre-determined master. So in a way, hbase.zookeeper.quorum is what replaces hbase.master FWIW that was done in 0.20.0 which was released in September of 2009, so hbase.master has been r

Re: You Are Dead Exception due to promotion failure

This line: [CMS-concurrent-mark: 12.929/88.767 secs] [Times: user=14.30 sys=3.74, real=88.77 secs] Is suspicious. Are you swapping? J-D On Mon, Oct 7, 2013 at 8:34 AM, prakash kadel wrote: > Also, >why is the CMS not kicking in early, i have set XX:+ > UseCMSInitiatingOccupancyOnly??? > >

Re: Upcoming HBase bay area user and dev meetups

While we're on the topic of upcoming meetups, there's also a meetup at Facebook's NYC office the week of Strata/Hadoop World (10/28). There's still room for about 50 attendees. http://www.meetup.com/HBase-NYC/events/135434632/ J-D On Mon, Oct 7, 2013 at 2:10 PM, Enis Söztutar wrote: > Hi guys

Re: You Are Dead Exception due to promotion failure

On Tue, Oct 8, 2013 at 7:09 AM, prakash kadel >wrote: > > > thanks, > > > > yup, it seems so. I have 48 gb memory. i see it swaps at that point. > > > > btw, why is the CMS not kicking in early? do you have any idea? > > > > sincerely > &g

Re: HBase Random Read latency > 100ms

On Wed, Oct 9, 2013 at 10:59 AM, Vladimir Rodionov wrote: > I can't say for SCR. There is a possibility that the feature is broken, of > course. > But the fact that hbase.regionserver.checksum.verify does not affect > performance means that OS caches > effectively HDFS checksum files. > See "OS c

Re: ethernet channel bonding experiences

Hi Oliver, Unless you are network-bound you shouldn't see an improvement, verify that first. J-D On Mon, Mar 19, 2012 at 8:58 AM, Oliver Meyn (GBIF) wrote: > Hi all, > > I've been experimenting with PerformanceEvaluation in the last weeks and on a > whim thought I'd give channel bonding a try

Re: hbase schema row_key only

> 1) As I understand column family internally represents a file , but in my > case I enter nothing to the column family ( actually I enter empty > qualifier:value pairs). >     I can't find any details about how rowkeys is stores. Any information > about storing rowkeys will be very helpful. Right

Re: Region server crashes

On Sun, Mar 25, 2012 at 1:23 AM, Lior Schachter wrote: > Hi all, > We use hbase 0.9.2. We recently started to experience region servers You mean 0.90.2? Or 0.92.0? > crashed under heavy load (2-3 different servers crashes eah load). > Seems like missing block in HDFS causes a full GC and regions

Re: Migrating Tables with primary key and foreign key from mysql to Hbae

Hi, HBase is not a relational database, it doesn't have foreign keys or constraints. I'd suggest you familiarize yourself with HBase by reading the refence manual[1] or buying the book[2]. Regards, J-D 1. http://hbase.apache.org/book/book.html 2. http://www.amazon.com/dp/1449396100 On Tue, Mar

Re: Starting Abnormally After Shutting Down For Some Time

Hi Bing, Two questions: - Can you look at the master log and see what's preventing the master from starting? - Did you change dfs.data.dir and dfs.name.dir in hdfs-site.xml? By default it writes to /tmp which can get cleaned up. J-D On Tue, Mar 27, 2012 at 12:52 PM, Bing Li wrote: > Dear all,

Re: Region server shutting down due to HDFS error

Any chance we can see what happened before that too? Usually you should see a lot more HDFS spam before getting that all the datanodes are bad. J-D On Wed, Mar 28, 2012 at 4:28 AM, Eran Kutner wrote: > Hi, > > We have region server sporadically stopping under load due supposedly to > errors writ

Re: Region server shutting down due to HDFS error

2012-03-28 00:15:05,359 WARN > org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Error > getting node's version in CLOSIN > G state, aborting close of > gs_users,731481|Sn쒪㝨眳ԫ䂣⫰==,1331226388691.29929cb2200b3541ead85e17b836ade5. > > -eran > > > > On Wed, Mar 28, 2012 at 18:38, Jean-Daniel Cryans wrote: >

Re: Doubt at hbase configuration

The timeout is on the client side, not the server side. J-D On Fri, Mar 30, 2012 at 12:11 AM, Balaji k wrote: > Hi, >          I was running coprocessor AggregationProtocol, it got socket > timeout exception. Somebody said to set  "hbase.rpc.timeout" to larger > value. > I added the following  i

Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

This is a client-side configuration so if your mapred-site.xml is _not_ on your classpath when you start the bulk load, it's not going to pick it up. So either have that file on your classpath, or put it in whatever other configuration file you have. J-D On Fri, Mar 30, 2012 at 2:52 PM, anil gupt

Re: HBase bulk loader doing speculative execution when it set to false in mapred-site.xml

nks for the quick reply, Jean. Is there any link where i can find the > name of all client-side configuration for HBase? > > ~Anil > > On Fri, Mar 30, 2012 at 3:01 PM, Jean-Daniel Cryans > wrote: > >> This is a client-side configuration so if your mapred-site.xml is >&

Re: Too many hlogs

On Mon, Apr 2, 2012 at 12:27 PM, Miles Spielberg wrote: > > Our region server are each hosting ~270 regions. Our writes are extremely > well distributed (our HBase keys are output from a hash function) and small > (~100s of bytes). I believe that the writes are being so well distributed > across

Re: Too many hlogs

On Mon, Apr 2, 2012 at 2:18 PM, Miles Spielberg wrote: > So it sounds like with our write pattern (highly distributed, all regions > being written to simultaneously), we should be trying to keep number of > regions down to 32 (or whatever hbase.regionserver.maxlogs is set to). I > suppose we could

Re: Region Server down when use export tools to backup tabls

The log says that the region server tried to talk to the region server "dp7.abcd.com" and it timed out after 60 seconds, and that happened during a split which is pretty bad. As the log says: org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Abort; we got an error after point-of-no-retu

Re: determine whole-row delete in Coprocessor.preDelete

The same way you tell HBase to delete a whole row, no columns are specified. J-D On Thu, Apr 5, 2012 at 6:32 PM, NNever wrote: > Dear all, > > when trigger a preDelete in Coprocessor, is there any possible to determine > this Delete is going to remove a row ( not just a family or qualifier, but

Re: Schema Updates: what do you do today?

Still doing it old school unless I have to alter a bunch of tables for the same reason, like enabling replication. J-D On Mon, Apr 9, 2012 at 9:08 AM, Ian Varley wrote: > All: > > I'm doing a little research into various ways to apply schema modifications > to an HBase cluster. Anybody care to

Re: Is htable.delete(List) transactional?

Simple answer: it's not transactional. J-D On Mon, Apr 16, 2012 at 4:28 PM, Haijia Zhou wrote: > Very simple question as the subject shows: > Is htable.delete(List) transactional? > Say if I am to delete 1000 rows and in the middle of deletion some error > occurs, then will the whole deletion op

Re: Storing extremely large size file

Make sure the config is changed client-side not server-side. Also you might not want to store 12MB values in HBase. J-D On Tue, Apr 17, 2012 at 6:06 AM, vishnupriyaa wrote: > > I want to save a file of size 12MB but an exception occuring like this > KeyValue size too large. > I have set the val

Re: Storing extremely large size file

Yes, you fine tuned it properly :) But in general I wouldn't recommend it to new users. J-D On Tue, Apr 17, 2012 at 10:47 AM, Jack Levin wrote: > Whats wrong with that size?  We store > 15MB routinely into our image hbase. > > -Jack > > On Tue, Apr 17, 2012 at 10:46

Re: I cannot find the hbase configuration templates

Yeah I'm not sure why that page is not in the reference guide but it should be moved and fixed since that file has been removed a long time ago. I created https://issues.apache.org/jira/browse/HBASE-5822 J-D On Tue, Apr 17, 2012 at 11:09 PM, wrote: > I asked the same question on Stackoverflow:

Re: Need help on using hbase on EC2

Would it be possible for you to pastebin a much bigger portion of the hbase log? Thx, J-D On Tue, Apr 17, 2012 at 10:35 AM, Xin Liu wrote: > Hi there, > > I setup hadoop and hbase on top of EC2 in Pseudo-distributed mode. I > can use hbase shell to connect. However, when I use java client to >

Re: HBase Cyclic Replication Issue: some data are missing in the replication for intensive write

> I will try your suggestion today with a master-slave replication enabled from > Cluster A -> Cluster B. Please do. > Last Friday, I tried to limit the variability/the moving part of the > replication components. I reduced the size of Cluster B to have only 1 > regionserver and having Cluster

Re: Understanding responseTooSlow in HBase regionserver log

You are right. FWIW I cannot find operationTooSlow in the source code, not sure what's going on there. J-D On Thu, Apr 26, 2012 at 10:55 AM, Mike Spreitzer wrote: > I found http://hbase.apache.org/book/ops.monitoring.html but am confused > about the distinction between operationTooSlow and resp

Re: Are minor compaction and major compaction different in HBase 0.92?

I think the 0.92 code has a way to promote minor into major compactions, feel free to checkout the code (also it should be present in your logs). J-D On Wed, Apr 25, 2012 at 11:48 PM, yonghu wrote: > Hello, > > My HBase version is 0.92.0. And I find that when I use minor > compaction and major c

Re: region size

No, it's a direct read using a block index which is in memory. J-D On Wed, May 2, 2012 at 2:29 PM, Paul Mackles wrote: > I think the answer to this is "no", but I am hoping someone with more > experience can confirm this… we are on hbase 0.90.4 (from cdh3u2). Some of > our storefiles have grow

Re: multi-data center support

A single HBase instance doesn't work across datacenters, maybe that's why you haven't found any documentation. HBase does have replication between clusters, see http://hbase.apache.org/replication.html J-D On Thu, May 3, 2012 at 11:10 AM, Marco Villalobos wrote: > I have not found any documenta

Re: multi-data center support

d want counters to work across data-center, with the > correct increment eventually merging. > > On Thu, May 3, 2012 at 11:26 AM, Jean-Daniel Cryans > wrote: >> A single HBase instance doesn't work across datacenters, maybe that's >> why you haven't found any docu

Re: multi-data center support

a map > reduce job can aggregate them later, then replicate? > > I hope something like that works. > > On Thu, May 3, 2012 at 1:23 PM, Jean-Daniel Cryans > wrote: >> Since 0.92 you can replicate in a Master-Master fashion if you want, >> just set each cluster to be the s

Re: Handling Hostname changes in HBase

Those bugs were taken care of, just shutdown the processes and restart them after the machine has the new configuration. J-D On Wed, May 9, 2012 at 3:00 AM, kisalay wrote: > Hi, > > I had a doubt about the procedure for hostname change of master / > regionservers. Can we achieve this without cle

Re: Looking for a single row - HTable.get(Get) or Scan(Get)

What Bryan said, also Scan(Get) is used internally in the region server code so that's probably why that constructor exists. But as a user you need to create the scanner, do next() then close it so using a straight Get saves 2 rpc. J-D On Wed, May 9, 2012 at 1:43 PM, Bryan Beaudreault wrote: >

Re: Switching existing table to Snappy possible?

Just alter the families, the old store files will get converted during compaction later on. J-D On Wed, May 9, 2012 at 2:06 PM, Otis Gospodnetic wrote: > Hi, > > Based on the example on > http://hbase.apache.org/book/snappy.compression.html and some > search-hadoop.com searches I'm guessing it

Re: Switching existing table to Snappy possible?

h this.  This is a good thing for everybody to >> know. >> >> >> >> >> >> On 5/9/12 5:08 PM, "Jean-Daniel Cryans" wrote: >> >> >Just alter the families, the old store files will get converted during >> >compact

Re: Switching existing table to Snappy possible?

;> >>>  enable 't' >>>> >>> (Make sure to test the table. And do not remove away the old codec >>> immediately. You need to wait until the whole of the table's regions >>> have major compacted, leaving no old-codec-encoded store file traces

Re: Hbase DeleteAll is not working

Please don't cross-post, your question is about HBase not MapReduce itself so I put mapreduce-user@ in BCC. 0.20.3 is, relatively to the age of the project, as old as my grand-mother so you should consider upgrading to 0.90 or 0.92 which are both pretty stable. I'm curious about the shell's behav

Re: Regions can split, but can they combine?

They won't merge automatically. For a manual process see http://hbase.apache.org/book.html#ops.regionmgt.merge J-D On Mon, May 14, 2012 at 12:25 PM, Tom Brown wrote: > I know that regions can split (either manually, or automatically), but > is there any process whereby regions that have previou

Re: HBase mapreduce sink - using a custom TableReducer to pass in Puts

My first guess would be to check if all the KVs using the same qualifier, because then it's basically the same cell 10 times. J-D On Mon, May 14, 2012 at 6:50 PM, Ben Kim wrote: > Hello! > > I'm writing a mapreduce code to read a SequenceFile and write it to hbase > table. > Normally, or what hb

Re: client timeouts after upgrading to 0.92

This means that the servers aren't responding in 60 seconds to the clients, I believe this is new from 0.90 so it could be that you were used to have long-running requests. If not, check what's going on with those servers at the address given in the exception message. J-D On Thu, May 17, 2012 at

Re: Can we store a HBase Result object using Put

How exactly are you building the Put? It doesn't have a constructor that can take byte[] and figure out how it should use it, it only takes a row key (meaning that if you do new Put(Result.getBytes().get()), you're passing the whole thing as a row key which is wrong). In the HBase code we do the R

Re: About HBase Memstore Flushes

On Wed, May 23, 2012 at 2:33 PM, Alex Baranau wrote: > Talked to J-D (and source code). It turned out that > when hbase.regionserver.global.memstore.lowerLimit is reached flushes are > forced without blocking reads (of course, > if hbase.regionserver.global.memstore.upperLimit is not hit). Makes p

Re: Append and Put

It's a facility so that you don't have to read+write in order to add something to a value. With Append the read is done in the region server before the write, also it solves the problem where you could have a race when there are multiple appenders. J-D On Tue, May 22, 2012 at 8:51 PM, NNever wro

Re: Append and Put

On Wed, May 23, 2012 at 8:11 PM, NNever wrote: > Thanks J-D. > > so it means 'Append' keeps write-lock only and 'Put' keeps > write-lock/read-lock both? Yeah... not at all. First, there's no read lock. Then Put is just a Put, it takes a write lock. Append is a read+write operation, still just use

Re: question on filters

What you need is a secondary index and HBase doesn't have that. For some tips see: http://hbase.apache.org/book.html#secondary.indexes J-D On Thu, May 24, 2012 at 5:06 PM, jack chrispoo wrote: > Hi, > > I'm new to HBase and I have a question about using filters. I know that I > can use filters w

Re: improve performance of a MapReduce job with HBase input

TIF should be configured via TableMapReduceUtil.initTableMapperJob which takes a Scan object. J-D On Fri, May 25, 2012 at 11:30 AM, Ey-Chih chow wrote: > Thanks.  Since we use TableInputFormat in our map/reduce job.  The scan > object is created inside TableInputFormat.  Is there any way to get

Re: Problems with scan after lot of Puts

Care to share that TestPutScan? Just attach it in a pastebin Thx, J-D On Tue, May 29, 2012 at 6:13 AM, Ondřej Stašek wrote: > My program writes changes to HBase table by issuing lots of Puts (autoCommit > turned off, flush on end) and afterwards uses ResultScanner on whole table > to read all r

Re: Problems with scan after lot of Puts

1:37 PM, Ondřej Stašek wrote: > Here it is: > > http://pastebin.com/0AgsQjur > > > On 29.5.2012 22:44, Jean-Daniel Cryans wrote: >> >> Care to share that TestPutScan? Just attach it in a pastebin >> >> Thx, >> >> J-D >> >> On Tue, May 29

Re: Problems with scan after lot of Puts

: Expected value: value 001 005, got: value 0496107 005 That's a split so the ClientScanner did a reset on the start row. So I'm going to fix your code and see if I can get anything else. J-D On Wed, May 30, 2012 at 11:56 AM, Jean-Daniel Cryans wrote: > I'm running i

Re: Problems with scan after lot of Puts

NFO hbase.TestPutScan: Run 5 scan + del every 10th row > 12/05/31 10:44:33 ERROR hbase.TestPutScan: Expected value: value 0402040 > 005, got: value 0402041 004 > > It seems that 1 row was skipped during scan. Strange. > > I'll keep testing. > >  Ondrej Stasek &g

Re: hosts unreachables

What I'm seeing looks more like GC issues. Start reading this: http://hbase.apache.org/book.html#gc J-D On Thu, May 31, 2012 at 12:10 PM, Cyril Scetbon wrote: > I'm still having region nodes that crash :( > > You can see reasons on 2 region nodes : > > *node1 :* > > 2012-05-31 17:57:03,937 INFO

Re: hosts unreachables

Both, also you could bigger log snippets (post them on something like pastebin.com) and we could see more evidence of the issue. J-D On Thu, May 31, 2012 at 2:09 PM, Cyril Scetbon wrote: > On 5/31/12 11:00 PM, Jean-Daniel Cryans wrote: >> >> What I'm seeing looks more l

Re: HBase Client API: Why does HTable.get(get).getValue() return null if more than one family is added to the get instance?

See https://issues.apache.org/jira/browse/HBASE-6200 On Mon, Jun 11, 2012 at 11:23 AM, Desert R. wrote: > > I tried with 0.94 now.  I added the htablePool to simulate my other > multi-thread environment. Here's the code: http://pastebin.com/7TBA1XGh > If line 70 is commented it returns not null.

Re: How does scan work internally? Does it make use of multi-threading/replication?

A region is only served by 1 region server, and since HBase uses the HDFS client it doesn't have a view of the blocks layout. HBase currently doesn't even know about replication, it asks to read a file and gets some data coming from somewhere (that somewhere is determined by HDFS). Hope this helps

Re: How does scan work internally? Does it make use of multi-threading/replication?

On Mon, Jun 18, 2012 at 11:34 AM, IGZ Nick wrote: > Hi Jean, > > Thank you for your reply. So RS is a completely different entity when > compared to the datanode? Totally. > How does RS server the data? That's HBase 101, I recommend you read the guide http://hbase.apache.org/book/book.html or t

Re: How does scan work internally? Does it make use of multi-threading/replication?

s regions don't end up > being server by the same regionserver? No, AFAIK there's no reason to do it. J-D > > On Tue, Jun 19, 2012 at 12:11 AM, Jean-Daniel Cryans > wrote: > >> On Mon, Jun 18, 2012 at 11:34 AM, IGZ Nick wrote: >> > Hi Jean, >> > >

Re: RS unresponsive after series of deletes

Mass deleting in HBase is equivalent to mass inserting, it's just that the former doesn't have to write values out (just keys). Almost everything that applies to batch insert tunings applies to batch deleting. Now the error you get comes from this: https://issues.apache.org/jira/browse/HBASE-5190

Re: delete rows from hbase

In order to delete a row in HBase you need to know that it exists, so the way I'd go around this is running a MR job that scans and for each row that matches the filter would emit a Delete. Hope this helps, J-D On Mon, Jun 18, 2012 at 3:08 PM, Oleg Ruchovets wrote: > Hi , > I need to delete row

Re: delete rows from hbase

On Mon, Jun 18, 2012 at 4:13 PM, Oleg Ruchovets wrote: > Ok , I see. Is it possible to do it using one map/reduce job. Map phase >  will scan required rows using filter. Reduce phase use this row and delete > it from the table. > My question is it possible to execute delete using > reducers and no

Re: performance of Get from MR Job

This is a common but hard problem. I do not have a good answer. This issue with doing random reads for each line you are processing is that there's no way to batch them so you're basically doing this: - Open a socket to a region server - Send the request over the network - The region server seeks

Re: base running JVM6 and application server running JVM7

Maybe it's something else? What's the error? Thx, J-D On Tue, Jun 19, 2012 at 12:27 AM, Ben Cuthbert wrote: > All > > We have our application server running in Java 7 and hbase started and > running on Java 6 using the stumbleupon API. When we are trying to connect > from our compiled code on

Re: Increment Counters in HBase during MapReduce

This question was answered here already: http://mail-archives.apache.org/mod_mbox/hbase-user/201101.mbox/%3caanlktinnw2d7dmcyfu3ptv1hu_i3xqk_1pdsgd5nt...@mail.gmail.com%3E Counters are not idempotent, this can be hard to manage. J-D On Mon, Jun 18, 2012 at 5:49 PM, Sid Kumar wrote: > Hi everyon

<    1   2   3   4   5   6   7   8   9   10   >