You can utilize Scan#setTimeRange() method.
Cheers
On Jul 22, 2014, at 9:56 PM, Sachin wrote:
> Hello,
>
> I want to know how can I retrive rows which are inserted in last 7 days or in
> a perticular time period. I am putting current timestamp in row key while
> inserting data to hbase.
>
Hello,
I want to know how can I retrive rows which are inserted in last 7 days or in
a perticular time period. I am putting current timestamp in row key while
inserting data to hbase.
Hi Varun,
I am still learning HBase here, so the experts can point out the mistakes I
make. Your problem seems to be something which can be easily mapped to a
HBase table structure.
Firstly, never ever store references in HBase. It doesn't serve any purpose
and will just make your queries slower.
That's true. I never thought of it that way. Thanks for pointing it out.
Arun
On Jul 22, 2014 4:07 PM, "Ted Yu" wrote:
> When storing new lists using new columns, similar issue would arise, right
> ?
> In Ishan's words:
>
> bq. read all the columns and combine when reading
>
> The combining proc
You mean the snapshot(s) taken by the 0.94.x release ?
0.98 release is fully compatible in snapshot directory structure with that
of 0.94
Cheers
On Tue, Jul 22, 2014 at 7:01 PM, Qiang Tian wrote:
> what about snapshot?
>
>
> On Wed, Jul 23, 2014 at 7:49 AM, Guangle Fan wrote:
>
> > Hi, Ted yo
what about snapshot?
On Wed, Jul 23, 2014 at 7:49 AM, Guangle Fan wrote:
> Hi, Ted you are right, hfiles on 0.94 cluster are v2 by default. I
> actually don't need the convert tool.
>
> To migrate that data to another 0.96 cluster, does it work by copying over
> hfiles ? How to handle the meta
Migrating to another cluster is for keeping service running non-stop ?
Please read http://hbase.apache.org/book.html#upgrade0.96
Cheers
On Jul 22, 2014, at 3:49 PM, Guangle Fan wrote:
> Hi, Ted you are right, hfiles on 0.94 cluster are v2 by default. I
> actually don't need the convert tool.
I trying to use HBase to model a directory structure. Basically we have a fixed
set of nested directory structure that could store millions of files each. The
directory structure is accessed by users and every user has his/her own set.
Something like
user 1
- dir 1
- fi
Apache HBase 0.98.4 is now available for download. Get it from an Apache
mirror [1] or Maven repository.
The list of changes in this release can be found in the release notes [2]
or following this announcement.
Thanks to all who contributed to this release.
Best,
The HBase Dev Team
1. http://w
Hi, Ted you are right, hfiles on 0.94 cluster are v2 by default. I
actually don't need the convert tool.
To migrate that data to another 0.96 cluster, does it work by copying over
hfiles ? How to handle the meta data of the table in the new cluster ?
On Tue, Jul 22, 2014 at 3:46 PM, Ted Yu wro
Hi guys,
I see in the documentation java Scan api support reverse scan (so I think I
can fetch the latest X row) this feature propagated to the rest api? Can I
call like http:///database/query_*?reversed=true or something like that?
Thanks
b0c1
The subject of this thread says '9.4' - I assume you meant '0.94' release.
If so, please make correction in the subject.
Looking at hbase-default.xml from 0.94 branch:
hfile.format.version
2
The HFile format version to use for new files. Set this to 1 to
test
Ted Thanks for your answer!
Actually that wouldn't convert automatically. The problem is 0.94 is using
hfileV1, and that is online cluster that I can't shut down and do hbase
upgrade/check file, thus I can't convert the file to v2. The idea is copy
over the hfilleV1 to another 0.96 cluster and con
When storing new lists using new columns, similar issue would arise, right ?
In Ishan's words:
bq. read all the columns and combine when reading
The combining process applies to the multi-version approach as well.
Cheers
On Tue, Jul 22, 2014 at 12:32 PM, Arun Allamsetty wrote:
> Hi,
>
> Isn'
Guangle:
For 0.94 cluster, you can perform major compaction after which your HFiles
would be converted to HFileV2.
Cheers
On Tue, Jul 22, 2014 at 1:54 PM, Guangle Fan wrote:
> Thanks Nick!
> For some reason, I can't do a in-place upgrading for the cluster. Is there
> a tool there I can use to
Thanks Nick!
For some reason, I can't do a in-place upgrading for the cluster. Is there
a tool there I can use to convert hfileV1 to hfileV2 ?
On Tue, Jul 22, 2014 at 1:43 PM, Nick Dimiduk wrote:
> Hi Guangle,
>
> Please have a look at the online book, there's a section on upgrades. Also,
> p
Hi Guangle,
Please have a look at the online book, there's a section on upgrades. Also,
please consider upgrading to 0.98. The 0.96 line is in minimum maintenance
mode and 0.98 is considered the stable/production line.
http://hbase.apache.org/book.html#upgrading
Thanks,
Nick
On Tue, Jul 22, 20
We just solved a very similar issue with our cluster (yesterday!). I would
suggest you look at 2 things in particular:
- Is the network on your region server saturated? That would prevent
connections from being made
- See if the region server has any RPC handlers available when you get this
erro
Hi, All
I have hfiles in HBase 9.4 cluster that I want to migrate to HBase 9.6
cluster.
What's the proper process to handle this ?
Regards,
Guangle
Yes, that will work.
On Tue, Jul 22, 2014 at 12:39 PM, Libo Yu wrote:
> My cluster has a standby master process. If I restart it, will it reload
> data from zookeeper? If it does, I can shut down the current master and let
> standby master become the active one. Will that work?
>
> Libo
>
> Fro
My cluster has a standby master process. If I restart it, will it reload data
from zookeeper? If it does, I can shut down the current master and let standby
master become the active one. Will that work?
Libo
From: yu_l...@hotmail.com
To: user@hbase.apache.org
Subject: how to let hmaster read zo
Hi,
Isn't versioning used for an entirely different purpose? What if I screw up
a book name and then have to rewrite it? Then I'll have two versions for
the same book. Also, AFAIK the default number of versions is 1 on table
creation without additional parameters.
Thanks,
Arun
On Jul 22, 2014 12:
Andrew
You are right. These are m1.xlarge instance for our hbasetest cluster. Our
production cluster mostly use i2 instance. I will do some math and config
it accordingly to prevent these problem.
I reduced the tasktracker count and memory used to 1G and I will see
if/when the Java Heap Space fai
Hi,
If a author does not have hundreds of publications, you can directly write
in one column. Hence, your column will contain multiple data versions. The
default data version is 3 but you can send more.
On Tue, Jul 22, 2014 at 4:20 AM, Ishan Chhabra
wrote:
> Arun,
> You need to represent your
> The node has only 15G
memory.
EC2 m1.xlarge or m3.xlarge ? You might find some of the new types with more
memory have better price-performance value. If you are on EC2 and are
colocating mapreduce with HBase, you'll want more RAM *and* vCPU I think.
> But will that cause Java Heap Space
Andrew
Thanks for your answer! I think you are right. The node has only 15G
memory. We configured it to run RS with 12G. And then we configured 4
mapper and 4 reducer on each node, each to use 2G memory. So that probably
caused RS being killed by OOM.
*mapred.child.java.opts*-Xmx2048m
I have anot
Accidentally hit send too soon.
A good rule of thumb is the aggregate of all Java heaps (daemons like
DataNOde, RegionServer, NodeManager, etc. + the max allowed number of
mapreduce jobs * task heap setting) ... should fit into available RAM.
If you don't have enough available RAM, then you n
You need to better manage the colocation of the mapreduce runtime. In other
words, you are allowing mapreduce to grab too many node resources,
resulting in activation of the kernel's OOM killer. A good rule of thumb is
the aggregate of all Java heaps (daemons like DataNOde, RegionServer,
NodeManage
yes, we are using 94.7. The RS log has no error since the system killed it.
This is from the syslog
Jul 22 07:25:12 hbasetest-e-regionserver-684ab93a monit[10188]:
'hbasetest-e-regionserver-684ab93a' cpu user usage of 84.4% matches
resource limit [cpu user usage>70.0%]
Jul 22 07:25:12 hbasetest-e
You can leave your config value there.
Remember to record such change in a place for future reference - you may
change other cost parameter later.
The side-effects of this change partially depend on how you want your
cluster balanced. I suggest you go over the CostFunction's in
StochasticLoadBalan
That did the trick. I set it to 100 and regions are uniform now. Should I leave
it there? What are the side-effects of this change?
Thanks.
Brian
On Jul 22, 2014, at 11:28 AM, Ted Yu wrote:
> Here is code snippet from StochasticLoadBalancer
> w.r.t. TableSkewCostFunction :
>
>private sta
Have you tried including namespace in tablename parameter ?
Cheers
On Mon, Jul 21, 2014 at 11:35 PM, ch huang wrote:
> hi,maillist:
>
> i try import data to 0.96 hbase ,but when i use import tool ,it can
> import data to default namespace, how can i import it to diffent namespace?
>
Can you post region server log snippet prior to (and including) the OOME ?
Are you using 0.94 release ?
Cheers
On Tue, Jul 22, 2014 at 8:15 AM, Tianying Chang wrote:
> Hi
>
> I was running WALPlayer that output HFile for future bulkload. There are
> 6200 hlogs, and the total size is about 400
With 6 +1 votes and no -1 or 0 votes, the vote carries and RC0 will be
released as 0.98.4. I'll send the release announcement out shortly. Thanks
to all who tested the RC!
On Mon, Jul 14, 2014 at 8:38 PM, Andrew Purtell wrote:
> The 1st HBase 0.98.4 release candidate (RC0) is available for down
Here is code snippet from StochasticLoadBalancer
w.r.t. TableSkewCostFunction :
private static final String TABLE_SKEW_COST_KEY =
"hbase.master.balancer.stochastic.tableSkewCost";
private static final float DEFAULT_TABLE_SKEW_COST = 35;
TableSkewCostFunction(Configuration co
Hi
I was running WALPlayer that output HFile for future bulkload. There are
6200 hlogs, and the total size is about 400G.
The mapreduce job finished. But I saw two bad things:
1. More than half of RS died. I checked the syslog, it seems they are
killed by OOM. They also have very high CPU spike f
I don’t understand the logging output, but I do see a strange pattern. I’ll try
to summarize.
There are 5 RegionServers, call them rs1 through rs5. There are a total of 174
regions for the table in question,
with 69 in rs1. In the log output I see lines (greatly simplified) like the
following:
Thanks a lot. I will try it. Can I ask that if you used java to
transfer data before?
It always has the problem which in the following:
Thanks a lot for answering.
From: Shahab Yunus
Date: 2014-07-22 21:12
To: user@hbase.apache.org
Subject: Re: how c
The load balancer in 0.98 considers many factors when making balancing
decisions.
Can you take a look at the master log and look for balancer related lines ?
That would give you some clue.
Cheers
On Jul 22, 2014, at 5:03 AM, Brian Jeltema
wrote:
> I ran the balancer from hbase shell, but d
Can you explain a bit that what issue you are facing?
Sqoop's documentation explains quite clearly how to import data from MySQL
to Hbase. You can use those commands in script to automate the process.
http://sqoop.apache.org/docs/1.4.3/SqoopUserGuide.html#_selecting_the_data_to_import
Regards,
Sh
Ch:
HFileV2 is the file format used by 0.96
It is not a bug w.r.t. HFileV1 support.
Cheers
On Jul 22, 2014, at 1:27 AM, Esteban Gutierrez wrote:
> Hi,
>
> Are you getting any kind of error? even if ImportTsv
> uses HFileOutputFormat.configureIncrementalLoad() method, internally the
> HFileOut
Hi, guys
In the condition that I have built the enviroment of Hadoop, how can
I use sqoop transfer data from mysql.
I try some methods such as using some scripts but it doesn't work.Please
help me if you used it ago.Thaks a lot.
Best Regards
Amanda
+
I ran the balancer from hbase shell, but don’t see any change. Is there a way
to balance a specific table?
> bq. One RegionServer has 69 regions
>
> Can you run load balancer so that your regions are better balanced ?
>
> Cheers
>
>
> On Mon, Jul 21, 2014 at 6:56 AM, Brian Jeltema <
> brian.j
Please also see 17.8.8 under http://hbase.apache.org/book.html#ops.snapshots
Cheers
On Jul 22, 2014, at 1:57 AM, Wellington Chevreuil
wrote:
> I think you need to run a ZK instance apart from HBase. But if your main goal
> is to copy data from one cluster to another, you may use other options
-user@hbase (bcc), +cdh-user
Hello Pavel,
I'm moving your question to the cdh-u...@cloudera.org mailing list since
its more related to specific Hadoop distribution. However from the symptoms
it looks like there is some contention (probably in HDFS or something else)
that is causing the Region Ser
Jobs, running on this cluster, print exceptions:
java.util.concurrent.ExecutionException: java.net.SocketTimeoutException:
Call to ds-hadoop-wk01p.tcsbank.ru/10.218.64.11:60020 failed on socket
timeout exception: java.net.SocketTimeoutException: 6 millis timeout
while waiting for channel to be
I think you need to run a ZK instance apart from HBase. But if your main goal
is to copy data from one cluster to another, you may use other options, such as
CopyTable, Bulkload, or Export/Import tools. Replication will not copy data
already inserted previously on your source Hbase, it only repl
Hi,
Are you getting any kind of error? even if ImportTsv
uses HFileOutputFormat.configureIncrementalLoad() method, internally the
HFileOutputFormat2 is used:
https://github.com/apache/hbase/blob/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java#L360
cheers
hi,maillist:
i view the code of importTsv in hbase 0.96, when it do bulk output
,it still use HFileOutputFormat not HFileOutputFormat2 ,however ,in hbase
0.96 ,it dose not support HFileV1,
Is it a bug of importTsv ?
Hello,
Seems that you have a typo in the command line: -Dmporttsv.separator it
should be -Dimporttsv.separator
cheers,
esteban.
--
Cloudera, Inc.
On Tue, Jul 22, 2014 at 1:01 AM, ch huang wrote:
> hi,maillist:
>
> i test hbase 0.96.1.1 importtsv tool ,find it do not work with non tab
> fi
I don't know. I think the other parameter is more important:
net.core.somaxconn=1024 (original 128)
net.ipv4.tcp_synack_retries=2 (original 5)
Since I found many connections were in SYN_RECV status, my purpose of
changing these 2 parameters are:
net.ipv4.tcp_synack_retries: Reduce the waiting ti
hi,maillist:
i test hbase 0.96.1.1 importtsv tool ,find it do not work with non tab
field separator
# sudo -u hdfs hbase org.apache.hadoop.hbase.mapreduce.ImportTsv
-Dimporttsv.columns=HBASE_ROW_KEY,myco1,mycol2
"-Dmporttsv.separator=|" alex:mymy2 /tmp/alex_test
2014-07-22 15:55:59,746 INFO [m
Thanks for keeping us updated Rural!
I'm still curious why changing net.core.somaxconn in the kernel helped here
if you didn't change ipc.server.listen.queue.size. Perhaps that property is
in hdfs-site.xml or core-site.xml with a higher value?
cheers,
esteban.
--
Cloudera, Inc.
On Mon, Jul 2
53 matches
Mail list logo
