(via Tom White)
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
--
Todd
, where you need to think about
consequences over the system as a whole, and questions that go deeper in
its architecture. It's a well elaborated exam indeed.
Thanks again!
Cristofer
--
Todd Lipcon
Software Engineer, Cloudera
On Fri, Apr 13, 2012 at 9:06 PM, Stack st...@duboce.net wrote:
On Fri, Apr 13, 2012 at 8:02 PM, Todd Lipcon t...@cloudera.com wrote:
If you want to patch on the HBase side, you can edit HLog.java to
remove the checks for the sync method, and have it only call
hflush. It's only
.
On Thu, Apr 5, 2012 at 8:16 PM, Todd Lipcon t...@cloudera.com wrote:
Hi Placido,
Sounds like it might be related to HDFS-2379. Try updating to Hadoop
1.0.1 or CDH3u3 and you'll get a fix for that.
You can verify by grepping for BlockReport in your DN logs - if the
pauses on the hbase side
(they last up to 30 seconds on bursts of up to 3 minutes). Does anyone
have a clue as to what could be happening?
Thanks.
--
Todd Lipcon
Software Engineer, Cloudera
these two, got rid of all of the 20 and 40 ms response
times and dropped the average response time we measured from HBase by
more than half. Plus, we can push HBase a lot harder.
That had an effect on random read workload only Ron?
Thanks,
St.Ack
--
Todd Lipcon
Software Engineer
.
--
Todd Lipcon
Software Engineer, Cloudera
disk and network
have plenty of unused bandwidth. How could I diagnose where the read
bottleneck is?
Any help would be greatly appreciated :)
Thanks in advance!
--
Bharath Ravi
--
Bharath Ravi
--
Todd Lipcon
Software Engineer, Cloudera
could I diagnose where the read
bottleneck is?
Any help would be greatly appreciated :)
Thanks in advance!
--
Bharath Ravi
--
Todd Lipcon
Software Engineer, Cloudera
and dry (we could have done better messaging around these contribs it
seems) -- but nonetheless a timely reminder from the lads over at
Traackr (sorry to see you go George and crew).
St.Ack
--
Todd Lipcon
Software Engineer, Cloudera
to
be consistent even for the high rate counters. I am wondering if there is a
buffering issue with the htable scanner object if it is unable to obtain a
lock on the row it will default to the data on disk?
--
Todd Lipcon
Software Engineer, Cloudera
accesses.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
from.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
:
Andrey:
A quick search led me to https://github.com/toddlipcon where you would be
able to find:
https://github.com/toddlipcon/hadoop-common
Cheers
On Sat, Dec 24, 2011 at 12:39 PM, Andrey Stepachev oct...@gmail.com
wrote:
23 декабря 2011 г. 22:48 пользователь Todd Lipcon t...@cloudera.com
early next year.
-Todd
20 декабря 2011 г. 11:46 пользователь Todd Lipcon t...@cloudera.comнаписал:
Hi Andrey,
I don't think anyone has ever run Ceph in a production use case, much
less run it in a production use case with HBase.
The SPOF problem is currently being addressed
tasks?
--
Andrey.
--
Todd Lipcon
Software Engineer, Cloudera
in consistency and that it gives up
parition tolerance? My understanding is that failure of one data node
still doesn't impact client as they would re-adjust the list of
available data nodes.
--
Todd Lipcon
Software Engineer, Cloudera
the cloudera/patches/
directory.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
, the performance shall be quite similar.
Can anyone explain the 40X slowdown?
Thanks
Weihua
--
Todd Lipcon
Software Engineer, Cloudera
https://github.com/sonalgoyal/crux
Nube Technologies http://www.nubetech.co
http://in.linkedin.com/in/sonalgoyal
--
Todd Lipcon
Software Engineer, Cloudera
I haven't gotten many responses so far. If there doesn't seem to be
much interest, I may not spend the time to organize.
If you're feeling too busy to answer the full survey, feel free to
just reply with a +1 so I know there's some interest!
-Todd
On Fri, Aug 26, 2011 at 3:33 PM, Todd Lipcon t
questions.
Thanks! Looking forward to seeing everyone, and meeting new folks there.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
as well as Linux's
behavior with all local filesystems I'm aware of. As far as I know
it's the case with most NFS filers as well. Hence the existence of
explicit fencing commands accessible via remote administrative
interfaces on NetApps, for example.
-Todd
--
Todd Lipcon
Software Engineer
that at the intersection of outputs A B will be the values
from B.
Thanks,
Jacques
--
Todd Lipcon
Software Engineer, Cloudera
?
Nope, we run it here on our dev cluster and didn't encounter any issue
(with the code or node failure).
Thanks again for your help. I'm following up with the MapR guys as
well.
Good idea!
J-D
--
Have a Nice Day!
Lohit
--
Todd Lipcon
Software Engineer, Cloudera
-- i.e. I
wouldn't bring it up on the public mailing list.
That seems to be important information for not just the original poster but
others who may have the same problem.
What is the consensus on that?
On Mon, Jul 25, 2011 at 11:45 AM, Todd Lipcon t...@cloudera.com wrote:
Another answer
could recommend that, too. I
wouldn't have a problem with any of the above.
-Todd
On Mon, Jul 25, 2011 at 1:00 PM, Todd Lipcon t...@cloudera.com wrote:
On Mon, Jul 25, 2011 at 11:55 AM, Ted Dunning tdunn...@maprtech.com
wrote:
Todd,
Good to have you weigh in on this. You provide
-
From: Todd Lipcon t...@cloudera.com
To: user@hbase.apache.org
Cc:
Sent: Monday, July 25, 2011 11:45 AM
Subject: Re: Monitoring
On Mon, Jul 25, 2011 at 11:28 AM, Ted Dunning tdunn...@maprtech.com
wrote:
I am very sympathetic here. Also, somewhat linguistically challenged
In that, 't1', 'f1' is table and family in HBase.
No data write in 'output' folder.
Could you give me some advice?
Thank you in advance.
On Tue, Jun 14, 2011 at 10:44 AM, Todd Lipcon t...@cloudera.com wrote:
On Mon, Jun 13, 2011 at 8:17 PM, King JKing beuk...@gmail.com wrote:
Dear all
Program.
--
Prashant Sharma
Development Engineer
Pramati Technologies
Begumpet
Hare Krishna
--
Todd Lipcon
Software Engineer, Cloudera
completebulkload tool
Sounds about right. I don't know what the .dbf format is, but if you can
make an InputFormat that supports them, you can write a mapper to translate
from those records into HBase Puts, and then use HFileOutputFormat and bulk
loads just like ImportTsv.
-Todd
--
Todd Lipcon
/property
property
namehbase.zookeeper.quorum/name
valueserver1,serve3,server5/value
/property
cheers
Byambajargal
--
Todd Lipcon
Software Engineer, Cloudera
experience with JVMs up on
CentOS 5.5.
St.Ack
--
Todd Lipcon
Software Engineer, Cloudera
On Wed, May 25, 2011 at 1:42 PM, Todd Lipcon t...@cloudera.com wrote:
Hi Wayne,
Looks like your RAM might be oversubscribed. Could you paste your
hbase-site.xml and hbase-env.sh files? Also looks like you have some
strange GC settings on (eg perm gen collection which we don't really
need
by the
concurrent collector are run on machines with small numbers of
processors (e.g., 1 or 2). [See
http://www.oracle.com/technetwork/java/javase/gc-tuning-6-140523.html#icms
]
Don't you have 2 processors per machine?
St.Ack
--
Todd Lipcon
Software Engineer, Cloudera
in
error, please notify us immediately by e-mail and delete the message and
any
attachments from your system.
--
Todd Lipcon
Software Engineer, Cloudera
by splitting regions and distributing them around multiple
servers. If you have one super-hot row, it should fit in cache and give you
some 20k reads/second. If you need more reads/sec on a single row than
that, you'll need to add your own caching layer in front.
-Todd
--
Todd Lipcon
Software Engineer
issue is that fsreads for RS get really slow when that happens).
-Jack
--
Todd Lipcon
Software Engineer, Cloudera
wrote:
my yourkit version expired :)... but here is the jstack when it
happens: http://pastebin.com/5v6mHg3t
On Mon, May 2, 2011 at 1:00 PM, Todd Lipcon t...@cloudera.com wrote:
On Mon, May 2, 2011 at 12:56 PM, Jack Levin magn...@gmail.com wrote:
Tried removing yourkit and run on javasun
prefer to have quicker access to
the
data in HBase.
- Adam
--
Todd Lipcon
Software Engineer, Cloudera
in the doc (I'll do the rest).
Thanks,
St.Ack
Read more...
--
Todd Lipcon
Software Engineer, Cloudera
.
The importtsv program doesn't appear to be picking up mapred-site.xml, then.
Are you sure it's valid XML? You can try xmllint to verify. Perhaps attach
it here?
-Todd
Do you have suggestions for any other features/options that I should check?
--- On Mon, 4/25/11, Todd Lipcon t...@cloudera.com wrote
: map 0% reduce 0%
Apr 22, 2011 9:35:41 AM org.apache.hadoop.mapred.JobClient
monitorAndPrintJob
INFO: Job complete: job_local_0001
Apr 22, 2011 9:35:41 AM org.apache.hadoop.mapred.Counters log
INFO: Counters: 0
Thanks,
Eric
--
Todd Lipcon
Software Engineer, Cloudera
continue to cherry-pick bug
fixes back into the CDH3 version of HBase for our quarterly update
releases (CDH3u1, CDH3u2, etc).
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
...@1and1.ro
IM:diancu@united.domain
--
Todd Lipcon
Software Engineer, Cloudera
, Xiyun
--
Todd Lipcon
Software Engineer, Cloudera
-tp31329123p31329123.html
Sent from the HBase User mailing list archive at Nabble.com.
--
Todd Lipcon
Software Engineer, Cloudera
)
at java.lang.Thread.run(Thread.java:680)
Just want to make sure I'm not missing something before I start
re-compiling
those libs.
-GS
--
Todd Lipcon
Software Engineer, Cloudera
the task
runs?
St.Ack
--
Todd Lipcon
Software Engineer, Cloudera
(Is 10 the default connections
for zk?). Is it possible its not reading hbase-site.xml when the task
runs?
St.Ack
--
Todd Lipcon
Software Engineer, Cloudera
to come up with a
workaround against zookeeper connection leaks in HBase MR jobs.
thank you very much.
-Dmitriy
--
Todd Lipcon
Software Engineer, Cloudera
now the hadoop.tmp.dir
cannot be written.
-geoff
--
Todd Lipcon
Software Engineer, Cloudera
to iptables, what is the
best way to disable?
Generally haven't tuned tcp cookies. For iptables, it's usually
enabled by some script in /etc/init.d - you can use chkconfig or
update-rc.d to disable depending on OS.
-Todd
-Jack
On Sat, Mar 12, 2011 at 8:24 PM, Todd Lipcon t...@cloudera.com
and all regionservers went down, is this xciever issue?
property
namedfs.datanode.max.xcievers/name
value12047/value
/property
this is what I have, should I set it higher?
-Jack
--
Todd Lipcon
Software Engineer, Cloudera
)
--
Todd Lipcon
Software Engineer, Cloudera
Grey's recommendation.
So there isn't a penalty incurred from increasing the max region size
from 256MB to 5GB?
--
Todd Lipcon
Software Engineer, Cloudera
The Apache HBase team is happy to announce the general availability of HBase
0.90.1, available from your Apache mirror of choice:
http://www.apache.org/dyn/closer.cgi/hbase/
[at the time of this writing, not all mirrors have updated yet -- please
pick a different mirror if your first choice does
in java.library.path
My questions are
1. Are the two items I was given compatible?
2. If so, what could I be doing wrong?
Thanks
-Pete
--
Todd Lipcon
Software Engineer, Cloudera
Thanks,
Charan
On Thu, Feb 3, 2011 at 10:22 PM, Todd Lipcon t...@cloudera.com wrote:
Does the wiki really recommend that? Got a link handy?
On Thu, Feb 3, 2011 at 10:20 PM, charan kumar charan.ku...@gmail.com
wrote:
Todd,
That did the trick. I think the wiki should
(HRegionServer.java:1765)
...
5 more
--
Todd Lipcon
Software Engineer, Cloudera
:06 PM, Charan K charan.ku...@gmail.com wrote:
Thanks Todd.. I will try it out ..
On Feb 3, 2011, at 1:43 PM, Todd Lipcon t...@cloudera.com wrote:
Hi Charan,
Your GC settings are way off - 6m newsize will promote way too much to
the
oldgen.
Try this:
-XX
it somewhere I
can pull? I'll try it over here.
St.Ack
--
Todd Lipcon
Software Engineer, Cloudera
] :: resolving dependencies ::
com.hadoop.gplcompression#Hadoop-GPL-Compression;working@caiss01a
[ivy:resolve] confs: [common]
Would anyone have a pre-build 32bit LZO jar available?
Thanks
-Pete
--
Todd Lipcon
Software Engineer, Cloudera
very frequently?
Thanks in advance for any ideas of what could be causing
this.
--
Todd Lipcon
Software Engineer, Cloudera
. This is important for us because we don't want to have a burst
increase in our disk usage.
Thanks and regards,
- Ashish
--
Todd Lipcon
Software Engineer, Cloudera
on any Sun or
OpenJDK as far as I know.
(Probably the wrong list for this, but: is there a officially supported JVM
version for CDH3?)
We recommend the Sun 1.6 =u16 but not u18
-Todd
On 13 jan 2011, at 07:42, Todd Lipcon wrote:
On Wed, Jan 12, 2011 at 5:01 PM, Tatsuya Kawano tatsuya6
to the appropriate location, but the server
assignment did not happen until I restarted HBase. Is this a consequence of
the master rewrite?
V
--
Todd Lipcon
Software Engineer, Cloudera
to deal with ZK or META manually in any
way, and if any of that stuff changes you'll be using a supported public
API.
-Todd
On 1/13/11 10:30 AM, Todd Lipcon t...@cloudera.com wrote:
Hey Vidhya,
Nicolas actually did the multi-column-family patch for trunk a few weeks
ago, so no need to upload
fine. The problem
is HBase LZO compression. The region server process is the one with the
memory leak...
(Sorry for dumb question Friso) But HBase is leaking because we make
use of the Compression API in a manner that produces leaks?
Thanks,
St.Ack
--
Todd Lipcon
Software Engineer
Can someone who is having this issue try checking out the following git
branch and rebuilding LZO?
https://github.com/toddlipcon/hadoop-lzo/tree/realloc
This definitely stems one leak of a 64KB directbuffer on every reinit.
-Todd
On Wed, Jan 12, 2011 at 2:12 PM, Todd Lipcon t...@cloudera.com
column has 10 values
at
the
bottom, we would end up scanning and filtering 0 records
and
throwing them away, which seems inefficient.
Blooms+filters?
St.Ack
--
Todd Lipcon
Software Engineer, Cloudera
, compactions and splits are quite common,
though,
because we do read-modify-write cycles a lot. Anyone else doing
update
heavy work with LZO?
Cheers,
Friso
On 4 jan 2011, at 01:54, Todd Lipcon wrote:
Fishy. Are your cells particularly large? Or have you tuned the
HFile
simply missing?
Thanks.
-Jack
--
Todd Lipcon
Software Engineer, Cloudera
directory has a only a _temporary directory
with my outputs in each subdirectory. That doesn't seem be be what the
loadtable script expects (a column family directory with HFiles). Can
someone tell me if what I am doing makes sense in general or how to do
this properly? Thanks!
--
Todd Lipcon
large enough to make do? Having
short lived direct byte buffers is apparently a discouraged practice. If
anyone can provide some pointers on what to look out for, I could invest
some time in creating a patch.
Thanks,
Friso
--
Todd Lipcon
Software Engineer, Cloudera
pauses are acceptable, go to 128m or larger. You can also tune SurvivorRatio
down and use a larger new size for some workloads, but it's a little messy
to figure this out.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
ideas?
Friso
On 3 jan 2011, at 19:18, Todd Lipcon wrote:
Hi Friso,
Which OS are you running? Particularly, which version of glibc?
Can you try running with the environment variable MALLOC_ARENA_MAX=1 set?
Thanks
-Todd
On Mon, Jan 3, 2011 at 8:15 AM, Friso van Vollenhoven
why it wouldn't work for any case? Is
there
any architectural change that is necessary for these to work well that is
not present in 0.20.6?
Thanks,
Hari
--
Todd Lipcon
Software Engineer, Cloudera
swap space: 8 050 880 kbytes
--
Todd Lipcon
Software Engineer, Cloudera
FileOutputFormat.setCompressOutput/setOutputCompressorClass() on
the
Job; in order to turn on compression.
Does that sound right?
Marc
On Thu, Dec 23, 2010 at 2:34 PM, Todd Lipcon t...@cloudera.com wrote:
You beat me to it, Lars! Was writing a response when some family arrived
for
the holidays
publicity and even
help drive candidates if you're recruiting for teams that use HBase!
The page is here:
http://wiki.apache.org/hadoop/Hbase/PoweredBy
Thanks. and happy holidays!
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
[kswapd0]
That server is dual-core, and great for natural region serving, but
meta interaction implies a lot of fast transactions, which would be
better served on 8 Core Box.
-Jack
--
Todd Lipcon
Software Engineer, Cloudera
,1292936189782.714fc4e266abca11f578fd90a3561337.
--
Todd Lipcon
Software Engineer, Cloudera
to the local cluster first.
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
.
- Piet Hein (via Tom White)
--
Todd Lipcon
Software Engineer, Cloudera
and clarification. I guess GC does blow (especially
when writing things like databases and filesystems).
Right now I will dive into GC tuning once more and probably lower the number
of reducers on the insert jobs.
Thanks,
Friso
--
Todd Lipcon
Software Engineer, Cloudera
with such number of rows are used with HBase?
Thanks
Alexey
--
View this message in context:
http://old.nabble.com/HBase-with-trillions-rows-tp30482335p30482335.html
Sent from the HBase User mailing list archive at Nabble.com.
--
Todd Lipcon
Software Engineer, Cloudera
, memstoreSize=33,
compactionQueueSize=15,
usedHeap=856, maxHeap=2198, blockCacheSize=366779472,
blockCacheFree=87883088, blockCacheCount=5494,
blockCacheHitRatio=0
2010-12-15 20:01:20,571 INFO
org.apache.hadoop.ipc.HBaseServer:
Stopping server on 60020
Thanks,
Sandy
--
Todd
!
-Original Message-
From: Todd Lipcon [mailto:t...@cloudera.com]
Sent: Friday, December 17, 2010 13:54
To: user@hbase.apache.org
Subject: Re: Simple OOM crash?
Hi Sandy,
I've seen that error on github as well. Try using the git:// URL instead of
the http:// URL. The http transport
24
hours later. They have a info:regioninfo column but no info:server. Is
there a way to force assignment of these regions?
--
Todd Lipcon
Software Engineer, Cloudera
? Is it stable? Is it just us and the way
we set it up?
Also, would upgrading to 0.89 (cdh3b3) help?
Thanks,
Anze
--
Todd Lipcon
Software Engineer, Cloudera
@ SU
- Matthieu Lieber @ Datameer (building a connector)
- Lars George - at Cloudera - writing HBase book
- Gary Helmling @ Trend Micro - been working on coprocessors
- Mingjie Lai @ Trend Micro
- Todd Lipcon @ Cloudera
- Joshua Ho @ Trend Micro
- Eugene Koontz @ Trend Micro - working on ZK
recipient, you are hereby notified that any dissemination of this
communication is strictly prohibited. If you have received this communication
in
error, please notify us immediately by e-mail and delete the message and any
attachments from your system.
--
Todd Lipcon
Software Engineer
and use that for Start/Stop.
Thanks
-Pete
--
Todd Lipcon
Software Engineer, Cloudera
www.tis.bz.it.
--
Todd Lipcon
Software Engineer, Cloudera
has. Thanks so much!
-Sean
--
Todd Lipcon
Software Engineer, Cloudera
release right now. And
there is also the version that SU provides on githug, which
has the advantage of being heavily used in a production
environment by people who know what they're doing.
Any advise on this anyone?
Thanks,
Friso
--
Todd Lipcon
On Mon, Nov 22, 2010 at 3:44 PM, Andrew Purtell apurt...@apache.org wrote:
On Mon, 11/22/10, Todd Lipcon t...@cloudera.com wrote:
Once 0.90 is released, we plan on spending a week or two to suss
out any possible integration issues, and then release CDH3b4
including 0.90.
I'm sure
(starting with 5).
We already read
http://www.cloudera.com/blog/2010/03/clouderas-support-team-shares-some-basic-hardware-recommendations/
.
We would appreciate your feedback on our proposed configuration.
Regards Oleg Lior
--
Todd Lipcon
Software Engineer, Cloudera
supported that option, but
in HBase 0.20.0 the new HFile format was developed which is always block
based.
Of course nothing stops you from storing compressed records if you see fit!
HBase does fine with binary values.
Thanks
-Todd
--
Todd Lipcon
Software Engineer, Cloudera
1 - 100 of 153 matches
Mail list logo