FileSplit clarification

I wanted to get clarification on the start parameter. If I understand correctly, it's the byte offset from the beginning of the file. /** Constructs a split with host information * * @param file the file name * @param start the position of the first byte in the file to process *

Hadoop Custom InputFormat (SequenceFileInputFormat vs FileInputFormat)

I'm working with a single image file that consists of headers and a multitude of different of data segment types (each data segment having its own sub-header that contains meta data). Example file layout: | Header | Seg A-1 Sub-Header | Seg A-1 Data | Seg A-2 SubHdr | Seg A-2 Data | Seg B-1

Re: ext4 on a hadoop cluster datanodes

option heavily tunes down the number of inodes that get created at mkfs time, so you might have to tweak that. I haven't seen any specific problems in my environment that would indicate the options to be bad, horrible, or causing unexpected explosions. Try it out and let us know! Travis -- Travis

Re: decommissioning disks on a data node

/2011/03/hadoop-facter-and-the-puppet-marionette/ http://www.ghostar.org/2013/05/using-cobbler-with-a-fast-file-system-creation-snippet-for-kickstart-post-install/ Cheers, Travis -- Travis Campbell tra...@ghostar.org

Re: decommissioning disks on a data node

those missing blocks are and no longer treat them as under-replicated. Note: You'll still get some re-replication occurring for the blocks that lived on the drive you removed. But it's only a drive's worth of blocks, not a whole datanode. Travis -- Travis Campbell tra...@ghostar.org

Re: ext4 on a hadoop cluster datanodes

On Tue, Oct 7, 2014 at 7:11 PM, Colin Kincaid Williams disc...@uw.edu wrote: Hi Travis, Are you using SSDs or spinning disks in your configuration? We use 7200rpm NLSAS drives from various manufacturers including (primarily) Seagate, Hitachi, and I believe there's a handful of Fujitsu

Re: Hadoop and RAID 5

On Sun, Oct 5, 2014 at 4:17 PM, Ulul had...@ulul.org wrote: Hi Travis Thank you for your detailed answer and for honoring my question with a blog entry :-) No problem. I had been meaning to write something up. Thanks for the prod. :-) I will look into bus quiescing with admins

Re: ext4 on a hadoop cluster datanodes

can only sustain maybe a quarter of that in the same disk configuration. Travis On Mon, Oct 6, 2014 at 4:46 PM, Colin Kincaid Williams disc...@uw.edu wrote: Hi, I'm trying to figure out what are more ideal settings for using ext4 on hadoop cluster datanodes. From the hadoop site its

Re: Hadoop and RAID 5

. We only use RAID1 for the OS drives, simply because we want to reduce the recovery period associated with a system failure. No reason to re-install a system and have to replicate data back onto it if we don't have to. Cheers, Travis -- Travis Campbell tra...@ghostar.org

indexing example

Can anyone point me to an example of indexing data via map/reduce with lucene? I know this is fairly common in the Hadoop universe. Thanks Travis

Re: HBase Stack

I don't think you would want to run all of this on the same machine, especially if your application/ data requirements are fairly large. On Tue, Nov 15, 2011 at 11:27 AM, Em mailformailingli...@yahoo.de wrote: Hello folks, seems like you deal here with HBase-questions. Below you will find

Re: HBase Stack

to commonly asked questions. From: Em mailformailingli...@yahoo.de To: common-user@hadoop.apache.org Sent: Tuesday, November 15, 2011 9:38 AM Subject: Re: HBase Stack Hi Travis, I think I wasn't very clear about my question: If the project grows, you

hadoop cluster on VM's

Is it recommended to install a hadoop cluster on a set of VM's that are all connected to a SAN? Thanks, Travis

Re: hadoop cluster on VM's

some ops documentation then have at it :) Matt -Original Message- From: Travis Camechis [mailto:camec...@gmail.com] Sent: Monday, August 15, 2011 12:45 PM To: common-user@hadoop.apache.org Subject: hadoop cluster on VM's Is it recommended to install a hadoop cluster

Re: hadoop cluster on VM's

documentation then have at it :) Matt -Original Message- From: Travis Camechis [mailto:camec...@gmail.com] Sent: Monday, August 15, 2011 12:45 PM To: common-user@hadoop.apache.org Subject: hadoop cluster on VM's Is it recommended to install a hadoop cluster on a set of VM's

Reduce stuck at 0%

it. Thanks for any help you can provide. Travis

Map-Reduce stuck at 0% map, 0% reduce

if it would help; I just didn't find a good way to format them here. Any help or suggestions would be appreciated. Thanks, Travis

Map-Reduce stuck at 0% map, 0% reduce

config files if it would help; I just didn't find a good way to format them here. Any help or suggestions would be appreciated. Thanks, Travis

Re: Map-Reduce stuck at 0% map, 0% reduce

talking to each other? What should mapred-site.xml look like for a typical cluster? Thanks, Travis On 5/3/2011 11:28 AM, Harsh J wrote: Hello Travis, On Tue, May 3, 2011 at 10:53 PM, Travis Bolingertboli...@gmail.com wrote: 11/05/03 10:44:39 INFO mapred.JobClient: Running job

Re: distributed cache exceeding local.cache.size

On Thu, Mar 31, 2011 at 3:25 PM, Allen Wittenauer a...@apache.org wrote: On Mar 31, 2011, at 11:45 AM, Travis Crawford wrote: Is anyone familiar with how the distributed cache deals when datasets larger than the total cache size are referenced? I've disabled the job that caused

distributed cache exceeding local.cache.size

cache deals when datasets larger than the total cache size are referenced? I've disabled the job that caused this situation but am wondering if I can configure things more defensively. Thanks! Travis

mismatched dfs.data.dir

: + e.getMessage()); } } if (dirs.size() 0) return new DataNode(conf, dirs); LOG.error(All directories in dfs.data.dir are invalid.); return null; } Thoughts? --travis

Re: Unbalanced disks - need to take down whole HDFS?

around, then restart the DataNode. You might be able to just replace the disk and have them unbalanced for a while though. As data is created and deleted things will naturally even out. --Travis Thanks, \EF -- Erik Forsberg forsb...@opera.com Developer, Opera Software - http://www.opera.com/

Python HDFS library

for working with files, and a filesystem class for interacting with non-file data. I don't recommend using this in production yet, and not all methods have been wrapped, but its usable and far enough along to demonstrate this approach. Thoughts/suggestions? --travis

Re: full disk woes

On Wed, Jul 21, 2010 at 2:01 PM, Allen Wittenauer awittena...@linkedin.com wrote: On Jul 21, 2010, at 12:45 PM, Travis Crawford wrote: Does anyone else run into machines with overfull disks? It was a common problem when I was at Yahoo!.  As the drives get more full, the NN starts getting

Re: Logger hierarchies in ZK?

://github.com/twitter/twitcher Short version is a single daemon sets your watches and triggers local actions when stuff happens. If your app doesn't speak ZK this might be a good solution. --travis On Tue, Jul 20, 2010 at 5:38 PM, ewhau...@gmail.com wrote: Has anyone experimented with storing

Re: zookeeper crash

. Or if the patch appears good I can make a new release and help test. Let me know :) --travis On Wed, Jun 16, 2010 at 3:25 PM, Flavio Junqueira f...@yahoo-inc.com wrote: I would recommend opening a separate jira issue. I'm not convinced the issues are the same, so I'd rather keep them separate

Zookeeper outage recap questions

is roughly scheduled to ship? Thanks! Travis

Re: Zookeeper outage recap questions

I've moved this thread to: https://issues.apache.org/jira/browse/ZOOKEEPER-801 --travis On Thu, Jul 1, 2010 at 12:37 AM, Patrick Hunt ph...@apache.org wrote: Hi Travis, as Flavio suggested would be great to get the logs. A few questions: 1) how did you eventually recover, restart

Re: ZKClient

On Tue, May 4, 2010 at 3:45 PM, Ted Dunning ted.dunn...@gmail.com wrote: Travis, Attachments are stripped from this mailing list.  Can you file a JIRA and put your attachment on that instead? Here is a link to get you started: https://issues.apache.org/jira/browse/ZOOKEEPER Whoops. Filed

Re: Misbehaving zk servers

. Basically the high/low watermark thing. Thoughts? --travis Patrick On 04/29/2010 09:34 AM, Mahadev Konar wrote: Hi Travis,  How many clients did you have connected to this server? Usually the default is 8K file descriptors. Did you have clients more than that? Also, if clients

Re: python client structure

On Wed, Apr 21, 2010 at 12:26 AM, Henry Robinson he...@cloudera.com wrote: Hi Travis - Great to see zkpython getting used. I'm glad you're finding the problems with the documentation - please do file JIRAs with anything you'd like to see improved (and I know there's a lot to improve

Re: Recovery issue - how to debug?

On Mon, Apr 19, 2010 at 2:15 PM, Ted Dunning ted.dunn...@gmail.com wrote: Can you attach the screen shot to the JIRA issue?  The mailing list strips these things. Oops. Updated jira: https://issues.apache.org/jira/browse/ZOOKEEPER-744 --travis On Mon, Apr 19, 2010 at 1:18 PM, Travis

Re: monitoring zookeeper

as capacity planning. --travis On Wed, Apr 14, 2010 at 4:50 PM, kishore g g.kish...@gmail.com wrote: Hi Travis, We do monitor zookeeper using JMX. We have a simple code which does the following - parse JMX output and convert the output into key value format. The nested properties

Re: a pig screencast on the basics

Agreed. Thanks for posting the video Matt! --travis On Sun, Jan 17, 2010 at 11:39 AM, Theo Hultberg t...@iconara.net wrote: really good, thanks! I agree that the foreach with nested operations is a tricky one to get your head around, but you make a really good job of explaining it. T

cast to tuple errors

the script to complete (although with not very useful output). Any thoughts on what the problem might be? Thanks! Travis

RE: Is the thrift server a likely bottleneck?

a thought, Hope this helps, Travis Hegner http://www.travishegner.com/ -Original Message- From: jdcry...@gmail.com [mailto:jdcry...@gmail.com] On Behalf Of Jean-Daniel Cryans Sent: Thursday, September 03, 2009 7:24 AM To: hbase-user@hadoop.apache.org; s...@defuze.org Subject: Re

RE: HBase data model question

What about a document-id, word-position, and word combination. With the proper combo all words in a single document would be located near each-other. Travis Hegner http://www.travishegner.com/ -Original Message- From: llpind [mailto:sonny_h...@hotmail.com] Sent: Monday, August 24, 2009

RE: HBase data model question

to retrieve the surrounding words of a given document ID, and word position. Just a thought, hope this helps. Travis Hegner http://www.travishegner.com/ -Original Message- From: llpind [mailto:sonny_h...@hotmail.com] Sent: Monday, August 24, 2009 2:37 PM To: hbase-user@hadoop.apache.org

RE: HBase data model question

. There are some indexing tools available to automate some of this for Hbase, but I'm not very versed in how to use them. I believe one is call IndexTable. Maybe someone with more experience there could jump in and offer a possible solution? Travis Hegner http://www.travishegner.com/ -Original

RE: Indexed Table in Hbase

with the tableindexed.IndexKeyGenerator that Gary mentioned. Thanks, Travis Hegner http://www.travishegner.com/ -Original Message- From: bharath vissapragada [mailto:bharathvissapragada1...@gmail.com] Sent: Monday, August 17, 2009 1:46 PM To: hbase-user@hadoop.apache.org Subject: Re

Re: Pass a Delete or a Put

some more new hbasers/hadoopers down the line. Travis Hegner http://www.travishegner.com/ -Original Message- From: Travis Hegner theg...@trilliumit.commailto:travis%20hegner%20%3ctheg...@trilliumit.com%3e Reply-to: hbase-user@hadoop.apache.org hbase-user@hadoop.apache.org, Hegner

Re: Pass a Delete or a Put

, and not having my reduce() called. I don't know if that tid bit aided in my solution, but since it worked I left it. Thanks, Travis Hegner http://www.travishegner.com/ -Original Message- From: Andrew Purtell apurt...@apache.orgmailto:andrew%20purtell%20%3capurt...@apache.org%3e

Re: Pass a Delete or a Put

on with the TableMapper.Context and TableReducer.Context, but whatever is calling TableOutputFormat$RecordWriter.write(Key, Value), is calling it with my MAP class output, instead of my REDUCE class output. Anything else I can check? Thanks, Travis Hegner http://www.travishegner.com/ -Original

Pass a Delete or a Put

()? I have also tried casting the value with: context.write(key, (Put)thisput); Any Ideas? Travis Hegner http://www.travishegner.com/ The information contained in this communication is confidential and is intended only for the use of the named recipient. Unauthorized use, disclosure

javadoc

Hi All, Forgive the simple question, as I am pretty new to java development, but what is the recommended way to execute javadoc from the trunk root? Is there a pre-written script to specify the overview and source tree? Thanks for any help, Travis Hegner http://www.travishegner.com

File System Upgrade...

, because it wants the file system to be upgraded. Is there any way around this chicken/egg problem? Thanks, Travis Hegner http://www.travishegner.com/ The information contained in this communication is confidential and is intended only for the use of the named

Re: File System Upgrade...

version? A script I should run or something to bring anything else up to date? Thanks for the tip, Travis Hegner http://www.travishegner.com/ -Original Message- From: stack st...@duboce.netmailto:stack%20%3cst...@duboce.net%3e To: hbase-user@hadoop.apache.org hbase-user

Re: TSocket: timed out reading 4 bytes from

: 96.01 kbytes Thanks again, Travis -Original Message- From: stack st...@duboce.netmailto:stack%20%3cst...@duboce.net%3e Reply-to: hbase-user@hadoop.apache.org hbase-user@hadoop.apache.org To: hbase-user@hadoop.apache.org hbase-user@hadoop.apache.orgmailto:%22hbase-u...@hadoop.apache.org%22

RE: TSocket: timed out reading 4 bytes from

very hard either). Thanks again, Travis Hegner http://www.travishegner.com/ From: saint@gmail.com [saint@gmail.com] On Behalf Of stack [st...@duboce.net] Sent: Thursday, July 16, 2009 1:10 PM To: hbase-user@hadoop.apache.org; Hegner, Travis Subject

Re: TSocket: timed out reading 4 bytes from

this same sequence of exceptions upon the initial attempt to put data. I made it through about 167 20 row puts before it did a region split and crashed with the attached exceptions. I am happy to provide anything else I can to assist in troubleshooting. Thanks, Travis Hegner http

TSocket: timed out reading 4 bytes from

even times out. Yet it still times out even though I can scan and count the table without issue. Any input or info is greatly appreciated. Thanks, Travis Hegner http://www.travishegner.com/

Re: TSocket: timed out reading 4 bytes from

, writing my first map-reduce jobs, then utilizing hbase as an input and output for those... Any recommended tutorials for that? Thanks again, Travis Hegner http://www.travishegner.com/ -Original Message- From: Hegner, Travis theg...@trilliumit.com Reply-to: hbase-user@hadoop.apache.org

Re: TSocket: timed out reading 4 bytes from

as 30 still having the same problem. The tutorial I followed to get the thrift client up and running was perhaps a little dated, so I will make sure my thrift client code is up to date. Any other suggestions? Travis Hegner http://www.travishegner.com/ -Original Message- From: Travis

Re: TSocket: timed out reading 4 bytes from

for the help, Travis Hegner http://www.travishegner.com/ -Original Message- From: Jonathan Gray jl...@streamy.com Reply-to: hbase-user@hadoop.apache.org hbase-user@hadoop.apache.org To: hbase-user@hadoop.apache.org hbase-user@hadoop.apache.org Subject: Re: TSocket: timed out reading 4 bytes

RE: how safe table permanently in hbase

'. Obviously, without the double quotes, and replace myTable with your own table name. Travis -Original Message- From: Muhammad Mudassar [mailto:mudassa...@gmail.com] Sent: Wednesday, July 08, 2009 9:34 AM To: hbase-user@hadoop.apache.org Subject: how safe table permanently in hbase Hi I m

Re: Hadoop streaming performance problem

in Hadoop streaming? Thanks, Lin -- Theodore Van Rooy http://greentheo.scroggles.com -- Travis Brady www.mochiads.com

Re: Possible memory leak in MapTask$MapOutputBuffer

any other way. My heap is more than large enough to support two 100M buffers. FYI, I added code to clear the comparator's buffer (see patch below), and a job that was failing with 650M heaps now succeeds with 512M. Travis Index: src/java/org/apache/hadoop/io/WritableComparator.java