I have sort of the same problem I feel. I've a large segment I cannot index because there's not enough room, nor -copyToLocal, local being the server. How can I instead -copyToLocal where local is my local terminal machine, instead of the server?
On Tue, Jun 7, 2011 at 9:30 PM, Joe Greenawalt <[email protected]>wrote: > Thanks, and I've seen this example. I think after I can connect, i'm ok, > but i'm not sure how to do it remotely. I'm writing a groovy script just > to > test connection and i'll paste it below so you can see what i'm trying to > do. > > @GrabResolver(name='org.apache.mahout.hadoop', root=' > > http://mymavenrepo/nexus/content/repositories/thirdparty') > > @Grab(group='org.apache.mahout.hadoop', module='hadoop-core', > > version='0.20.203.0') > > import org.apache.hadoop.hdfs.DFSClient > > import org.apache.hadoop.hdfs.DistributedFileSystem > > import org.apache.hadoop.conf.Configuration > > org.apache.hadoop.hdfs.protocol.HdfsFileStatus > > def DIR_HADOOP = "1.1.1.1"; > > def PORT_HADOOP = "9000"; > > def config = new Configuration() > > config.set("fs.default.name", DIR_HADOOP + ":" + PORT_HADOOP) //got > this > > from from some site > > def dfs = new DistributedFileSystem() > > def dfsClient = dfs.getClient() > > def fileInfo = dfsClient.getFileInfo("/DEV") > > println fileInfo.isDir() > > > On Tue, Jun 7, 2011 at 2:11 PM, Habermaas, William < > [email protected]> wrote: > > > Joe, > > > > Take a look at http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample > > > > It should give you an idea of how to read and write HDFS files. This page > > is somewhat old and the package names have changed a bit between versions > > but I hope it will get you on the right track. If you don't want to write > > code there are HDFS copy utilities that you can use instead from shell > > scripts. > > > > Bill > > > > -----Original Message----- > > From: Joe Greenawalt [mailto:[email protected]] > > Sent: Tuesday, June 07, 2011 1:38 PM > > To: [email protected] > > Subject: Re: remotely downloading file > > > > Bill, > > thanks for the reply, is there a resource that you have available that i > > can > > look at the correct way to connect remotely? > > I seem to be seeing conflicting ways on doing that. > > > > I'm looking at: > > > > > http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DFSClient.html > > > > > http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DistributedFileSystem.html > > > > But the examples i'm seeing are using the Configuration but i don't see > > that > > being used in those classes. > > > > Thanks again, > > > > Joe > > > > On Fri, Jun 3, 2011 at 5:05 PM, Habermaas, William < > > [email protected]> wrote: > > > > > You can access HDFS for reading and writing from other machines. The > API > > > works through the HDFS client which can be anywhere on the network and > > not > > > just on the namenode. You just have to have the Hadoop core jar with > your > > > application wherever it is going to run. > > > > > > Bill > > > > > > -----Original Message----- > > > From: Joe Greenawalt [mailto:[email protected]] > > > Sent: Friday, June 03, 2011 4:55 PM > > > To: [email protected] > > > Subject: remotely downloading file > > > > > > Hi, > > > We're interested in using HDFS to store several large file sets to be > > > available for download from our customers in the following paradigm: > > > > > > Customer <- | APPSERVER-CLUSTER {app1,app2,app3} | <- | HDFS | > > > > > > I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER > > > could > > > be done program-ably remotely after browsing the documentation. But > > after > > > reading the API, it seems that writing Java code to interface with HDFS > > > needs to happen locally? Is that correct? > > > > > > If it is correct, what is the best/recommended way to > > > deliver downloadables to the APPSERVERS (and vice versa) which are > hosted > > > in > > > the same network but on different machines? > > > > > > Thanks, > > > Joe > > > > > > -- Regards, K. Gabriele --- unchanged since 20/9/10 --- P.S. If the subject contains "[LON]" or the addressee acknowledges the receipt within 48 hours then I don't resend the email. subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x) < Now + 48h) ⇒ ¬resend(I, this). If an email is sent by a sender that is not a trusted contact or the email does not contain a valid code then the email is not received. A valid code starts with a hyphen and ends with "X". ∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈ L(-[a-z]+[0-9]X)).
