It looks like karmasphere can handle it: INFO com.karmasphere.studio.hadoop.netbeans.filesystem.CommonsVfsBrowserTransferHandler importData: Past filenode: FileNode for file:///Users/simpatico/Documents, state=Updated; type=folder INFO com.karmasphere.studio.hadoop.netbeans.filesystem.FileOperation$Type$2 run: Moving Files: 1 to FileNode for file:///Users/simpatico/Documents, state=Updated; type=folder
On Sun, Jul 31, 2011 at 8:41 PM, Gabriele Kahlout <[email protected]>wrote: > I have sort of the same problem I feel. > I've a large segment I cannot index because there's not enough room, nor > -copyToLocal, local being the server. > How can I instead -copyToLocal where local is my local terminal machine, > instead of the server? > > > On Tue, Jun 7, 2011 at 9:30 PM, Joe Greenawalt > <[email protected]>wrote: > >> Thanks, and I've seen this example. I think after I can connect, i'm ok, >> but i'm not sure how to do it remotely. I'm writing a groovy script just >> to >> test connection and i'll paste it below so you can see what i'm trying to >> do. >> >> @GrabResolver(name='org.apache.mahout.hadoop', root=' >> > http://mymavenrepo/nexus/content/repositories/thirdparty') >> > @Grab(group='org.apache.mahout.hadoop', module='hadoop-core', >> > version='0.20.203.0') >> > import org.apache.hadoop.hdfs.DFSClient >> > import org.apache.hadoop.hdfs.DistributedFileSystem >> > import org.apache.hadoop.conf.Configuration >> > org.apache.hadoop.hdfs.protocol.HdfsFileStatus >> > def DIR_HADOOP = "1.1.1.1"; >> > def PORT_HADOOP = "9000"; >> > def config = new Configuration() >> > config.set("fs.default.name", DIR_HADOOP + ":" + PORT_HADOOP) //got >> this >> > from from some site >> > def dfs = new DistributedFileSystem() >> > def dfsClient = dfs.getClient() >> > def fileInfo = dfsClient.getFileInfo("/DEV") >> > println fileInfo.isDir() >> >> >> On Tue, Jun 7, 2011 at 2:11 PM, Habermaas, William < >> [email protected]> wrote: >> >> > Joe, >> > >> > Take a look at http://wiki.apache.org/hadoop/HadoopDfsReadWriteExample >> > >> > It should give you an idea of how to read and write HDFS files. This >> page >> > is somewhat old and the package names have changed a bit between >> versions >> > but I hope it will get you on the right track. If you don't want to >> write >> > code there are HDFS copy utilities that you can use instead from shell >> > scripts. >> > >> > Bill >> > >> > -----Original Message----- >> > From: Joe Greenawalt [mailto:[email protected]] >> > Sent: Tuesday, June 07, 2011 1:38 PM >> > To: [email protected] >> > Subject: Re: remotely downloading file >> > >> > Bill, >> > thanks for the reply, is there a resource that you have available that i >> > can >> > look at the correct way to connect remotely? >> > I seem to be seeing conflicting ways on doing that. >> > >> > I'm looking at: >> > >> > >> http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DFSClient.html >> > >> > >> http://hadoop.apache.org/hdfs/docs/current/api/org/apache/hadoop/hdfs/DistributedFileSystem.html >> > >> > But the examples i'm seeing are using the Configuration but i don't see >> > that >> > being used in those classes. >> > >> > Thanks again, >> > >> > Joe >> > >> > On Fri, Jun 3, 2011 at 5:05 PM, Habermaas, William < >> > [email protected]> wrote: >> > >> > > You can access HDFS for reading and writing from other machines. The >> API >> > > works through the HDFS client which can be anywhere on the network and >> > not >> > > just on the namenode. You just have to have the Hadoop core jar with >> your >> > > application wherever it is going to run. >> > > >> > > Bill >> > > >> > > -----Original Message----- >> > > From: Joe Greenawalt [mailto:[email protected]] >> > > Sent: Friday, June 03, 2011 4:55 PM >> > > To: [email protected] >> > > Subject: remotely downloading file >> > > >> > > Hi, >> > > We're interested in using HDFS to store several large file sets to be >> > > available for download from our customers in the following paradigm: >> > > >> > > Customer <- | APPSERVER-CLUSTER {app1,app2,app3} | <- | HDFS | >> > > >> > > I had assumed that pulling the file from HDFS to the APPSERVER-CLUSTER >> > > could >> > > be done program-ably remotely after browsing the documentation. But >> > after >> > > reading the API, it seems that writing Java code to interface with >> HDFS >> > > needs to happen locally? Is that correct? >> > > >> > > If it is correct, what is the best/recommended way to >> > > deliver downloadables to the APPSERVERS (and vice versa) which are >> hosted >> > > in >> > > the same network but on different machines? >> > > >> > > Thanks, >> > > Joe >> > > >> > >> > > > > -- > Regards, > K. Gabriele > > --- unchanged since 20/9/10 --- > P.S. If the subject contains "[LON]" or the addressee acknowledges the > receipt within 48 hours then I don't resend the email. > subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ > time(x) < Now + 48h) ⇒ ¬resend(I, this). > > If an email is sent by a sender that is not a trusted contact or the email > does not contain a valid code then the email is not received. A valid code > starts with a hyphen and ends with "X". > ∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈ > L(-[a-z]+[0-9]X)). > > -- Regards, K. Gabriele --- unchanged since 20/9/10 --- P.S. If the subject contains "[LON]" or the addressee acknowledges the receipt within 48 hours then I don't resend the email. subject(this) ∈ L(LON*) ∨ ∃x. (x ∈ MyInbox ∧ Acknowledges(x, this) ∧ time(x) < Now + 48h) ⇒ ¬resend(I, this). If an email is sent by a sender that is not a trusted contact or the email does not contain a valid code then the email is not received. A valid code starts with a hyphen and ends with "X". ∀x. x ∈ MyInbox ⇒ from(x) ∈ MySafeSenderList ∨ (∃y. y ∈ subject(x) ∧ y ∈ L(-[a-z]+[0-9]X)).
