For FileInputFormat, start with "_" is hidden file by default. you can write a custom PathFilter, and pass it to the InputFormat.
On Wed, Apr 3, 2013 at 5:58 PM, Harsh J <[email protected]> wrote: > You've been misled by the GUI you use, I'm afraid. Many DEs (Desktop > Environments) consider ~-suffix files as hidden but not the general > standards (try ls for example, or even shell expansions, it will > ignore . prefixes, but not ~ suffixes) :) > > To answer specifically though, no, the base FileInputFormat does not > recognize ~ today, but if you want it to, you can pass a custom path > filter to your InputFormat's implementation for when it calls the > listStatus method. > > On Wed, Apr 3, 2013 at 3:16 PM, Mohammad Tariq <[email protected]> wrote: > > Hello Harsh, > > > > Thank you for the response. I am sorry for being unclear. > > Actually I was talking about the backup files which end with "~" > > I mean these files are not visible normally, but my job is able to > > see them. Does FileInputFormat behave in the same way for "~" > > as it does in the case of "." and "_"? > > > > Thanks. > > > > Warm Regards, > > Tariq > > https://mtariq.jux.com/ > > cloudfront.blogspot.com > > > > > > On Wed, Apr 3, 2013 at 7:45 AM, Harsh J <[email protected]> wrote: > >> > >> Not quite sure if I got your question. These tidbits may help though, > >> from what I can understand: > >> > >> * LocalFileSystem's listing uses Java's APIs for file/dir listing, and > >> has no concept of what a hidden file is on its own. It retrieves the > >> whole list. > >> * MR's FileInputFormat (and normal derivatives) does filter away "." > >> and "_" starting path names, from added input paths to the job. > >> > >> On Wed, Apr 3, 2013 at 3:09 AM, Mohammad Tariq <[email protected]> > wrote: > >> > > >> > Warm Regards, > >> > Tariq > >> > https://mtariq.jux.com/ > >> > cloudfront.blogspot.com > >> > > >> > > >> > ---------- Forwarded message ---------- > >> > From: Mohammad Tariq <[email protected]> > >> > Date: Tue, Apr 2, 2013 at 5:16 PM > >> > Subject: MapReduce on Local files > >> > To: [email protected] > >> > > >> > > >> > Hello list, > >> > > >> > Is a MR job capable of reading even the hidden temp files > >> > present > >> > inside a directory located on my local FS?I have noticed this thing > >> > today > >> > for the first time because till now I never tried running MR jobs on > >> > local > >> > files. > >> > > >> > Thank you so much for your time? > >> > > >> > Warm Regards, > >> > Tariq > >> > https://mtariq.jux.com/ > >> > cloudfront.blogspot.com > >> > > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > -- > Harsh J >
