Hi Nicholas, month= is for Hive to auto discover the partitions. It's part of the url of my files.
Jianshi On Wed, Jun 18, 2014 at 11:52 PM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > Is that month= syntax something special, or do your files actually have > that string as part of their name? > > > > On Wed, Jun 18, 2014 at 2:25 AM, Jianshi Huang <jianshi.hu...@gmail.com> > wrote: > >> Hi all, >> >> Thanks for the reply. I'm using parquetFile as input, is that a problem? >> In hadoop fs -ls, the path (hdfs://domain/user/ >> jianshuang/data/parquet/table/month=2014*) will get list all the files. >> >> I'll test it again. >> >> Jianshi >> >> >> On Wed, Jun 18, 2014 at 2:23 PM, Jianshi Huang <jianshi.hu...@gmail.com> >> wrote: >> >>> Hi Andrew, >>> >>> Strangely in my spark (1.0.0 compiled against hadoop 2.4.0) log, it says >>> file not found. I'll try again. >>> >>> Jianshi >>> >>> >>> On Wed, Jun 18, 2014 at 12:36 PM, Andrew Ash <and...@andrewash.com> >>> wrote: >>> >>>> In Spark you can use the normal globs supported by Hadoop's FileSystem, >>>> which are documented here: >>>> http://hadoop.apache.org/docs/r2.3.0/api/org/apache/hadoop/fs/FileSystem.html#globStatus(org.apache.hadoop.fs.Path) >>>> >>>> >>>> On Wed, Jun 18, 2014 at 12:09 AM, MEETHU MATHEW <meethu2...@yahoo.co.in >>>> > wrote: >>>> >>>>> Hi Jianshi, >>>>> >>>>> I have used wild card characters (*) in my program and it worked.. >>>>> My code was like this >>>>> b = sc.textFile("hdfs:///path to file/data_file_2013SEP01*") >>>>> >>>>> Thanks & Regards, >>>>> Meethu M >>>>> >>>>> >>>>> On Wednesday, 18 June 2014 9:29 AM, Jianshi Huang < >>>>> jianshi.hu...@gmail.com> wrote: >>>>> >>>>> >>>>> It would be convenient if Spark's textFile, parquetFile, etc. can >>>>> support path with wildcard, such as: >>>>> >>>>> hdfs://domain/user/jianshuang/data/parquet/table/month=2014* >>>>> >>>>> Or is there already a way to do it now? >>>>> >>>>> Jianshi >>>>> >>>>> -- >>>>> Jianshi Huang >>>>> >>>>> LinkedIn: jianshi >>>>> Twitter: @jshuang >>>>> Github & Blog: http://huangjs.github.com/ >>>>> >>>>> >>>>> >>>> >>> >>> >>> -- >>> Jianshi Huang >>> >>> LinkedIn: jianshi >>> Twitter: @jshuang >>> Github & Blog: http://huangjs.github.com/ >>> >> >> >> >> -- >> Jianshi Huang >> >> LinkedIn: jianshi >> Twitter: @jshuang >> Github & Blog: http://huangjs.github.com/ >> > > -- Jianshi Huang LinkedIn: jianshi Twitter: @jshuang Github & Blog: http://huangjs.github.com/
