Thanks Nick, DB - that was helpful.
On Mon, Oct 13, 2014 at 5:44 PM, DB Tsai <[email protected]> wrote:
> For now, with SparkSPARK-3462 parquet pushdown for unionAll PR, you
> can do the following for unionAll schemaRDD.
>
> val files = Array("hdfs://file1.parquet", "hdfs://file2.parquet",
> "hdfs://file3.parquet")
> val rdds = paths.map(hc.parquetFile(_))
>
> val unionedRDD = {
> var temp = rdds(0)
> for (i <- 1 until rdds.length) {
> temp = temp.unionAll(rdds(i))
> }
> temp
> }
>
> Sincerely,
>
> DB Tsai
> -------------------------------------------------------
> My Blog: https://www.dbtsai.com
> LinkedIn: https://www.linkedin.com/in/dbtsai
>
>
> On Mon, Oct 13, 2014 at 7:22 PM, Nicholas Chammas
> <[email protected]> wrote:
> > Right now I believe the only supported option is to pass a
> comma-delimited
> > list of paths.
> >
> > I've opened SPARK-3928: Support wildcard matches on Parquet files to
> request
> > this feature.
> >
> > Nick
> >
> > On Mon, Oct 13, 2014 at 12:21 PM, Sadhan Sood <[email protected]>
> wrote:
> >>
> >> How can we read all parquet files in a directory in spark-sql. We are
> >> following this example which shows a way to read one file:
> >>
> >> // Read in the parquet file created above. Parquet files are
> >> self-describing so the schema is preserved.
> >> // The result of loading a Parquet file is also a SchemaRDD.
> >> val parquetFile = sqlContext.parquetFile("people.parquet")
> >>
> >> //Parquet files can also be registered as tables and then used in SQL
> >> statements.
> >> parquetFile.registerTempTable("parquetFile")
> >
> >
>