For now, with SparkSPARK-3462 parquet pushdown for unionAll PR, you
can do the following for unionAll schemaRDD.

  val files = Array("hdfs://file1.parquet", "hdfs://file2.parquet",
"hdfs://file3.parquet")
  val rdds = paths.map(hc.parquetFile(_))

  val unionedRDD = {
    var temp = rdds(0)
    for (i <- 1 until rdds.length) {
      temp = temp.unionAll(rdds(i))
    }
    temp
  }

Sincerely,

DB Tsai
-------------------------------------------------------
My Blog: https://www.dbtsai.com
LinkedIn: https://www.linkedin.com/in/dbtsai


On Mon, Oct 13, 2014 at 7:22 PM, Nicholas Chammas
<[email protected]> wrote:
> Right now I believe the only supported option is to pass a comma-delimited
> list of paths.
>
> I've opened SPARK-3928: Support wildcard matches on Parquet files to request
> this feature.
>
> Nick
>
> On Mon, Oct 13, 2014 at 12:21 PM, Sadhan Sood <[email protected]> wrote:
>>
>> How can we read all parquet files in a directory in spark-sql. We are
>> following this example which shows a way to read one file:
>>
>> // Read in the parquet file created above.  Parquet files are
>> self-describing so the schema is preserved.
>> // The result of loading a Parquet file is also a SchemaRDD.
>> val parquetFile = sqlContext.parquetFile("people.parquet")
>>
>> //Parquet files can also be registered as tables and then used in SQL
>> statements.
>> parquetFile.registerTempTable("parquetFile")
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to