Figured how to do it. Hence thought of sharing in case if someone is
interested.

import parquet.column.ColumnReader
import parquet.filter.ColumnRecordFilter._
import parquet.filter.ColumnPredicates._
import parquet.hadoop.{ParquetOutputFormat, ParquetInputFormat}
import org.apache.hadoop.mapred.JobConf
import parquet.bytes.BytesInput
import parquet.pig.TupleReadSupport;
import org.apache.pig.data.Tuple;
val conf = new JobConf()
conf.set("parquet.pig.schema","id:int,name:chararray")
ParquetInputFormat.setReadSupportClass(conf,classOf[TupleReadSupport])
val file =
sc.newAPIHadoopFile("<path>/part-m-00000.parquet",classOf[ParquetInputFormat[Tuple]],classOf[Void],classOf[Tuple],conf).map(x=>(x._2.get(0),x._2.get(1))).collect

Regards,
SB


On Sat, Apr 26, 2014 at 3:31 PM, suman bharadwaj <[email protected]>wrote:

> Hi All,
>
> We have written PIG Jobs which outputs the data in parquet format.
>
> For eg:
>
> register parquet-column-1.3.1.jar;
> register parquet-common-1.3.1.jar;
> register parquet-format-2.0.0.jar;
> register parquet-hadoop-1.3.1.jar;
> register parquet-pig-1.3.1.jar;
> register parquet-encoding-1.3.1.jar;
>
> A =load 'path' using PigStorage('\t') as (in:int,name:chararray);
> store A into 'output_path' using parquet.pig.ParquetStorer();
>
> Now how do i read this parquet file in SPARK ?
>
> Thanks in advance.
>
> Regards,
> SB
>

Reply via email to