[Spark-SQL] - Creating Hive Metastore Parquet table from Avro schema

Hello Everyone, I have my parquet files stored on HDFS. I am trying to create a table in Hive Metastore from Spark SQL. I have an Avro schema file from which I generated the parquet files. I am doing the following to create the table. 1) Firstly create an Avro dummy table from the schema file

Recommended way to serialize Hadoop Writables' in Spark

Hi, Is there any recommended way of serializing Hadoop Writables' in Spark? Here is my problem. Question1: I have a pair RDD which is created by reading a SEQ[LongWritable, BytesWritable]: RDD[(LongWritable, BytesWritable)] I have these two settings set in spark conf. spark.serializer=org.apach

Recommended way to serialize Hadoop Writables' in Spark

Hi, Is there any recommended way of serializing Hadoop Writables' in Spark? Here is my problem. Question1: I have a pair RDD which is created by reading a SEQ[LongWritable, BytesWritable]: RDD[(LongWritable, BytesWritable)] I have these two settings set in spark conf. spark.serializer=org.apach