Hi,
I have an use case where I want to write my data into Hive using Parquet file 
format. I want to perform all this using Cascading framework. I came across the 
Cascading Parquet-Avro scheme (To convert data in Parquet format) & HiveTap (TO 
Write the data in Hive Table) which does this job. But the problem I am facing 
while doing this I was not able to write the Date datatype in Avro.

So I tried the same thing by converting the date into long values (as Avro 
doesn't provide Date provision in it schema) but it didn't worked for me as my 
Hive table has field with Date datatype. It gave me exception "LongWritable 
cannot cast to DateWritable".

So is there any way through which I can write the date in Hive Table through 
Parquet-Avro?

Below is my code for the same:
public class Test {

       public static void main(String[] args) throws IOException {

              Properties properties = new Properties();
              String tableName = "datetest";
              CoercibleType dateType = new DateType("yyyy-MM-dd");

              Fields field = new Fields("f1", "f2", "f3").applyTypes(new Type[] 
{
                           String.class, String.class, dateType });

              String[] columnNames = { "f1", "f2", "f3" };
              String[] columnTypes = { "string", "string", "date" };

              Tap source = new Hfs(new TextDelimited(field, false, ","),
                           "data/file2.txt");

              HiveTableDescriptor tableDesc = new HiveTableDescriptor("default",
              tableName, columnNames, columnTypes, new String[] {}, ",");

              HiveTap hiveTap = new HiveTap(tableDesc, new ParquetAvroScheme(
              new Schema.Parser().parse(ParquetAvro2.class.getClassLoader()
              .getResourceAsStream("avro2.avsc"))));


              Pipe p = new Pipe("pipe");

              FlowDef flowDef = FlowDef.flowDef().addSource(p, source)
                           .addTailSink(p, hiveTap);

              properties = new Properties();
              AppProps.setApplicationName(properties,
                           "cascading hive integration demo");

              new 
Hadoop2MR1FlowConnector(properties).connect(flowDef).complete();
       }
}

avro2.avsc(Avro Schema):
{
     "type": "record",
     "name": "avro",
     "fields": [
       { "name": "f1", "type": "string" },
       { "name": "f2", "type": "string" },
      {"name": "f3", "type":"long"}
     ]
}



Thanks,
Bhavesh
**************************************Disclaimer******************************************
 This e-mail message and any attachments may contain confidential information 
and is for the sole use of the intended recipient(s) only. Any views or 
opinions presented or implied are solely those of the author and do not 
necessarily represent the views of BitWise. If you are not the intended 
recipient(s), you are hereby notified that disclosure, printing, copying, 
forwarding, distribution, or the taking of any action whatsoever in reliance on 
the contents of this electronic information is strictly prohibited. If you have 
received this e-mail message in error, please immediately notify the sender and 
delete the electronic message and any attachments.BitWise does not accept 
liability for any virus introduced by this e-mail or any attachments. 
********************************************************************************************

Reply via email to