Re: Spark Streaming saveAsNewAPIHadoopFiles

2014-10-07 Thread Abraham Jacob
Hi All, Continuing on this discussion... Is there a good reason why the def of "saveAsNewAPIHadoopFiles" in org/apache/spark/streaming/api/java/JavaPairDStream.scala is defined like this - def saveAsNewAPIHadoopFiles( prefix: String, suffix: String, keyClass: Class[_], val

Re: Spark Streaming saveAsNewAPIHadoopFiles

2014-10-06 Thread Abraham Jacob
Sean, Thanks a ton Sean... This is exactly what I was looking for. As mentioned in the code - // This horrible, separate declaration is necessary to appease the compiler @SuppressWarnings("unchecked") Class> outputFormatClass = (Class>) (Class) SequenceFileOutputFormat.class; wri

Re: Spark Streaming saveAsNewAPIHadoopFiles

2014-10-06 Thread Sean Owen
Here's an example: https://github.com/OryxProject/oryx/blob/master/oryx-lambda/src/main/java/com/cloudera/oryx/lambda/BatchLayer.java#L131 On Mon, Oct 6, 2014 at 7:39 PM, Abraham Jacob wrote: > Hi All, > > Would really appreciate from the community if anyone has implemented the > saveAsNewAPIHad

Spark Streaming saveAsNewAPIHadoopFiles

2014-10-06 Thread Abraham Jacob
Hi All, Would really appreciate from the community if anyone has implemented the saveAsNewAPIHadoopFiles method in "Java" found in the org.apache.spark.streaming.api.java.JavaPairDStream Any code snippet or online link would be greatly appreciated. Regards, Jacob