sequenceFile() calls hadoopFile() where:
val confBroadcast = broadcast(new
SerializableConfiguration(hadoopConfiguration))
You can set the parameter in sc.hadoopConfiguration before calling
sc.sequenceFile().
Cheers
On Sun, Jun 28, 2015 at 9:23 PM, ÐΞ€ρ@Ҝ (๏̯͡๏) <[email protected]> wrote:
> I can do this
>
> val hadoopConf = new Configuration(sc.hadoopConfiguration)
>
> * hadoopConf.set("mapreduce.input.fileinputformat.split.maxsize",
> "67108864")*
>
> sc.newAPIHadoopFile(
>
> path + "/*.avro",
>
> classOf[AvroKeyInputFormat[GenericRecord]],
>
> classOf[AvroKey[GenericRecord]],
>
> classOf[NullWritable],
>
> hadoopConf)
>
>
> But i cant do the same with
>
> sc.sequenceFile("path", classOf[Text], classOf[Text])
> How can i achieve the same with sequenceFile
> --
> Deepak
>
>
