Let me crash this thread to suggest this *might* be related to this problem I'm trying to solve: https://issues.apache.org/jira/browse/SPARK-4196
Basically the question there is: this blank Configuration object gets made on the driver in the saveAsNewAPIHadoopFiles call, and seems to need to be serialized to use it in foreachRDD. This fails for me and at least 2 other users I know. But I feel like I am missing something. If you're investigating handling of Configuration when enabling checkpointing with the getOrCreate method, have a look and see if you have any comments vis-a-vis this JIRA. On Tue, Nov 4, 2014 at 5:48 PM, Marcelo Vanzin <van...@cloudera.com> wrote: > On Tue, Nov 4, 2014 at 9:34 AM, Cody Koeninger <c...@koeninger.org> wrote: >> 2. Is there a reason StreamingContext.getOrCreate defaults to a blank >> hadoop configuration rather than >> org.apache.spark.deploy.SparkHadoopUtil.get.conf, >> which would pull values from spark config? > > This is probably something I overlooked when I changed the rest of the > code to use SparkHadoopUtil.get.conf. Feel free to send a PR for it, > we should get it into 1.2 so that all code creates configuration > objects in a consistent way. > > -- > Marcelo > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
