Re: setting inputMetrics in HadoopRDD#compute()

2014-07-26 Thread Reynold Xin
That makes sense, Sandy. When you add the patch, can you make sure you comment inline on why the fallback is needed? On Sat, Jul 26, 2014 at 11:46 AM, Sandy Ryza sandy.r...@cloudera.com wrote: I'm working on a patch that switches this stuff out with the Hadoop FileSystem StatisticsData,

Re: setting inputMetrics in HadoopRDD#compute()

2014-07-26 Thread Kay Ousterhout
Reynold you're totally right, as discussed offline -- I didn't think about the limit use case when I wrote this. Sandy, is it easy to fix this as part of your patch to use StatisticsData? If not, I can fix it in a separate patch. On Sat, Jul 26, 2014 at 12:12 PM, Reynold Xin

Re: SparkContext startup time out

2014-07-26 Thread Anand Avati
I am bumping into this problem as well. I am trying to move to akka 2.3.x from 2.2.x in order to port to Scala 2.11 - only akka 2.3.x is available in Scala 2.11. All 2.2.x akka works fine, and all 2.3.x akka give the following exception in new SparkContext. Still investigating why..