Help us teach Spark and grow the Spark community

Dear Spark Devs, Ameet Talwalkar, Brian Clapper, and I are teaching several large free MOOCs this year on Apache Spark and we are looking for participants from the community who would like to help by being Teaching Assistants for the

Spark streaming Kafka receiver WriteAheadLog question

Hi, Is it possible for Kafka receiver generated WriteAheadLogBackedBlockRDD to hold corresponded Kafka offset range so that during recovery the RDD can refer back to Kafka queue instead of paying the cost of write ahead log? I guess there must be a reason here. Could anyone please help me

Proposal of closing some PRs which at least one of committers suggested so

Hi all, I realised that there are many open PRs and it is somehow problematic after the past discussion ( http://apache-spark-developers-list.1001551.n3.nabble.com/auto-closing-pull-requests-that-have-been-inactive-gt-30-days-td17208.html ). ​ I looked through them PR by PR and could make a list

Re: executor delay in Spark

Glad to hear that the problem was solvable! I have not seen delays of this type for later stages in jobs run by spark-submit, but I do not think it impossible if your stage has no lineage dependence on other RDDs. I'm CC'ing the dev list to report of other users observing load imbalance caused by

Re: RFC: Remove "HBaseTest" from examples?

I'd suggest that the hbase-downstreamer project[1] is a better place for folks to see these examples. There's already an example for spark streaming that does not rely on any of the new goodness in the hbase-spark module[2]. Granted, it uses the Spark Java APIs[3], but we'd be glad to have a

Re: Possible deadlock in registering applications in the recovery mode

Hi guys, any update on this? Best On Wed, Apr 20, 2016 at 3:00 AM, Niranda Perera wrote: > Hi Reynold, > > I have created a JIRA for this [1]. I have also created a PR for the same > issue [2]. > > Would be very grateful if you could look into this, because this is a