Maintainer for Mesos

Hi Spark devs, I'm interested in having a committer look at a PR [1] for Mesos, but there's not an entry for Mesos in the maintainers specialties on the wiki [2]. Which Spark committers have expertise in the Mesos features? Thanks! Andrew [1] https://github.com/apache/spark/pull/3074 [2] https

Re: jenkins redirect down (but jenkins is up!), lots of potential

The pull request builder and SCM-polling builds appear to be working fine, but the links in pull request comments won't work because the AMP Lab webserver is still down. In the meantime, though, you can continue to access Jenkins through https://hadrian.ist.berkeley.edu/jenkins/ On Mon, Jan 5, 20

Re: Spark Streaming Data flow graph

Hey François, Well, at a high-level here is what I thought about the diagram. - ReceiverSupervisor handles only one Receiver. - BlockGenerator is part of ReceiverSupervisor not ReceivedBlockHandler - The blocks are inserted in BlockManager and if activated, WriteAheadLogManager in parallel, not t

Re: Spark UI history job duration is wrong

Thanks for reporting this - it definitely sounds like a bug. Please open a JIRA for it. My guess is that we define the start or end time of the job based on the current time instead of looking at data encoded in the underlying event stream. That would cause it to not work properly when loading from

Spark UI history job duration is wrong

Hello, I'm using Spark 1.2.0 and when running an application, if I go into the UI and then in the job tab ("/jobs/") the jobs duration are relevant and the posted durations looks ok. However when I open the history ("history/app-/jobs/") for that job, the duration are wrong showing milliseconds i

Fwd: ApacheCon North America 2015 Call For Papers

FYI, ApacheCon North America call for papers is up. Matei > Begin forwarded message: > > Date: January 5, 2015 at 9:40:41 AM PST > From: Rich Bowen > Reply-To: dev > To: dev > Subject: ApacheCon North America 2015 Call For Papers > > Fellow ASF enthusiasts, > > We now have less than a month

jenkins redirect down (but jenkins is up!), lots of potential

UC Berkeley had some major maintenance done this past weekend, and long story short, not everything came back. our primary webserver's NFS is down and that means we're not serving websites, meaning that the redirect to jenkins is failing. jenkins is still up, and building some jobs, but we will p

Re: python converter in HBaseConverter.scala(spark/examples)

Absolutely; as I mentioned by all means submit a PR - I just wanted to point out that any specific converter is not "officially" supported, although the interface is of course. I'm happy to review a PR just ping me when ready. — Sent from Mailbox On Mon, Jan 5, 2015 at 7:06 PM, Ted Yu wrote

Re: python converter in HBaseConverter.scala(spark/examples)

HBaseConverter is in Spark source tree. Therefore I think it makes sense for this improvement to be accepted so that the example is more useful. Cheers On Mon, Jan 5, 2015 at 7:54 AM, Nick Pentreath wrote: > Hey > > These converters are actually just intended to be examples of how to set > up a

RE: python converter in HBaseConverter.scala(spark/examples)

We are planning to support HBase as a "native" data source to Spark SQL in 1.3 (SPARK-3880). More details will come soon. -Original Message- From: Ted Yu [mailto:yuzhih...@gmail.com] Sent: Monday, January 05, 2015 7:37 AM To: tgbaggio Cc: dev@spark.apache.org Subject: Re: python conver

Re: python converter in HBaseConverter.scala(spark/examples)

Hey  These converters are actually just intended to be examples of how to set up a custom converter for a specific input format. The converter interface is there to provide flexibility where needed, although with the new SparkSQL data store interface the intention is that most common use cases

Re: python converter in HBaseConverter.scala(spark/examples)

In my opinion this would be useful - there was another thread where returning only the value of first column in the result was mentioned. Please create a SPARK JIRA and a pull request. Cheers On Mon, Jan 5, 2015 at 6:42 AM, tgbaggio wrote: > Hi, > > In HBaseConverter.scala > < > https://githu

Re: Registering custom metrics

Hi Gerard, Thanks for the answer! I had a good look at it, but I couldn't figure out whether one can use that to emit metrics from your application code. Suppose I wanted to monitor the rate of bytes I produce, like so: stream .map { input => val bytes = produce(input)

python converter in HBaseConverter.scala(spark/examples)

Hi, In HBaseConverter.scala , the python converter HBaseResultToStringConverter return only the value of first column in the result. In my opinion, it limits t

Re: Tuning Spark Streaming jobs

Hi Tim, First of all, let m wish you a happy and fulfilling New Year. Sorry for the delay in my response. I was out for the xmas break. I've added my thoughts to the ticket from the perspective of a streaming Job. @TD: What do you think? -kr, Gerard. On Tue, Dec 23, 2014 at 8:02 PM, Timothy Che

Re: Registering custom metrics

Hi, Yes, I managed to create a register custom metrics by creating an implementation of org.apache.spark.metrics.source.Source and registering it to the metrics subsystem. Source is [Spark] private, so you need to create it under a org.apache.spark package. In my case, I'm dealing with Spark Str