Re: Welcoming two new committers

[Cheng Lian]

Awesome! Congrats and welcome!!

Cheng

On Tue, Feb 9, 2016 at 2:55 AM, Shixiong(Ryan) Zhu 
wrote:

> Congrats!!! Herman and Wenchen!!!
>
>
> On Mon, Feb 8, 2016 at 10:44 AM, Luciano Resende 
> wrote:
>
>>
>>
>> On Mon, Feb 8, 2016 at 9:15 AM, Matei Zaharia 
>> wrote:
>>
>>> Hi all,
>>>
>>> The PMC has recently added two new Spark committers -- Herman van Hovell
>>> and Wenchen Fan. Both have been heavily involved in Spark SQL and Tungsten,
>>> adding new features, optimizations and APIs. Please join me in welcoming
>>> Herman and Wenchen.
>>>
>>> Matei
>>>
>>
>> Congratulations !!!
>>
>> --
>> Luciano Resende
>> http://people.apache.org/~lresende
>> http://twitter.com/lresende1975
>> http://lresende.blogspot.com/
>>
>
>

pull request template

[Reynold Xin]

Github introduced a new feature today that allows projects to define
templates for pull requests. I pushed a very simple template to the
repository:

https://github.com/apache/spark/blob/master/.github/PULL_REQUEST_TEMPLATE


Over time I think we can see how this works and perhaps add a small
checklist to the pull request template so contributors are reminded every
time they submit a pull request the important things to do in a pull
request (e.g. having proper tests).



## What changes were proposed in this pull request?

(Please fill in changes proposed in this fix)


## How was the this patch tested?

(Please explain how this patch was tested. E.g. unit tests, integration
tests, manual tests)


(If this patch involves UI changes, please attach a screenshot; otherwise,
remove this)

Re: Welcoming two new committers

[Cheng Lian]

Awesome! Congrats and welcome!!

On 2/9/16 2:55 AM, Shixiong(Ryan) Zhu wrote:

Congrats!!! Herman and Wenchen!!!

On Mon, Feb 8, 2016 at 10:44 AM, Luciano Resende > wrote:




On Mon, Feb 8, 2016 at 9:15 AM, Matei Zaharia
> wrote:

Hi all,

The PMC has recently added two new Spark committers -- Herman
van Hovell and Wenchen Fan. Both have been heavily involved in
Spark SQL and Tungsten, adding new features, optimizations and
APIs. Please join me in welcoming Herman and Wenchen.

Matei


Congratulations !!!

-- 
Luciano Resende

http://people.apache.org/~lresende

http://twitter.com/lresende1975
http://lresende.blogspot.com/

FYI: github is getting DDOSed

[shane knapp]

this may cause builds to timeout on the git fetch much more than usual[1].

https://status.github.com/messages

just thought people might want to know...

shane

1 -- this actually happens pretty often, sadly.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: SparkOscope: Enabling Spark Optimization through Cross-stack Monitoring and Visualization

[Stavros Kontopoulos]

Cool work! I will have a look to the project.

Cheers

On Fri, Feb 5, 2016 at 11:09 AM, Pete Robbins  wrote:

> Yiannis,
>
> I'm interested in what you've done here as I was looking for ways to allow
> the Spark UI to display custom metrics in a pluggable way without having to
> modify the Spark source code. It would be good to see if we could have
> modify your code to add extension points into the UI so we could configure
> sources of the additional metrics. So for instance rather than creating
> events from your HDFS files I would like to have a module that is pulling
> in system/jvm metrics that are in eg Elasticsearch.
>
> Do any of the Spark committers have any thoughts on this?
>
> Cheers,
>
>
> On 3 February 2016 at 15:26, Yiannis Gkoufas  wrote:
>
>> Hi all,
>>
>> I just wanted to introduce some of my recent work in IBM Research around
>> Spark and especially its Metric System and Web UI.
>> As a quick overview of our contributions:
>> We have a created a new type of Sink for the metrics ( HDFSSink ) which
>> captures the metrics into HDFS,
>> We have extended the metrics reported by the Executors to include
>> OS-level metrics regarding CPU, RAM, Disk IO, Network IO utilizing the
>> Hyperic Sigar library
>> We have extended the Web UI for the completed applications to visualize
>> any of the above metrics the user wants to.
>> The above functionalities can be configured in the metrics.properties and
>> spark-defaults.conf files.
>> We have recorded a small demo that shows those capabilities which you can
>> find here : https://ibm.app.box.com/s/vyaedlyb444a4zna1215c7puhxliqxdg
>> There is a blog post which gives more details on the functionality here:
>> *www.spark.tc/sparkoscope-enabling-spark-optimization-through-cross-stack-monitoring-and-visualization-2/*
>> 
>> and also there is a public repo where anyone can try it:
>> *https://github.com/ibm-research-ireland/sparkoscope*
>> 
>>
>> I would really appreciate any feedback or advice regarding this work.
>> Especially if you think it's worth it to upstream to the official Spark
>> repository.
>>
>> Thanks a lot!
>>
>
>


-- 

[JIRA] (SPARK-540) Add API to customize in-memory representation of RDDs

[Anonymous (JIRA)]

Title: Message Title


 
 
 
 

 
 
 

 
   
 Anonymous started work on  SPARK-540  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
Change By: 
 Anonymous  
 
 
Status: 
 Open In Progress  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian JIRA (v7.2.0-OD-01-031#72000-sha1:c6b05a6)  
 
 

 
   
 

  
 

  
 

   

Re: SPARK_WORKER_MEMORY in Spark Standalone - conf.getenv vs System.getenv?

[Sean Owen]

Here he's referring to a line of code that calls SparkConf.getenv vs
System.getenv, but the former calls the latter. In neither case does
it read from a props file.

On Tue, Feb 16, 2016 at 10:44 PM, Igor Costa  wrote:
> Actually answering the first question:
>
> Is there a reason to use conf to read SPARK_WORKER_MEMORY not
> System.getenv as for the other env vars?
>
> You can use the properties file to change the amount, System.getenv would be
> bad when you have for example other things running on the JVM which will
> cause conflict on some parts.
>  Defined usage in properties files is more convenience for custom UI to be
> made available.
>
> On Sat, Feb 13, 2016 at 8:38 PM, Sean Owen  wrote:
>>
>> Yes you said it is only set in a props file, but why do you say that?
>> because the resolution of your first question is that this is not
>> differently handled.
>>
>> On Fri, Feb 12, 2016 at 11:11 PM, Jacek Laskowski  wrote:
>> > On Fri, Feb 12, 2016 at 11:08 PM, Sean Owen  wrote:
>> >> I think that difference in the code is just an oversight. They
>> >> actually do the same thing.
>> >
>> > Correct. Just meant to know the reason if there was any.
>> >
>> >> Why do you say this property can only be set in a file?
>> >
>> > I said that conf/spark-defaults.conf can *not* be used to set
>> > spark.worker.ui.port property and wondered why is so? It'd be nice to
>> > have it for settings (not use workarounds like
>> > SPARK_WORKER_OPTS=-Dspark.worker.ui.port=21212). Just spot it and
>> > thought I'd ask if it needs to be cleaned up or improved.
>> >
>> > Jacek
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
>> For additional commands, e-mail: dev-h...@spark.apache.org
>>
>

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: [build system] additional jenkins downtime next thursday

[shane knapp]

the security release has been delayed until next wednesday morning,
and i'll be doing the upgrade first thing thursday morning.

i'll update everyone when i get more information.

thanks!

shane

On Thu, Feb 11, 2016 at 10:19 AM, shane knapp  wrote:
> there's a big security patch coming out next week, and i'd like to
> upgrade our jenkins installation so that we're covered.  it'll be
> around 8am, again, and i'll send out more details about the upgrade
> when i get them.
>
> thanks!
>
> shane

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org