Heap Settings for History Server

Hi folks, I couldn't find much literature on this so I figured I could ask here. Does anyone have experience in tuning the memory settings and interval times of the Spark History Server? Let's say I have 500 applications at 0.5 G each with a *spark.history.fs.update.interval* of 400s. Is there

Re: Thoughts on release cadence?

Is now the best time for submitting a SPIP for a feature targeting the 2.3 release? On Mon, Jul 31, 2017 at 12:22 PM, Sean Owen wrote: > Done at https://spark.apache.org/versioning-policy.html > > On Mon, Jul 31, 2017 at

Re: Thoughts on release cadence?

Done at https://spark.apache.org/versioning-policy.html On Mon, Jul 31, 2017 at 6:22 PM Reynold Xin wrote: > We can just say release in December, and code freeze mid Nov? > > On Mon, Jul 31, 2017 at 10:14 AM, Sean Owen wrote: > >> Will do, just was

Re: Tests failing with run-tests.py SyntaxError

(for the wider dev audience) there's an apache issue open for this, which i have commented on. https://issues.apache.org/jira/browse/SPARK-21573 i have a workaround and will be getting this sorted asap. On Fri, Jul 28, 2017 at 7:27 PM, Hyukjin Kwon wrote: > Or maybe in

Re: [VOTE] [SPIP] SPARK-18085: Better History Server scalability

Adding my own +1 (binding). On Mon, Jul 31, 2017 at 10:27 AM, Marcelo Vanzin wrote: > Hey all, > > Following the SPIP process, I'm putting this SPIP up for a vote. It's > been open for comments as an SPIP for about 3 weeks now, and had been > open without the SPIP label for

[VOTE] [SPIP] SPARK-18085: Better History Server scalability

Hey all, Following the SPIP process, I'm putting this SPIP up for a vote. It's been open for comments as an SPIP for about 3 weeks now, and had been open without the SPIP label for about 9 months before that. There has been no new feedback since it was tagged as an SPIP, so I'm assuming all the

Re: Thoughts on release cadence?

We can just say release in December, and code freeze mid Nov? On Mon, Jul 31, 2017 at 10:14 AM, Sean Owen wrote: > Will do, just was waiting for some feedback. I'll give approximate dates > towards the end of the year. It isn't binding or anything. > > > On Mon, Jul 31,

Re: Thoughts on release cadence?

Will do, just was waiting for some feedback. I'll give approximate dates towards the end of the year. It isn't binding or anything. On Mon, Jul 31, 2017, 18:06 Michael Armbrust wrote: > +1, should we update https://spark.apache.org/versioning-policy.html ? > > On Sun,

Re: Thoughts on release cadence?

+1, should we update https://spark.apache.org/versioning-policy.html ? On Sun, Jul 30, 2017 at 3:34 PM, Reynold Xin wrote: > This is reasonable ... +1 > > > On Sun, Jul 30, 2017 at 2:19 AM, Sean Owen wrote: > >> The project had traditionally posted some