Re: Does long-lived SparkContext hold on to executor resources?
I would be cautious regarding use of spark.cleaner.ttl, as it can lead to confusing error messages if time-based cleaning deletes resources that are still needed. See my comment at https://issues.apache.org/jira/browse/SPARK-5594?focusedCommentId=14486034&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14486034 On Mon, May 11, 2015 at 10:17 AM, Ganelin, Ilya wrote: > Also check out the spark.cleaner.ttl property. Otherwise, you will > accumulate shuffle metadata in the memory of the driver. > > > > Sent with Good (www.good.com) > > > > -Original Message- > *From: *Silvio Fiorito [silvio.fior...@granturing.com] > *Sent: *Monday, May 11, 2015 01:03 PM Eastern Standard Time > *To: *stanley; user@spark.apache.org > *Subject: *Re: Does long-lived SparkContext hold on to executor resources? > > You want to look at dynamic resource allocation, here: > http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation > > > > > > On 5/11/15, 11:23 AM, "stanley" wrote: > > >I am building an analytics app with Spark. I plan to use long-lived > >SparkContexts to minimize the overhead for creating Spark contexts, which > in > >turn reduces the analytics query response time. > > > >The number of queries that are run in the system is relatively small each > >day. Would long lived contexts hold on to the executor resources when > there > >is no queries running? Is there a way to free executor resources in this > >type of use cases? > > > > > > > >-- > >View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Does-long-lived-SparkContext-hold-on-to-executor-resources-tp22848.html > >Sent from the Apache Spark User List mailing list archive at Nabble.com. > > > >- > >To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > >For additional commands, e-mail: user-h...@spark.apache.org > > > > -- > > The information contained in this e-mail is confidential and/or > proprietary to Capital One and/or its affiliates. The information > transmitted herewith is intended only for use by the individual or entity > to which it is addressed. If the reader of this message is not the > intended recipient, you are hereby notified that any review, > retransmission, dissemination, distribution, copying or other use of, or > taking of any action in reliance upon this information is strictly > prohibited. If you have received this communication in error, please > contact the sender and delete the material from your computer. >
RE: Does long-lived SparkContext hold on to executor resources?
Also check out the spark.cleaner.ttl property. Otherwise, you will accumulate shuffle metadata in the memory of the driver. Sent with Good (www.good.com) -Original Message- From: Silvio Fiorito [silvio.fior...@granturing.com<mailto:silvio.fior...@granturing.com>] Sent: Monday, May 11, 2015 01:03 PM Eastern Standard Time To: stanley; user@spark.apache.org Subject: Re: Does long-lived SparkContext hold on to executor resources? You want to look at dynamic resource allocation, here: http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation On 5/11/15, 11:23 AM, "stanley" wrote: >I am building an analytics app with Spark. I plan to use long-lived >SparkContexts to minimize the overhead for creating Spark contexts, which in >turn reduces the analytics query response time. > >The number of queries that are run in the system is relatively small each >day. Would long lived contexts hold on to the executor resources when there >is no queries running? Is there a way to free executor resources in this >type of use cases? > > > >-- >View this message in context: >http://apache-spark-user-list.1001560.n3.nabble.com/Does-long-lived-SparkContext-hold-on-to-executor-resources-tp22848.html >Sent from the Apache Spark User List mailing list archive at Nabble.com. > >- >To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >For additional commands, e-mail: user-h...@spark.apache.org > The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
Re: Does long-lived SparkContext hold on to executor resources?
You want to look at dynamic resource allocation, here: http://spark.apache.org/docs/latest/job-scheduling.html#dynamic-resource-allocation On 5/11/15, 11:23 AM, "stanley" wrote: >I am building an analytics app with Spark. I plan to use long-lived >SparkContexts to minimize the overhead for creating Spark contexts, which in >turn reduces the analytics query response time. > >The number of queries that are run in the system is relatively small each >day. Would long lived contexts hold on to the executor resources when there >is no queries running? Is there a way to free executor resources in this >type of use cases? > > > >-- >View this message in context: >http://apache-spark-user-list.1001560.n3.nabble.com/Does-long-lived-SparkContext-hold-on-to-executor-resources-tp22848.html >Sent from the Apache Spark User List mailing list archive at Nabble.com. > >- >To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >For additional commands, e-mail: user-h...@spark.apache.org >
