[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16683600#comment-16683600
]
sam commented on SPARK-2243:
Big bonus of being able to create and shutdown SparkContexts is to be able to
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16384687#comment-16384687
]
ASF GitHub Bot commented on SPARK-2243:
---
Github user asfgit closed the pull request at:
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16201364#comment-16201364
]
ASF GitHub Bot commented on SPARK-2243:
---
Github user 561152 commented on the issue:
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16198062#comment-16198062
]
ASF GitHub Bot commented on SPARK-2243:
---
GitHub user 561152 opened a pull request:
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15424709#comment-15424709
]
Leonid commented on SPARK-2243:
---
If you have a necessity to work with lots of Spark contexts, you can turn
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15420467#comment-15420467
]
Stephen Boesch commented on SPARK-2243:
---
Given this were not going to be fixed: please update
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15314470#comment-15314470
]
ASF GitHub Bot commented on SPARK-2243:
---
Github user bbuild11 commented on the issue:
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103187#comment-15103187
]
Sean Owen commented on SPARK-2243:
--
I don't think this itself is a good argument for lots of contexts.
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103272#comment-15103272
]
Sean Owen commented on SPARK-2243:
--
You say you're concerned with over-utilizing a cluster for steps that
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103298#comment-15103298
]
Sean Owen commented on SPARK-2243:
--
Yes, dynamic allocation operates in units of executors. You can make
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103268#comment-15103268
]
Richard Marscher commented on SPARK-2243:
-
I fail to see how dynamic allocation would help, can
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103289#comment-15103289
]
Jason Hubbard commented on SPARK-2243:
--
I think what Richard is referring to and what we have seen as
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103348#comment-15103348
]
Jason Hubbard commented on SPARK-2243:
--
Dynamic allocation allows executors to be started and stopped
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103419#comment-15103419
]
Jason Hubbard commented on SPARK-2243:
--
There is a limit to how small you can make the heap before
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103451#comment-15103451
]
Sean Owen commented on SPARK-2243:
--
What do you mean? each context operates independently no matter where
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103429#comment-15103429
]
Sean Owen commented on SPARK-2243:
--
But the question isn't multiple JVMs or multiple contexts -- it's
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103476#comment-15103476
]
Jason Hubbard commented on SPARK-2243:
--
It seems feasible to me to implement such functionality and
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103446#comment-15103446
]
Jason Hubbard commented on SPARK-2243:
--
The difference between N contexts in N JVMs and N contexts in
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15103407#comment-15103407
]
Sean Owen commented on SPARK-2243:
--
Executors have the same heap size, yes, but the total heap available
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=14935614#comment-14935614
]
Richard Marscher commented on SPARK-2243:
-
To add another potential use case. We run several
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502139#comment-14502139
]
Marko Bonaci commented on SPARK-2243:
-
Your fleet of hosts shares the same JVM?
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502326#comment-14502326
]
Swaranga Sarma commented on SPARK-2243:
---
No, they are different physical/virtual
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494527#comment-14494527
]
Josh Rosen commented on SPARK-2243:
---
This was done in SPARK-4180, which was included in
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392475#comment-14392475
]
Sudharma Puranik commented on SPARK-2243:
-
[~jahubba] : Running on the seperate
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392474#comment-14392474
]
Sudharma Puranik commented on SPARK-2243:
-
[~jahubba] : Running on the seperate
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392804#comment-14392804
]
Jason Hubbard commented on SPARK-2243:
--
I don't think programmatically spinning up
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392860#comment-14392860
]
Sean Owen commented on SPARK-2243:
--
[~sams] the {{SparkContext}} constructor will throw
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392616#comment-14392616
]
sam commented on SPARK-2243:
[~srowen]
The real issue here and use case, is to be able to
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392780#comment-14392780
]
sam commented on SPARK-2243:
// sam in this particular case, can you simply set both of these
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392798#comment-14392798
]
sam commented on SPARK-2243:
What I would suggest though, is putting an `assert`/`requires`
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392848#comment-14392848
]
sam commented on SPARK-2243:
Yup, a singleton would make sense, it's creation is side
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392721#comment-14392721
]
Sean Owen commented on SPARK-2243:
--
[~sams] in this particular case, can you simply set
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392519#comment-14392519
]
Sudharma Puranik commented on SPARK-2243:
-
[~sowen] : My reply was for Jason
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14392499#comment-14392499
]
Sean Owen commented on SPARK-2243:
--
The best thing to do is state your use case. Not a
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391770#comment-14391770
]
Jason Hubbard commented on SPARK-2243:
--
Apologizing for being flippant is a bit of an
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391623#comment-14391623
]
Neelesh Srinivas Salian commented on SPARK-2243:
I hit this error. Simply
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391629#comment-14391629
]
Sean Owen commented on SPARK-2243:
--
Sorry to be flippant but really the answer is to not
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357545#comment-14357545
]
Jason Hubbard commented on SPARK-2243:
--
It also mentions changing the broadcast
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357591#comment-14357591
]
Peter Rudenko commented on SPARK-2243:
--
Unfortunatelly it doesn't work in
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14357463#comment-14357463
]
Craig Lukasik commented on SPARK-2243:
--
The Spark Job Server uses multiple
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14351198#comment-14351198
]
Alex Feinberg commented on SPARK-2243:
--
Interested in this as well: while there are
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14293019#comment-14293019
]
Aniket Bhatnagar commented on SPARK-2243:
-
I am also interested in having this
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284335#comment-14284335
]
Manoj Samel commented on SPARK-2243:
Is there a target release for this ?
Support
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119479#comment-14119479
]
Reynold Xin commented on SPARK-2243:
Pushing this for later since Hive on Spark no
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061767#comment-14061767
]
wangfei commented on SPARK-2243:
What do you mean of but it's something we could support
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061620#comment-14061620
]
wangfei commented on SPARK-2243:
Spark Job Server may new multiple SparkContext in one jvm
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14061698#comment-14061698
]
Patrick Wendell commented on SPARK-2243:
I think the Spark JobServer shares a
[
https://issues.apache.org/jira/browse/SPARK-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14046574#comment-14046574
]
Patrick Wendell commented on SPARK-2243:
This is not supported - but it's
48 matches
Mail list logo
