[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16300926#comment-16300926
]
Xuefu Zhang commented on SPARK-22765:
-
Did some benchmarking with a set of 20 queries on upfront
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298634#comment-16298634
]
Xuefu Zhang commented on SPARK-22765:
-
bq: at least based on this one experiment up front allocation
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16298589#comment-16298589
]
Thomas Graves commented on SPARK-22765:
---
ok so at least based on this one experiment up front
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297914#comment-16297914
]
Xuefu Zhang commented on SPARK-22765:
-
Alright, I tested upfront allocation and its combinations with
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297645#comment-16297645
]
Xuefu Zhang commented on SPARK-22765:
-
Haven't got a chance to try upfront allocation. Tried one
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297549#comment-16297549
]
Thomas Graves commented on SPARK-22765:
---
with SPARK-21656 does upfront allocation vs exponential
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297529#comment-16297529
]
Xuefu Zhang commented on SPARK-22765:
-
{quote}
SPARK-21656 and the dynamic allocation should handle
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297453#comment-16297453
]
Nan Zhu commented on SPARK-22765:
-
I took a look at the code, one of the possibilities is as following:
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297346#comment-16297346
]
Xuefu Zhang commented on SPARK-22765:
-
I'm not 100% positive, but that seems to be what I saw. I will
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297335#comment-16297335
]
Thomas Graves commented on SPARK-22765:
---
SPARK-21656 and the dynamic allocation should handle that,
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297316#comment-16297316
]
Xuefu Zhang commented on SPARK-22765:
-
Actually I meant across parallel stages (those connected to
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297307#comment-16297307
]
Thomas Graves commented on SPARK-22765:
---
yes between stages becomes a problem with lower timeout,
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16297276#comment-16297276
]
Xuefu Zhang commented on SPARK-22765:
-
Hi [~CodingCat], yes, we adapted both #1 and #2, but we still
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16296811#comment-16296811
]
Thomas Graves commented on SPARK-22765:
---
[~CodingCat] with SPARK-21656 executors shouldn't timeout
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295958#comment-16295958
]
Nan Zhu commented on SPARK-22765:
-
[~xuefuz] Regarding this, "The symptom is that newly allocated
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16295814#comment-16295814
]
Xuefu Zhang commented on SPARK-22765:
-
As an update, I managed to backport SPARK-21656, among a few
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292594#comment-16292594
]
Thomas Graves commented on SPARK-22765:
---
I'm not sure how mr style and 4-core executor go together.
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16292048#comment-16292048
]
Xuefu Zhang commented on SPARK-22765:
-
[~tgraves], I think it would help if SPARK-21656 can make a
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289793#comment-16289793
]
Thomas Graves commented on SPARK-22765:
---
ok so before you do anything else I would suggest trying
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289726#comment-16289726
]
Xuefu Zhang commented on SPARK-22765:
-
Yes, we are using Hive on Spark. Our Spark version is 1.6.1,
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289700#comment-16289700
]
Thomas Graves commented on SPARK-22765:
---
ok so its basically they idle timeout during DAG computing
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289676#comment-16289676
]
Xuefu Zhang commented on SPARK-22765:
-
Hi [~tgraves], Thanks for your input.
In our busy, heavily
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16289613#comment-16289613
]
Thomas Graves commented on SPARK-22765:
---
why doesn't idle timeout very small < 5 seconds work? If
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288414#comment-16288414
]
Xuefu Zhang commented on SPARK-22765:
-
I wouldn't say that MR is static, at lease not static in
[
https://issues.apache.org/jira/browse/SPARK-22765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288353#comment-16288353
]
Sean Owen commented on SPARK-22765:
---
I'm not clear why this needs another allocation scheme. You say
25 matches
Mail list logo
