[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20770 @ajithme so have you got some time to recheck? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20770 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user squito commented on the issue: https://github.com/apache/spark/pull/20770 took a quick look, agree with shivaram's observations, you've got to handle `shuffleIdToMapStage` which will not be so easy. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user AjithShetty2489 commented on the issue: https://github.com/apache/spark/pull/20770 @shivaram yes, you are right. Let me recheck --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user AjithShetty2489 commented on the issue: https://github.com/apache/spark/pull/20770 @Ngone51 Thanks for the question. I am not quite sure of actual implementation of Triemap but i see https://www.scala-lang.org/api/2.12.3/scala/collection/concurrent/TrieMap.html https://stackoverflow.com/questions/45566633/whats-the-difference-between-scala-triemap-and-java-concurrenthashmap https://stackoverflow.com/questions/29499381/what-is-a-triemap-and-what-is-its-advantages-disadvantages-compared-to-a-hashmap and conclude trieMap is better for concurrent scenarios. but i measure the performance of ConcurrentHashMap vs TrieMap with basic put and get operation, it seems TrieMap is slower than ConcurrentHashmap --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user markhamstra commented on the issue: https://github.com/apache/spark/pull/20770 @squito is the master of DAGSchedulerSuite, and can provide you the best advice on changing or adding to the existing DAGSchedulerSuite. I'll be back from skiing next week and try to find some time to look at this. Hopefully @kayousterhout can find some time too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/20770 @AjithShetty2489 I'm not sure just changing these two maps is sufficient ? For example createResultStage could in turn create all the parent stages and the parents stages could be ShuffleMapStage which in turn means that the map `shuffleIdToMapStage` would need to be protected and we'll also need to ensure that the MapOutputTracker `registerShuffle` is thread safe --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user shivaram commented on the issue: https://github.com/apache/spark/pull/20770 cc @kayousterhout @markhamstra --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user AjithShetty2489 commented on the issue: https://github.com/apache/spark/pull/20770 For UT, i see DAGSchedulerSuite currently does not have same behaviour as DAGScheduler as its events posted to DAGSchedulerEventProcessLoopTester are running in same thread unlike DAGScheduler where posted events are processed in separate thread . Can i modify this.? or write a separate suite.? Pls suggest --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20770 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20770 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
