[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-74728480 Here is the new PR https://github.com/apache/spark/pull/4653 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-74453703 Yep, looks like it's still here: https://amplab.cs.berkeley.edu/jenkins/view/Spark/job/Spark-1.3-SBT/AMPLAB_JENKINS_BUILD_PROFILE=hadoop1.0,label=centos/93/testReport/junit/org.apache.spark.util/AkkaUtilsSuite/remote_fetch_ssl_on___untrusted_server/ The `org.apache.spark.util.AkkaUtilsSuite.remote fetch ssl on - untrusted server` test failed with the following exception: ``` sbt.ForkMain$ForkError: Expected exception java.util.concurrent.TimeoutException to be thrown, but akka.actor.ActorNotFound was thrown. at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:496) at org.scalatest.FunSuite.newAssertionFailedException(FunSuite.scala:1555) at org.scalatest.Assertions$class.intercept(Assertions.scala:1004) at org.scalatest.FunSuite.intercept(FunSuite.scala:1555) at org.apache.spark.util.AkkaUtilsSuite$$anonfun$8.apply$mcV$sp(AkkaUtilsSuite.scala:373) at org.apache.spark.util.AkkaUtilsSuite$$anonfun$8.apply(AkkaUtilsSuite.scala:349) at org.apache.spark.util.AkkaUtilsSuite$$anonfun$8.apply(AkkaUtilsSuite.scala:349) at org.scalatest.Transformer$$anonfun$apply$1.apply$mcV$sp(Transformer.scala:22) at org.scalatest.OutcomeOf$class.outcomeOf(OutcomeOf.scala:85) at org.scalatest.OutcomeOf$.outcomeOf(OutcomeOf.scala:104) at org.scalatest.Transformer.apply(Transformer.scala:22) at org.scalatest.Transformer.apply(Transformer.scala:20) at org.scalatest.FunSuiteLike$$anon$1.apply(FunSuiteLike.scala:166) at org.scalatest.Suite$class.withFixture(Suite.scala:1122) at org.scalatest.FunSuite.withFixture(FunSuite.scala:1555) at org.scalatest.FunSuiteLike$class.invokeWithFixture$1(FunSuiteLike.scala:163) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) at org.scalatest.FunSuiteLike$$anonfun$runTest$1.apply(FunSuiteLike.scala:175) at org.scalatest.SuperEngine.runTestImpl(Engine.scala:306) at org.scalatest.FunSuiteLike$class.runTest(FunSuiteLike.scala:175) at org.apache.spark.util.AkkaUtilsSuite.org$scalatest$BeforeAndAfterEach$$super$runTest(AkkaUtilsSuite.scala:37) at org.scalatest.BeforeAndAfterEach$class.runTest(BeforeAndAfterEach.scala:255) at org.apache.spark.util.AkkaUtilsSuite.runTest(AkkaUtilsSuite.scala:37) at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208) at org.scalatest.FunSuiteLike$$anonfun$runTests$1.apply(FunSuiteLike.scala:208) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:413) at org.scalatest.SuperEngine$$anonfun$traverseSubNodes$1$1.apply(Engine.scala:401) at scala.collection.immutable.List.foreach(List.scala:318) at org.scalatest.SuperEngine.traverseSubNodes$1(Engine.scala:401) at org.scalatest.SuperEngine.org$scalatest$SuperEngine$$runTestsInBranch(Engine.scala:396) at org.scalatest.SuperEngine.runTestsImpl(Engine.scala:483) at org.scalatest.FunSuiteLike$class.runTests(FunSuiteLike.scala:208) at org.scalatest.FunSuite.runTests(FunSuite.scala:1555) at org.scalatest.Suite$class.run(Suite.scala:1424) at org.scalatest.FunSuite.org$scalatest$FunSuiteLike$$super$run(FunSuite.scala:1555) at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212) at org.scalatest.FunSuiteLike$$anonfun$run$1.apply(FunSuiteLike.scala:212) at org.scalatest.SuperEngine.runImpl(Engine.scala:545) at org.scalatest.FunSuiteLike$class.run(FunSuiteLike.scala:212) at org.apache.spark.util.AkkaUtilsSuite.org$scalatest$BeforeAndAfterAll$$super$run(AkkaUtilsSuite.scala:37) at org.scalatest.BeforeAndAfterAll$class.liftedTree1$1(BeforeAndAfterAll.scala:257) at org.scalatest.BeforeAndAfterAll$class.run(BeforeAndAfterAll.scala:256) at org.apache.spark.util.AkkaUtilsSuite.run(AkkaUtilsSuite.scala:37) at org.scalatest.tools.Framework.org$scalatest$tools$Framework$$runSuite(Framework.scala:462) at org.scalatest.tools.Framework$ScalaTestTask.execute(Framework.scala:671) at sbt.ForkMain$Run$2.call(ForkMain.java:294) at sbt.ForkMain$Run$2.call(ForkMain.java:284) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: sbt.ForkMain$ForkError: Actor not found for: ActorSelection[Anchor(akka.ssl.tcp://spark@localhost:50364/), Path(/user/MapOutputTracker)] at
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user dragos commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-74072152 The race condition seems [still there](https://ci.typesafe.com/view/Spark/job/spark-nightly-build/31/testReport/junit/org.apache.spark.util/AkkaUtilsSuite/remote_fetch_ssl_on___untrusted_server/). @jacek-lewandowski I'm not 100% sure, but it doesn't seem to be related to the timeout this time. The stack trace indicates that the `Identity` message returned without finding any actor. Does that fall into the real failure category? In that case, there's another race condition in this test. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-73115605 I see, so you are worried about false negatives. For now, let's increase the timeout then. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-73118448 @pwendell so is this pr enough (according to what @JoshRosen said?) or should i create another one for master? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-73119222 Actually can you close this PR? I already merged but it doesn't close correclty when someone merges into a topic branch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-73119185 This one is fine. I can take care of getting it in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4343 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-72858032 @pwendell I think the problem with catching either exception is that the `ActorNotFoundException` is also thrown in other situations - in this particular test we want to prove that actor system A cannot connect to the other actor system B because A doesn't trust B, and this results in timeout. This is even lower communication layer than SASL. On the other hand `ActorNotFoundException` is thrown in case of timeout, in case there is no actor with that name or in case of authentication problems - completely different situations because they occur when we are successfully connected to the other actor system and that remote actor system refuses to return the reference to the requested actor. Therefore, `ActorNotFoundException` when not related to the timeout should cause the test to fail because it means that we successfully connected to untrusted actor system. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-72798850 @jacek-lewandowski how about just catching either exception? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-72748719 [Test build #26675 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26675/consoleFull) for PR 4343 at commit [`b9ba47e`](https://github.com/apache/spark/commit/b9ba47e635cb31d3a66c7417cb0048edd667ed42). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class SimpleFunctionRegistry(val caseSensitive: Boolean) extends FunctionRegistry ` * `class StringKeyHashMap[T](normalizer: (String) = String) ` * `case class MultiAlias(child: Expression, names: Seq[String])` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-72748732 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26675/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
GitHub user jacek-lewandowski opened a pull request: https://github.com/apache/spark/pull/4343 SPARK-5548: Fixed a race condition in AkkaUtilsSuite You can merge this pull request into a Git repository by running: $ git pull https://github.com/jacek-lewandowski/spark SPARK-5548-1.3 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4343.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4343 commit b9ba47e635cb31d3a66c7417cb0048edd667ed42 Author: Jacek Lewandowski lewandowski.ja...@gmail.com Date: 2015-02-03T21:08:36Z SPARK-5548: Fixed a race condition in AkkaUtilsSuite --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-72735186 Once approved, I'll create another PR for master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-72735757 [Test build #26675 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26675/consoleFull) for PR 4343 at commit [`b9ba47e`](https://github.com/apache/spark/commit/b9ba47e635cb31d3a66c7417cb0048edd667ed42). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user vanzin commented on a diff in the pull request:
https://github.com/apache/spark/pull/4343#discussion_r24040441
--- Diff: core/src/test/scala/org/apache/spark/util/AkkaUtilsSuite.scala ---
@@ -371,7 +371,7 @@ class AkkaUtilsSuite extends FunSuite with
LocalSparkContext with ResetSystemPro
AkkaUtils.address(AkkaUtils.protocol(slaveSystem), spark,
localhost, boundPort, MapOutputTracker))
val timeout = AkkaUtils.lookupTimeout(conf)
intercept[TimeoutException] {
- slaveTracker.trackerActor =
Await.result(selection.resolveOne(timeout), timeout)
+ slaveTracker.trackerActor =
Await.result(selection.resolveOne(timeout * 2), timeout)
--- End diff --
Could you explain the the PR's comment (= the commit message) how
increasing a timeout fixes a race condition, instead of just making it less
likely?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-72743098 @jacek-lewandowski Since master and branch-1.3 are pretty much in sync at this point, feel free to open your PR directly against master; I can handle backports myself as part of the merge process, since our merge script makes it really easy to do cherry-picks. It's still worth opening separate backport PRs if the branches have diverged significantly, but if they haven't it's generally easier to open PRs against master and have the committer handle the ports. If you'd like a patch to be merged into multiple branches, just leave a comment in the PR description so the committer knows where to merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
