[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-185966335 **[Test build #51502 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51502/consoleFull)** for PR 11140 at commit [`23c853f`](https://github.com/apache/spark/commit/23c853f762ca995abf4702fc4881909ea88fd399). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-185966344 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-185966348 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51502/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-185965746 **[Test build #51502 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51502/consoleFull)** for PR 11140 at commit [`23c853f`](https://github.com/apache/spark/commit/23c853f762ca995abf4702fc4881909ea88fd399). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user GayathriMurali closed the pull request at: https://github.com/apache/spark/pull/11140 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user GayathriMurali commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-184842753 @JoshRosen @zsxwing I am currently trying to fix a null pointer exception. But on digging deeper, I see that there is no control over the daemon thread. It could start removing file objects even before it is added to the queue and there is no way to kill it. https://commons.apache.org/proper/commons-io/apidocs/src-html/org/apache/commons/io/FileCleaningTracker.html --- This implements a cleaner and more predictable Phantom Reference thread. I can write up a document on what needs to be changed to incorporate something like this, but can someone let me know if this approach would work. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-184512589 No comment on the contents of this PR (since I haven't looked at them), but I did want to note that I think that the pull request description is a little thin here. Could you add a concise summary of the changes here, their impact on the code, and motivation for why we're doing this? This helps reviewers / readers know what to focus on and also helps future readers by allowing them to understand the gist of this change without having to read the entire JIRA / discussion. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-184091762 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-184091767 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51294/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-184091233 **[Test build #51294 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51294/consoleFull)** for PR 11140 at commit [`44b16af`](https://github.com/apache/spark/commit/44b16af2b803a9d9eea2dcbfcd631b5899f9314d). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-184071981 **[Test build #51294 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51294/consoleFull)** for PR 11140 at commit [`44b16af`](https://github.com/apache/spark/commit/44b16af2b803a9d9eea2dcbfcd631b5899f9314d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-184064541 **[Test build #51293 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51293/consoleFull)** for PR 11140 at commit [`7f9dffc`](https://github.com/apache/spark/commit/7f9dffc0e98fe12a7c4e6c2373048da147df01ee). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-184064595 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51293/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-184064594 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-184043728 **[Test build #51293 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51293/consoleFull)** for PR 11140 at commit [`7f9dffc`](https://github.com/apache/spark/commit/7f9dffc0e98fe12a7c4e6c2373048da147df01ee). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183227808 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183227812 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51175/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183227646 **[Test build #51175 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51175/consoleFull)** for PR 11140 at commit [`837252a`](https://github.com/apache/spark/commit/837252a74ec87e8f1ac07e80406bf0410c9088d7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11140#discussion_r52812148
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -892,27 +911,30 @@ private[spark] class PythonBroadcast(@transient var
path: String) extends Serial
private def readObject(in: ObjectInputStream): Unit =
Utils.tryOrIOException {
val dir = new File(Utils.getLocalDir(SparkEnv.get.conf))
val file = File.createTempFile("broadcast", "", dir)
+phantomReferences += new FilePhantomReference(file, queue)
path = file.getAbsolutePath
val out = new FileOutputStream(file)
Utils.tryWithSafeFinally {
Utils.copyStream(in, out)
} {
- out.close()
+ in.close();
--- End diff --
Why change this line? I think this line makes PythonBroadcastSuite fail.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11140#discussion_r52812799
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -892,27 +911,30 @@ private[spark] class PythonBroadcast(@transient var
path: String) extends Serial
private def readObject(in: ObjectInputStream): Unit =
Utils.tryOrIOException {
val dir = new File(Utils.getLocalDir(SparkEnv.get.conf))
val file = File.createTempFile("broadcast", "", dir)
+phantomReferences += new FilePhantomReference(file, queue)
path = file.getAbsolutePath
val out = new FileOutputStream(file)
Utils.tryWithSafeFinally {
Utils.copyStream(in, out)
} {
- out.close()
+ in.close();
}
}
- /**
- * Delete the file once the object is GCed.
+ /** Create a seperate daemon thread
+ * to remove phantomreferences from queue and invoke cleanup
*/
- override def finalize() {
-if (!path.isEmpty) {
- val file = new File(path)
- if (file.exists()) {
-if (!file.delete()) {
- logWarning(s"Error deleting ${file.getPath}")
-}
+val referenceThread = new Thread(threadName){
--- End diff --
This will create a thread for each `PythonBroadcast`. We should avoid that.
Maybe use a global thread to do that? However, this will leak a thread even
if it's not used any more (e.g., after stopping SparkContext).
Maybe `finalize` is the best option.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user GayathriMurali commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183537578 @zsxwing I dont have a constructor to do object initialization for the FilePhantomReferences class. I am guessing that could also be causing python broadcast to fail with null pointer exception. I am going to fix that. I will also remove the change on 920(out.close), I tried removing that line earlier to test something and added the wrong one. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183198830 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183201556 **[Test build #51175 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51175/consoleFull)** for PR 11140 at commit [`837252a`](https://github.com/apache/spark/commit/837252a74ec87e8f1ac07e80406bf0410c9088d7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user GayathriMurali commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183141770  Test build failed spark unit tests. But I am not able to see any failures in the build results. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183136045 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51134/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183135618 **[Test build #51134 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51134/consoleFull)** for PR 11140 at commit [`837252a`](https://github.com/apache/spark/commit/837252a74ec87e8f1ac07e80406bf0410c9088d7). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183136043 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-183088846 **[Test build #51134 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51134/consoleFull)** for PR 11140 at commit [`837252a`](https://github.com/apache/spark/commit/837252a74ec87e8f1ac07e80406bf0410c9088d7). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182506639 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182506635 **[Test build #51038 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51038/consoleFull)** for PR 11140 at commit [`0af18af`](https://github.com/apache/spark/commit/0af18af63e0359c8971bf1f8eeac3f0e8c137070). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182506641 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51038/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182529110 **[Test build #51044 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51044/consoleFull)** for PR 11140 at commit [`1b3f732`](https://github.com/apache/spark/commit/1b3f732df753b0569137f34b6a4c4c8d5773c440). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182521814 **[Test build #51041 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51041/consoleFull)** for PR 11140 at commit [`7068425`](https://github.com/apache/spark/commit/7068425c27e086040dd73e73ec079e8da749754d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182577329 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182577111 **[Test build #51044 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51044/consoleFull)** for PR 11140 at commit [`1b3f732`](https://github.com/apache/spark/commit/1b3f732df753b0569137f34b6a4c4c8d5773c440). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182577331 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51044/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182506287 **[Test build #51038 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51038/consoleFull)** for PR 11140 at commit [`0af18af`](https://github.com/apache/spark/commit/0af18af63e0359c8971bf1f8eeac3f0e8c137070). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182522659 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182522650 **[Test build #51041 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51041/consoleFull)** for PR 11140 at commit [`7068425`](https://github.com/apache/spark/commit/7068425c27e086040dd73e73ec079e8da749754d). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182522662 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51041/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11140#discussion_r52544162
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -18,12 +18,17 @@
package org.apache.spark.api.python
import java.io._
+import java.lang.ref.PhantomReference
+import java.lang.ref.ReferenceQueue
import java.net._
import java.util.{ArrayList => JArrayList, Collections, List => JList, Map
=> JMap}
import scala.collection.JavaConverters._
import scala.collection.mutable
+import scala.collection.mutable.ListBuffer
import scala.language.existentials
+import scala.language.postfixOps
+import scala.ref
--- End diff --
what's this import for?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11140#discussion_r52544228
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -874,10 +890,14 @@ private class PythonAccumulatorParam(@transient
private val serverHost: String,
private[spark] class PythonBroadcast(@transient var path: String) extends
Serializable
with Logging {
- /**
- * Read data from disks, then copy it to `out`
- */
- private def writeObject(out: ObjectOutputStream): Unit =
Utils.tryOrIOException {
+ val queue = new ReferenceQueue[File]()
+ val phantomReferences = new ListBuffer[FilePhantomReference]()
+ val threadName = "WeakReference"
+/**
--- End diff --
can you fix the indentation here, and add proper blank lines (line before
the doc, not after)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/11140#discussion_r52544259
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -889,30 +909,32 @@ private[spark] class PythonBroadcast(@transient var
path: String) extends Serial
/**
* Write data into disk, using randomly generated name.
*/
- private def readObject(in: ObjectInputStream): Unit =
Utils.tryOrIOException {
+ private def readObject(in: ObjectInputStream): Unit =
Utils.tryOrIOException {
--- End diff --
fix indentation
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182634439 @GayathriMurali thanks for doing this, but can you please fix all the style violations you have? The changes should indent the code blocks consistently. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
GitHub user GayathriMurali opened a pull request: https://github.com/apache/spark/pull/11140 SPARK-12729 PhantomReferences to replace Finalize() PhantomReference to replace finalize() method in Python Broadcast. You can merge this pull request into a Git repository by running: $ git pull https://github.com/GayathriMurali/spark SPARK-12729 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/11140.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #11140 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user holdenk commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182112607 Thanks for making this PR - at first glance there are some minor styling issues that might be good to fix (see our style guide at https://cwiki.apache.org/confluence/display/SPARK/Spark+Code+Style+Guide ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182069521 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/11140#discussion_r52374340
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -892,27 +911,31 @@ private[spark] class PythonBroadcast(@transient var
path: String) extends Serial
private def readObject(in: ObjectInputStream): Unit =
Utils.tryOrIOException {
val dir = new File(Utils.getLocalDir(SparkEnv.get.conf))
val file = File.createTempFile("broadcast", "", dir)
+
+phantomReferences += new FilePhantomReference(file,queue)
+
path = file.getAbsolutePath
val out = new FileOutputStream(file)
Utils.tryWithSafeFinally {
Utils.copyStream(in, out)
} {
- out.close()
+ in.close();
}
}
-
- /**
- * Delete the file once the object is GCed.
- */
- override def finalize() {
-if (!path.isEmpty) {
- val file = new File(path)
- if (file.exists()) {
-if (!file.delete()) {
- logWarning(s"Error deleting ${file.getPath}")
+ /** Create a seperate daemon thread to remove phantomreferences from
queue and invoke cleanup */
+
+ val referenceThread= new Thread {
+ setDaemon(true)
+ override def run() {
+try {
+ val ref=queue.remove().asInstanceOf[FilePhantomReference]
+ phantomReferences-=ref
+
+} catch{
+ case e: Exception => logError(s"Error removing reference",e)
}
}
-}
- }
+}.start()
+
--- End diff --
whitespace
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user holdenk commented on a diff in the pull request:
https://github.com/apache/spark/pull/11140#discussion_r52374381
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -865,6 +869,17 @@ private class PythonAccumulatorParam(@transient
private val serverHost: String,
}
}
}
+/**
+ * Create a class that extends PhantomReference
+ */
+
+private[spark] class FilePhantomReference(@transient var f:File, var q:
ReferenceQueue[File]) extends PhantomReference(f,q){
+
+ private def cleanup()
--- End diff --
check indentation here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182128137 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182132144 **[Test build #51003 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51003/consoleFull)** for PR 11140 at commit [`14876ee`](https://github.com/apache/spark/commit/14876ee2e549c998753b577e8148832dbc4822f5). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user GayathriMurali commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182127382 Fixed styling issues. Please review --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182132682 **[Test build #51003 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51003/consoleFull)** for PR 11140 at commit [`14876ee`](https://github.com/apache/spark/commit/14876ee2e549c998753b577e8148832dbc4822f5). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182132693 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51003/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182132689 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user GayathriMurali commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182136487 Recent commit fixes all the styling bugs reported. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182138833 **[Test build #51006 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51006/consoleFull)** for PR 11140 at commit [`b6c9caf`](https://github.com/apache/spark/commit/b6c9cafa80f8bdc2540ae4e0a2172b968ad553b4). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182139322 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51006/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182139321 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182139316 **[Test build #51006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51006/consoleFull)** for PR 11140 at commit [`b6c9caf`](https://github.com/apache/spark/commit/b6c9cafa80f8bdc2540ae4e0a2172b968ad553b4). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182205390 **[Test build #51024 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51024/consoleFull)** for PR 11140 at commit [`a400264`](https://github.com/apache/spark/commit/a40026432c92462f5e50916bd64160829114e48d). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182206121 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182206130 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/51024/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/11140#issuecomment-182206105 **[Test build #51024 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/51024/consoleFull)** for PR 11140 at commit [`a400264`](https://github.com/apache/spark/commit/a40026432c92462f5e50916bd64160829114e48d). * This patch **fails Scala style tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: SPARK-12729 PhantomReferences to replace Final...
Github user zsxwing commented on a diff in the pull request:
https://github.com/apache/spark/pull/11140#discussion_r52418903
--- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala
---
@@ -889,30 +910,31 @@ private[spark] class PythonBroadcast(@transient var
path: String) extends Serial
/**
* Write data into disk, using randomly generated name.
*/
- private def readObject(in: ObjectInputStream): Unit =
Utils.tryOrIOException {
+ private def readObject(in: ObjectInputStream): Unit =
Utils.tryOrIOException {
val dir = new File(Utils.getLocalDir(SparkEnv.get.conf))
-val file = File.createTempFile("broadcast", "", dir)
+val file = File.createTempFile("broadcast", "", dir)
+phantomReferences += new FilePhantomReference(file, queue)
path = file.getAbsolutePath
val out = new FileOutputStream(file)
Utils.tryWithSafeFinally {
Utils.copyStream(in, out)
} {
- out.close()
+ in.close();
}
}
-
- /**
- * Delete the file once the object is GCed.
- */
- override def finalize() {
-if (!path.isEmpty) {
- val file = new File(path)
- if (file.exists()) {
-if (!file.delete()) {
- logWarning(s"Error deleting ${file.getPath}")
+ /** Create a seperate daemon thread to remove phantomreferences from
queue and invoke cleanup */
+
+ val referenceThread = new Thread {
--- End diff --
nit: set a thread name to help debug
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
