[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47497190 Thanks. I'm merging this in master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/1245 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user jerryshao commented on a diff in the pull request:
https://github.com/apache/spark/pull/1245#discussion_r14333421
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -96,15 +98,15 @@ class HashPartitioner(partitions: Int) extends
Partitioner {
* the value of `partitions`.
*/
class RangePartitioner[K : Ordering : ClassTag, V](
-partitions: Int,
+var partitions: Int,
--- End diff --
Hi Reynold, thanks for your comments, will this field `partitions` be used
in executor side, I think this field can be transient according to my
knowledge, am I miss something?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47495454 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47495446 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47497119 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16257/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47497117 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1245#discussion_r14322912
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -96,15 +98,15 @@ class HashPartitioner(partitions: Int) extends
Partitioner {
* the value of `partitions`.
*/
class RangePartitioner[K : Ordering : ClassTag, V](
-partitions: Int,
+var partitions: Int,
--- End diff --
actually we probably want to make partitions just a constructor field, and
have an internal variable private var numPartitions, so this class remains
immutable from the user's point of view.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user rxin commented on a diff in the pull request:
https://github.com/apache/spark/pull/1245#discussion_r14322913
--- Diff: core/src/main/scala/org/apache/spark/Partitioner.scala ---
@@ -96,15 +98,15 @@ class HashPartitioner(partitions: Int) extends
Partitioner {
* the value of `partitions`.
*/
class RangePartitioner[K : Ordering : ClassTag, V](
-partitions: Int,
+var partitions: Int,
--- End diff --
make sure you replace references to partitions with numPartitions once you
add the field
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47443388 This LGTM, besides @rxin's earlier comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
GitHub user jerryshao opened a pull request: https://github.com/apache/spark/pull/1245 [SPARK-2104] Fix task serializing issues when sort with Java non serializable class Details can be see in [SPARK-2104](https://issues.apache.org/jira/browse/SPARK-2104). This work is based on Reynold's work, add some unit tests to validate the issue. @rxin , would you please take a look at this PR, thanks a lot. You can merge this pull request into a Git repository by running: $ git pull https://github.com/jerryshao/apache-spark SPARK-2104 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/1245.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1245 commit 47d763cc817dc1fe05e7caf1bf8357a5c427a256 Author: jerryshao saisai.s...@intel.com Date: 2014-06-27T08:23:21Z Fix task serializing issue when sort with Java non serializable class commit 2b41917714dc2c33c5cf0d544945a8a651360c2b Author: jerryshao saisai.s...@intel.com Date: 2014-06-27T09:14:26Z Minor changes --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47324256 Merged build started. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47324241 Merged build triggered. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47327948 Merged build finished. All automated tests passed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47327950 All automated tests passed. Refer to this link for build results: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/16205/ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] spark pull request: [SPARK-2104] Fix task serializing issues when ...
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/1245#issuecomment-47409432 This looks good to me, but somebody else should review thsi too since I wrote half of the code here ... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
