[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-73291095 I've cherry-picked this into `branch-1.3` (1.3.0) as well. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4258 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-73220358 This looks like a logical change, since it makes sense to use the provided default classloader to register user classes, just as it's already used to create the serializer. Given reviews from 3 other people and confirmation this does what's intended, it seems like a good fix to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72786916 [Test build #26707 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26707/consoleFull) for PR 4258 at commit [`73b719f`](https://github.com/apache/spark/commit/73b719f5bc7b69fca8d51cb8b991074dc92e50ed). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72786922 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26707/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72780813 [Test build #26707 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26707/consoleFull) for PR 4258 at commit [`73b719f`](https://github.com/apache/spark/commit/73b719f5bc7b69fca8d51cb8b991074dc92e50ed). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72780420 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72543087 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26528/ Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72543067 [Test build #26528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26528/consoleFull) for PR 4258 at commit [`73b719f`](https://github.com/apache/spark/commit/73b719f5bc7b69fca8d51cb8b991074dc92e50ed). * This patch **fails Spark unit tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72533654 [Test build #26528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26528/consoleFull) for PR 4258 at commit [`73b719f`](https://github.com/apache/spark/commit/73b719f5bc7b69fca8d51cb8b991074dc92e50ed). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72405667 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user sryza commented on a diff in the pull request:
https://github.com/apache/spark/pull/4258#discussion_r23855340
--- Diff:
core/src/main/scala/org/apache/spark/serializer/KryoSerializer.scala ---
@@ -97,7 +87,9 @@ class KryoSerializer(conf: SparkConf)
// Use the default classloader when calling the user registrator.
Thread.currentThread.setContextClassLoader(classLoader)
// Register classes given through spark.kryo.classesToRegister.
- classesToRegister.foreach { clazz = kryo.register(clazz) }
+ classesToRegister.split(',')
--- End diff --
Can we do the splitting and filtering during initialization so that we
don't incur them for each task?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user lianhuiwang commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72298982 @sryza thank you. I have updated for your comment. Can you review again? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72298961 [Test build #26447 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26447/consoleFull) for PR 4258 at commit [`73b719f`](https://github.com/apache/spark/commit/73b719f5bc7b69fca8d51cb8b991074dc92e50ed). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72301751 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26447/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-72301748 [Test build #26447 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26447/consoleFull) for PR 4258 at commit [`73b719f`](https://github.com/apache/spark/commit/73b719f5bc7b69fca8d51cb8b991074dc92e50ed). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds no public classes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-71964178 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/26281/ Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-71958741 [Test build #26281 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26281/consoleFull) for PR 4258 at commit [`64cf306`](https://github.com/apache/spark/commit/64cf3069457de9ba303faa0018a82ac3ec5eb6a0). * This patch merges cleanly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user zzcclp commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-71959063 LGTM, I test this with spark-branch 1.2, it works correctly. :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
GitHub user lianhuiwang opened a pull request: https://github.com/apache/spark/pull/4258 [SPARK-5470][Core]use defaultClassLoader to load classes of classesToRegister in KryoSeria... Now KryoSerializer load classes of classesToRegister at the time of its initialization. when we set spark.kryo.classesToRegister=class1, it will throw SparkException(Failed to load class to register with Kryo. because in KryoSerializer's initialization, classLoader cannot include class of user's jars. we need to use defaultClassLoader of Serializer in newKryo(), because executor will reset defaultClassLoader of Serializer after Serializer's initialization. You can merge this pull request into a Git repository by running: $ git pull https://github.com/lianhuiwang/spark SPARK-5470 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/spark/pull/4258.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #4258 commit 64cf3069457de9ba303faa0018a82ac3ec5eb6a0 Author: lianhuiwang lianhuiwan...@gmail.com Date: 2015-01-29T02:25:21Z use defaultClassLoader to load classes of classesToRegister in KryoSerializer --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark pull request: [SPARK-5470][Core]use defaultClassLoader to lo...
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4258#issuecomment-71964173 [Test build #26281 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/26281/consoleFull) for PR 4258 at commit [`64cf306`](https://github.com/apache/spark/commit/64cf3069457de9ba303faa0018a82ac3ec5eb6a0). * This patch **passes all tests**. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * `class LogisticRegression(JavaEstimator, HasFeaturesCol, HasLabelCol, HasPredictionCol, HasMaxIter,` * `class LogisticRegressionModel(JavaModel):` * `class Tokenizer(JavaTransformer, HasInputCol, HasOutputCol):` * `class HashingTF(JavaTransformer, HasInputCol, HasOutputCol, HasNumFeatures):` * `class Param(object):` * `class Params(Identifiable):` * `template = '''class Has$Name(Params):` * `class HasMaxIter(Params):` * `class HasRegParam(Params):` * `class HasFeaturesCol(Params):` * `class HasLabelCol(Params):` * `class HasPredictionCol(Params):` * `class HasInputCol(Params):` * `class HasOutputCol(Params):` * `class HasNumFeatures(Params):` * `class Estimator(Params):` * `class Transformer(Params):` * `class Pipeline(Estimator):` * `class PipelineModel(Transformer):` * `class Identifiable(object):` * `class JavaWrapper(Params):` * `class JavaEstimator(Estimator, JavaWrapper):` * `class JavaTransformer(Transformer, JavaWrapper):` * `class JavaModel(JavaTransformer):` --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
