[GitHub] spark pull request: [SPARK-5366][EC2] Check the mode of private ke...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4162#issuecomment-71035537 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-4654][CORE] Clean up DAGScheduler getMi...

2015-01-22 Thread lianhuiwang
Github user lianhuiwang commented on a diff in the pull request: https://github.com/apache/spark/pull/4134#discussion_r23381894 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -349,34 +349,7 @@ class DAGScheduler( } private def

[GitHub] spark pull request: [SPARK-5367][SQL] Support star expression in u...

2015-01-22 Thread scwf
GitHub user scwf opened a pull request: https://github.com/apache/spark/pull/4163 [SPARK-5367][SQL] Support star expression in udf now spark sql does not support star expression in udf, run the following sql by spark-sql will get error ``` select concat( * ) from src ``

[GitHub] spark pull request: [SPARK-5367][SQL] Support star expression in u...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4163#issuecomment-71039264 [Test build #25969 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25969/consoleFull) for PR 4163 at commit [`da1da09`](https://githu

[GitHub] spark pull request: SPARK-5308 [BUILD] MD5 / SHA1 hash format does...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4161#issuecomment-71039411 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: SPARK-5308 [BUILD] MD5 / SHA1 hash format does...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4161#issuecomment-71039399 [Test build #25968 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25968/consoleFull) for PR 4161 at commit [`e25eff8`](https://gith

[GitHub] spark pull request: [SPARK-4943][SPARK-5251][SQL] Allow table name...

2015-01-22 Thread scwf
Github user scwf commented on the pull request: https://github.com/apache/spark/pull/4062#issuecomment-71040474 ping @marmbrus --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature en

[GitHub] spark pull request: [SPARK-2691][Mesos] Support for Mesos DockerIn...

2015-01-22 Thread tarnfeld
Github user tarnfeld commented on the pull request: https://github.com/apache/spark/pull/3074#issuecomment-71040782 I've got a branch based on the current spark master that cherry-picks (and fixes conflicts) these changes. I'm in the process of testing it now, and will open a differen

[GitHub] spark pull request: [SPARK-4382] Add locations parameter to Twitte...

2015-01-22 Thread viirya
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/3246#issuecomment-71042483 Hi @srowen, I refactor the codes for the comments. If you have time to review it, that would be great. Thanks. --- If your project is set up for it, you can reply to this

[GitHub] spark pull request: [SPARK-5365][MLlib] Refactor KMeans to reduce ...

2015-01-22 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/4159#issuecomment-71046761 LGTM. Merged into master. Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-5365][MLlib] Refactor KMeans to reduce ...

2015-01-22 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4159 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-3541][MLLIB] New ALS implementation wit...

2015-01-22 Thread mengxr
Github user mengxr commented on the pull request: https://github.com/apache/spark/pull/3720#issuecomment-71047509 @srowen @coderxiang Do you have more comments? I'm thinking about merging this and then port nonnegative support. After that, we can replace the ALS implementation under "

[GitHub] spark pull request: [SPARK-4382] Add locations parameter to Twitte...

2015-01-22 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/3246#discussion_r23388400 --- Diff: external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterInputDStream.scala --- @@ -85,9 +93,14 @@ class TwitterReceiver(

[GitHub] spark pull request: [SPARK-4382] Add locations parameter to Twitte...

2015-01-22 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/3246#discussion_r23388449 --- Diff: external/twitter/src/main/scala/org/apache/spark/streaming/twitter/TwitterUtils.scala --- @@ -53,7 +67,7 @@ object TwitterUtils { * @param j

[GitHub] spark pull request: [SPARK-4382] Add locations parameter to Twitte...

2015-01-22 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3246#issuecomment-71050223 I think this is a lot better without all the overloads, yes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-5367][SQL] Support star expression in u...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4163#issuecomment-71051891 [Test build #25969 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25969/consoleFull) for PR 4163 at commit [`da1da09`](https://gith

[GitHub] spark pull request: [SPARK-5367][SQL] Support star expression in u...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4163#issuecomment-71051906 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-4382] Add locations parameter to Twitte...

2015-01-22 Thread viirya
Github user viirya commented on the pull request: https://github.com/apache/spark/pull/3246#issuecomment-71052031 Fixed in new commit. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3541][MLLIB] New ALS implementation wit...

2015-01-22 Thread srowen
Github user srowen commented on the pull request: https://github.com/apache/spark/pull/3720#issuecomment-71053076 @mengxr Given how familiar you are with this implementation, and the tests, I can only be pretty sure it works. I didn't see any style issues, and thought through some of

[GitHub] spark pull request: [SPARK-5347][CORE] Change FileSplit to InputSp...

2015-01-22 Thread sryza
Github user sryza commented on the pull request: https://github.com/apache/spark/pull/4150#issuecomment-71055028 I think this is a duplicate of #4050, which only adds support for `CombineFileSplit`s. We shouldn't add support for generic `InputSplit`s because many input formats do not

[GitHub] spark pull request: [SPARK-2669] [yarn] Distribute client configur...

2015-01-22 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/4142#issuecomment-71055700 Hi @redbaron , > IMHO Spark should mimic what Hadoop MR is doing, which is building a Configuration object and then serializing into xml That has the limi

[GitHub] spark pull request: SPARK-5370. [YARN] Remove some unnecessary syn...

2015-01-22 Thread sryza
GitHub user sryza opened a pull request: https://github.com/apache/spark/pull/4164 SPARK-5370. [YARN] Remove some unnecessary synchronization in YarnAlloca... ...tor You can merge this pull request into a Git repository by running: $ git pull https://github.com/sryza/spark sand

[GitHub] spark pull request: SPARK-5370. [YARN] Remove some unnecessary syn...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4164#issuecomment-71056567 [Test build #25970 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25970/consoleFull) for PR 4164 at commit [`0c8d736`](https://githu

[GitHub] spark pull request: [SPARK-3562]Periodic cleanup event logs

2015-01-22 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2471#issuecomment-71058558 I'm not a committer so I can't merge the patch. But it has merge conflicts now, so that at least needs to be fixed. --- If your project is set up for it, you can reply to

[GitHub] spark pull request: SPARK-5370. [YARN] Remove some unnecessary syn...

2015-01-22 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/4164#issuecomment-71058662 changes look fine, waiting on jenkins. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project d

[GitHub] spark pull request: [MLLIB] SPARK-5362 (4526, 2372) Gradient and O...

2015-01-22 Thread avulanov
Github user avulanov commented on the pull request: https://github.com/apache/spark/pull/4152#issuecomment-71062463 Some unrelated (streaming kafka) test failed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pr

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4155#issuecomment-71065082 [Test build #25971 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25971/consoleFull) for PR 4155 at commit [`f135a8e`](https://githu

[GitHub] spark pull request: SPARK-5370. [YARN] Remove some unnecessary syn...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4164#issuecomment-71068119 [Test build #25970 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25970/consoleFull) for PR 4164 at commit [`0c8d736`](https://gith

[GitHub] spark pull request: SPARK-5370. [YARN] Remove some unnecessary syn...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4164#issuecomment-71068132 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-5063] More helpful error messages for s...

2015-01-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/3884#discussion_r23397600 --- Diff: core/src/main/scala/org/apache/spark/SparkContext.scala --- @@ -955,6 +977,11 @@ class SparkContext(config: SparkConf) extends Logging with Execut

[GitHub] spark pull request: [SPARK-5063] More helpful error messages for s...

2015-01-22 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/3884#discussion_r23397684 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -76,10 +76,25 @@ import org.apache.spark.util.random.{BernoulliSampler, PoissonSampler,

[GitHub] spark pull request: [SPARK-5012][MLLib][PySpark]Python API for Gau...

2015-01-22 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4059#discussion_r23399715 --- Diff: examples/src/main/python/mllib/gaussian_mixture_model.py --- @@ -0,0 +1,65 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-5012][MLLib][PySpark]Python API for Gau...

2015-01-22 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4059#discussion_r23399730 --- Diff: examples/src/main/python/mllib/gaussian_mixture_model.py --- @@ -0,0 +1,65 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-5012][MLLib][PySpark]Python API for Gau...

2015-01-22 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4059#discussion_r23399766 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala --- @@ -280,6 +280,48 @@ class PythonMLLibAPI extends Serializable {

[GitHub] spark pull request: [SPARK-5012][MLLib][PySpark]Python API for Gau...

2015-01-22 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4059#discussion_r23399719 --- Diff: examples/src/main/python/mllib/gaussian_mixture_model.py --- @@ -0,0 +1,65 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-5012][MLLib][PySpark]Python API for Gau...

2015-01-22 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4059#discussion_r23399756 --- Diff: examples/src/main/python/mllib/gaussian_mixture_model.py --- @@ -0,0 +1,65 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-5012][MLLib][PySpark]Python API for Gau...

2015-01-22 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4059#discussion_r23399765 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/api/python/PythonMLLibAPI.scala --- @@ -280,6 +280,48 @@ class PythonMLLibAPI extends Serializable {

[GitHub] spark pull request: [SPARK-5012][MLLib][PySpark]Python API for Gau...

2015-01-22 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4059#discussion_r23399804 --- Diff: python/pyspark/mllib/clustering.py --- @@ -86,6 +86,68 @@ def train(cls, rdd, k, maxIterations=100, runs=1, initializationMode="k-means||"

[GitHub] spark pull request: [SPARK-5012][MLLib][PySpark]Python API for Gau...

2015-01-22 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4059#discussion_r23399808 --- Diff: python/pyspark/mllib/clustering.py --- @@ -86,6 +86,68 @@ def train(cls, rdd, k, maxIterations=100, runs=1, initializationMode="k-means||"

[GitHub] spark pull request: [SPARK-5012][MLLib][PySpark]Python API for Gau...

2015-01-22 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4059#discussion_r23399900 --- Diff: python/pyspark/mllib/clustering.py --- @@ -86,6 +86,68 @@ def train(cls, rdd, k, maxIterations=100, runs=1, initializationMode="k-means||"

[GitHub] spark pull request: [SPARK-5012][MLLib][PySpark]Python API for Gau...

2015-01-22 Thread mengxr
Github user mengxr commented on a diff in the pull request: https://github.com/apache/spark/pull/4059#discussion_r23399968 --- Diff: examples/src/main/python/mllib/gaussian_mixture_model.py --- @@ -0,0 +1,65 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one

[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

2015-01-22 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/3920#discussion_r23403050 --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala --- @@ -23,15 +23,27 @@ import org.apache.spark.api.python.

[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

2015-01-22 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/3920#discussion_r23403193 --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala --- @@ -23,15 +23,27 @@ import org.apache.spark.api.python.

[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

2015-01-22 Thread MLnick
Github user MLnick commented on a diff in the pull request: https://github.com/apache/spark/pull/3920#discussion_r23403597 --- Diff: examples/src/main/python/hbase_inputformat.py --- @@ -16,6 +16,7 @@ # import sys +import ast from pyspark import SparkC

[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

2015-01-22 Thread MLnick
Github user MLnick commented on the pull request: https://github.com/apache/spark/pull/3920#issuecomment-71085426 @GenTang overall looks fine to me - just noted a few minor comments / questions, and in particular it would be good to expand on the sample data in the example just to sho

[GitHub] spark pull request: SPARK-5370. [YARN] Remove some unnecessary syn...

2015-01-22 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/4164 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4155#issuecomment-71086649 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4155#issuecomment-71086637 **[Test build #25971 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25971/consoleFull)** for PR 4155 at commit [`f135a8e`](https://git

[GitHub] spark pull request: SPARK-5357: Update commons-codec version to 1....

2015-01-22 Thread MattWhelan
Github user MattWhelan commented on the pull request: https://github.com/apache/spark/pull/4153#issuecomment-71088861 @srowen I actually tried that. It turns out there are some pretty bad issues with the classloader used for that feature at present. I've got another PR for that comi

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread MattWhelan
GitHub user MattWhelan opened a pull request: https://github.com/apache/spark/pull/4165 SPARK-5358: Rework the classloader impelementation. The fundamental issue is that you can't change the delegation scheme without overriding loadClass (rather than findClass). And, if you o

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4165#issuecomment-71089809 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread nchammas
Github user nchammas commented on the pull request: https://github.com/apache/spark/pull/4165#issuecomment-71092175 I'll leave it to a maintainer to confirm, but I believe that Java 6 support is an official commitment of the Spark project. Dropping that support would be a major decisi

[GitHub] spark pull request: SPARK-5308 [BUILD] MD5 / SHA1 hash format does...

2015-01-22 Thread nchammas
Github user nchammas commented on a diff in the pull request: https://github.com/apache/spark/pull/4161#discussion_r23407106 --- Diff: dev/create-release/create-release.sh --- @@ -122,8 +122,14 @@ if [[ ! "$@" =~ --package-only ]]; then for file in $(find . -type f) do

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/4165#issuecomment-71093445 The 1.7 dependency will probably be shot down pretty quickly. But one way you could work around it in Scala is to have a factory method to create instances of this classlo

[GitHub] spark pull request: [SPARK-3541][MLLIB] New ALS implementation wit...

2015-01-22 Thread coderxiang
Github user coderxiang commented on the pull request: https://github.com/apache/spark/pull/3720#issuecomment-71094167 @mengxr the logic and style also look good to me. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If yo

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/4165#discussion_r23408475 --- Diff: core/src/main/java/org/apache/spark/classloader/GreedyUrlClassLoader.java --- @@ -0,0 +1,61 @@ +package org.apache.spark.classloader; --- End

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4155#issuecomment-71096349 [Test build #25972 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25972/consoleFull) for PR 4155 at commit [`abc7db4`](https://githu

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread MattWhelan
Github user MattWhelan commented on a diff in the pull request: https://github.com/apache/spark/pull/4165#discussion_r23411517 --- Diff: core/src/main/java/org/apache/spark/classloader/GreedyUrlClassLoader.java --- @@ -0,0 +1,61 @@ +package org.apache.spark.classloader; ---

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread MattWhelan
Github user MattWhelan commented on the pull request: https://github.com/apache/spark/pull/4165#issuecomment-71105198 @vanzin The way the registration method works is also really weird. It takes no params, you'll notice. In ClassLoader, it calls a Reflection native method to ac

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/4165#issuecomment-71105956 @MattWhelan I see. Still, from reading the docs, it's not clear why you have to call that method. What happens if you don't? Does the JVM do locking for you (potentially s

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/4165#issuecomment-71106397 @MattWhelan I see. Still, I'm not sure we need to call that method at all. From the docs: In environments in which the delegation model is not strictly hierar

[GitHub] spark pull request: [SPARK-2996] Implement userClassPathFirst for ...

2015-01-22 Thread MattWhelan
Github user MattWhelan commented on a diff in the pull request: https://github.com/apache/spark/pull/3233#discussion_r23413276 --- Diff: core/src/main/scala/org/apache/spark/executor/ExecutorURLClassLoader.scala --- @@ -32,36 +35,40 @@ private[spark] trait MutableURLClassLoader ext

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-71106555 [Test build #25974 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25974/consoleFull) for PR 4129 at commit [`bca1080`](https://githu

[GitHub] spark pull request: SPARK-5308 [BUILD] MD5 / SHA1 hash format does...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4161#issuecomment-71106554 [Test build #25973 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25973/consoleFull) for PR 4161 at commit [`70d09d0`](https://githu

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread MattWhelan
Github user MattWhelan commented on the pull request: https://github.com/apache/spark/pull/4165#issuecomment-71107518 @vanzin There needs to be some locking. The 1.6 version of loadClass is simply a synchronized method (on 'this'). That's a viable alternative, and the way I'll

[GitHub] spark pull request: [SPARK-2996] Implement userClassPathFirst for ...

2015-01-22 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3233#discussion_r23413919 --- Diff: core/src/main/scala/org/apache/spark/executor/ExecutorURLClassLoader.scala --- @@ -32,36 +35,40 @@ private[spark] trait MutableURLClassLoader extends

[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

2015-01-22 Thread GenTang
Github user GenTang commented on a diff in the pull request: https://github.com/apache/spark/pull/3920#discussion_r23413908 --- Diff: examples/src/main/scala/org/apache/spark/examples/pythonconverters/HBaseConverters.scala --- @@ -23,15 +23,27 @@ import org.apache.spark.api.python

[GitHub] spark pull request: [SPARK-5090][examples] The improvement of pyth...

2015-01-22 Thread GenTang
Github user GenTang commented on a diff in the pull request: https://github.com/apache/spark/pull/3920#discussion_r23414030 --- Diff: examples/src/main/python/hbase_inputformat.py --- @@ -16,6 +16,7 @@ # import sys +import ast from pyspark import Spark

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-71108212 [Test build #25975 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25975/consoleFull) for PR 4129 at commit [`8b088b2`](https://githu

[GitHub] spark pull request: [SPARK-2996] Implement userClassPathFirst for ...

2015-01-22 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/3233#discussion_r23414288 --- Diff: core/src/main/scala/org/apache/spark/executor/ExecutorURLClassLoader.scala --- @@ -32,36 +35,40 @@ private[spark] trait MutableURLClassLoader extends

[GitHub] spark pull request: [SPARK-5361]python tuple not supported while c...

2015-01-22 Thread wingchen
Github user wingchen commented on the pull request: https://github.com/apache/spark/pull/4146#issuecomment-7633 Found a good way to reproduce it: ``` from pyspark.rdd import RDD dl = [ (u'2', {u'director': u'David Lean'}), (u'7', {u'director':

[GitHub] spark pull request: Spark 3883: SSL support for HttpServer and Akk...

2015-01-22 Thread jacek-lewandowski
Github user jacek-lewandowski commented on the pull request: https://github.com/apache/spark/pull/3571#issuecomment-71112931 @vanzin it is low hanging fruit to achieve that --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well.

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread MattWhelan
GitHub user MattWhelan opened a pull request: https://github.com/apache/spark/pull/4166 SPARK-5358: Rework the classloader impelementation. The fundamental issue is that you can't change the delegation scheme without overriding loadClass (rather than findClass). And, if you o

[GitHub] spark pull request: Spark 3883: SSL support for HttpServer and Akk...

2015-01-22 Thread jacek-lewandowski
Github user jacek-lewandowski commented on a diff in the pull request: https://github.com/apache/spark/pull/3571#discussion_r23417469 --- Diff: core/src/main/scala/org/apache/spark/deploy/ApplicationDescription.scala --- @@ -28,5 +28,14 @@ private[spark] class ApplicationDescriptio

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4166#issuecomment-71115098 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your pro

[GitHub] spark pull request: [SPARK-2996] Implement userClassPathFirst for ...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3233#issuecomment-71115199 [Test build #25976 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25976/consoleFull) for PR 3233 at commit [`fbb8ab5`](https://githu

[GitHub] spark pull request: SPARK-5358: Rework the classloader impelementa...

2015-01-22 Thread MattWhelan
Github user MattWhelan commented on the pull request: https://github.com/apache/spark/pull/4166#issuecomment-71115860 BTW, I spent a few minutes pondering the deadlock scenario, and the delegation changes you see here. I'm pretty sure we're safe with 1.6-style coarse locking, because

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4155#issuecomment-71115931 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4155#issuecomment-71115924 **[Test build #25972 timed out](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25972/consoleFull)** for PR 4155 at commit [`abc7db4`](https://git

[GitHub] spark pull request: Spark 3883: SSL support for HttpServer and Akk...

2015-01-22 Thread jacek-lewandowski
Github user jacek-lewandowski commented on a diff in the pull request: https://github.com/apache/spark/pull/3571#discussion_r23418026 --- Diff: core/src/test/scala/org/apache/spark/deploy/worker/WorkerTest.scala --- @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software F

[GitHub] spark pull request: Spark 3883: SSL support for HttpServer and Akk...

2015-01-22 Thread jacek-lewandowski
Github user jacek-lewandowski commented on a diff in the pull request: https://github.com/apache/spark/pull/3571#discussion_r23417992 --- Diff: core/src/test/scala/org/apache/spark/deploy/worker/WorkerTest.scala --- @@ -0,0 +1,56 @@ +/* + * Licensed to the Apache Software F

[GitHub] spark pull request: [SPARK-2996] Implement userClassPathFirst for ...

2015-01-22 Thread MattWhelan
Github user MattWhelan commented on the pull request: https://github.com/apache/spark/pull/3233#issuecomment-71116000 Sorry, "cache" was vague. It's often called that, but what it really refers to is the call to findLoadedClass(name) near the top of the loadClass method in ClassLoade

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-01-22 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23418148 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DeterministicKafkaInputDStream.scala --- @@ -0,0 +1,123 @@ +/* + * L

[GitHub] spark pull request: [SPARK-2996] Implement userClassPathFirst for ...

2015-01-22 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/3233#issuecomment-71116585 @MattWhelan actually I'm not calling `findClass` at all, exactly because I ran into the linkage error with the original code. So my change is pretty much the same thing yo

[GitHub] spark pull request: [SPARK-5361]python tuple not supported while c...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4146#issuecomment-71116623 [Test build #25977 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25977/consoleFull) for PR 4146 at commit [`4cf1187`](https://githu

[GitHub] spark pull request: [SPARK-5361]python tuple not supported while c...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4146#issuecomment-71116741 [Test build #25977 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25977/consoleFull) for PR 4146 at commit [`4cf1187`](https://gith

[GitHub] spark pull request: [SPARK-5361]python tuple not supported while c...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4146#issuecomment-71116745 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-4879] Use the Spark driver to authorize...

2015-01-22 Thread mccheah
Github user mccheah commented on the pull request: https://github.com/apache/spark/pull/4155#issuecomment-71116954 Looks like the tests timed out. This change is probably a large performance bottleneck, as communication back to the driver on every commit task is expensive? --- If yo

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-71117106 [Test build #25974 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25974/consoleFull) for PR 4129 at commit [`bca1080`](https://gith

[GitHub] spark pull request: SPARK-5308 [BUILD] MD5 / SHA1 hash format does...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4161#issuecomment-71117062 [Test build #25973 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25973/consoleFull) for PR 4161 at commit [`70d09d0`](https://gith

[GitHub] spark pull request: [SPARK-5361]python tuple not supported while c...

2015-01-22 Thread wingchen
Github user wingchen commented on the pull request: https://github.com/apache/spark/pull/4146#issuecomment-71117026 @JoshRosen Added in a test case. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project d

[GitHub] spark pull request: SPARK-5308 [BUILD] MD5 / SHA1 hash format does...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4161#issuecomment-71117069 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-71117113 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-5361]python tuple not supported while c...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4146#issuecomment-71117363 [Test build #25978 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25978/consoleFull) for PR 4146 at commit [`126be6b`](https://githu

[GitHub] spark pull request: [SPARK-5361]python tuple not supported while c...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4146#issuecomment-71117488 [Test build #25978 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25978/consoleFull) for PR 4146 at commit [`126be6b`](https://gith

[GitHub] spark pull request: [SPARK-5361]python tuple not supported while c...

2015-01-22 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/4146#issuecomment-71117491 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/25

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-01-22 Thread koeninger
Github user koeninger commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23418815 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DeterministicKafkaInputDStream.scala --- @@ -0,0 +1,123 @@ +/* + * License

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-01-22 Thread harishreedharan
Github user harishreedharan commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23418917 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/DeterministicKafkaInputDStream.scala --- @@ -0,0 +1,123 @@ +/* + * L

[GitHub] spark pull request: Spark 3883: SSL support for HttpServer and Akk...

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/3571#issuecomment-71118034 [Test build #25979 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25979/consoleFull) for PR 3571 at commit [`054c3d8`](https://githu

[GitHub] spark pull request: [SPARK-5337][Mesos][Standalone] respect spark....

2015-01-22 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/4129#issuecomment-71118466 [Test build #25975 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/25975/consoleFull) for PR 4129 at commit [`8b088b2`](https://gith

<    1   2   3   4   >