[GitHub] spark pull request: [SPARK-2974] [SPARK-2975] Fix two bugs related...

2014-08-19 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2002#discussion_r16398417 --- Diff: core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala --- @@ -62,7 +62,7 @@ private[spark] class PythonRDD( val env =

[GitHub] spark pull request: [SPARK-2974] [SPARK-2975] Fix two bugs related...

2014-08-19 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/2002#discussion_r16398508 --- Diff: core/src/main/scala/org/apache/spark/storage/DiskBlockManager.scala --- @@ -131,10 +132,9 @@ private[spark] class

[GitHub] spark pull request: [SPARK-2974] [SPARK-2975] Fix two bugs related...

2014-08-19 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2002#issuecomment-52595071 Hey Josh - this LGTM with some minor comments. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SQL] add note of use synchronizedMap in SQLCo...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1996#issuecomment-52595173 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18813/consoleFull) for PR 1996 at commit

[GitHub] spark pull request: [SPARK-3103] [PySpark] fix saveAsTextFile() wi...

2014-08-19 Thread davies
Github user davies closed the pull request at: https://github.com/apache/spark/pull/2018 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-3103] [PySpark] fix saveAsTextFile() wi...

2014-08-19 Thread davies
Github user davies commented on the pull request: https://github.com/apache/spark/pull/2018#issuecomment-52595281 merged --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-2974] [SPARK-2975] Fix two bugs related...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2002#issuecomment-52595493 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18814/consoleFull) for PR 2002 at commit

[GitHub] spark pull request: [SPARK-2790] [PySpark] fix zip with serializer...

2014-08-19 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/1894#issuecomment-52595899 My impression from the StackOverflow example was that both RDDs had the _same_ number of items, since `f.count()` and `ind.count()` were both 52. It doesn't look like

[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...

2014-08-19 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52595993 Pulled this from the jenkins log ``` 14/08/18 22:52:57.452 INFO BlockManager: Found block broadcast_13 locally 14/08/18 22:52:57.453 ERROR Executor:

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52596008 cc @shivaram @mosharaf --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52596109 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18816/consoleFull) for PR 2030 at commit

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/2030#discussion_r16399340 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -27,41 +29,87 @@ import org.apache.spark.io.CompressionCodec

[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...

2014-08-19 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/1545#discussion_r16399357 --- Diff: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala --- @@ -902,14 +913,19 @@ class DAGScheduler( } val stage =

Re: [GitHub] spark pull request: [SPARK-3045] [SPARK-3046] Make Serializer inte...

2014-08-19 Thread Debasish Das
With the fixes, I could run it fine on top of branch-1.0 On master when running on YARN I am getting another KryoException: Exception in thread main org.apache.spark.SparkException: Job aborted due to stage failure: Task 247 in stage 52.0 failed 4 times, most recent failure: Lost task 247.3 in

[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...

2014-08-19 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1958#issuecomment-52597090 unit test is failing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/2030#discussion_r16399641 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -27,41 +29,87 @@ import org.apache.spark.io.CompressionCodec

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1935#issuecomment-52597474 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18817/consoleFull) for PR 1935 at commit

[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52597470 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18818/consoleFull) for PR 1545 at commit

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52597491 Ran into some task failures when testing this commit on EC2 with the SchedulerThroughputTest: ``` 14/08/19 07:01:24 WARN scheduler.TaskSetManager: Lost

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread shivaram
Github user shivaram commented on a diff in the pull request: https://github.com/apache/spark/pull/2030#discussion_r16399782 --- Diff: core/src/main/scala/org/apache/spark/broadcast/TorrentBroadcast.scala --- @@ -109,99 +137,30 @@ private[spark] class TorrentBroadcast[T:

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1935#issuecomment-52597531 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18817/consoleFull) for PR 1935 at commit

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread shivaram
Github user shivaram commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52598348 @rxin -- Nice work in reducing this to 2 RPCs. The patch looks good in terms of maintaining the same functionality as before. I'll wait for the Snappy fix and for

[GitHub] spark pull request: [SPARK-2790] [PySpark] fix zip with serializer...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1894#issuecomment-52598488 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18820/consoleFull) for PR 1894 at commit

[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52598493 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18822/consoleFull) for PR 1545 at commit

[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...

2014-08-19 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1958#discussion_r16400150 --- Diff: external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala --- @@ -0,0 +1,206 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1723#issuecomment-52598483 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18821/consoleFull) for PR 1723 at commit

[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...

2014-08-19 Thread mubarak
Github user mubarak commented on the pull request: https://github.com/apache/spark/pull/1723#issuecomment-52598991 *Streaming example:* ![screen shot 2014-08-19 at 12 26 23 am](https://cloud.githubusercontent.com/assets/668134/3962878/189b730e-2772-11e4-8fcc-79ce694a36f0.png)

[GitHub] spark pull request: [SPARK-3074] [PySpark] support groupByKey() wi...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1977#issuecomment-52599265 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18823/consoleFull) for PR 1977 at commit

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1935#issuecomment-52599287 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18824/consoleFull) for PR 1935 at commit

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1935#issuecomment-52599351 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18824/consoleFull) for PR 1935 at commit

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52599404 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18816/consoleFull) for PR 2030 at commit

[GitHub] spark pull request: [SPARK-3074] [PySpark] support groupByKey() wi...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1977#issuecomment-52599524 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18815/consoleFull) for PR 1977 at commit

[GitHub] spark pull request: [SPARK-2974] [SPARK-2975] Fix two bugs related...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2002#issuecomment-52599590 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18814/consoleFull) for PR 2002 at commit

[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52599671 **Tests timed out** after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SQL] add note of use synchronizedMap in SQLCo...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1996#issuecomment-52599963 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18813/consoleFull) for PR 1996 at commit

[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...

2014-08-19 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1958#discussion_r16400756 --- Diff: external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala --- @@ -0,0 +1,206 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread rxin
Github user rxin commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52600292 Ok I pushed a new version that should've addressed all the comments and fixed the bug. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52600436 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18825/consoleFull) for PR 2030 at commit

[GitHub] spark pull request: [SPARK-3063][SQL] ExistingRdd should convert M...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1963#issuecomment-52600825 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18826/consoleFull) for PR 1963 at commit

[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...

2014-08-19 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1958#discussion_r16401133 --- Diff: external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala --- @@ -0,0 +1,206 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...

2014-08-19 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1958#discussion_r16401169 --- Diff: external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala --- @@ -0,0 +1,206 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...

2014-08-19 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1958#discussion_r16401236 --- Diff: external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala --- @@ -0,0 +1,206 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...

2014-08-19 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1958#discussion_r16401267 --- Diff: external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala --- @@ -0,0 +1,206 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...

2014-08-19 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1958#discussion_r16401356 --- Diff: external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala --- @@ -0,0 +1,206 @@ +/* + * Licensed to the

[GitHub] spark pull request: [SPARK-3114] [PySpark] Fix Python UDFs in Spar...

2014-08-19 Thread marmbrus
Github user marmbrus commented on the pull request: https://github.com/apache/spark/pull/2027#issuecomment-52601883 Thanks guys! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-2974] [SPARK-2975] Fix two bugs related...

2014-08-19 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2002#issuecomment-52601986 Just tested this with YARN on HDP-2.1 - Hadoop 2.4 and it worked fine in both `yarn-master` and `yarn-client` modes (ran the SparkPi example). --- If your project is

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52602155 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18827/consoleFull) for PR 2030 at commit

[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1723#issuecomment-52602246 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18821/consoleFull) for PR 1723 at commit

[GitHub] spark pull request: [SPARK-3054][STREAMING] Add unit tests for Spa...

2014-08-19 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1958#discussion_r16401593 --- Diff: external/flume-sink/src/test/scala/org/apache/spark/streaming/flume/sink/SparkSinkSuite.scala --- @@ -0,0 +1,206 @@ +/* + * Licensed to the

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1935#issuecomment-52602565 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18828/consoleFull) for PR 1935 at commit

[GitHub] spark pull request: [WIP][SPARK-2554][SQL] CountDistinct and SumDi...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1935#issuecomment-52602693 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18828/consoleFull) for PR 1935 at commit

[GitHub] spark pull request: add spark.shuffle.spill.batchSize and fix the ...

2014-08-19 Thread CrazyJvm
GitHub user CrazyJvm opened a pull request: https://github.com/apache/spark/pull/2031 add spark.shuffle.spill.batchSize and fix the value of spark.shuffle.manager ```scala private val serializerBatchSize = sparkConf.getLong(spark.shuffle.spill.batchSize, 1) ``` add

[GitHub] spark pull request: [SPARK-2929][SQL] Refactored Thrift server and...

2014-08-19 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/1856#issuecomment-52603023 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-2790] [PySpark] fix zip with serializer...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1894#issuecomment-52603153 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18820/consoleFull) for PR 1894 at commit

[GitHub] spark pull request: add spark.shuffle.spill.batchSize and fix the ...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2031#issuecomment-52603467 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18831/consoleFull) for PR 2031 at commit

[GitHub] spark pull request: [SPARK-2964] [SQL] Improve spark-sql and start...

2014-08-19 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/1886#issuecomment-52603458 @sarutak Would you mind to add a description of fixing the `-S | --silent` option issue in the PR description? This issue is actually more important than leveraging

[GitHub] spark pull request: [SPARK-2929][SQL] Refactored Thrift server and...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1856#issuecomment-52603484 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18832/consoleFull) for PR 1856 at commit

[GitHub] spark pull request: [SPARK-3074] [PySpark] support groupByKey() wi...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1977#issuecomment-52603829 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18823/consoleFull) for PR 1977 at commit

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52604743 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18833/consoleFull) for PR 2030 at commit

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52604677 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18825/consoleFull) for PR 2030 at commit

[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...

2014-08-19 Thread mubarak
Github user mubarak commented on the pull request: https://github.com/apache/spark/pull/1723#issuecomment-52605797 ``` [info] - call sites report correct locations *** FAILED *** [info] [apply] did not equal [makeRDD] (SparkContextInfoSuite.scala:85) ``` CallSite

[GitHub] spark pull request: [WIP][SPARK-3036][SPARK-3037][SQL] Add MapType...

2014-08-19 Thread ueshin
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/2032 [WIP][SPARK-3036][SPARK-3037][SQL] Add MapType/ArrayType containing null value support to Parquet. JIRA: - https://issues.apache.org/jira/browse/SPARK-3036 -

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52606422 Benchmarked as of 0d8ed5b and the results aren't conclusively faster than `master`; the good news is that we've narrowed the gap that I saw earlier between `master`

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52606590 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18827/consoleFull) for PR 2030 at commit

[GitHub] spark pull request: [WIP][SPARK-3036][SPARK-3037][SQL] Add MapType...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2032#issuecomment-52606749 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18834/consoleFull) for PR 2032 at commit

[GitHub] spark pull request: [SPARK-2873] [SQL] using ExternalAppendOnlyMap...

2014-08-19 Thread guowei2
Github user guowei2 commented on the pull request: https://github.com/apache/spark/pull/2029#issuecomment-52606844 @marmbrus what should i give the outputs about the benchmarks? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-2849 / 2914] Handle certain Spark confi...

2014-08-19 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/1845#discussion_r16403594 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -1421,3 +1421,24 @@ private[spark] object Utils extends Logging { }

[GitHub] spark pull request: [SPARK-2849 / 2914] Handle certain Spark confi...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52608216 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18836/consoleFull) for PR 1845 at commit

[GitHub] spark pull request: [SPARK-2849 / 2914] Handle certain Spark confi...

2014-08-19 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52608223 @pwendell Thanks for your feedback. I think I have addressed all your comments. In particular, I have fixed the precedence order of the various properties that were

[GitHub] spark pull request: add some shuffle configurations in doc

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2031#issuecomment-52609052 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18831/consoleFull) for PR 2031 at commit

[GitHub] spark pull request: [SPARK-2298] Encode stage attempt in SparkList...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1545#issuecomment-52609251 **Tests timed out** after a configured wait of `120m`. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

[GitHub] spark pull request: [SPARK-3119] Re-implementation of TorrentBroad...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2030#issuecomment-52610271 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18833/consoleFull) for PR 2030 at commit

[GitHub] spark pull request: [SPARK-2929][SQL] Refactored Thrift server and...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1856#issuecomment-52610520 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18832/consoleFull) for PR 1856 at commit

[GitHub] spark pull request: [SPARK-2929][SQL] Refactored Thrift server and...

2014-08-19 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/1856#issuecomment-52610965 With a significant longer timeout (1 minute), `CliSuite` passes while `HiveThriftServer2Suite` still timeout. Trying an even larger timeout to see whether the test

[GitHub] spark pull request: [SPARK-2964] [SQL] Fix the -S and --silent opt...

2014-08-19 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/1886#discussion_r16405208 --- Diff: sbin/start-thriftserver.sh --- @@ -43,36 +47,9 @@ function usage { $FWDIR/bin/spark-class $CLASS --help 21 | grep -v $pattern 12 }

[GitHub] spark pull request: [SPARK-2929][SQL] Refactored Thrift server and...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1856#issuecomment-52611508 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18837/consoleFull) for PR 1856 at commit

[GitHub] spark pull request: [SPARK-2964] [SQL] Fix the -S and --silent opt...

2014-08-19 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/1886#issuecomment-52611702 LGTM except the typo in `sbin/start-thrfitserver.sh`. @pwendell @marmbrus This is good to go once the typo is fixed and passes Jenkins. --- If your project

[GitHub] spark pull request: add some shuffle configurations in doc

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2031#issuecomment-52612670 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18835/consoleFull) for PR 2031 at commit

[GitHub] spark pull request: [SPARK-2849] Handle driver configs separately ...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1845#issuecomment-52613410 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18836/consoleFull) for PR 1845 at commit

[GitHub] spark pull request: [WIP][SPARK-3036][SPARK-3037][SQL] Add MapType...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2032#issuecomment-52613443 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18834/consoleFull) for PR 2032 at commit

[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...

2014-08-19 Thread tdas
Github user tdas commented on the pull request: https://github.com/apache/spark/pull/1723#issuecomment-52615460 Oh yes, that failure makes sense. Since we are capturing all Spark classes, it is not able to identify the callsite inside SparkContextInfoSuite. This is because the

[GitHub] spark pull request: [SPARK-2964] [SQL] Fix the -S and --silent opt...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1886#issuecomment-52615671 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18838/consoleFull) for PR 1886 at commit

[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...

2014-08-19 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1723#discussion_r16407131 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala --- @@ -106,6 +106,27 @@ abstract class DStream[T: ClassTag] ( /**

[GitHub] spark pull request: [SPARK-1853] Show Streaming application code c...

2014-08-19 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/1723#discussion_r16407308 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/DStream.scala --- @@ -106,6 +106,27 @@ abstract class DStream[T: ClassTag] ( /**

[GitHub] spark pull request: [SPARK-2929][SQL] Refactored Thrift server and...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1856#issuecomment-52618569 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18837/consoleFull) for PR 1856 at commit

[GitHub] spark pull request: [SPARK-2964] [SQL] Fix the -S and --silent opt...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1886#issuecomment-52620020 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18838/consoleFull) for PR 1886 at commit

[GitHub] spark pull request: [SPARK-3106] *Race Condition Issue* Fix the or...

2014-08-19 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2019#issuecomment-52621410 This change can resolve being threw ClosedChannelException, CancelledKeyException and warning message Corresponding SendingConnectionManagerId not found and All

[GitHub] spark pull request: [SPARK-3106] *Race Condition Issue* Fix the or...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2019#issuecomment-52621462 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18839/consoleFull) for PR 2019 at commit

[GitHub] spark pull request: [SPARK-2199] [mllib] topic modeling

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1269#issuecomment-52623723 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18840/consoleFull) for PR 1269 at commit

[GitHub] spark pull request: [SPARK-2199] [mllib] topic modeling

2014-08-19 Thread akopich
Github user akopich commented on the pull request: https://github.com/apache/spark/pull/1269#issuecomment-52624693 @chazchandler Topics are stored in `phi` field `GlobalParameterType` (`GlobalParameters` or `RobustGlobalParameters` depending whether `PLSA` or `RobustPLSA` is

[GitHub] spark pull request: [SPARK-3106] *Race Condition Issue* Fix the or...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2019#issuecomment-52626545 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18839/consoleFull) for PR 2019 at commit

[GitHub] spark pull request: [SPARK-3106] *Race Condition Issue* Fix the or...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2019#issuecomment-52627997 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18842/consoleFull) for PR 2019 at commit

[GitHub] spark pull request: [SPARK-2199] [mllib] topic modeling

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1269#issuecomment-52629464 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18840/consoleFull) for PR 1269 at commit

[GitHub] spark pull request: [SPARK-2974] [SPARK-2975] Fix two bugs related...

2014-08-19 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/2002#discussion_r16412610 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -449,12 +449,68 @@ private[spark] object Utils extends Logging { }

[GitHub] spark pull request: [SPARK-2974] [SPARK-2975] Fix two bugs related...

2014-08-19 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2002#issuecomment-52630453 on testing yarn did you just verify it ran or did you verify it is using the correct yarn directories? Its fairly simple looking at the logs to see the latter. Of

[GitHub] spark pull request: [SPARK-2165] spark on yarn: add support for se...

2014-08-19 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/1279#issuecomment-52630875 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: [SPARK-3110][YARN] Add a ha mode in YARN mod...

2014-08-19 Thread tgravescs
Github user tgravescs commented on the pull request: https://github.com/apache/spark/pull/2024#issuecomment-52631207 Can you perhaps filing the jiras with overall goal and design first if there are more parts to this. --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: [SPARK-2165] spark on yarn: add support for se...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/1279#issuecomment-52631256 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18843/consoleFull) for PR 1279 at commit

[GitHub] spark pull request: [SPARK-3106] *Race Condition Issue* Fix the or...

2014-08-19 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2019#issuecomment-52635149 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18842/consoleFull) for PR 2019 at commit

[GitHub] spark pull request: [SPARK-2974] [SPARK-2975] Fix two bugs related...

2014-08-19 Thread tgravescs
Github user tgravescs commented on a diff in the pull request: https://github.com/apache/spark/pull/2002#discussion_r16415449 --- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala --- @@ -449,12 +449,68 @@ private[spark] object Utils extends Logging { }

  1   2   3   4   5   >