[GitHub] spark pull request: [SPARK-5388] Provide a stable application subm...

2015-02-04 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4216#discussion_r24138570 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala --- @@ -300,6 +349,22 @@ private[spark] class SparkSubmitArguments(args

[GitHub] spark pull request: [SPARK-5388] Provide a stable application subm...

2015-02-04 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4216#discussion_r24138924 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/ui/MasterPage.scala --- @@ -73,6 +73,14 @@ private[spark] class MasterPage(parent: MasterWebUI

[GitHub] spark pull request: [SPARK-4874] [CORE] Collect record count metri...

2015-02-04 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4067#issuecomment-72975805 Hey @ksakellis - I did a pretty thorough review here, any chance you could take a look? Most of the things were small. The only major thing is that I think if we

[GitHub] spark pull request: [SPARK-5388] Provide a stable application subm...

2015-02-04 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4216#discussion_r24137865 --- Diff: core/src/main/scala/org/apache/spark/deploy/LocalSparkCluster.scala --- @@ -43,9 +47,11 @@ class LocalSparkCluster(numWorkers: Int, coresPerWorker

[GitHub] spark pull request: [SPARK-5388] Provide a stable application subm...

2015-02-04 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4216#discussion_r24138163 --- Diff: core/src/main/scala/org/apache/spark/deploy/rest/StandaloneRestClient.scala --- @@ -0,0 +1,280 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-5388] Provide a stable application subm...

2015-02-04 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4216#discussion_r24138670 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/MasterMessages.scala --- @@ -36,7 +36,7 @@ private[master] object MasterMessages

[GitHub] spark pull request: SPARK-5607: Update to Kryo 2.24.0 to avoid inc...

2015-02-04 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4383#issuecomment-73001586 Let's give this a shot and see if we witness the exception again. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-5388] Provide a stable application subm...

2015-02-04 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4216#discussion_r24144401 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -372,12 +453,28 @@ object SparkSubmit { sysProps

[GitHub] spark pull request: [SPARK-5388] Provide a stable application subm...

2015-02-04 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4216#discussion_r24143599 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -84,21 +94,92 @@ object SparkSubmit { if (appArgs.verbose

[GitHub] spark pull request: [SPARK-5388] Provide a stable application subm...

2015-02-04 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4216#discussion_r24143935 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -372,12 +453,28 @@ object SparkSubmit { sysProps

[GitHub] spark pull request: [SPARK-5574] use given name prefix in dir

2015-02-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4344#issuecomment-72779281 LGTM too --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-5574] use given name prefix in dir

2015-02-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4344#issuecomment-72798456 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-4939] revive offers periodically in Loc...

2015-02-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4147#issuecomment-72783843 @kayousterhout sure - that seems fine to me! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-5341] Use maven coordinates as dependen...

2015-02-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4215#issuecomment-72798586 I merged this - thanks Burak! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-5548: Fixed a race condition in AkkaUtil...

2015-02-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4343#issuecomment-72798850 @jacek-lewandowski how about just catching either exception? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23988445 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,174 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23988871 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/OffsetRange.scala --- @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23989829 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,174 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23989786 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,174 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23990370 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,174 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23988318 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,174 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23988994 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/Leader.scala --- @@ -0,0 +1,46 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23988976 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/OffsetRange.scala --- @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23989107 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/OffsetRange.scala --- @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23989943 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,249 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-5341] Use maven coordinates as dependen...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4215#discussion_r24046019 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -25,6 +25,18 @@ import scala.collection.mutable.{ArrayBuffer, HashMap, Map

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r24029555 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,174 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r24030535 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/OffsetRange.scala --- @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r24029025 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,174 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r24029286 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,174 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r24029463 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,249 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r24029652 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/OffsetRange.scala --- @@ -0,0 +1,69 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-5341] Use maven coordinates as dependen...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4215#discussion_r24049531 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -475,6 +500,199 @@ object SparkSubmit

[GitHub] spark pull request: [SPARK-5341] Use maven coordinates as dependen...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4215#discussion_r24049602 --- Diff: core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala --- @@ -475,6 +500,199 @@ object SparkSubmit

[GitHub] spark pull request: [SPARK-5341] Use maven coordinates as dependen...

2015-02-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4215#issuecomment-72763888 LGTM pending tests. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5158] [core] [security] Spark standalon...

2015-02-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4106#issuecomment-72763704 Hey @mccheah - if you are too busy I think it's fine to let it slip past 1.3, given that there are still several unknowns. --- If your project is set up for it, you

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3798#issuecomment-72617088 I took a pass through the public API. I'm not very familiar with Kafka so it was somewhat slow going. However, some reactions: 1. We should try to tighten

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-03 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23990592 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,174 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-4874] [CORE] Collect record count metri...

2015-02-02 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4067#discussion_r23951392 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -47,9 +49,13 @@ private[spark] class CacheManager(blockManager: BlockManager

[GitHub] spark pull request: SPARK-5542: Decouple publishing, packaging, an...

2015-02-02 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/4319 SPARK-5542: Decouple publishing, packaging, and tagging in release script These are some changes to the build script to allow parts of it to be run independently. This has already been tested

[GitHub] spark pull request: [SPARK-5388] Provide a stable application subm...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4216#issuecomment-72554547 @sryza maybe that is the source of some confusion. This is not intended as a third party extension point at this time. This exists because we want some very simple

[GitHub] spark pull request: [SPARK-4987] [SQL] parquet timestamp type supp...

2015-02-02 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3820#discussion_r23972062 --- Diff: pom.xml --- @@ -149,6 +149,7 @@ scala.binary.version2.10/scala.binary.version jline.version${scala.version}/jline.version

[GitHub] spark pull request: [SPARK-4987] [SQL] parquet timestamp type supp...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3820#issuecomment-72567388 Dependency looks fine to me, thanks for running it by. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: Add a config option to print DAG.

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4257#issuecomment-72568187 Fair, but the issue is in some cases (e.g. GraphX) the printed representation of the DAG can be many hundreds of lines long. could that potentially explode the output

[GitHub] spark pull request: [SPARK-3778] newAPIHadoopRDD doesn't properly ...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4292#issuecomment-72572163 LGTM and seems very straightforward. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: Spark 3883: SSL support for HttpServer and Akk...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3571#issuecomment-72560603 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4874] [CORE] Collect record count metri...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4067#issuecomment-72428547 @rxin yeah that seems good. @ksakellis one other thing I realized which is a little confusing, right now we report the bytes as read from a cached RDD as soon

[GitHub] spark pull request: [SQL] Improve DataFrame API error reporting

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4296#issuecomment-72584164 @rxin this test failure was spurious. It is safe to merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark pull request: SPARK-3996: Add jetty servlet and continuation...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4323#issuecomment-72593339 @JoshRosen sounds good. In the mean time I'll likely still pull this in, since it's blocking perf testing and some other stuff. --- If your project is set up

[GitHub] spark pull request: [SPARK-4964] [Streaming] Exactly-once semantic...

2015-02-02 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3798#discussion_r23983277 --- Diff: external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala --- @@ -144,4 +150,174 @@ object KafkaUtils

[GitHub] spark pull request: [SPARK-5154] [PySpark] [Streaming] Kafka strea...

2015-02-02 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3715#discussion_r23977407 --- Diff: make-distribution.sh --- @@ -188,6 +188,7 @@ echo Build flags: $@ $DISTDIR/RELEASE # Copy jars cp $SPARK_HOME/assembly/target/scala

[GitHub] spark pull request: [SPARK-5154] [PySpark] [Streaming] Kafka strea...

2015-02-02 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3715#discussion_r23977374 --- Diff: external/kafka-assembly/pom.xml --- @@ -0,0 +1,106 @@ +?xml version=1.0 encoding=UTF-8? +!-- + ~ Licensed to the Apache Software

[GitHub] spark pull request: SPARK-5542: Decouple publishing, packaging, an...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4319#issuecomment-72587409 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SQL] Improve DataFrame API error reporting

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4296#issuecomment-72583465 Jenkins, test this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: SPARK-3996: Add jetty servlet and continuation...

2015-02-02 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/4323 SPARK-3996: Add jetty servlet and continuations. These are needed transitively from the other Jetty libraries we include. It was not picked up by unit tests because we disable the UI. You

[GitHub] spark pull request: SPARK-5542: Decouple publishing, packaging, an...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4319#issuecomment-72579094 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: SPARK-5542: Decouple publishing, packaging, an...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4319#issuecomment-72593258 Just an internal clean-up, so I'm gonna pull this in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-3778] newAPIHadoopRDD doesn't properly ...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4292#issuecomment-72604443 @JoshRosen that's a great point and could cause regressing behavior that would be really hard for users to diagnose. @tgravescs. What about deferring the injection

[GitHub] spark pull request: [SPARK-3778] newAPIHadoopRDD doesn't properly ...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4292#issuecomment-72604619 We could also just leave it as-is and then do something like that if we find this is encountered by users. --- If your project is set up for it, you can reply

[GitHub] spark pull request: [SPARK-1405] [mllib] Latent Dirichlet Allocati...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4047#issuecomment-72579348 Jenkins, retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-4874] [CORE] Collect record count metri...

2015-02-02 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4067#discussion_r23913204 --- Diff: core/src/main/scala/org/apache/spark/CacheManager.scala --- @@ -47,9 +49,13 @@ private[spark] class CacheManager(blockManager: BlockManager

[GitHub] spark pull request: [SPARK-4874] [CORE] Collect record count metri...

2015-02-02 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4067#issuecomment-72425732 Having a count does seem like a good idea, but I think the current approach of adding function calls for each iteration is too expensive. What about using a counting

[GitHub] spark pull request: SPARK-5492. Thread statistics can break with o...

2015-02-01 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4305#issuecomment-72417193 LGTM pending tests. Thanks, Sandy --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-3996]: Shade Jetty in Spark deliverable...

2015-02-01 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4285#issuecomment-72407319 Okay - let's try this for take 2. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-5353] Log failures in REPL class loadin...

2015-02-01 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4130#issuecomment-72408916 I can merge it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5341] Use maven coordinates as dependen...

2015-02-01 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4215#discussion_r23907239 --- Diff: core/pom.xml --- @@ -225,6 +225,16 @@ scopetest/scope /dependency dependency + groupIdorg.apache.ivy/groupId

[GitHub] spark pull request: [SPARK-5208][DOC] Add more documentation to Ne...

2015-02-01 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4012#issuecomment-72408184 Okay @sarutak can you close this issue then? Looks like we intentionally left these out for now --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: Add a config option to print DAG.

2015-02-01 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4257#issuecomment-72408451 @rxin I have noticed that very few users know about `toDebugString`. Maybe we should open a JIRA to add better documentation for that function (i.e. discuss

[GitHub] spark pull request: [SPARK-5353] Log failures in REPL class loadin...

2015-02-01 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4130#issuecomment-72408322 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: [SPARK-3996]: Shade Jetty in Spark deliverable...

2015-01-30 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4285#issuecomment-72249880 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5474][Build]curl should support URL red...

2015-01-29 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4263#issuecomment-72081223 @witgo can you update other cases in the codebase where we do this? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-5466] Add explicit guava dependencies w...

2015-01-29 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4272#issuecomment-72081625 Thanks for figuring this out @vanzin! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-5366][EC2] Check the mode of private ke...

2015-01-29 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4162#issuecomment-72113135 Hey on this one - doesn't ssh itself give a good error message if the key file does not have the correct permissions? Should we just be propagating the SSH message

[GitHub] spark pull request: [WIP] [SPARK-3996]: Shade Jetty in Spark deliv...

2015-01-29 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4252#issuecomment-72114193 @vanzin I've addressed your additional comments. Mind taking a look? --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [SPARK-5158] [core] [security] Spark standalon...

2015-01-29 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4106#discussion_r23816550 --- Diff: core/src/main/scala/org/apache/spark/deploy/StandaloneSparkHadoopUtil.scala --- @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-5158] [core] [security] Spark standalon...

2015-01-29 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4106#discussion_r23816373 --- Diff: core/src/main/scala/org/apache/spark/deploy/StandaloneSparkHadoopUtil.scala --- @@ -0,0 +1,76 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-3778] newAPIHadoopRDD doesn't properly ...

2015-01-29 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2676#issuecomment-72133280 @tgravescs mind brining it up to date? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark pull request: [SPARK-3778] newAPIHadoopRDD doesn't properly ...

2015-01-29 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2676#issuecomment-72133319 I bumped the severity per @harishreedharan's commnet. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark pull request: [SPARK-3778] newAPIHadoopRDD doesn't properly ...

2015-01-29 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/2676#issuecomment-72133263 Hey guys - sorry don't block on my comment. If you all think this looks good, just merge it. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: [SPARK-5307] SerializationDebugger

2015-01-29 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4098#issuecomment-72133550 @rxin LGTM - I took a quick look through, not an expert on this but I think it's good. --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request: [SPARK-3996]: Shade Jetty in Spark deliverable...

2015-01-29 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/4285 [SPARK-3996]: Shade Jetty in Spark deliverables (v2 of this patch with a fix that was only relevant for the maven build). This patch piggy-back's on @vanzin's work to simplify the Guava

[GitHub] spark pull request: [SPARK-5415] bump sbt to version to 0.13.7

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4211#issuecomment-71810049 Sure, let's do it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-5434] [EC2] Preserve spaces in EC2 path

2015-01-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4224#discussion_r23677054 --- Diff: ec2/spark-ec2 --- @@ -20,6 +20,6 @@ # Preserve the user's CWD so that relative paths are passed correctly to #+ the underlying

[GitHub] spark pull request: [SPARK-5428]: Declare the 'assembly' module at...

2015-01-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/4232#discussion_r23675195 --- Diff: pom.xml --- @@ -105,6 +104,8 @@ moduleexternal/zeromq/module moduleexamples/module modulerepl/module

[GitHub] spark pull request: [SPARK-5154] [PySpark] [Streaming] Kafka strea...

2015-01-28 Thread pwendell
Github user pwendell commented on a diff in the pull request: https://github.com/apache/spark/pull/3715#discussion_r23672024 --- Diff: make-distribution.sh --- @@ -188,6 +188,7 @@ echo Build flags: $@ $DISTDIR/RELEASE # Copy jars cp $SPARK_HOME/assembly/target/scala

[GitHub] spark pull request: [HOTFIX] Rename the identifier of driver from ...

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4245#issuecomment-71810748 This could provide a usability regression for people since the name now appears differently in the UI. Is there any way we can intercept this somewhere downstream

[GitHub] spark pull request: [WIP] [SPARK-3996]: Shade Jetty in Spark deliv...

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4252#issuecomment-71919125 Changed approach to only shade inside of core jar per @vanzin's suggestion. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark pull request: [WIP] [SPARK-3996]: Shade Jetty in Spark deliv...

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4252#issuecomment-71912406 Hey @vanzin - here the jetty classes show up in both core and streaming. Is that a problem? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: SPARK-1934 [CORE] this reference escape to ...

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4225#issuecomment-71912133 Thanks @srowen and @zsxwing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-5388] Provide a stable application subm...

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4216#issuecomment-71982680 Hey so in case it's not clear, the issue is that because Akka makes no guarantees whatsoever about compatibility, we need to define our own protocol if we want

[GitHub] spark pull request: SPARK-5458. Refer to aggregateByKey instead of...

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4251#issuecomment-71898475 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request: SPARK-5458. Refer to aggregateByKey instead of...

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4251#issuecomment-71895372 I think this text also appears in `python/pyspark/rdd.py` - maybe we should udate that as well? --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request: SPARK-3996: Shade Jetty in Spark deliverables.

2015-01-28 Thread pwendell
Github user pwendell closed the pull request at: https://github.com/apache/spark/pull/3130 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark pull request: SPARK-3996: Shade Jetty in Spark deliverables.

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/3130#issuecomment-71903029 Closing this to re-open with a new version. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [WIP] [SPARK-3996]: Shade Jetty in Spark deliv...

2015-01-28 Thread pwendell
GitHub user pwendell opened a pull request: https://github.com/apache/spark/pull/4252 [WIP] [SPARK-3996]: Shade Jetty in Spark deliverables This patch piggy-back's on @vanzin's work to simplify the Guava shading, and adds Jetty as a shaded library in Spark. Other than adding

[GitHub] spark pull request: [WIP] [SPARK-3996]: Shade Jetty in Spark deliv...

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4252#issuecomment-71903345 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [WIP] [SPARK-3996]: Shade Jetty in Spark deliv...

2015-01-28 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4252#issuecomment-71907448 @vanzin - would you be able to take a look? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: SPARK-5308 [BUILD] MD5 / SHA1 hash format does...

2015-01-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4161#issuecomment-71699359 Thanks Sean - pulling this in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request: [SPARK-5430] move treeReduce and treeAggregate...

2015-01-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4228#issuecomment-71710396 @mengxr if we are going to add this as a first class API, can we have it in Java and Python too? Also, /cc to @rxin to also vet whether we want this in the core API. My

[GitHub] spark pull request: [SPARK-5430] move treeReduce and treeAggregate...

2015-01-27 Thread pwendell
Github user pwendell commented on the pull request: https://github.com/apache/spark/pull/4228#issuecomment-71712072 I don't think we should do it separately (it sets a bad precedent), but if you are too busy, we can try to find someone in the community to do all three. It's pretty

<    4   5   6   7   8   9   10   11   12   13   >