[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-59469758 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21849/consoleFull) for PR 2833 at commit

[GitHub] spark pull request: [SPARK-3961] [MLlib] [PySpark] Python API for ...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2819#issuecomment-59469759 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21850/consoleFull) for PR 2819 at commit

[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-59469751 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/390/consoleFull) for PR 2833 at commit

[GitHub] spark pull request: [SPARK-3812] [BUILD] Adapt maven build to publ...

2014-10-17 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/2673#issuecomment-59469813 We can definitely install other modules, I am afraid if the resultant(effective) pom(s) will carry reference to parent pom. Let me try that out. --- If your

[GitHub] spark pull request: [SPARK-3736] Workers reconnect when disassocia...

2014-10-17 Thread ash211
Github user ash211 commented on a diff in the pull request: https://github.com/apache/spark/pull/2828#discussion_r19002923 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala --- @@ -362,9 +372,19 @@ private[spark] class Worker( } }

[GitHub] spark pull request: [SPARK-3721] [PySpark] broadcast objects large...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2659#issuecomment-59470191 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2497#issuecomment-59470644 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21843/consoleFull) for PR 2497 at commit

[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2816#issuecomment-59470604 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21846/consoleFull) for PR 2816 at commit

[GitHub] spark pull request: [SPARK-3959][SPARK-3960][SQL] SqlParser fails ...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2816#issuecomment-59470610 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2497#issuecomment-59470648 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3952] [Streaming] [PySpark] add Python ...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2808#issuecomment-59473908 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3952] [Streaming] [PySpark] add Python ...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2808#issuecomment-59473901 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21847/consoleFull) for PR 2808 at commit

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-17 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/2753#discussion_r19004339 --- Diff: core/src/main/scala/org/apache/spark/network/netty/NettyBlockFetcher.scala --- @@ -0,0 +1,92 @@ +/* + * Licensed to the Apache Software

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-17 Thread aarondav
Github user aarondav commented on a diff in the pull request: https://github.com/apache/spark/pull/2753#discussion_r19004341 --- Diff: core/src/main/scala/org/apache/spark/network/BlockFetchingListener.scala --- @@ -31,7 +34,7 @@ trait BlockFetchingListener extends EventListener {

[GitHub] spark pull request: [SPARK-3961] [MLlib] [PySpark] Python API for ...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2819#issuecomment-59474327 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

2014-10-17 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-59474302 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3904] [SQL] add constant objectinspecto...

2014-10-17 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2762#issuecomment-59474277 Thank you @gvramana , I've updated the code as you suggested. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-3961] [MLlib] [PySpark] Python API for ...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2819#issuecomment-59474321 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21850/consoleFull) for PR 2819 at commit

[GitHub] spark pull request: [SPARK-3911] [SQL] HiveSimpleUdf can not be op...

2014-10-17 Thread chenghao-intel
Github user chenghao-intel commented on the pull request: https://github.com/apache/spark/pull/2771#issuecomment-59474335 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-59474641 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-59474636 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21849/consoleFull) for PR 2833 at commit

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-59474679 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21852/consoleFull) for PR 2753 at commit

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-17 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-59475022 @rxin Thanks for the preliminary pass. I've updated the PR to include UploadBlocks and to make communication bidirectionally possible (though this functionality is not

[GitHub] spark pull request: [SPARK-3935][Core] log the number of records t...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2791#issuecomment-59475586 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21853/consoleFull) for PR 2791 at commit

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-59475786 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...

2014-10-17 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/2824#issuecomment-59475885 Hi @JoshRosen , I just add a configuration that can bypass the NIO way of copying stream. Would you mind taking a look at it? --- If your project is set up for it,

[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2824#issuecomment-59475982 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21854/consoleFull) for PR 2824 at commit

[GitHub] spark pull request: [spark-3907][sql] add truncate table support

2014-10-17 Thread tianyi
Github user tianyi commented on a diff in the pull request: https://github.com/apache/spark/pull/2770#discussion_r19004814 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala --- @@ -121,7 +121,8 @@ private[hive] object HiveQl { // Commands that we do

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

2014-10-17 Thread wangxiaojing
Github user wangxiaojing commented on a diff in the pull request: https://github.com/apache/spark/pull/2765#discussion_r19004975 --- Diff: streaming/src/main/scala/org/apache/spark/streaming/dstream/FileInputDStream.scala --- @@ -207,6 +220,9 @@ class FileInputDStream[K:

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/2497#discussion_r19005481 --- Diff: yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnStableUtils.scala --- @@ -0,0 +1,54 @@ +/* + * Licensed to the Apache

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/2497#discussion_r19005493 --- Diff: core/src/main/scala/org/apache/spark/ui/JettyUtils.scala --- @@ -148,14 +146,19 @@ private[spark] object JettyUtils extends Logging {

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2497#issuecomment-59478273 Ok I merged this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3916] [Streaming] discover new appended...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2806#issuecomment-59478342 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/389/consoleFull)** for PR 2806 at commit

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

2014-10-17 Thread jerryshao
Github user jerryshao commented on the pull request: https://github.com/apache/spark/pull/2765#issuecomment-59478360 Can we just check the time of file, not directory to filter out some unqualified files, I'm not sure about this. cc @tdas , mind taking a look at this? ---

[GitHub] spark pull request: [SPARK-3965] Ensure that Spark assembly for ha...

2014-10-17 Thread dajac
Github user dajac commented on the pull request: https://github.com/apache/spark/pull/2822#issuecomment-59478665 I used following command: ```bash mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -Phive -DskipTests clean package ``` You'll see that

[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-59478889 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/390/consoleFull)** for PR 2833 at commit

[GitHub] spark pull request: [SPARK-3916] [Streaming] discover new appended...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2806#issuecomment-59480946 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/391/consoleFull) for PR 2806 at commit

[GitHub] spark pull request: [SPARK-3721] [PySpark] broadcast objects large...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2659#issuecomment-59481093 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/392/consoleFull) for PR 2659 at commit

[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2681#issuecomment-59481089 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/393/consoleFull) for PR 2681 at commit

[GitHub] spark pull request: [spark-3907][sql] add truncate table support

2014-10-17 Thread adrian-wang
Github user adrian-wang commented on a diff in the pull request: https://github.com/apache/spark/pull/2770#discussion_r19006488 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala --- @@ -121,7 +121,8 @@ private[hive] object HiveQl { // Commands that we

[GitHub] spark pull request: [SPARK-3935][Core] log the number of records t...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2791#issuecomment-59482693 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21853/consoleFull) for PR 2791 at commit

[GitHub] spark pull request: [SPARK-3935][Core] log the number of records t...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2791#issuecomment-59482698 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3900][YARN] ApplicationMaster's shutdow...

2014-10-17 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2755#issuecomment-59483403 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3870] EOL character enforcement

2014-10-17 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2726#issuecomment-59483367 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3657] yarn alpha YarnRMClientImpl throw...

2014-10-17 Thread sarutak
Github user sarutak commented on the pull request: https://github.com/apache/spark/pull/2728#issuecomment-59483505 test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2824#issuecomment-59483494 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21854/consoleFull) for PR 2824 at commit

[GitHub] spark pull request: [SPARK-3948][Shuffle]Fix stream corruption bug...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2824#issuecomment-59483499 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [spark-3907][sql] add truncate table support

2014-10-17 Thread wangxiaojing
Github user wangxiaojing commented on a diff in the pull request: https://github.com/apache/spark/pull/2770#discussion_r19007078 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveQl.scala --- @@ -121,7 +121,8 @@ private[hive] object HiveQl { // Commands that we

[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2520#issuecomment-59483851 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21855/consoleFull) for PR 2520 at commit

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-59484057 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21852/consoleFull) for PR 2753 at commit

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-59484064 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [spark-3586][streaming]Support nested director...

2014-10-17 Thread wangxiaojing
Github user wangxiaojing commented on the pull request: https://github.com/apache/spark/pull/2765#issuecomment-59485316 @jerryshao @tdas First,According to the depth to check all the directory ,then filter the directory if the modification time more then the ignore time.Is this

[GitHub] spark pull request: [SPARK-3916] [Streaming] discover new appended...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2806#issuecomment-59486879 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/391/consoleFull) for PR 2806 at commit

[GitHub] spark pull request: Added possibility to directly install python p...

2014-10-17 Thread ziky90
GitHub user ziky90 opened a pull request: https://github.com/apache/spark/pull/2836 Added possibility to directly install python packages on EC2 Goal of this PR is to simplify the way how to install Python packages for PySpark on EC2. It installs selected packages directly to all

[GitHub] spark pull request: [SPARK-3989]Added possibility to directly inst...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2836#issuecomment-59487823 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-3133] embed small object in broadcast t...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2681#issuecomment-59488832 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/393/consoleFull) for PR 2681 at commit

[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2520#issuecomment-59491682 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [SPARK-3677] [BUILD] [YARN] pom.xml and SparkB...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2520#issuecomment-59491673 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21855/consoleFull) for PR 2520 at commit

[GitHub] spark pull request: SPARK-1830 Deploy failover, Make Persistence e...

2014-10-17 Thread ScrapCodes
Github user ScrapCodes commented on the pull request: https://github.com/apache/spark/pull/771#issuecomment-59493430 @aarondav So you are asking for another `read()` method, which end users can override. So that we can have a default implementation for `readPersistedData` ? --- If

[GitHub] spark pull request: [SPARK-3721] [PySpark] broadcast objects large...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2659#issuecomment-59493977 **[Tests timed out](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/392/consoleFull)** for PR 2659 at commit

[GitHub] spark pull request: [SPARK-3736] Workers reconnect when disassocia...

2014-10-17 Thread CodingCat
Github user CodingCat commented on a diff in the pull request: https://github.com/apache/spark/pull/2828#discussion_r19011614 --- Diff: core/src/main/scala/org/apache/spark/deploy/worker/Worker.scala --- @@ -362,9 +372,19 @@ private[spark] class Worker( } }

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-17 Thread zsxwing
Github user zsxwing commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-59499788 retest this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [Spark-3822] Ability to add/delete executors f...

2014-10-17 Thread PraveenSeluka
Github user PraveenSeluka commented on the pull request: https://github.com/apache/spark/pull/2798#issuecomment-59502779 Thanks @andrewor14 for taking time to test this out. - Regarding `AutoscaleServer` is Yarn Specific = Good point. Here is a proposal to fix this issue so

[GitHub] spark pull request: [Spark-3822] Ability to add/delete executors f...

2014-10-17 Thread PraveenSeluka
Github user PraveenSeluka commented on the pull request: https://github.com/apache/spark/pull/2798#issuecomment-59506099 On using `CoarseGrainedSchedulerBackend`, here are some thoughts - It already receives `StopExecutor, RemoveExecutor` messages. If we add `AddExecutor,

[GitHub] spark pull request: [SPARK-3969][SQL] Optimizer should have a supe...

2014-10-17 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/2825#discussion_r19014836 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ExpressionOptimizationSuite.scala --- @@ -30,7 +30,7 @@ class

[GitHub] spark pull request: [SPARK-3969][SQL] Optimizer should have a supe...

2014-10-17 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/2825#discussion_r19014954 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -28,7 +28,9 @@ import

[GitHub] spark pull request: [SPARK-3969][SQL] Optimizer should have a supe...

2014-10-17 Thread liancheng
Github user liancheng commented on the pull request: https://github.com/apache/spark/pull/2825#issuecomment-59506552 Two minor comments. This LGTM, thanks :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread devldevelopment
GitHub user devldevelopment opened a pull request: https://github.com/apache/spark/pull/2837 Initial time estimator with new column for remaining time. SPARK-576: A first approach to display estimated time remaining. Initially this is done by calculating the average completion rate

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2837#issuecomment-59507810 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-59533135 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/394/consoleFull) for PR 2833 at commit

[GitHub] spark pull request: [SPARK-3721] [PySpark] broadcast objects large...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2659#issuecomment-59533144 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/395/consoleFull) for PR 2659 at commit

[GitHub] spark pull request: SPARK-1830 Deploy failover, Make Persistence e...

2014-10-17 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/771#issuecomment-59533468 Yeah, that sounds good, as long as the API that they have to implement doesn't require them to import any of the *Info classes. --- If your project is set up for it,

[GitHub] spark pull request: [SPARK-3985] [Examples] fix file path using os...

2014-10-17 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/2834#discussion_r19026522 --- Diff: examples/src/main/python/sql.py --- @@ -48,7 +48,7 @@ # A JSON dataset is pointed to by path. # The path can be either a

[GitHub] spark pull request: [Spark-3822] Ability to add/delete executors f...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2798#issuecomment-59540816 Actually my proposal is more like the following. `CoarseGrainedSchedulerBackend` will define what you propose as the `Autoscaling trait`, and each of its subclasses

[GitHub] spark pull request: [SPARK-3982] [Streaming] [PySpark] Python API:...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2833#issuecomment-59542965 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/394/consoleFull) for PR 2833 at commit

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-17 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-59546336 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-17 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/2732#discussion_r19030942 --- Diff: yarn/common/src/main/scala/org/apache/spark/deploy/yarn/ClientBase.scala --- @@ -485,7 +485,20 @@ private[spark] trait ClientBase extends Logging {

[GitHub] spark pull request: [SPARK-3877][YARN] Throw an exception when app...

2014-10-17 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2732#issuecomment-59546652 LGTM pending Jenkins being happy. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2837#issuecomment-59546842 Hey @devldevelopment can you add [SPARK-576] to the title? See how other PRs are opened. --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/2837#discussion_r19031066 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala --- @@ -28,30 +28,31 @@ import org.apache.spark.util.Utils /** Page

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2837#issuecomment-59546853 ok to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/2837#discussion_r19031103 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala --- @@ -179,8 +182,21 @@ private[ui] class StageTableBase( td

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/2837#discussion_r19031171 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala --- @@ -172,6 +174,7 @@ private[ui] class StageTableBase(

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on a diff in the pull request: https://github.com/apache/spark/pull/2837#discussion_r19031133 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StageTable.scala --- @@ -146,6 +147,7 @@ private[ui] class StageTableBase( if (finishTime

[GitHub] spark pull request: [SPARK-3721] [PySpark] broadcast objects large...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2659#issuecomment-59547048 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/395/consoleFull) for PR 2659 at commit

[GitHub] spark pull request: [SPARK-3979] [yarn] Use fs's default replicati...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2831#issuecomment-59547329 @tgravescs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request: [SPARK-3935][Core] log the number of records t...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2791#issuecomment-59547539 Ok thanks I merge --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request: [SPARK-3935][Core] log the number of records t...

2014-10-17 Thread andrewor14
Github user andrewor14 commented on the pull request: https://github.com/apache/spark/pull/2791#issuecomment-59547676 Hey @jackylk do you have a JIRA account? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2837#issuecomment-59547709 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21856/consoleFull) for PR 2837 at commit

[GitHub] spark pull request: [SPARK-3935][Core] log the number of records t...

2014-10-17 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/2791 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2837#issuecomment-59548133 [QA tests have finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/21856/consoleFull) for PR 2837 at commit

[GitHub] spark pull request: Initial time estimator with new column for rem...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2837#issuecomment-59548137 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark pull request: [Spark-3822] Ability to add/delete executors f...

2014-10-17 Thread PraveenSeluka
Github user PraveenSeluka commented on the pull request: https://github.com/apache/spark/pull/2798#issuecomment-59549322 @andrewor14 I have got a full picture of what you are planning to do there. The details are exactly the same as this PR. I have tried to keep the auto-scaling

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-17 Thread aarondav
Github user aarondav commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-59550672 @rxin RPC unit tests added, this is good to go on my side (and will turn off by default right before merge). --- If your project is set up for it, you can reply to

[GitHub] spark pull request: [SPARK-3952] [Streaming] [PySpark] add Python ...

2014-10-17 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2808#issuecomment-59550837 It looks like there's a missing tag or space or something, because now the markup after the connection pool section is messed up:

[GitHub] spark pull request: [SPARK-3855][SQL] Preserve the result attribut...

2014-10-17 Thread SparkQA
Github user SparkQA commented on the pull request: https://github.com/apache/spark/pull/2717#issuecomment-59551302 [QA tests have started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/396/consoleFull) for PR 2717 at commit

[GitHub] spark pull request: [SPARK-3989]Added possibility to directly inst...

2014-10-17 Thread JoshRosen
Github user JoshRosen commented on the pull request: https://github.com/apache/spark/pull/2836#issuecomment-59552268 I like the idea behind this, but I'm worried about adding even more stuff to the `spark-ec2` script, especially since I think this use case could be addressed by a

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-17 Thread vanzin
Github user vanzin commented on the pull request: https://github.com/apache/spark/pull/2497#issuecomment-59552443 Hmmm. Weird how these PRs on branch-1.1 don't get automatically closed. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request: [SPARK-3606] [yarn] Correctly configure AmIpFi...

2014-10-17 Thread vanzin
Github user vanzin closed the pull request at: https://github.com/apache/spark/pull/2497 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request: [SPARK-3453] Netty-based BlockTransferService,...

2014-10-17 Thread AmplabJenkins
Github user AmplabJenkins commented on the pull request: https://github.com/apache/spark/pull/2753#issuecomment-59552944 Test FAILed. Refer to this link for build results (access rights to CI server needed):

  1   2   3   >