[GitHub] spark pull request #18711: [SPARK-21506][DOC]The description of "spark.execu...

2017-07-25 Thread 10110346
Github user 10110346 commented on a diff in the pull request: https://github.com/apache/spark/pull/18711#discussion_r129251166 --- Diff: docs/configuration.md --- @@ -1103,10 +1103,10 @@ Apart from these, the following properties are also available, and may be useful The

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18702#discussion_r129251081 --- Diff: sql/log4j.properties --- @@ -0,0 +1,24 @@ +# --- End diff -- Oh, actually, I think I can get rid of this. Let me try. ---

[GitHub] spark pull request #18730: [SPARK-21527][CORE] Use buffer limit in order to ...

2017-07-25 Thread caneGuy
GitHub user caneGuy opened a pull request: https://github.com/apache/spark/pull/18730 [SPARK-21527][CORE] Use buffer limit in order to use JAVA NIO Util's buffercache ## What changes were proposed in this pull request? Right now, ChunkedByteBuffer#writeFully do not slice

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #79930 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79930/testReport)** for PR 18555 at commit

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18555 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18555 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79930/ Test FAILed. ---

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18702 @srowen, so, the current structure is like ... ``` . ├── docs └── sql ├── create-docs.sh # Genenerates HTMLs. ├──

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #79932 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79932/testReport)** for PR 18555 at commit

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18555 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79932/ Test FAILed. ---

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18555 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18702 Oh I see, it's for consistency. Well OK stay consistent with R/Python then. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

2017-07-25 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/18513 Thanks @sethah @hhbyyh for the review. I updated the behavior doc string as suggested. Any other comments? cc @srowen @jkbradley @yanboliang --- If your project is set up for it, you can

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18513 **[Test build #79934 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79934/testReport)** for PR 18513 at commit

[GitHub] spark pull request #18731: [SPARK-20990][SQL] Read all JSON documents in fil...

2017-07-25 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/18731 [SPARK-20990][SQL] Read all JSON documents in files when multiline mode is on ## What changes were proposed in this pull request? The PR improves the JSON parsing so that now all the

[GitHub] spark issue #18726: [MINOR][CORE][TEST]Repeat stop SparkContext in ExecutorA...

2017-07-25 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18726 IIUC I think close context again which is already closed is fine to Spark, though neither your code nor the previous one can handle spark context creation failure related object leakage. --- If

[GitHub] spark issue #18728: [SPARK-21524] [ML] unit test fix: ValidatorParamsSuiteHe...

2017-07-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18728 Hm, is the more basic error here that `object ValidatorParamsSuiteHelpers extends SparkFunSuite` ? it seems like it's trying to leverage the temp dir cleanup function in the superclass, but it will

[GitHub] spark pull request #18711: [SPARK-21506][DOC]The description of "spark.execu...

2017-07-25 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/18711#discussion_r129248960 --- Diff: docs/configuration.md --- @@ -1103,10 +1103,10 @@ Apart from these, the following properties are also available, and may be useful The

[GitHub] spark issue #18726: [MINOR][CORE][TEST]Repeat stop SparkContext in ExecutorA...

2017-07-25 Thread heary-cao
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/18726 @jerryshao I think close Sparkcontext again that is little significance. Because they are closed again. spark prompt `SparkContext already stopped.` --- If your project is

[GitHub] spark issue #18651: [SPARK-21383][Core] Fix the YarnAllocator allocates more...

2017-07-25 Thread djvulee
Github user djvulee commented on the issue: https://github.com/apache/spark/pull/18651 I update the code, please take a look at @vanzin @tgravescs --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18702 (I got rid of log4j file by avoiding directly accessing to JVM) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...

2017-07-25 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18723 Thank you. 1. makes sense since memory is allocated by 2x at that point. While I looked at a [PR](https://github.com/netty/netty/pull/464) for netty, I cannot understand why 16 was used.

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18702 (sorry, I just edited the structure above to be more correct) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #18723: [SPARK-21517][CORE] Avoid copying memory when transfer c...

2017-07-25 Thread caneGuy
Github user caneGuy commented on the issue: https://github.com/apache/spark/pull/18723 @kiszk Actually, i am confused with default value 16 too. Yes, it occurs in specific scenario.In our case, it was large size block data which caused this issue. --- If your project is set up

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18702 Thank you @srowen. I will double check and clean up minor nits and will ping you within tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #18711: [SPARK-21506][DOC]The description of "spark.executor.cor...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18711 **[Test build #79925 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79925/testReport)** for PR 18711 at commit

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18702#discussion_r129250088 --- Diff: sql/log4j.properties --- @@ -0,0 +1,24 @@ +# --- End diff -- This is used -

[GitHub] spark issue #18722: [SPARK-21498][Examples] quick start -> one py demo have ...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18722 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #18726: [MINOR][CORE][TEST]Repeat stop SparkContext in ExecutorA...

2017-07-25 Thread jerryshao
Github user jerryshao commented on the issue: https://github.com/apache/spark/pull/18726 I don't see a significant overhead here by calling `stop()` again. I don't see a strong reason to change the code here AFAIK. --- If your project is set up for it, you can reply to this email

[GitHub] spark issue #18711: [SPARK-21506][DOC]The description of "spark.executor.cor...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18711 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79925/ Test PASSed. ---

[GitHub] spark issue #18711: [SPARK-21506][DOC]The description of "spark.executor.cor...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18711 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

2017-07-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18702#discussion_r129245068 --- Diff: sql/gen-sql-markdown.py --- @@ -0,0 +1,96 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

2017-07-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18702#discussion_r129245196 --- Diff: docs/_plugins/copy_api_dirs.rb --- @@ -150,4 +150,31 @@ cp("../R/pkg/DESCRIPTION", "api") end + if not

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

2017-07-25 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/18702#discussion_r129244802 --- Diff: sql/log4j.properties --- @@ -0,0 +1,24 @@ +# --- End diff -- Pardon, is a log4j.properties really needed here? what reads it?

[GitHub] spark issue #18722: [SPARK-21498][Examples] quick start -> one py demo have ...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18722 **[Test build #3851 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3851/testReport)** for PR 18722 at commit

[GitHub] spark issue #18610: [SPARK-21386] ML LinearRegression supports warm start fr...

2017-07-25 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/18610 There was a _lot_ of discussion around the `KMeans` initial model. But here we seem to be just ignoring certain issues. Such as, does the `initialModel` override the param settings (e.g. reg param,

[GitHub] spark issue #18651: [SPARK-21383][Core] Fix the YarnAllocator allocates more...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18651 **[Test build #79928 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79928/testReport)** for PR 18651 at commit

[GitHub] spark issue #18651: [SPARK-21383][Core] Fix the YarnAllocator allocates more...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18651 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79928/ Test PASSed. ---

[GitHub] spark issue #18651: [SPARK-21383][Core] Fix the YarnAllocator allocates more...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18651 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18702 In any case, it is easy. I was just thinking of matching it up with, like, R and Python which are, up to my knowledge, something like the below: **R**: ``` . ├── docs

[GitHub] spark issue #18729: [SPARK-21526] [MLlib] Add support to ML LogisticRegressi...

2017-07-25 Thread MLnick
Github user MLnick commented on the issue: https://github.com/apache/spark/pull/18729 Hi there, thanks for this. However, please see #18610 and related JIRA tickets. The solution is slightly more complex and needs to be done in a more generic way. I suggest that this be updated once

[GitHub] spark pull request #18722: [SPARK-21498][Examples] quick start -> one py dem...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18722#discussion_r129244806 --- Diff: docs/quick-start.md --- @@ -421,16 +421,15 @@ $ YOUR_SPARK_HOME/bin/spark-submit \ Lines with a: 46, Lines with b: 23 {%

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #79926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79926/testReport)** for PR 18555 at commit

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18555 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18555 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79926/ Test FAILed. ---

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #79926 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79926/testReport)** for PR 18555 at commit

[GitHub] spark issue #18503: [SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a multip...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18503 **[Test build #79929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79929/testReport)** for PR 18503 at commit

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #79932 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79932/testReport)** for PR 18555 at commit

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/18702 Yeah the latter is what I had in mind but I don't know, is it hard? it does seem more natural to generate the output directly into its destination, I think, unless I'm overlooking something else the

[GitHub] spark pull request #18711: [SPARK-21506][DOC]The description of "spark.execu...

2017-07-25 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/18711#discussion_r129246022 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -580,7 +580,13 @@ private[deploy] class Master( * The number of

[GitHub] spark pull request #18722: [SPARK-21498][Examples] quick start -> one py dem...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18722#discussion_r129248977 --- Diff: docs/quick-start.md --- @@ -421,7 +421,7 @@ $ YOUR_SPARK_HOME/bin/spark-submit \ Lines with a: 46, Lines with b: 23 {% endhighlight

[GitHub] spark issue #18722: [SPARK-21498][Examples] quick start -> one py demo have ...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18722 **[Test build #3851 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3851/testReport)** for PR 18722 at commit

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18555 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #79927 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79927/testReport)** for PR 18555 at commit

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18555 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79927/ Test FAILed. ---

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #79930 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79930/testReport)** for PR 18555 at commit

[GitHub] spark issue #18711: [SPARK-21506][DOC]The description of "spark.executor.cor...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18711 **[Test build #79925 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79925/testReport)** for PR 18711 at commit

[GitHub] spark issue #18503: [SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a multip...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18503 **[Test build #79924 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79924/testReport)** for PR 18503 at commit

[GitHub] spark issue #18730: [SPARK-21527][CORE] Use buffer limit in order to use JAV...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18730 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #14151: [SPARK-16496][SQL] Add wholetext as option for reading t...

2017-07-25 Thread ScrapCodes
Github user ScrapCodes commented on the issue: https://github.com/apache/spark/pull/14151 @sameeragarwal Do you think this change still makes sense? Can I improve it somehow? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as

[GitHub] spark pull request #18711: [SPARK-21506][DOC]The description of "spark.execu...

2017-07-25 Thread jerryshao
Github user jerryshao commented on a diff in the pull request: https://github.com/apache/spark/pull/18711#discussion_r129245415 --- Diff: core/src/main/scala/org/apache/spark/deploy/master/Master.scala --- @@ -580,7 +580,13 @@ private[deploy] class Master( * The number of

[GitHub] spark issue #18693: [SPARK-21491][GraphX] Enhance GraphX performance: breakO...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18693 **[Test build #3850 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3850/testReport)** for PR 18693 at commit

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18702#discussion_r129251387 --- Diff: sql/gen-sql-markdown.py --- @@ -0,0 +1,96 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +#

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #79927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79927/testReport)** for PR 18555 at commit

[GitHub] spark pull request #18503: [SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a...

2017-07-25 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/18503#discussion_r129257673 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala --- @@ -363,7 +363,8 @@ private[state]

[GitHub] spark issue #18709: [SPARK-21504] [SQL] Add spark version info into table me...

2017-07-25 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18709 LGTM except one comment. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18702 I addressed the log4j comment for now first (I got rid of log4j file by avoiding directly accessing to JVM) --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #18693: [SPARK-21491][GraphX] Enhance GraphX performance: breakO...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18693 **[Test build #3850 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3850/testReport)** for PR 18693 at commit

[GitHub] spark pull request #18722: [SPARK-21498][Examples] quick start -> one py dem...

2017-07-25 Thread lizhaoch
Github user lizhaoch commented on a diff in the pull request: https://github.com/apache/spark/pull/18722#discussion_r129246403 --- Diff: docs/quick-start.md --- @@ -421,16 +421,15 @@ $ YOUR_SPARK_HOME/bin/spark-submit \ Lines with a: 46, Lines with b: 23 {% endhighlight

[GitHub] spark pull request #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation ...

2017-07-25 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18702#discussion_r129254786 --- Diff: docs/_plugins/copy_api_dirs.rb --- @@ -150,4 +150,31 @@ cp("../R/pkg/DESCRIPTION", "api") end + if not

[GitHub] spark issue #18503: [SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a multip...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18503 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18503: [SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a multip...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18503 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79924/ Test FAILed. ---

[GitHub] spark pull request #18709: [SPARK-21504] [SQL] Add spark version info into t...

2017-07-25 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/18709#discussion_r129259658 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/client/HiveClientImpl.scala --- @@ -932,6 +934,7 @@ private[hive] object HiveClientImpl {

[GitHub] spark issue #18651: [SPARK-21383][Core] Fix the YarnAllocator allocates more...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18651 **[Test build #79928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79928/testReport)** for PR 18651 at commit

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18702 **[Test build #79931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79931/testReport)** for PR 18702 at commit

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #79933 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79933/testReport)** for PR 18555 at commit

[GitHub] spark issue #18731: [SPARK-20990][SQL] Read all JSON documents in files when...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18731 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #18388: [SPARK-21175] Reject OpenBlocks when memory shortage on ...

2017-07-25 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/18388 thanks, merging to master! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #16765: [SPARK-19425][SQL] Make ExtractEquiJoinKeys support UDT ...

2017-07-25 Thread joosterman
Github user joosterman commented on the issue: https://github.com/apache/spark/pull/16765 @alg-jmx Issue is fixed for Spark 2.2.0 according the ticket here: https://issues.apache.org/jira/browse/SPARK-19425 --- If your project is set up for it, you can reply to this email and have

[GitHub] spark issue #18503: [SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a multip...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18503 **[Test build #79929 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79929/testReport)** for PR 18503 at commit

[GitHub] spark issue #18503: [SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a multip...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18503 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79929/ Test PASSed. ---

[GitHub] spark pull request #18726: [MINOR][CORE][TEST]Repeat stop SparkContext in Ex...

2017-07-25 Thread heary-cao
Github user heary-cao closed the pull request at: https://github.com/apache/spark/pull/18726 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18702 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79931/ Test PASSed. ---

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18702 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #79933 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79933/testReport)** for PR 18555 at commit

[GitHub] spark issue #18731: [SPARK-20990][SQL] Read all JSON documents in files when...

2017-07-25 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/18731 cc @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark issue #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF and Ja...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17848 **[Test build #79936 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79936/testReport)** for PR 17848 at commit

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18513 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18513 **[Test build #79934 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79934/testReport)** for PR 18513 at commit

[GitHub] spark issue #18513: [SPARK-13969][ML] Add FeatureHasher transformer

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18513 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79934/ Test PASSed. ---

[GitHub] spark pull request #18388: [SPARK-21175] Reject OpenBlocks when memory short...

2017-07-25 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18388 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark pull request #18722: [SPARK-21498][Examples] quick start -> one py dem...

2017-07-25 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18722 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #18702: [SPARK-21485][SQL][DOCS] Spark SQL documentation generat...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18702 **[Test build #79931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79931/testReport)** for PR 18702 at commit

[GitHub] spark issue #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF and Ja...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17848 **[Test build #79935 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79935/testReport)** for PR 17848 at commit

[GitHub] spark issue #18388: [SPARK-21175] Reject OpenBlocks when memory shortage on ...

2017-07-25 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/18388 @tgravescs Thanks for help. > I think we should expand the description of the config to say what happens when the limit is hit. Since its not using real flow control a user might set

[GitHub] spark pull request #18610: [SPARK-21386] ML LinearRegression supports warm s...

2017-07-25 Thread yanboliang
Github user yanboliang commented on a diff in the pull request: https://github.com/apache/spark/pull/18610#discussion_r129324027 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/LinearRegression.scala --- @@ -365,7 +398,11 @@ class LinearRegression @Since("1.3.0")

[GitHub] spark issue #18707: [SPARK-21503][UI]: Spark UI shows incorrect task status ...

2017-07-25 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18707 **[Test build #79937 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79937/testReport)** for PR 18707 at commit

[GitHub] spark pull request #18707: [SPARK-21503][UI]: Spark UI shows incorrect task ...

2017-07-25 Thread pgandhi999
Github user pgandhi999 commented on a diff in the pull request: https://github.com/apache/spark/pull/18707#discussion_r129330779 --- Diff: core/src/main/scala/org/apache/spark/ui/exec/ExecutorsTab.scala --- @@ -140,6 +140,8 @@ class ExecutorsListener(storageStatusListener:

[GitHub] spark issue #18651: [SPARK-21383][Core] Fix the YarnAllocator allocates more...

2017-07-25 Thread tgravescs
Github user tgravescs commented on the issue: https://github.com/apache/spark/pull/18651 LGTM. @vanzin anything further? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17848: [SPARK-20586] [SQL] Add deterministic to ScalaUDF and Ja...

2017-07-25 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/17848 let's leave the java UDF API unchanged and think about whether we should add java UDF API in `functions` later. @gatorsmile can you update the PR title? Thanks! --- If your project is set up

[GitHub] spark issue #18503: [SPARK-21271][SQL] Ensure Unsafe.sizeInBytes is a multip...

2017-07-25 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18503 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

  1   2   3   >