[GitHub] spark issue #20210: [SPARK-23009][PYTHON] Fix for non-str col names to creat...

2018-01-10 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20210 Hi, @HyukjinKwon . It seems that branch-2.3 doesn't have this. Could you merge to branch-2.3, too? --- - To

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20096 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20056: [SPARK-22878] [CORE] Count totalDroppedEvents for LiveLi...

2018-01-10 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20056 I have the same question as @jiangxb1987 , what is the situation where you'd use this metric? the jira doesn't say either. seems like existing metrics mostly cover this? ---

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20096 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85926/ Test PASSed. ---

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20096 **[Test build #85926 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85926/testReport)** for PR 20096 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85931/ Test PASSed. ---

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19885 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #85931 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85931/testReport)** for PR 19885 at commit

[GitHub] spark issue #20223: [SPARK-23020][core] Fix races in launcher code, test.

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20223 **[Test build #85929 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85929/testReport)** for PR 20223 at commit

[GitHub] spark issue #19885: [SPARK-22587] Spark job fails if fs.defaultFS and applic...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19885 **[Test build #85931 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85931/testReport)** for PR 19885 at commit

[GitHub] spark issue #20174: [SPARK-22951][SQL] fix aggregation after dropDuplicates ...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20174 **[Test build #85930 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85930/testReport)** for PR 20174 at commit

[GitHub] spark issue #20192: [SPARK-22994][k8s] Use a single image for all Spark cont...

2018-01-10 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20192 If there's no more feedback I'll merge this later today. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data

2018-01-10 Thread brad-kaiser
Github user brad-kaiser commented on the issue: https://github.com/apache/spark/pull/19041 Lol, no worries. Thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20223: [SPARK-23020][core] Fix races in launcher code, t...

2018-01-10 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/20223 [SPARK-23020][core] Fix races in launcher code, test. The race in the code is because the handle might update its state to the wrong state if the connection handling thread is still

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20013 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85924/ Test PASSed. ---

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20013 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20013 **[Test build #85924 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85924/testReport)** for PR 20013 at commit

[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data

2018-01-10 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/19041 > Is there anything else you need for this PR? An extra day on my work week... --- - To unsubscribe, e-mail:

[GitHub] spark issue #19041: [SPARK-21097][CORE] Add option to recover cached data

2018-01-10 Thread brad-kaiser
Github user brad-kaiser commented on the issue: https://github.com/apache/spark/pull/19041 Hey @vanzin just wanted to check in on this. Is there anything else you need for this PR? Thanks! --- - To unsubscribe,

[GitHub] spark issue #20218: [SPARK-23000] [TEST-HADOOP2.6] Fix Flaky test suite Data...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20218 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85925/ Test PASSed. ---

[GitHub] spark issue #20218: [SPARK-23000] [TEST-HADOOP2.6] Fix Flaky test suite Data...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20218 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20218: [SPARK-23000] [TEST-HADOOP2.6] Fix Flaky test suite Data...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20218 **[Test build #85925 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85925/testReport)** for PR 20218 at commit

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20222 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20222 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85923/ Test FAILed. ---

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20222 **[Test build #85923 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85923/testReport)** for PR 20222 at commit

[GitHub] spark issue #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregation func...

2018-01-10 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/19872 @ueshin and @HyukjinKwon Thanks much for the review. I have addressed the latest comment. @ueshin I think UDAF that supports partial aggregation can be build on top of this. The

[GitHub] spark issue #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregation func...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19872 **[Test build #85928 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85928/testReport)** for PR 19872 at commit

[GitHub] spark pull request #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregati...

2018-01-10 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r160779041 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala --- @@ -15,12 +15,30 @@ * limitations under the

[GitHub] spark pull request #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregati...

2018-01-10 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r160779007 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/python/AggregateInPandasExec.scala --- @@ -0,0 +1,152 @@ +/* + * Licensed to

[GitHub] spark pull request #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregati...

2018-01-10 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r160778894 --- Diff: python/pyspark/sql/tests.py --- @@ -4052,6 +4045,386 @@ def test_unsupported_types(self):

[GitHub] spark pull request #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregati...

2018-01-10 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r160778794 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/CheckAnalysis.scala --- @@ -153,11 +153,20 @@ trait CheckAnalysis extends

[GitHub] spark pull request #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregati...

2018-01-10 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r160778766 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/SparkStrategies.scala --- @@ -271,9 +272,14 @@ abstract class SparkStrategies extends

[GitHub] spark pull request #20096: [SPARK-22908] Add kafka source and sink for conti...

2018-01-10 Thread zsxwing
Github user zsxwing commented on a diff in the pull request: https://github.com/apache/spark/pull/20096#discussion_r160774506 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/continuous/EpochCoordinator.scala --- @@ -39,6 +39,15 @@ private[continuous]

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20222 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20222 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85922/ Test FAILed. ---

[GitHub] spark pull request #20174: [SPARK-22951][SQL] fix aggregation after dropDupl...

2018-01-10 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/20174#discussion_r160772523 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/optimizer/ReplaceOperatorSuite.scala --- @@ -198,6 +198,20 @@ class

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20222 **[Test build #85922 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85922/testReport)** for PR 20222 at commit

[GitHub] spark pull request #20174: [SPARK-22951][SQL] fix aggregation after dropDupl...

2018-01-10 Thread liancheng
Github user liancheng commented on a diff in the pull request: https://github.com/apache/spark/pull/20174#discussion_r160770992 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameAggregateSuite.scala --- @@ -666,4 +665,16 @@ class DataFrameAggregateSuite extends

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20222 Wait dev/release-tag.sh does this automatically though. I just want to make sure we are not missing things (like R/DESCRIPTION) I think maybe we should run a subset of

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread felixcheungu
Github user felixcheungu commented on the issue: https://github.com/apache/spark/pull/20222 @felixcheung --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20153: [SPARK-22392][SQL] data source v2 columnar batch ...

2018-01-10 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/20153#discussion_r160764573 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsScanColumnarBatch.java --- @@ -0,0 +1,51 @@ +/* + * Licensed to the

[GitHub] spark issue #20174: [SPARK-22951][SQL] fix aggregation after dropDuplicates ...

2018-01-10 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20174 I see, thnaks for your answer @liancheng --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20174: [SPARK-22951][SQL] fix aggregation after dropDuplicates ...

2018-01-10 Thread liancheng
Github user liancheng commented on the issue: https://github.com/apache/spark/pull/20174 @mgaido91 We can't because we do not know whether there are any input rows or not. For example: ```scala val df1 = spark.range(10).select() val df2 = spark.range(10).filter($"id"

[GitHub] spark issue #20174: [SPARK-22951][SQL] fix aggregation after dropDuplicates ...

2018-01-10 Thread liufengdb
Github user liufengdb commented on the issue: https://github.com/apache/spark/pull/20174 @mgaido91 Your proposal and current approach are both with one line change. Since the issue is actually related to the hash aggregate implementation, I think it is reasonable to include it in the

[GitHub] spark issue #20174: [SPARK-22951][SQL] fix aggregation after dropDuplicates ...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20174 **[Test build #85927 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85927/testReport)** for PR 20174 at commit

[GitHub] spark pull request #20174: [SPARK-22951][SQL] fix aggregation after dropDupl...

2018-01-10 Thread liufengdb
Github user liufengdb commented on a diff in the pull request: https://github.com/apache/spark/pull/20174#discussion_r160757582 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -1221,7 +1221,12 @@ object

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20096 **[Test build #85926 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85926/testReport)** for PR 20096 at commit

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-10 Thread jose-torres
Github user jose-torres commented on the issue: https://github.com/apache/spark/pull/20096 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20218: [SPARK-23000] [TEST-HADOOP2.6] Fix Flaky test suite Data...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20218 **[Test build #85925 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85925/testReport)** for PR 20218 at commit

[GitHub] spark issue #20218: [SPARK-23000] [TEST-HADOOP2.6] Fix Flaky test suite Data...

2018-01-10 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20218 Since this is a flakiness issue, I retriggered it. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #20218: [SPARK-23000] [TEST-HADOOP2.6] Fix Flaky test suite Data...

2018-01-10 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20218 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20221: [SPARK-23019][Core] Wait until SparkContext.stop(...

2018-01-10 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20221 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #18991: [SPARK-21783][SQL] Turn on ORC filter push-down by defau...

2018-01-10 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18991 @gatorsmile . I don't have any numbers for PPD=false. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #20219: [SPARK-23025][SQL] Support Null type in scala reflection

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20219 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20219: [SPARK-23025][SQL] Support Null type in scala reflection

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20219 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85921/ Test PASSed. ---

[GitHub] spark issue #20219: [SPARK-23025][SQL] Support Null type in scala reflection

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20219 **[Test build #85921 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85921/testReport)** for PR 20219 at commit

[GitHub] spark issue #20221: [SPARK-23019][Core] Wait until SparkContext.stop() finis...

2018-01-10 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20221 Merging to master / 2.3. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20221: [SPARK-23019][Core] Wait until SparkContext.stop() finis...

2018-01-10 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20221 Oh, wait, yes, this should fix SPARK-23019 (since that's caused by the context being still alive). I'll keep investigating SPARK-23020 separately. ---

[GitHub] spark pull request #20153: [SPARK-22392][SQL] data source v2 columnar batch ...

2018-01-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20153#discussion_r160747322 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsScanColumnarBatch.java --- @@ -0,0 +1,51 @@ +/* + * Licensed to

[GitHub] spark pull request #20153: [SPARK-22392][SQL] data source v2 columnar batch ...

2018-01-10 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20153#discussion_r160746791 --- Diff: sql/core/src/main/java/org/apache/spark/sql/sources/v2/reader/SupportsScanColumnarBatch.java --- @@ -0,0 +1,51 @@ +/* + * Licensed to

[GitHub] spark issue #20221: [SPARK-23019][Core] Wait until SparkContext.stop() finis...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20221 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85919/ Test PASSed. ---

[GitHub] spark issue #20221: [SPARK-23019][Core] Wait until SparkContext.stop() finis...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20221 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20221: [SPARK-23019][Core] Wait until SparkContext.stop() finis...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20221 **[Test build #85919 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85919/testReport)** for PR 20221 at commit

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20013 **[Test build #85924 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85924/testReport)** for PR 20013 at commit

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20222 Oh I see, they're just not listed on https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/ . I believe I can fix that. ---

[GitHub] spark pull request #20222: [SPARK-23028] Bump master branch version to 2.4.0...

2018-01-10 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/20222#discussion_r160741170 --- Diff: project/MimaExcludes.scala --- @@ -34,6 +34,10 @@ import com.typesafe.tools.mima.core.ProblemFilters._ */ object MimaExcludes {

[GitHub] spark pull request #20222: [SPARK-23028] Bump master branch version to 2.4.0...

2018-01-10 Thread JoshRosen
Github user JoshRosen commented on a diff in the pull request: https://github.com/apache/spark/pull/20222#discussion_r160740904 --- Diff: project/MimaExcludes.scala --- @@ -34,6 +34,10 @@ import com.typesafe.tools.mima.core.ProblemFilters._ */ object MimaExcludes {

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread JoshRosen
Github user JoshRosen commented on the issue: https://github.com/apache/spark/pull/20222 We should already be set up for 2.3.x builds in AMPLab Jenkins. For example: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test/job/spark-branch-2.3-test-maven-hadoop-2.6/ ---

[GitHub] spark issue #20210: [SPARK-23009][PYTHON] Fix for non-str col names to creat...

2018-01-10 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/20210 Thanks @ueshin and @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20213: [SPARK-23018][PYTHON] Fix createDataFrame from Pandas ti...

2018-01-10 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/20213 Thanks @ueshin and @HyukjinKwon ! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20072: [SPARK-22790][SQL] add a configurable factor to describe...

2018-01-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20072 cc @CodingCat --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #18991: [SPARK-21783][SQL] Turn on ORC filter push-down by defau...

2018-01-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18991 What is the performance number when turning it on, compared with the off mode? --- - To unsubscribe, e-mail:

[GitHub] spark issue #20219: [SPARK-23025][SQL] Support Null type in scala reflection

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20219 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85920/ Test PASSed. ---

[GitHub] spark issue #20219: [SPARK-23025][SQL] Support Null type in scala reflection

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20219 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20219: [SPARK-23025][SQL] Support Null type in scala reflection

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20219 **[Test build #85920 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85920/testReport)** for PR 20219 at commit

[GitHub] spark issue #20203: [SPARK-22577] [core] executor page blacklist status shou...

2018-01-10 Thread attilapiros
Github user attilapiros commented on the issue: https://github.com/apache/spark/pull/20203 cc @tgravescs --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20151 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85918/ Test PASSed. ---

[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20151 **[Test build #85918 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85918/testReport)** for PR 20151 at commit

[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20151 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85917/ Test PASSed. ---

[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20151 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20151: [SPARK-22959][PYTHON] Configuration to select the module...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20151 **[Test build #85917 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85917/testReport)** for PR 20151 at commit

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20222 **[Test build #85923 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85923/testReport)** for PR 20222 at commit

[GitHub] spark pull request #20221: [SPARK-23019][Core] Wait until SparkContext.stop(...

2018-01-10 Thread gengliangwang
Github user gengliangwang commented on a diff in the pull request: https://github.com/apache/spark/pull/20221#discussion_r160719518 --- Diff: core/src/test/java/org/apache/spark/launcher/SparkLauncherSuite.java --- @@ -133,6 +134,10 @@ public void testInProcessLauncher() throws

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20222 `R/pkg/DESCRIPTION` needs an update too. I'll update the release process. @JoshRosen or @shaneknapp looks like we also need to have set up the branch-2.3 tests in Jenkins. Right now it's

[GitHub] spark issue #20208: [SPARK-23007][SQL][TEST] Add schema evolution test suite...

2018-01-10 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20208 Also, ping @rxin , too. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands,

[GitHub] spark issue #18991: [SPARK-21783][SQL] Turn on ORC filter push-down by defau...

2018-01-10 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/18991 Yes, @cloud-fan . I added the same test coverage for ORC in Apache Spark. Sorry, @gatorsmile . I always turned on PPD, so there is no perf number for PPD=false. ---

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20222 @srowen Yeah. I am still in travel mode. : ) Please help do it. Thanks! --- - To unsubscribe, e-mail:

[GitHub] spark issue #20195: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20195 Thanks! Merged to 2.2 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20199: [Spark-22967][TESTS]Fix VersionSuite's unit tests...

2018-01-10 Thread Ngone51
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/20199#discussion_r160709123 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/VersionsSuite.scala --- @@ -842,6 +842,7 @@ class VersionsSuite extends SparkFunSuite

[GitHub] spark pull request #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregati...

2018-01-10 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r160708666 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/PythonUDF.scala --- @@ -15,12 +15,30 @@ * limitations under the

[GitHub] spark pull request #19872: [SPARK-22274][PYTHON][SQL] User-defined aggregati...

2018-01-10 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/19872#discussion_r160708246 --- Diff: python/pyspark/sql/tests.py --- @@ -511,7 +517,6 @@ def test_udf_with_order_by_and_limit(self): my_copy = udf(lambda x: x,

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20222 Yeah ideally that's part of the branching process. I think it could be documented in the "Preparing Spark for Release" section of `release-process.md` in `spark-website`. Open up a separate PR for

[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

2018-01-10 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19802 I see, yeah that's a fine point. Don't rethrow then. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #19802: [SPARK-22594][CORE] Handling spark-submit and master ver...

2018-01-10 Thread Jiri-Kremser
Github user Jiri-Kremser commented on the issue: https://github.com/apache/spark/pull/19802 > I think it's probably OK if you also re-throw that exception for now. You're just giving more info then. Hmm, In the original code the exception wasn't re-thrown. Do you really

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20222 **[Test build #85922 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85922/testReport)** for PR 20222 at commit

[GitHub] spark issue #20222: [SPARK-23028] Bump master branch version to 2.4.0-SNAPSH...

2018-01-10 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20222 cc @yhuai @sameeragarwal @JoshRosen @rxin --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20222: [SPARK-23028] Bump master branch version to 2.4.0...

2018-01-10 Thread gatorsmile
GitHub user gatorsmile opened a pull request: https://github.com/apache/spark/pull/20222 [SPARK-23028] Bump master branch version to 2.4.0-SNAPSHOT ## What changes were proposed in this pull request? This patch bumps the master branch version to `2.4.0-SNAPSHOT`. ## How

[GitHub] spark issue #20219: [SPARK-23025][SQL] Support Null type in scala reflection

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20219 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85916/ Test FAILed. ---

[GitHub] spark issue #20219: [SPARK-23025][SQL] Support Null type in scala reflection

2018-01-10 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20219 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

<    1   2   3   4   5   >