[GitHub] spark pull request #20535: [SPARK-23341][SQL] define some standard options f...

2018-04-23 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20535#discussion_r183470867 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -193,10 +196,13 @@ class DataFrameReader private[sql](sparkSession:

[GitHub] spark issue #18903: [SPARK-21590][SS]Window start time should support negati...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18903 **[Test build #4154 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4154/testReport)** for PR 18903 at commit [`07e98e7`](https://github.com/apache/spark/commit/

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21123 **[Test build #89724 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89724/testReport)** for PR 21123 at commit [`968799c`](https://github.com/apache/spark/commit/9

[GitHub] spark pull request #20923: [SPARK-23807][BUILD] Add Hadoop 3.1 profile with ...

2018-04-23 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/20923#discussion_r183470258 --- Diff: pom.xml --- @@ -2671,6 +2671,15 @@ + + hadoop-3.1 --- End diff -- +1 for skipping H

[GitHub] spark issue #19881: [SPARK-22683][CORE] Add a executorAllocationRatio parame...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19881 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #19881: [SPARK-22683][CORE] Add a executorAllocationRatio parame...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19881 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89712/ Test PASSed. ---

[GitHub] spark issue #19881: [SPARK-22683][CORE] Add a executorAllocationRatio parame...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19881 **[Test build #89712 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89712/testReport)** for PR 19881 at commit [`3b1dddc`](https://github.com/apache/spark/commit/3

[GitHub] spark issue #20701: [SPARK-23528][ML] Add numIter to ClusteringSummary

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20701 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20701: [SPARK-23528][ML] Add numIter to ClusteringSummary

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20701 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89722/ Test PASSed. ---

[GitHub] spark issue #21127: [SPARK-24052][CORE][UI] Add spark version information on...

2018-04-23 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21127 The Spark version already shows next to the Spark logo on every page in the UI? --- - To unsubscribe, e-mail: reviews-unsubscr...

[GitHub] spark issue #20701: [SPARK-23528][ML] Add numIter to ClusteringSummary

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20701 **[Test build #89722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89722/testReport)** for PR 20701 at commit [`59fef4e`](https://github.com/apache/spark/commit/5

[GitHub] spark issue #21128: [SPARK-24053][CORE] Support add subdirectory named as us...

2018-04-23 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/21128 > When we have multiple users on the same cluster Then the staging directory would be created under the respective user already (it's created under the user's home directory). I have no idea

[GitHub] spark issue #21018: [SPARK-23880][SQL] Do not trigger any jobs for caching d...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21018 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21018: [SPARK-23880][SQL] Do not trigger any jobs for caching d...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21018 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89718/ Test PASSed. ---

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21123 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21018: [SPARK-23880][SQL] Do not trigger any jobs for caching d...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21018 **[Test build #89718 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89718/testReport)** for PR 21018 at commit [`f5f8fbf`](https://github.com/apache/spark/commit/f

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21123 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89717/ Test PASSed. ---

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21123 **[Test build #89717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89717/testReport)** for PR 21123 at commit [`8301756`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #21128: [SPARK-24053][CORE] Support add subdirectory named as us...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21128 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89715/ Test PASSed. ---

[GitHub] spark issue #21128: [SPARK-24053][CORE] Support add subdirectory named as us...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21128 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21128: [SPARK-24053][CORE] Support add subdirectory named as us...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21128 **[Test build #89715 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89715/testReport)** for PR 21128 at commit [`b319052`](https://github.com/apache/spark/commit/b

[GitHub] spark issue #21072: [SPARK-23973][SQL] Remove consecutive Sorts

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21072 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21072: [SPARK-23973][SQL] Remove consecutive Sorts

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21072 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89721/ Test PASSed. ---

[GitHub] spark issue #21072: [SPARK-23973][SQL] Remove consecutive Sorts

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21072 **[Test build #89721 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89721/testReport)** for PR 21072 at commit [`e7391f3`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21123 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21123 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2595/ Tes

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21123 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21123 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89716/ Test PASSed. ---

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21123 **[Test build #89732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89732/testReport)** for PR 21123 at commit [`35656bd`](https://github.com/apache/spark/commit/35

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21123 **[Test build #89716 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89716/testReport)** for PR 21123 at commit [`8369618`](https://github.com/apache/spark/commit/8

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21130 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21130 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89731/ Test PASSed. ---

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21130 **[Test build #89731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89731/testReport)** for PR 21130 at commit [`7f75f5e`](https://github.com/apache/spark/commit/7

[GitHub] spark pull request #20946: [SPARK-23565] [SQL] New error message for structu...

2018-04-23 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/20946#discussion_r183447816 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/OffsetSeq.scala --- @@ -39,7 +39,9 @@ case class OffsetSeq(offsets: Seq[Opt

[GitHub] spark pull request #20946: [SPARK-23565] [SQL] New error message for structu...

2018-04-23 Thread xuanyuanking
Github user xuanyuanking commented on a diff in the pull request: https://github.com/apache/spark/pull/20946#discussion_r183447988 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/streaming/OffsetSeqLogSuite.scala --- @@ -125,6 +125,19 @@ class OffsetSeqLogSuite ex

[GitHub] spark issue #21124: [SPARK-23004][SS] Ensure StateStore.commit is called onl...

2018-04-23 Thread brkyvz
Github user brkyvz commented on the issue: https://github.com/apache/spark/pull/21124 LGTM! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apach

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21130 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21130 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89729/ Test PASSed. ---

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21130 **[Test build #89729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89729/testReport)** for PR 21130 at commit [`443034a`](https://github.com/apache/spark/commit/4

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21130 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21130 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2594/ Tes

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21130 **[Test build #89731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89731/testReport)** for PR 21130 at commit [`7f75f5e`](https://github.com/apache/spark/commit/7f

[GitHub] spark issue #20933: [SPARK-23817][SQL]Migrate ORC file format read path to d...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20933 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89719/ Test FAILed. ---

[GitHub] spark issue #20933: [SPARK-23817][SQL]Migrate ORC file format read path to d...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20933 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21106: [SPARK-23711][SQL][WIP] Add fallback logic for UnsafePro...

2018-04-23 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21106 I think it's very hard to unify the entry point(input type) for all the code generators. E.g. some use `Seq[Expression]` as input, some use `Seq[DataType]`. I'd like to make `CodegenObjec

[GitHub] spark issue #20933: [SPARK-23817][SQL]Migrate ORC file format read path to d...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20933 **[Test build #89719 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89719/testReport)** for PR 20933 at commit [`635a6a2`](https://github.com/apache/spark/commit/6

[GitHub] spark pull request #21107: [SPARK-24044][PYTHON] Explicitly print out skippe...

2018-04-23 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21107#discussion_r183430497 --- Diff: python/pyspark/ml/tests.py --- @@ -2136,17 +2136,23 @@ class ImageReaderTest2(PySparkTestCase): @classmethod def setUpClass(cl

[GitHub] spark issue #21113: [MINOR][DOCS] Fix comments of SQLExecution#withExecution...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21113 **[Test build #89730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89730/testReport)** for PR 21113 at commit [`6aceb43`](https://github.com/apache/spark/commit/6a

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-04-23 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21082 Will take a close look soon within this weekend as well. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org F

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21130 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2593/ Tes

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21130 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21113: [MINOR][DOCS] Fix comments of SQLExecution#withExecution...

2018-04-23 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21113 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@s

[GitHub] spark pull request #21107: [SPARK-24044][PYTHON] Explicitly print out skippe...

2018-04-23 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/21107#discussion_r183428356 --- Diff: python/pyspark/ml/tests.py --- @@ -2136,17 +2136,23 @@ class ImageReaderTest2(PySparkTestCase): @classmethod def setUpClass(c

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21130 **[Test build #89729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89729/testReport)** for PR 21130 at commit [`443034a`](https://github.com/apache/spark/commit/44

[GitHub] spark issue #21130: [SPARK-24054][R] Add array_position function / element_a...

2018-04-23 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21130 cc @felixcheung, can you take a look please when you are available? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.a

[GitHub] spark pull request #21130: [SPARK-24054][R] Add array_position function / el...

2018-04-23 Thread HyukjinKwon
GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/21130 [SPARK-24054][R] Add array_position function / element_at functions ## What changes were proposed in this pull request? This PR proposes to add array_position and element_at in R side t

[GitHub] spark issue #21106: [SPARK-23711][SQL][WIP] Add fallback logic for UnsafePro...

2018-04-23 Thread hvanhovell
Github user hvanhovell commented on the issue: https://github.com/apache/spark/pull/21106 Yeah, let's move forward slowly. I am still pondering about the what the right abstraction here would look like; this looks promising though. Can you try to unify this class with `UnsafeP

[GitHub] spark pull request #21082: [SPARK-22239][SQL][Python] Enable grouped aggrega...

2018-04-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21082#discussion_r183412509 --- Diff: python/pyspark/sql/functions.py --- @@ -2301,10 +2301,12 @@ def pandas_udf(f=None, returnType=None, functionType=None): The returned sc

[GitHub] spark pull request #21082: [SPARK-22239][SQL][Python] Enable grouped aggrega...

2018-04-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/21082#discussion_r183412270 --- Diff: python/pyspark/sql/functions.py --- @@ -2301,10 +2301,12 @@ def pandas_udf(f=None, returnType=None, functionType=None): The returned sc

[GitHub] spark issue #20998: [SPARK-23888][CORE] correct the comment of hasAttemptOnH...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20998 **[Test build #89728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89728/testReport)** for PR 20998 at commit [`e44d80b`](https://github.com/apache/spark/commit/e4

[GitHub] spark issue #19404: [SPARK-21760] [Streaming] Fix for Structured streaming t...

2018-04-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19404 @steveloughran what do you think of this? flushing sounds safe but is there a performance impact here if done on every `serialize`? --- ---

[GitHub] spark pull request #21107: [SPARK-24044][PYTHON] Explicitly print out skippe...

2018-04-23 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21107#discussion_r183409737 --- Diff: python/pyspark/ml/tests.py --- @@ -2136,17 +2136,23 @@ class ImageReaderTest2(PySparkTestCase): @classmethod def setUpClass(cl

[GitHub] spark issue #20677: Event time can't be greater then processing time. 12:21,...

2018-04-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20677 I agree. The (12:14, dog) event at 12:12 also happens "early". I suppose it could have been intentional, if the idea is to illustrate disagreement about time between the event producer and stream pro

[GitHub] spark pull request #21107: [SPARK-24044][PYTHON] Explicitly print out skippe...

2018-04-23 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/21107#discussion_r183409329 --- Diff: python/run-tests.py --- @@ -152,65 +172,17 @@ def parse_opts(): return opts -def _check_dependencies(python_exec, modul

[GitHub] spark pull request #20146: [SPARK-11215][ML] Add multiple columns support to...

2018-04-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/20146#discussion_r183408802 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -130,21 +161,53 @@ class StringIndexer @Since("1.4.0") ( @Since(

[GitHub] spark issue #21125: [Spark-24024][ML] Fix poisson deviance calculations in G...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21125 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/89723/ Test PASSed. ---

[GitHub] spark issue #21125: [Spark-24024][ML] Fix poisson deviance calculations in G...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21125 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark pull request #20998: [SPARK-23888][CORE] correct the comment of hasAtt...

2018-04-23 Thread Ngone51
Github user Ngone51 commented on a diff in the pull request: https://github.com/apache/spark/pull/20998#discussion_r183408481 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -287,7 +287,7 @@ private[spark] class TaskSetManager( None

[GitHub] spark issue #21125: [Spark-24024][ML] Fix poisson deviance calculations in G...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21125 **[Test build #89723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89723/testReport)** for PR 21125 at commit [`da53b1a`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #21082: [SPARK-22239][SQL][Python] Enable grouped aggregate pand...

2018-04-23 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/21082 Hey @HyukjinKwon @ueshin @BryanCutler I've fixed the tests and I think the PR is in good shape for review now. Could you please take a look when you have time? Thanks! ---

[GitHub] spark issue #21106: [SPARK-23711][SQL][WIP] Add fallback logic for UnsafePro...

2018-04-23 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/21106 @hvanhovell Any more comments on the current design? If not, I will apply this to all places that we create unsafe projection. --- --

[GitHub] spark issue #20146: [SPARK-11215][ML] Add multiple columns support to String...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20146 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #20146: [SPARK-11215][ML] Add multiple columns support to String...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20146 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2592/ Tes

[GitHub] spark issue #19694: [SPARK-22470][DOC][SQL] functions.hash is also used inte...

2018-04-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19694 This is true, and probably will stay true, but is it something a caller needs to know and that we want to document as a guarantee? --- --

[GitHub] spark pull request #20998: [SPARK-23888][CORE] correct the comment of hasAtt...

2018-04-23 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20998#discussion_r183406175 --- Diff: core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala --- @@ -287,7 +287,7 @@ private[spark] class TaskSetManager( None

[GitHub] spark issue #20146: [SPARK-11215][ML] Add multiple columns support to String...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20146 **[Test build #89727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89727/testReport)** for PR 20146 at commit [`ed35d87`](https://github.com/apache/spark/commit/ed

[GitHub] spark issue #19887: [SPARK-21168] KafkaRDD should always set kafka clientId.

2018-04-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19887 Seems reasonable; maybe @koeninger has a thought --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18903: [SPARK-21590][SS]Window start time should support negati...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18903 **[Test build #4154 has started](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/4154/testReport)** for PR 18903 at commit [`07e98e7`](https://github.com/apache/spark/commit/0

[GitHub] spark pull request #20146: [SPARK-11215][ML] Add multiple columns support to...

2018-04-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/20146#discussion_r183404967 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -130,21 +161,53 @@ class StringIndexer @Since("1.4.0") ( @Since(

[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-04-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20659 @wangyum you can close this experiment? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands

[GitHub] spark issue #21127: [SPARK-24052][CORE][UI] Add spark version information on...

2018-04-23 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/21127 We already have SparkBuildInfo for this purpose. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21072: [SPARK-23973][SQL] Remove consecutive Sorts

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21072 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21072: [SPARK-23973][SQL] Remove consecutive Sorts

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21072 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2591/ Tes

[GitHub] spark issue #21072: [SPARK-23973][SQL] Remove consecutive Sorts

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21072 **[Test build #89726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89726/testReport)** for PR 21072 at commit [`e2f4d4d`](https://github.com/apache/spark/commit/e2

[GitHub] spark pull request #21072: [SPARK-23973][SQL] Remove consecutive Sorts

2018-04-23 Thread mgaido91
Github user mgaido91 commented on a diff in the pull request: https://github.com/apache/spark/pull/21072#discussion_r183399564 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -736,12 +736,29 @@ object EliminateSorts extends Rule

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21123 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2590/ Tes

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21123 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21123 **[Test build #89725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89725/testReport)** for PR 21123 at commit [`c02a2b4`](https://github.com/apache/spark/commit/c0

[GitHub] spark pull request #21123: [SPARK-24045][SQL]Create base class for file data...

2018-04-23 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21123#discussion_r183398030 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/FileDataSourceV2.scala --- @@ -27,15 +27,13 @@ import org.apache.spark.sq

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21123 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/2589/ Tes

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21123 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional comma

[GitHub] spark issue #21123: [SPARK-24045][SQL]Create base class for file data source...

2018-04-23 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21123 **[Test build #89724 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/89724/testReport)** for PR 21123 at commit [`968799c`](https://github.com/apache/spark/commit/96

[GitHub] spark pull request #21018: [SPARK-23880][SQL] Do not trigger any jobs for ca...

2018-04-23 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/21018#discussion_r183395070 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/columnar/InMemoryRelation.scala --- @@ -55,56 +42,39 @@ object InMemoryRelation { priv

[GitHub] spark issue #20980: [SPARK-23589][SQL] ExternalMapToCatalyst should support ...

2018-04-23 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20980 ok, Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark

[GitHub] spark pull request #20146: [SPARK-11215][ML] Add multiple columns support to...

2018-04-23 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/20146#discussion_r183393215 --- Diff: mllib/src/main/scala/org/apache/spark/ml/feature/StringIndexer.scala --- @@ -130,21 +161,53 @@ class StringIndexer @Since("1.4.0") ( @Since(

[GitHub] spark pull request #21072: [SPARK-23973][SQL] Remove consecutive Sorts

2018-04-23 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/21072#discussion_r183392394 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -736,12 +736,29 @@ object EliminateSorts extends Rul

[GitHub] spark issue #20923: [SPARK-23807][BUILD] Add Hadoop 3.1 profile with relevan...

2018-04-23 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/20923 @vanzin : The followup to this is #21066; I could move the compile time changes there but if you are going to have POMs playing with dependencies, seems best to have it all in one place...the

[GitHub] spark pull request #21125: [Spark-24024][ML] Fix poisson deviance calculatio...

2018-04-23 Thread tengpeng
Github user tengpeng commented on a diff in the pull request: https://github.com/apache/spark/pull/21125#discussion_r183386685 --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala --- @@ -495,8 +495,8 @@ class GeneralizedLinearRegres

[GitHub] spark pull request #21125: [Spark-24024][ML] Fix poisson deviance calculatio...

2018-04-23 Thread tengpeng
Github user tengpeng commented on a diff in the pull request: https://github.com/apache/spark/pull/21125#discussion_r183386571 --- Diff: mllib/src/main/scala/org/apache/spark/ml/regression/GeneralizedLinearRegression.scala --- @@ -782,8 +782,12 @@ object GeneralizedLinearRegressio

[GitHub] spark pull request #21125: [Spark-24024][ML] Fix poisson deviance calculatio...

2018-04-23 Thread tengpeng
Github user tengpeng commented on a diff in the pull request: https://github.com/apache/spark/pull/21125#discussion_r183386022 --- Diff: mllib/src/test/scala/org/apache/spark/ml/regression/GeneralizedLinearRegressionSuite.scala --- @@ -495,8 +495,8 @@ class GeneralizedLinearRegres

<    1   2   3   4   5   6   >