[GitHub] spark pull request #20096: [SPARK-22908] Add kafka source and sink for conti...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20096#discussion_r160005666 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaWriteTask.scala --- @@ -74,8 +56,49 @@ private[kafka010] class

[GitHub] spark pull request #20096: [SPARK-22908] Add kafka source and sink for conti...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20096#discussion_r159985929 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaOffsetReader.scala --- @@ -145,6 +149,19 @@ private[kafka010] class

[GitHub] spark pull request #20096: [SPARK-22908] Add kafka source and sink for conti...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20096#discussion_r160008897 --- Diff: external/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaContinuousSuite.scala --- @@ -0,0 +1,248 @@ +/* + * Licensed to

[GitHub] spark issue #20146: [SPARK-11215][ML] Add multiple columns support to String...

2018-01-05 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20146 Another workaround is, we can add some rows into iris dataset and make the three values in Species column not frequency equal anymore. For example, we add three more rows into iris. Now the

[GitHub] spark issue #20163: [SPARK-22966][PySpark] Spark SQL should handle Python UD...

2018-01-05 Thread rednaxelafx
Github user rednaxelafx commented on the issue: https://github.com/apache/spark/pull/20163 Thanks for all of your comments, @HyukjinKwon and @icexelloss ! I'd like to wait for more discussions / suggestions on whether or not we want a behavior change that makes this reproducer

[GitHub] spark issue #20146: [SPARK-11215][ML] Add multiple columns support to String...

2018-01-05 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/20146 Seems to me we can't set string indexer order for R glm. A workaround is to encode the Species manually first. Then let R glm and spark.glm to fit the encoded Species column, instead of the

[GitHub] spark issue #20135: [SPARK-22937][SQL] SQL elt output binary for binary inpu...

2018-01-05 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20135 @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/20166 @cloud-fan --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r160004385 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -110,107 +114,240 @@ private[spark] class AppStatusStore( if

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20142 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85735/ Test PASSed. ---

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20142 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20142 **[Test build #85735 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85735/testReport)** for PR 20142 at commit

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159988995 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -110,107 +114,240 @@ private[spark] class AppStatusStore( if

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159992789 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -110,107 +114,240 @@ private[spark] class AppStatusStore( if

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159990658 --- Diff: core/src/main/scala/org/apache/spark/status/AppStatusStore.scala --- @@ -110,107 +114,240 @@ private[spark] class AppStatusStore( if

[GitHub] spark issue #20170: [SPARK-22960][K8S] Revert use of ARG base_image in image...

2018-01-05 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20170 LGTM. I'll merge when tests finish even though this doesn't affect them. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20142 **[Test build #85735 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85735/testReport)** for PR 20142 at commit

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20142 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20142 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85732/ Test FAILed. ---

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20142 **[Test build #85732 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85732/testReport)** for PR 20142 at commit

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20142 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85731/ Test FAILed. ---

[GitHub] spark issue #20170: [SPARK-22960][K8S] Revert use of ARG base_image in image...

2018-01-05 Thread foxish
Github user foxish commented on the issue: https://github.com/apache/spark/pull/20170 LGTM, thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20142 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20142 **[Test build #85731 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85731/testReport)** for PR 20142 at commit

[GitHub] spark issue #20170: [SPARK-22960][K8S] Revert use of ARG base_image in image...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20170 **[Test build #85734 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85734/testReport)** for PR 20170 at commit

[GitHub] spark pull request #20170: [SPARK-22960][K8S] Revert use of ARG base_image i...

2018-01-05 Thread liyinan926
GitHub user liyinan926 opened a pull request: https://github.com/apache/spark/pull/20170 [SPARK-22960][K8S] Revert use of ARG base_image in images ## What changes were proposed in this pull request? This PR reverts the `ARG base_image` before `FROM` in the images of driver,

[GitHub] spark issue #20169: [SPARK-17088][hive] Fix 'sharesHadoopClasses' option whe...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20169 **[Test build #85733 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85733/testReport)** for PR 20169 at commit

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20142 **[Test build #85732 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85732/testReport)** for PR 20142 at commit

[GitHub] spark pull request #20169: [SPARK-17088][hive] Fix 'sharesHadoopClasses' opt...

2018-01-05 Thread vanzin
GitHub user vanzin opened a pull request: https://github.com/apache/spark/pull/20169 [SPARK-17088][hive] Fix 'sharesHadoopClasses' option when creating client. Because the call to the constructor of HiveClientImpl crosses class loader boundaries, different versions of the same

[GitHub] spark pull request #20142: [SPARK-22930][PYTHON][SQL] Improve the descriptio...

2018-01-05 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/20142#discussion_r159994714 --- Diff: python/pyspark/sql/tests.py --- @@ -3950,6 +3974,33 @@ def test_vectorized_udf_timestamps_respect_session_timezone(self):

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20142 **[Test build #85731 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85731/testReport)** for PR 20142 at commit

[GitHub] spark issue #20154: [SPARK-22960][k8s] Make build-push-docker-images.sh more...

2018-01-05 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20154 That kinda sucks. It means the base image cannot have a tag so working with multiple Spark versions will be a little weird. Anyway, feel free to open a PR to revert that part. ---

[GitHub] spark issue #11994: [SPARK-14151] Expose metrics Source and Sink interface

2018-01-05 Thread CodingCat
Github user CodingCat commented on the issue: https://github.com/apache/spark/pull/11994 @jerryshao I mean we also need to provide BaseReporter trait ```scala trait Sink { protected val reporter: BaseReporter = createReporter() def createReporter():

[GitHub] spark issue #20154: [SPARK-22960][k8s] Make build-push-docker-images.sh more...

2018-01-05 Thread liyinan926
Github user liyinan926 commented on the issue: https://github.com/apache/spark/pull/20154 @vanzin it seems using `ARG` in `FROM` is only supported since Docker version 17.06. It does not work for earlier versions of Docker. See

[GitHub] spark pull request #20142: [SPARK-22930][PYTHON][SQL] Improve the descriptio...

2018-01-05 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/20142#discussion_r159987954 --- Diff: python/pyspark/sql/tests.py --- @@ -3950,6 +3974,33 @@ def test_vectorized_udf_timestamps_respect_session_timezone(self):

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20142 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20142 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85729/ Test PASSed. ---

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20142 **[Test build #85729 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85729/testReport)** for PR 20142 at commit

[GitHub] spark issue #20097: [SPARK-22912] v2 data source support in MicroBatchExecut...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20097 **[Test build #85730 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85730/testReport)** for PR 20097 at commit

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159985442 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala --- @@ -240,31 +240,35 @@ class StreamingQueryManager

[GitHub] spark pull request #20096: [SPARK-22908] Add kafka source and sink for conti...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20096#discussion_r159984700 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaContinuousReader.scala --- @@ -0,0 +1,224 @@ +/* + * Licensed to

[GitHub] spark pull request #20142: [SPARK-22930][PYTHON][SQL] Improve the descriptio...

2018-01-05 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/20142#discussion_r159983977 --- Diff: python/pyspark/sql/tests.py --- @@ -3950,6 +3974,33 @@ def test_vectorized_udf_timestamps_respect_session_timezone(self):

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread jose-torres
Github user jose-torres commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159983634 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -392,6 +443,21 @@ class

[GitHub] spark pull request #20096: [SPARK-22908] Add kafka source and sink for conti...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20096#discussion_r159983463 --- Diff: external/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/CachedKafkaConsumer.scala --- @@ -261,6 +261,10 @@ private[kafka010] case

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20142 **[Test build #85729 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85729/testReport)** for PR 20142 at commit

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159978428 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -392,6 +443,21 @@ class MicroBatchExecution(

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159980348 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/DataStreamReader.scala --- @@ -166,19 +167,31 @@ final class DataStreamReader

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159981422 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/streaming/StreamingQueryManager.scala --- @@ -240,31 +240,35 @@ class StreamingQueryManager

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159980158 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -33,15 +40,21 @@ class MicroBatchExecution(

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159978319 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -392,6 +443,21 @@ class MicroBatchExecution(

[GitHub] spark issue #20142: [SPARK-22930][PYTHON][SQL] Improve the description of Ve...

2018-01-05 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/20142 I added the test. @gatorsmile do you have to take a look or let me know who should I ping for review? --- - To unsubscribe,

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159961186 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -357,31 +400,39 @@ class MicroBatchExecution(

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159960120 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -236,14 +264,27 @@ class MicroBatchExecution(

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159980863 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Relation.scala --- @@ -35,6 +35,16 @@ case class

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159981274 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/sources/RateStreamSourceV2.scala --- @@ -28,17 +28,38 @@ import

[GitHub] spark pull request #20097: [SPARK-22912] v2 data source support in MicroBatc...

2018-01-05 Thread tdas
Github user tdas commented on a diff in the pull request: https://github.com/apache/spark/pull/20097#discussion_r159958467 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/MicroBatchExecution.scala --- @@ -33,15 +40,21 @@ class MicroBatchExecution(

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20096 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85728/ Test PASSed. ---

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20096 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20096 **[Test build #85728 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85728/testReport)** for PR 20096 at commit

[GitHub] spark pull request #20132: [SPARK-13030][ML] Follow-up cleanups for OneHotEn...

2018-01-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20132 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20132: [SPARK-13030][ML] Follow-up cleanups for OneHotEncoderEs...

2018-01-05 Thread jkbradley
Github user jkbradley commented on the issue: https://github.com/apache/spark/pull/20132 Thanks! Merging with master and branch-2.3 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19848: [SPARK-22162] Executors and the driver should use consis...

2018-01-05 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/19848 Done. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159955747 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/test/SharedSQLContext.scala --- @@ -17,4 +17,22 @@ package org.apache.spark.sql.test

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159954825 --- Diff: core/src/test/scala/org/apache/spark/SparkFunSuite.scala --- @@ -27,19 +27,55 @@ import org.apache.spark.util.AccumulatorContext /**

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159954589 --- Diff: core/src/test/scala/org/apache/spark/SparkFunSuite.scala --- @@ -27,19 +27,55 @@ import org.apache.spark.util.AccumulatorContext /**

[GitHub] spark pull request #19893: [SPARK-16139][TEST] Add logging functionality for...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/19893#discussion_r159955132 --- Diff: core/src/test/scala/org/apache/spark/ThreadAudit.scala --- @@ -0,0 +1,126 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20166 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85727/ Test PASSed. ---

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20166 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20166 **[Test build #85727 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85727/testReport)** for PR 20166 at commit

[GitHub] spark issue #20168: SPARK-22730 Add ImageSchema support for non-integer imag...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20168 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20168: SPARK-22730 Add ImageSchema support for non-integ...

2018-01-05 Thread tomasatdatabricks
GitHub user tomasatdatabricks opened a pull request: https://github.com/apache/spark/pull/20168 SPARK-22730 Add ImageSchema support for non-integer image formats ## What changes were proposed in this pull request? Added functionality to handle all OpenCV modes to ImageSchema:

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20166 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85726/ Test FAILed. ---

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20166 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20166 **[Test build #85726 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85726/testReport)** for PR 20166 at commit

[GitHub] spark pull request #20013: [SPARK-20657][core] Speed up rendering of the sta...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20013#discussion_r159943385 --- Diff: core/src/main/scala/org/apache/spark/status/LiveEntity.scala --- @@ -119,118 +121,115 @@ private class LiveTask( import

[GitHub] spark issue #20096: [SPARK-22908] Add kafka source and sink for continuous p...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20096 **[Test build #85728 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85728/testReport)** for PR 20096 at commit

[GitHub] spark issue #20013: [SPARK-20657][core] Speed up rendering of the stages pag...

2018-01-05 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20013 > I wonder which changes double the disk usage? It's the new indices, more explicitly the values, not the keys. I tried changing the disk layout to write all the indices in a new namespace

[GitHub] spark pull request #20147: [SPARK-22940][SQL] HiveExternalCatalogVersionsSui...

2018-01-05 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20147 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20147: [SPARK-22940][SQL] HiveExternalCatalogVersionsSui...

2018-01-05 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20147#discussion_r159939270 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/HiveExternalCatalogVersionsSuite.scala --- @@ -85,6 +93,34 @@ class

[GitHub] spark issue #20155: [SPARK-22961][REGRESSION] Constant columns should genera...

2018-01-05 Thread gengliangwang
Github user gengliangwang commented on the issue: https://github.com/apache/spark/pull/20155 A late LGTM! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20154: [SPARK-22960][k8s] Make build-push-docker-images.sh more...

2018-01-05 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/20154 ARGs can have default values, so we could do that if we decide to use the Docker Hub infra. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20167: Allow providing Mesos principal & secret via files (SPAR...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20167 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20167: Allow providing Mesos principal & secret via file...

2018-01-05 Thread rvesse
GitHub user rvesse opened a pull request: https://github.com/apache/spark/pull/20167 Allow providing Mesos principal & secret via files (SPARK-16501) ## What changes were proposed in this pull request? This commit modifies the Mesos submission client to allow the principal

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85725/ Test FAILed. ---

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20165 **[Test build #85725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85725/testReport)** for PR 20165 at commit

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20166 **[Test build #85727 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85727/testReport)** for PR 20166 at commit

[GitHub] spark issue #20166: [SPARK-22973][SQL] Fix incorrect results of Casting Map ...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20166 **[Test build #85726 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85726/testReport)** for PR 20166 at commit

[GitHub] spark pull request #20166: [SPARK-22973][SQL] Fix incorrect results of Casti...

2018-01-05 Thread maropu
GitHub user maropu opened a pull request: https://github.com/apache/spark/pull/20166 [SPARK-22973][SQL] Fix incorrect results of Casting Map to String ## What changes were proposed in this pull request? This pr fixed the issue when casting maps into strings; ``` scala>

[GitHub] spark issue #20163: [SPARK-22966][PySpark] Spark SQL should handle Python UD...

2018-01-05 Thread icexelloss
Github user icexelloss commented on the issue: https://github.com/apache/spark/pull/20163 I ran some experiments: ``` py_date = udf(datetime.date, DateType()) py_timestamp = udf(datetime.datetime, TimestampType()) ``` This works correctly ```

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85723/ Test FAILed. ---

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20165 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20165: [SPARK-22972] Couldn't find corresponding Hive SerDe for...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20165 **[Test build #85723 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85723/testReport)** for PR 20165 at commit

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159900404 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159900297 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends

[GitHub] spark pull request #20165: [SPARK-22972] Couldn't find corresponding Hive Se...

2018-01-05 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20165#discussion_r159900236 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/orc/HiveOrcSourceSuite.scala --- @@ -62,6 +63,22 @@ class HiveOrcSourceSuite extends

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2018-01-05 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19893 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/85722/ Test PASSed. ---

[GitHub] spark issue #19893: [SPARK-16139][TEST] Add logging functionality for leaked...

2018-01-05 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19893 **[Test build #85722 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/85722/testReport)** for PR 19893 at commit

[GitHub] spark pull request #19247: [Spark-21996][SQL] read files with space in name ...

2018-01-05 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/19247#discussion_r159893158 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/FileStreamSource.scala --- @@ -233,7 +233,7 @@ class FileStreamSource(

<    1   2   3   >