[GitHub] spark pull request #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviou...

2018-02-04 Thread ashashwat
GitHub user ashashwat opened a pull request: https://github.com/apache/spark/pull/20503 [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for Rows. ## What changes were proposed in this pull request? Fix \_\_repr\_\_ behaviour for Rows. Rows \_\_repr\_\_ assumes

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #87048 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87048/testReport)** for PR 18555 at commit

[GitHub] spark issue #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperat...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20498 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18555 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20501: [SPARK-22430][Docs] Unknown tag warnings when building R...

2018-02-04 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/20501 I see this too and Roxygen 6.0.1 seems common now. Is there any value to the tags? I don't think these removals can cause much merge conflict. It can go in master right? ---

[GitHub] spark issue #20373: [SPARK-23159][PYTHON] Update cloudpickle to v0.4.2 plus ...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20373 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20473 **[Test build #87046 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87046/testReport)** for PR 20473 at commit

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20473 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for R...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20503 BTW, does non-string field names work in this namedtuple way? --- - To unsubscribe, e-mail:

[GitHub] spark issue #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for R...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20503 Check if it's `unicode` and convert, etc. might also work .. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20499: [SPARK-23328][PYTHON] Disallow default value None in na....

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20499 **[Test build #87051 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87051/testReport)** for PR 20499 at commit

[GitHub] spark pull request #20456: [SPARK-22624][PYSPARK] Expose range partitioning ...

2018-02-04 Thread xubo245
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/spark/pull/20456#discussion_r165838751 --- Diff: python/pyspark/sql/dataframe.py --- @@ -667,6 +667,51 @@ def repartition(self, numPartitions, *cols): else: raise

[GitHub] spark pull request #18555: [SPARK-21353][CORE]add checkValue in spark.intern...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18555#discussion_r165838720 --- Diff: core/src/main/scala/org/apache/spark/internal/config/package.scala --- @@ -231,6 +315,9 @@ package object config { private[spark] val

[GitHub] spark issue #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for R...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20503 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperat...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20498 **[Test build #87050 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87050/testReport)** for PR 20498 at commit

[GitHub] spark issue #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperat...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20498 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/575/

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18555 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87048/ Test PASSed. ---

[GitHub] spark issue #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for R...

2018-02-04 Thread ashashwat
Github user ashashwat commented on the issue: https://github.com/apache/spark/pull/20503 @HyukjinKwon Here is what I tried: ``` # Code: return "" % ", ".join(fields.encode("utf8") for fields in self) >>> Row (u"아", "11") # Fails for

[GitHub] spark pull request #20502: [SPARK-23330][WebUI] Spark UI SQL executions page...

2018-02-04 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/20502#discussion_r165847251 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala --- @@ -179,7 +179,7 @@ private[ui] abstract class

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18555 retest this please --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18555 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20456: [SPARK-22624][PYSPARK] Expose range partitioning shuffle...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20456 **[Test build #87049 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87049/testReport)** for PR 20456 at commit

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18555 Seems https://github.com/apache/spark/pull/18555#discussion_r126293557 is missed. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for R...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20503 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeti...

2018-02-04 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/20498#discussion_r165844386 --- Diff: sql/core/src/test/resources/sql-tests/inputs/typeCoercion/native/decimalArithmeticOperations.sql --- @@ -48,8 +48,9 @@ select

[GitHub] spark pull request #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeti...

2018-02-04 Thread wangyum
Github user wangyum commented on a diff in the pull request: https://github.com/apache/spark/pull/20498#discussion_r165844408 --- Diff: sql/core/src/test/resources/sql-tests/inputs/typeCoercion/native/decimalArithmeticOperations.sql --- @@ -74,7 +75,8 @@ select

[GitHub] spark issue #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for R...

2018-02-04 Thread ashashwat
Github user ashashwat commented on the issue: https://github.com/apache/spark/pull/20503 @HyukjinKwon Do you mean something like `Row (a=1, b=2, c=3)` or `Row (1="Alice", 2=11)`? Former works fine, latter fails with `SyntaxError: keyword can't be an expression`. ---

[GitHub] spark issue #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for R...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20503 I think it still makes sense to produce a repr anyway because we successfully can create the instance for now but .. let me take a closer look within few days for sure. ---

[GitHub] spark issue #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperat...

2018-02-04 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/20498 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for R...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20503 `unicode(fields).encode("utf8")`: in this case, we will try to decode it by system default encoding first and then encode it by udf-8 if the input is `str` (bytes). So, for example, I think

[GitHub] spark issue #20499: [SPARK-23328][PYTHON] Disallow default value None in na....

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20499 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20499: [SPARK-23328][PYTHON] Disallow default value None in na....

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20499 **[Test build #87051 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87051/testReport)** for PR 20499 at commit

[GitHub] spark issue #20502: [SPARK-23330][WebUI] Spark UI SQL executions page throws...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20502 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20502: [SPARK-23330][WebUI] Spark UI SQL executions page throws...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20502 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87043/ Test PASSed. ---

[GitHub] spark issue #20502: [SPARK-23330][WebUI] Spark UI SQL executions page throws...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20502 **[Test build #87043 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87043/testReport)** for PR 20502 at commit

[GitHub] spark issue #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for R...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20503 I meant things like this: ```python >>> from pyspark.sql import Row >>> RowClass = Row(1) >>> RowClass("a") Row(1='a') ``` ```python >>>

[GitHub] spark issue #20503: [SPARK-23299][SQL][PYSPARK] Fix __repr__ behaviour for R...

2018-02-04 Thread ashashwat
Github user ashashwat commented on the issue: https://github.com/apache/spark/pull/20503 @HyukjinKwon `return "" % ", ".join("%s" % (fields) for fields in self)` takes care of everything. ``` >>> Row ("aa", 11) >>> Row (u"아", 11)

[GitHub] spark issue #20499: [SPARK-23328][PYTHON] Disallow default value None in na....

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20499 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/576/

[GitHub] spark issue #20499: [SPARK-23328][PYTHON] Disallow default value None in na....

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20499 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20499: [SPARK-23328][PYTHON] Disallow default value None in na....

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20499 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87051/ Test PASSed. ---

[GitHub] spark issue #20501: [SPARK-22430][Docs] Unknown tag warnings when building R...

2018-02-04 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/20501 It was not used in the current way we have the export and a change in 6.x started warning about this. Jenkins is not running Roxygen 6 though. There are pending tasks to upgrade. IMO

[GitHub] spark pull request #20495: [SPARK-23327] [SQL] Update the description and te...

2018-02-04 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20495#discussion_r165855003 --- Diff: python/pyspark/sql/functions.py --- @@ -1705,10 +1705,12 @@ def unhex(col): @ignore_unicode_prefix @since(1.5) def length(col):

[GitHub] spark issue #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperat...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20498 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87050/ Test PASSed. ---

[GitHub] spark issue #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperat...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20498 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20495: [SPARK-23327] [SQL] Update the description and te...

2018-02-04 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/20495#discussion_r165855729 --- Diff: python/pyspark/sql/functions.py --- @@ -1705,10 +1705,12 @@ def unhex(col): @ignore_unicode_prefix @since(1.5) def length(col):

[GitHub] spark issue #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperat...

2018-02-04 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/20498 LGTM Thanks! Merged to master/2.3! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tes...

2018-02-04 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20487#discussion_r165855092 --- Diff: python/pyspark/sql/utils.py --- @@ -115,18 +115,30 @@ def toJArray(gateway, jtype, arr): def require_minimum_pandas_version():

[GitHub] spark pull request #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tes...

2018-02-04 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20487#discussion_r165855123 --- Diff: python/pyspark/sql/dataframe.py --- @@ -1923,6 +1923,9 @@ def toPandas(self): 02 Alice 15Bob

[GitHub] spark issue #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperat...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20498 **[Test build #87050 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87050/testReport)** for PR 20498 at commit

[GitHub] spark pull request #20498: [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeti...

2018-02-04 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20498 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20495: [SPARK-23327] [SQL] Update the description and te...

2018-02-04 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/20495#discussion_r165854982 --- Diff: python/pyspark/sql/functions.py --- @@ -1705,10 +1705,12 @@ def unhex(col): @ignore_unicode_prefix @since(1.5) def length(col):

[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support t...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20504 **[Test build #87052 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87052/testReport)** for PR 20504 at commit

[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support t...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20504 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20505: [SPARK-23251][SQL] Add checks for collection elem...

2018-02-04 Thread michalsenkyr
GitHub user michalsenkyr opened a pull request: https://github.com/apache/spark/pull/20505 [SPARK-23251][SQL] Add checks for collection element Encoders Implicit methods of `SQLImplicits` providing Encoders for collections did not check for Encoders for their elements. This

[GitHub] spark pull request #19054: [SPARK-18067] Avoid shuffling child if join keys ...

2018-02-04 Thread eyalfa
Github user eyalfa commented on a diff in the pull request: https://github.com/apache/spark/pull/19054#discussion_r165860433 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala --- @@ -220,45 +220,99 @@ case class

[GitHub] spark pull request #19054: [SPARK-18067] Avoid shuffling child if join keys ...

2018-02-04 Thread eyalfa
Github user eyalfa commented on a diff in the pull request: https://github.com/apache/spark/pull/19054#discussion_r165861581 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/exchange/EnsureRequirements.scala --- @@ -220,45 +220,99 @@ case class

[GitHub] spark issue #20505: [SPARK-23251][SQL] Add checks for collection element Enc...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20505 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to su...

2018-02-04 Thread wangyum
GitHub user wangyum opened a pull request: https://github.com/apache/spark/pull/20504 [SPARK-23332][SQL] Update SQLQueryTestSuite to support test hive mode ## What changes were proposed in this pull request? Update `SQLQueryTestSuite` to support test hive mode. ##

[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support t...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20504 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/577/

[GitHub] spark issue #20505: [SPARK-23251][SQL] Add checks for collection element Enc...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20505 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tes...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20487#discussion_r165865284 --- Diff: python/pyspark/sql/dataframe.py --- @@ -1923,6 +1923,9 @@ def toPandas(self): 02 Alice 15Bob

[GitHub] spark pull request #20502: [SPARK-23330][WebUI] Spark UI SQL executions page...

2018-02-04 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/20502#discussion_r165872846 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala --- @@ -179,7 +179,7 @@ private[ui] abstract class

[GitHub] spark issue #20164: [SPARK-22971][ML] OneVsRestModel should use temporary Ra...

2018-02-04 Thread zhengruifeng
Github user zhengruifeng commented on the issue: https://github.com/apache/spark/pull/20164 @WeichenXu123 Yes, my concern is that it is confusing if the transform failure is caused by column conflict by a ‘invisible’ column. @srowen Agree that it is not perfect if we

[GitHub] spark issue #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tests for ...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20487 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tes...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20487#discussion_r165873632 --- Diff: pom.xml --- @@ -185,6 +185,10 @@ 2.8 1.8 1.0.0 +

[GitHub] spark issue #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tests for ...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20487 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/581/

[GitHub] spark issue #20164: [SPARK-22971][ML] OneVsRestModel should use temporary Ra...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20164 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/582/

[GitHub] spark issue #20164: [SPARK-22971][ML] OneVsRestModel should use temporary Ra...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20164 **[Test build #87058 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87058/testReport)** for PR 20164 at commit

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20473 Will merge this one if there's no more comments in few days. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tes...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20487#discussion_r165873671 --- Diff: python/setup.py --- @@ -100,6 +100,11 @@ def _supports_symlinks(): file=sys.stderr) exit(-1) +# If

[GitHub] spark pull request #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tes...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20487#discussion_r165873582 --- Diff: python/pyspark/sql/tests.py --- @@ -2794,7 +2792,6 @@ def count_bucketed_cols(names, table="pyspark_bucket"): def

[GitHub] spark issue #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tests for ...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20487 **[Test build #87057 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87057/testReport)** for PR 20487 at commit

[GitHub] spark issue #20164: [SPARK-22971][ML] OneVsRestModel should use temporary Ra...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20164 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20502: [SPARK-23330][WebUI] Spark UI SQL executions page...

2018-02-04 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/20502#discussion_r165876103 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/ui/AllExecutionsPage.scala --- @@ -179,7 +179,7 @@ private[ui] abstract class

[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support a...

2018-02-04 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/20504 After SPARK-21646, `hive/binaryComparison.sql.out`, `hive/decimalPrecision.sql.out` and `hive/promoteStrings.sql.out` seems like this:

[GitHub] spark issue #20492: [SPARK-23310][CORE] Turn off read ahead input stream for...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20492 **[Test build #87059 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87059/testReport)** for PR 20492 at commit

[GitHub] spark pull request #16099: [SPARK-18665][SQL] set statement state to "ERROR"...

2018-02-04 Thread BruceXu1991
Github user BruceXu1991 commented on a diff in the pull request: https://github.com/apache/spark/pull/16099#discussion_r165876866 --- Diff: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkExecuteStatementOperation.scala --- @@ -241,6 +241,8 @@

[GitHub] spark pull request #20487: [SPARK-23319][TESTS] Explicitly skips PySpark tes...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20487#discussion_r165865310 --- Diff: python/pyspark/sql/utils.py --- @@ -115,18 +115,30 @@ def toJArray(gateway, jtype, arr): def require_minimum_pandas_version():

[GitHub] spark issue #18555: [SPARK-21353][CORE]add checkValue in spark.internal.conf...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18555 **[Test build #87053 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87053/testReport)** for PR 18555 at commit

[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support a...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20504 **[Test build #87052 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87052/testReport)** for PR 20504 at commit

[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support a...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20504 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support a...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20504 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87052/ Test FAILed. ---

[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support a...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20504 **[Test build #87054 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87054/testReport)** for PR 20504 at commit

[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support a...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20504 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/578/

[GitHub] spark issue #20504: [SPARK-23332][SQL] Update SQLQueryTestSuite to support a...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20504 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20473 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20473 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/579/

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20473 **[Test build #87055 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87055/testReport)** for PR 20473 at commit

[GitHub] spark pull request #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyA...

2018-02-04 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20473#discussion_r165869377 --- Diff: python/run-tests.py --- @@ -151,6 +152,67 @@ def parse_opts(): return opts +def _check_dependencies(python_exec,

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20473 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20473 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/580/

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20473 **[Test build #87056 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87056/testReport)** for PR 20473 at commit

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20473 **[Test build #87055 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87055/testReport)** for PR 20473 at commit

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20473 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20473 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87055/ Test FAILed. ---

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20473 **[Test build #87056 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/87056/testReport)** for PR 20473 at commit

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20473 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20473: [SPARK-23300][TESTS] Prints out if Pandas and PyArrow ar...

2018-02-04 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20473 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/87056/ Test PASSed. ---

[GitHub] spark pull request #20472: [SPARK-22751][ML]Improve ML RandomForest shuffle ...

2018-02-04 Thread lucio-yz
Github user lucio-yz commented on a diff in the pull request: https://github.com/apache/spark/pull/20472#discussion_r165872418 --- Diff: mllib/src/main/scala/org/apache/spark/ml/tree/impl/RandomForest.scala --- @@ -1001,11 +1002,22 @@ private[spark] object RandomForest extends

[GitHub] spark pull request #20492: [SPARK-23310][CORE] Turn off read ahead input str...

2018-02-04 Thread sitalkedia
Github user sitalkedia commented on a diff in the pull request: https://github.com/apache/spark/pull/20492#discussion_r165874317 --- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeSorterSpillReader.java --- @@ -77,7 +77,7 @@ public

  1   2   >