[GitHub] spark pull request #20795: [SPARK-23486]cache the function name from the cat...

2018-03-16 Thread kevinyu98
Github user kevinyu98 commented on a diff in the pull request: https://github.com/apache/spark/pull/20795#discussion_r175248943 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala --- @@ -1192,11 +1195,23 @@ class Analyzer( * @see

[GitHub] spark issue #20841: [SPARK-23706][PYTHON] spark.conf.get(value, default=None...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20841 **[Test build #88336 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88336/testReport)** for PR 20841 at commit

[GitHub] spark issue #20841: [SPARK-23706][PYTHON] spark.conf.get(value, default=None...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20841 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88336/ Test FAILed. ---

[GitHub] spark issue #20841: [SPARK-23706][PYTHON] spark.conf.get(value, default=None...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20841 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20841: [SPARK-23706][PYTHON] spark.conf.get(value, default=None...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20841 **[Test build #88336 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88336/testReport)** for PR 20841 at commit

[GitHub] spark issue #20841: [SPARK-23706][PYTHON] spark.conf.get(value, default=None...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20841 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20841: [SPARK-23706][PYTHON] spark.conf.get(value, default=None...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20841 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1578/

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19222 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19222 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88334/ Test PASSed. ---

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19222 **[Test build #88334 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88334/testReport)** for PR 19222 at commit

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20838 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88332/ Test PASSed. ---

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20838 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20838 **[Test build #88332 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88332/testReport)** for PR 20838 at commit

[GitHub] spark pull request #20841: [SPARK-23706][PYTHON] spark.conf.get(value, defau...

2018-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20841#discussion_r175248117 --- Diff: python/pyspark/sql/tests.py --- @@ -2504,6 +2504,10 @@ def test_conf(self): spark.conf.unset("bogo")

[GitHub] spark pull request #20841: [SPARK-23706][PYTHON] spark.conf.get(value, defau...

2018-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20841#discussion_r175248095 --- Diff: python/pyspark/sql/tests.py --- @@ -2504,6 +2504,10 @@ def test_conf(self): spark.conf.unset("bogo")

[GitHub] spark issue #20841: [SPARK-23706][PYTHON] spark.conf.get(value, default=None...

2018-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20841 Ah, I got the point. So, you mean a dictionary-like approach - `get("non-existant-key")` should return `None` always instead of throwing an error, language-specifically? I think we might

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20847 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20847 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88335/ Test FAILed. ---

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20847 **[Test build #88335 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88335/testReport)** for PR 20847 at commit

[GitHub] spark issue #20785: [SPARK-23640][CORE] Fix hadoop config may override spark...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20785 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88331/ Test PASSed. ---

[GitHub] spark issue #20785: [SPARK-23640][CORE] Fix hadoop config may override spark...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20785 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20785: [SPARK-23640][CORE] Fix hadoop config may override spark...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20785 **[Test build #88331 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88331/testReport)** for PR 20785 at commit

[GitHub] spark issue #20208: [SPARK-23007][SQL][TEST] Add schema evolution test suite...

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20208 After merging this PR, I'm going to make a PR on SQL part for Apache Spark 2.4. --- - To unsubscribe, e-mail:

[GitHub] spark issue #20208: [SPARK-23007][SQL][TEST] Add schema evolution test suite...

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20208 @gatorsmile . Yes. It's officially supported Hive like the following compatible cases which means upgrading wider types. As you see in my PR, Apache Spark 2.3 also supports, but the degree of

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20847 **[Test build #88335 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88335/testReport)** for PR 20847 at commit

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20847 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20847 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1577/

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20847 Retest this please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20841: [SPARK-23706][PYTHON] spark.conf.get(value, default=None...

2018-03-16 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/20841 Yeah, true that is how the Scala side works so I suppose that is best, but I kind of view this api similar to the python `dict.get` which returns None and won't raise an error, and that would

[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20659 I added more specific comments about ORC parts, @wangyum . --- - To unsubscribe, e-mail:

[GitHub] spark pull request #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/20659#discussion_r175243600 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilterSuite.scala --- @@ -22,7 +22,7 @@ import java.sql.{Date,

[GitHub] spark pull request #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/20659#discussion_r175243593 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcSerializer.scala --- @@ -17,11 +17,11 @@ package

[GitHub] spark pull request #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/20659#discussion_r175243579 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcDeserializer.scala --- @@ -17,9 +17,9 @@ package

[GitHub] spark pull request #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/20659#discussion_r175243585 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFilters.scala --- @@ -17,9 +17,9 @@ package

[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20659 @wangyum . Please read again my previous [comment](https://github.com/apache/spark/pull/20659/files#r172587976). Although I support your efforts, Spark can provide ORC data

[GitHub] spark issue #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hint in p...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20788 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hint in p...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20788 **[Test build #88333 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88333/testReport)** for PR 20788 at commit

[GitHub] spark issue #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hint in p...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20788 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88333/ Test PASSed. ---

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20726 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19222 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20726 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88329/ Test PASSed. ---

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19222 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1576/

[GitHub] spark issue #19222: [SPARK-10399][CORE][SQL] Introduce multiple MemoryBlocks...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19222 **[Test build #88334 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88334/testReport)** for PR 19222 at commit

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20726 **[Test build #88329 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88329/testReport)** for PR 20726 at commit

[GitHub] spark issue #20846: [SPARK-5498][SQL][FOLLOW] add schema to table partition

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20846 @liutang123 , Spark should not do this kind of risky thing. Hive 2.3.2 also disallows incompatible schema changes like the following. ```sql hive> CREATE TABLE test_par(a string)

[GitHub] spark issue #20788: [SPARK-23647][PYTHON][SQL] Adds more types for hint in p...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20788 **[Test build #88333 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88333/testReport)** for PR 20788 at commit

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20838 **[Test build #88332 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88332/testReport)** for PR 20838 at commit

[GitHub] spark issue #20846: [SPARK-5498][SQL][FOLLOW] add schema to table partition

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/20846 @liutang123 . Did you test this with the latest Apache Spark 2.3? Apache Spark 2.3 works without any problem with your example. ```scala scala> sql("create table test_par(a string)

[GitHub] spark issue #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20838 ok to test --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #20838: [SPARK-23698] Resolve undefined names in Python 3

2018-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/20838#discussion_r175240674 --- Diff: dev/create-release/releaseutils.py --- @@ -149,7 +152,11 @@ def get_commits(tag): if not is_valid_author(author):

[GitHub] spark issue #20785: [SPARK-23640][CORE] Fix hadoop config may override spark...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20785 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1575/

[GitHub] spark pull request #20846: [SPARK-5498][SQL][FOLLOW] add schema to table par...

2018-03-16 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/20846#discussion_r175240453 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/catalog/interface.scala --- @@ -99,7 +99,8 @@ case class CatalogTablePartition(

[GitHub] spark issue #20785: [SPARK-23640][CORE] Fix hadoop config may override spark...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20785 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20785: [SPARK-23640][CORE] Fix hadoop config may override spark...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20785 **[Test build #88331 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88331/testReport)** for PR 20785 at commit

[GitHub] spark issue #20841: [SPARK-23706][PYTHON] spark.conf.get(value, default=None...

2018-03-16 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/20841 I think we should better match Python side behaviour to Scala side in general and throw a better exception if possible. I think the application conf was done like that because there wasn't

[GitHub] spark pull request #20669: [SPARK-22839][K8S] Remove the use of init-contain...

2018-03-16 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20669#discussion_r175238936 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -53,14 +53,10 @@ fi case "$SPARK_K8S_CMD" in

[GitHub] spark pull request #20669: [SPARK-22839][K8S] Remove the use of init-contain...

2018-03-16 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/20669#discussion_r175238547 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -53,14 +53,10 @@ fi case "$SPARK_K8S_CMD" in

[GitHub] spark pull request #20669: [SPARK-22839][K8S] Remove the use of init-contain...

2018-03-16 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20669#discussion_r175237767 --- Diff: resource-managers/kubernetes/docker/src/main/dockerfiles/spark/entrypoint.sh --- @@ -53,14 +53,10 @@ fi case "$SPARK_K8S_CMD" in

[GitHub] spark pull request #20669: [SPARK-22839][K8S] Remove the use of init-contain...

2018-03-16 Thread mccheah
Github user mccheah commented on a diff in the pull request: https://github.com/apache/spark/pull/20669#discussion_r175237354 --- Diff: resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala --- @@ -79,6 +79,12 @@ private[spark] object Config

[GitHub] spark issue #20669: [SPARK-22839][K8S] Remove the use of init-container for ...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20669 Kubernetes integration test status failure URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1558/ ---

[GitHub] spark issue #20659: [DO-NOT-MERGE] Try to update Hive to 2.3.2

2018-03-16 Thread wangyum
Github user wangyum commented on the issue: https://github.com/apache/spark/pull/20659 Thanks everyone, we can move to [SPARK-23710](https://issues.apache.org/jira/browse/SPARK-23710) to discuss. --- - To

[GitHub] spark issue #20848: [SPARK-23623][SS] Avoid concurrent use of cached consume...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20848 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20848: [SPARK-23623][SS] Avoid concurrent use of cached consume...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20848 **[Test build #88330 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88330/testReport)** for PR 20848 at commit

[GitHub] spark issue #20848: [SPARK-23623][SS] Avoid concurrent use of cached consume...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20848 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88330/ Test PASSed. ---

[GitHub] spark issue #18982: [SPARK-21685][PYTHON][ML] PySpark Params isSet state sho...

2018-03-16 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/18982 So between https://github.com/apache/spark/pull/20410 & this one which path do we want to go down? --- - To unsubscribe,

[GitHub] spark issue #20410: [SPARK-23234][ML][PYSPARK] Remove setting defaults on Ja...

2018-03-16 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/20410 @BryanCutler I think it would be an OK assumption to make that the default values should be the same between Python and Scala provided we maybe added it to the params docs? What does @HyukjinKwon /

[GitHub] spark issue #20669: [SPARK-22839][K8S] Remove the use of init-container for ...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20669 Kubernetes integration test starting URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-spark-integration/1558/ ---

[GitHub] spark issue #20669: [SPARK-22839][K8S] Remove the use of init-container for ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20669 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1574/

[GitHub] spark issue #20669: [SPARK-22839][K8S] Remove the use of init-container for ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20669 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20701: [SPARK-23528][ML] Add numIter to ClusteringSummar...

2018-03-16 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/20701#discussion_r175236067 --- Diff: project/MimaExcludes.scala --- @@ -36,6 +36,11 @@ object MimaExcludes { // Exclude rules for 2.4.x lazy val v24excludes =

[GitHub] spark pull request #20701: [SPARK-23528][ML] Add numIter to ClusteringSummar...

2018-03-16 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/20701#discussion_r175235256 --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/ClusteringSummary.scala --- @@ -34,7 +34,8 @@ class ClusteringSummary private[clustering] (

[GitHub] spark pull request #20701: [SPARK-23528][ML] Add numIter to ClusteringSummar...

2018-03-16 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/20701#discussion_r175236013 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala --- @@ -36,8 +36,9 @@ import org.apache.spark.sql.{Row, SparkSession}

[GitHub] spark pull request #20701: [SPARK-23528][ML] Add numIter to ClusteringSummar...

2018-03-16 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/20701#discussion_r175234959 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala --- @@ -46,6 +47,10 @@ class KMeansModel @Since("2.4.0")

[GitHub] spark pull request #20701: [SPARK-23528][ML] Add numIter to ClusteringSummar...

2018-03-16 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/20701#discussion_r175235205 --- Diff: mllib/src/main/scala/org/apache/spark/ml/clustering/BisectingKMeans.scala --- @@ -312,4 +312,5 @@ class BisectingKMeansSummary

[GitHub] spark pull request #20701: [SPARK-23528][ML] Add numIter to ClusteringSummar...

2018-03-16 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/20701#discussion_r175235769 --- Diff: mllib/src/main/scala/org/apache/spark/mllib/clustering/KMeansModel.scala --- @@ -46,6 +47,10 @@ class KMeansModel @Since("2.4.0")

[GitHub] spark issue #20767: [SPARK-23623] [SS] Avoid concurrent use of cached consum...

2018-03-16 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/20767 Just to be clear, I am not saying that we *have to* move to this pool stuff. I am just saying that if we want to make this more robust, then we should try to use existing tools (after careful

[GitHub] spark issue #20767: [SPARK-23623] [SS] Avoid concurrent use of cached consum...

2018-03-16 Thread tedyu
Github user tedyu commented on the issue: https://github.com/apache/spark/pull/20767 Interesting. https://commons.apache.org/proper/commons-pool/apidocs/org/apache/commons/pool2/impl/BaseGenericObjectPool.html#getBorrowedCount() ---

[GitHub] spark issue #20767: [SPARK-23623] [SS] Avoid concurrent use of cached consum...

2018-03-16 Thread tdas
Github user tdas commented on the issue: https://github.com/apache/spark/pull/20767 @tedyu It was indeed hard to find :) But apache commons pool does expose metrics on idle/active counts. See

[GitHub] spark issue #20579: [SPARK-23372][SQL] Writing empty struct in parquet fails...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20579 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88327/ Test FAILed. ---

[GitHub] spark issue #20579: [SPARK-23372][SQL] Writing empty struct in parquet fails...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20579 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20579: [SPARK-23372][SQL] Writing empty struct in parquet fails...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20579 **[Test build #88327 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88327/testReport)** for PR 20579 at commit

[GitHub] spark issue #20669: [SPARK-22839][K8S] Remove the use of init-container for ...

2018-03-16 Thread ifilonenko
Github user ifilonenko commented on the issue: https://github.com/apache/spark/pull/20669 Results from integration testing: ``` Discovery starting. Discovery completed in 123 milliseconds. Run starting. Expected test count is: 8 KubernetesSuite: - Run SparkPi

[GitHub] spark issue #20848: [SPARK-23623][SS] Avoid concurrent use of cached consume...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20848 **[Test build #88330 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88330/testReport)** for PR 20848 at commit

[GitHub] spark issue #20848: [SPARK-23623][SS] Avoid concurrent use of cached consume...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20848 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution/1573/

[GitHub] spark issue #20848: [SPARK-23623][SS] Avoid concurrent use of cached consume...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20848 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #20848: [SPARK-23623][SS] Avoid concurrent use of cached ...

2018-03-16 Thread tdas
GitHub user tdas opened a pull request: https://github.com/apache/spark/pull/20848 [SPARK-23623][SS] Avoid concurrent use of cached consumers in CachedKafkaConsumer (branch-2.3) This is a backport of #20767 to branch 2.3 ## What changes were proposed in this pull request?

[GitHub] spark issue #20579: [SPARK-23372][SQL] Writing empty struct in parquet fails...

2018-03-16 Thread dilipbiswal
Github user dilipbiswal commented on the issue: https://github.com/apache/spark/pull/20579 @cloud-fan OK.. i was thinking of adding this check in each built in datasource like Text, CSV, Parquet, ORC, JSON etc. Just like we check it in Parquet with this PR. Would you have any

[GitHub] spark issue #16770: [SPARK-15009][PYTHON][ML] Construct a CountVectorizerMod...

2018-03-16 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/16770 Thanks @holdenk! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #16770: [SPARK-15009][PYTHON][ML] Construct a CountVector...

2018-03-16 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/16770#discussion_r175232475 --- Diff: python/pyspark/ml/tests.py --- @@ -1980,8 +1997,8 @@ def test_java_params(self): pyspark.ml.regression]

[GitHub] spark issue #20841: [SPARK-23706][PYTHON] spark.conf.get(value, default=None...

2018-03-16 Thread BryanCutler
Github user BryanCutler commented on the issue: https://github.com/apache/spark/pull/20841 Shouldn't it be the same as in the application conf here https://github.com/apache/spark/blob/master/python/pyspark/conf.py#L174? Here the default is `None` and it returns `None` if

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20847 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88322/ Test FAILed. ---

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20847 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20847: [SPARK-23644][CORE][UI][BACKPORT-2.3] Use absolute path ...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20847 **[Test build #88322 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88322/testReport)** for PR 20847 at commit

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20726 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88325/ Test PASSed. ---

[GitHub] spark pull request #20824: [SPARK-23683][SQL] FileCommitProtocol.instantiate...

2018-03-16 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/20824 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20726 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #20726: [SPARK-23574][SQL] Report SinglePartition in DataSourceV...

2018-03-16 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/20726 **[Test build #88325 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/88325/testReport)** for PR 20726 at commit

[GitHub] spark issue #20824: [SPARK-23683][SQL] FileCommitProtocol.instantiate() hard...

2018-03-16 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/20824 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #20770: [SPARK-23626][CORE] DAGScheduler blocked due to JobSubmi...

2018-03-16 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/20770 took a quick look, agree with shivaram's observations, you've got to handle `shuffleIdToMapStage` which will not be so easy. ---

[GitHub] spark issue #20742: [SPARK-23572][docs] Bring "security.md" up to date.

2018-03-16 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/20742 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/88324/ Test FAILed. ---

  1   2   3   4   5   >