[GitHub] spark pull request #18664: [SPARK-21375][PYSPARK][SQL][WIP] Add Date and Tim...

2017-08-15 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/18664#discussion_r133261665 --- Diff: python/pyspark/sql/tests.py --- @@ -3036,6 +3052,9 @@ def test_toPandas_arrow_toggle(self): pdf = df.toPandas() self.

[GitHub] spark issue #18918: [SPARK-21707][SQL]Improvement a special case for non-det...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18918 **[Test build #80693 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80693/testReport)** for PR 18918 at commit [`97a3270`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #18950: [SPARK-20589][Core][Scheduler] Allow limiting task concu...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18950 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18942: [BACKPORT-2.1][SPARK-19372][SQL] Fix throwing a Java exc...

2017-08-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18942 @poplav it looks good @gatorsmile Do you think it is ok for backport now? The previous commit included unnecessary changes. --- If your project is set up for it, you can reply to this email and h

[GitHub] spark issue #18950: [SPARK-20589][Core][Scheduler] Allow limiting task concu...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18950 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80690/ Test PASSed. ---

[GitHub] spark pull request #18933: [WIP][SPARK-21722][SQL][PYTHON] Enable timezone-a...

2017-08-15 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/18933#discussion_r133254340 --- Diff: python/pyspark/sql/tests.py --- @@ -2507,6 +2507,37 @@ def test_to_pandas(self): self.assertEquals(types[2], np.bool)

[GitHub] spark issue #18950: [SPARK-20589][Core][Scheduler] Allow limiting task concu...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18950 **[Test build #80690 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80690/testReport)** for PR 18950 at commit [`d3f8162`](https://github.com/apache/spark/commit/d

[GitHub] spark pull request #18933: [WIP][SPARK-21722][SQL][PYTHON] Enable timezone-a...

2017-08-15 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/18933#discussion_r133255672 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -912,6 +912,14 @@ object SQLConf { .intConf

[GitHub] spark issue #18940: [SPARK-21501] Change CacheLoader to limit entries based ...

2017-08-15 Thread redsanket
Github user redsanket commented on the issue: https://github.com/apache/spark/pull/18940 @kiszk wouldn't the updated release notes/docs take care of that, which configs can no longer be used and which are not. I don't mind adding a warning msg saying please use another cache.size inst

[GitHub] spark issue #11494: [SPARK-10399][CORE][SQL] Introduce OffHeapMemoryBlock to...

2017-08-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/11494 kindly ping @yzotov --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark pull request #18640: [SPARK-21422][BUILD] Depend on Apache ORC 1.4.0

2017-08-15 Thread dongjoon-hyun
Github user dongjoon-hyun commented on a diff in the pull request: https://github.com/apache/spark/pull/18640#discussion_r133255040 --- Diff: sql/core/pom.xml --- @@ -87,6 +87,16 @@ + org.apache.orc + orc-core + ${orc.classifier}

[GitHub] spark issue #16648: [SPARK-18016][SQL][CATALYST] Code Generation: Constant P...

2017-08-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/16648 kindly ping @bdrillard --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #18940: [SPARK-21501] Change CacheLoader to limit entries based ...

2017-08-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18940 @redsanket I am thinking about the case that the same configuration file, which explicitly sets a value (e.g. 4096) into `spark.shuffle.service.index.cache.entries`, is used in Spark 2.3. The user

[GitHub] spark issue #18488: [SPARK-21255][SQL][WIP] Fixed NPE when creating encoder ...

2017-08-15 Thread mike0sv
Github user mike0sv commented on the issue: https://github.com/apache/spark/pull/18488 @srowen @HyukjinKwon , retest this please :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featur

[GitHub] spark pull request #18930: [SPARK-21677][SQL] json_tuple throws NullPointExc...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18930#discussion_r133249207 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -2034,4 +2034,25 @@ class JsonSuite extends Que

[GitHub] spark pull request #18640: [SPARK-21422][BUILD] Depend on Apache ORC 1.4.0

2017-08-15 Thread omalley
Github user omalley commented on a diff in the pull request: https://github.com/apache/spark/pull/18640#discussion_r133248648 --- Diff: sql/core/pom.xml --- @@ -87,6 +87,16 @@ + org.apache.orc + orc-core + ${orc.classifier} --

[GitHub] spark issue #18949: [SPARK-12961][CORE][FOLLOW-UP] Remove wrapper code for S...

2017-08-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18949 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature

[GitHub] spark issue #18930: [SPARK-21677][SQL] json_tuple throws NullPointException ...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18930 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80688/ Test PASSed. ---

[GitHub] spark issue #18947: [SPARK-21721][SQL][Backport-2.1] Clear FileSystem delete...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18947 @viirya Could you close it? Thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enab

[GitHub] spark issue #18930: [SPARK-21677][SQL] json_tuple throws NullPointException ...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18930 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18421: [SPARK-21213][SQL] Support collecting partition-level st...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18421 This is just to make it consistent with the partition spec in our current INSERT statement. Could you justify why we need to make them inconsistent? Thanks! Also cc @sameeragarwal --

[GitHub] spark issue #18930: [SPARK-21677][SQL] json_tuple throws NullPointException ...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18930 **[Test build #80688 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80688/testReport)** for PR 18930 at commit [`ab16929`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #18849: [SPARK-21617][SQL] Store correct table metadata when alt...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18849 **[Test build #80694 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80694/testReport)** for PR 18849 at commit [`4a05b55`](https://github.com/apache/spark/commit/4a

[GitHub] spark issue #18266: [SPARK-20427][SQL] Read JDBC table use custom schema

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18266 Users should be allowed to specify the schema from the table properties by using DDL-like strings. --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark issue #18918: [SPARK-21707][SQL]Improvement a special case for non-det...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18918 **[Test build #80693 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80693/testReport)** for PR 18918 at commit [`97a3270`](https://github.com/apache/spark/commit/97

[GitHub] spark pull request #18266: [SPARK-20427][SQL] Read JDBC table use custom sch...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18266#discussion_r133239924 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala --- @@ -111,7 +111,22 @@ private[sql] case class JD

[GitHub] spark pull request #18266: [SPARK-20427][SQL] Read JDBC table use custom sch...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18266#discussion_r133239835 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala --- @@ -111,7 +111,22 @@ private[sql] case class JD

[GitHub] spark pull request #18266: [SPARK-20427][SQL] Read JDBC table use custom sch...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18266#discussion_r133239599 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JDBCRelation.scala --- @@ -111,7 +111,22 @@ private[sql] case class JD

[GitHub] spark pull request #18849: [SPARK-21617][SQL] Store correct table metadata w...

2017-08-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/18849#discussion_r133236830 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -1175,6 +1205,27 @@ private[spark] class HiveExternalCatalog(con

[GitHub] spark issue #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes fails fo...

2017-08-15 Thread eyalfa
Github user eyalfa commented on the issue: https://github.com/apache/spark/pull/18855 Funny enough, that's the approach I've chosen. On Aug 15, 2017 19:17, "Marcelo Vanzin" wrote: > *@vanzin* commented on this pull request. > -- >

[GitHub] spark pull request #18849: [SPARK-21617][SQL] Store correct table metadata w...

2017-08-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/18849#discussion_r133236101 --- Diff: sql/hive/src/main/scala/org/apache/spark/sql/hive/HiveExternalCatalog.scala --- @@ -342,6 +359,12 @@ private[spark] class HiveExternalCatalog(conf:

[GitHub] spark pull request #18266: [SPARK-20427][SQL] Read JDBC table use custom sch...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18266#discussion_r133235997 --- Diff: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/OracleIntegrationSuite.scala --- @@ -268,4 +275,44 @@ class OracleI

[GitHub] spark issue #18488: [SPARK-21255][SQL][WIP] Fixed NPE when creating encoder ...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18488 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18488: [SPARK-21255][SQL][WIP] Fixed NPE when creating encoder ...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18488 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80685/ Test FAILed. ---

[GitHub] spark issue #18488: [SPARK-21255][SQL][WIP] Fixed NPE when creating encoder ...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18488 **[Test build #80685 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80685/testReport)** for PR 18488 at commit [`fbdc599`](https://github.com/apache/spark/commit/f

[GitHub] spark pull request #18266: [SPARK-20427][SQL] Read JDBC table use custom sch...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/18266#discussion_r133235311 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala --- @@ -197,11 +197,13 @@ class DataFrameReader private[sql](sparkSession:

[GitHub] spark issue #18940: [SPARK-21501] Change CacheLoader to limit entries based ...

2017-08-15 Thread redsanket
Github user redsanket commented on the issue: https://github.com/apache/spark/pull/18940 @kiszk I dont think that would be ideal, it is better to backport the feature itself to a desired version or branch, having two conflicting configs for the same task is not ideal, if that is what

[GitHub] spark issue #18266: [SPARK-20427][SQL] Read JDBC table use custom schema

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18266 The example in the PR description looks a little bit confusing. ```Scala val dfRead = spark.read.schema(schema).jdbc(jdbcUrl, "tableWithCustomSchema", new Properties()) ``` C

[GitHub] spark issue #18926: [SPARK-21712] [PySpark] Clarify type error for Column.su...

2017-08-15 Thread nchammas
Github user nchammas commented on the issue: https://github.com/apache/spark/pull/18926 It's cleaner but less specific. Unless we branch on whether `startPos` and `length` are the same type, we will give the same error message for mixed types and for unsupported types. That seems like

[GitHub] spark issue #18943: [SPARK-21731][build][test-maven] Upgrade scalastyle to 0...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18943 **[Test build #80692 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80692/testReport)** for PR 18943 at commit [`b1e49fa`](https://github.com/apache/spark/commit/b1

[GitHub] spark issue #18951: [SPARK-21738] Thriftserver doesn't cancel jobs when sess...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18951 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feat

[GitHub] spark pull request #18951: [SPARK-21738] Thriftserver doesn't cancel jobs wh...

2017-08-15 Thread mgaido91
GitHub user mgaido91 opened a pull request: https://github.com/apache/spark/pull/18951 [SPARK-21738] Thriftserver doesn't cancel jobs when session is closed ## What changes were proposed in this pull request? When a session is closed the Thriftserver doesn't cancel the jobs

[GitHub] spark issue #18943: [SPARK-21731][build][test-maven] Upgrade scalastyle to 0...

2017-08-15 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/18943 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or

[GitHub] spark pull request #18943: [SPARK-21731][build] Upgrade scalastyle to 0.9.

2017-08-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/18943#discussion_r133233495 --- Diff: examples/src/main/scala/org/apache/spark/examples/ml/BucketedRandomProjectionLSHExample.scala --- @@ -21,9 +21,9 @@ package org.apache.spark.examp

[GitHub] spark pull request #18855: [SPARK-3151] [Block Manager] DiskStore.getBytes f...

2017-08-15 Thread vanzin
Github user vanzin commented on a diff in the pull request: https://github.com/apache/spark/pull/18855#discussion_r133232857 --- Diff: core/src/test/scala/org/apache/spark/storage/BlockManagerSuite.scala --- @@ -1415,6 +1415,79 @@ class BlockManagerSuite extends SparkFunSuite with

[GitHub] spark issue #18926: [SPARK-21712] [PySpark] Clarify type error for Column.su...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18926 ```Python if isinstance(startPos, int) and isinstance(length, int): jc = self._jc.substr(startPos, length) elif isinstance(startPos, Column) and isinstance(l

[GitHub] spark issue #18649: [SPARK-21395][SQL] Spark SQL hive-thriftserver doesn't r...

2017-08-15 Thread debugger87
Github user debugger87 commented on the issue: https://github.com/apache/spark/pull/18649 @dilipbiswal Thanks for your reply. In my eyes, there have been some mechanism or configuration to control the number of opening files generated by SQL Operation. e.g: ```

[GitHub] spark pull request #18907: [SPARK-18464][SQL][followup] support old table wh...

2017-08-15 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18907 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #18907: [SPARK-18464][SQL][followup] support old table which doe...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18907 Thanks! Merging to master. Hit conflicts when trying to merge to the previous versions. --- If your project is set up for it, you can reply to this email and have your reply appear on G

[GitHub] spark issue #18907: [SPARK-18464][SQL][followup] support old table which doe...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18907 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the fe

[GitHub] spark pull request #18933: [WIP][SPARK-21722][SQL][PYTHON] Enable timezone-a...

2017-08-15 Thread icexelloss
Github user icexelloss commented on a diff in the pull request: https://github.com/apache/spark/pull/18933#discussion_r133229705 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala --- @@ -912,6 +912,14 @@ object SQLConf { .intConf

[GitHub] spark issue #18622: [SPARK-21340] Bring pyspark BinaryClassificationMetrics ...

2017-08-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/18622 @srowen any comment on this PR? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled an

[GitHub] spark issue #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17373 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17373 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80689/ Test PASSed. ---

[GitHub] spark issue #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17373 **[Test build #80689 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80689/testReport)** for PR 17373 at commit [`eedc647`](https://github.com/apache/spark/commit/e

[GitHub] spark issue #18329: [SPARK-19909][SS] Disabling the usage of a temporary dir...

2017-08-15 Thread mgaido91
Github user mgaido91 commented on the issue: https://github.com/apache/spark/pull/18329 @zsxwing @tdas any comment on this? Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this featu

[GitHub] spark issue #18940: SPARK-21501 Change CacheLoader to limit entries based on...

2017-08-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18940 nit: title should be "`[SPARK-21501] ...`". --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18940: SPARK-21501 Change CacheLoader to limit entries based on...

2017-08-15 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/18940 I like this feature. For backward compatibility, how about referring to `spark.shuffle.service.index.cache.entries` only if `spark.shuffle.service.index.cache.entries` is explicitly declared.

[GitHub] spark pull request #18946: [SPARK-19471][SQL]AggregationIterator does not in...

2017-08-15 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/18946 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is ena

[GitHub] spark issue #18421: [SPARK-21213][SQL] Support collecting partition-level st...

2017-08-15 Thread mbasmanova
Github user mbasmanova commented on the issue: https://github.com/apache/spark/pull/18421 ping @gatorsmile --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so,

[GitHub] spark issue #18940: SPARK-21501 Change CacheLoader to limit entries based on...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18940 **[Test build #80691 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80691/testReport)** for PR 18940 at commit [`e9afdf7`](https://github.com/apache/spark/commit/e9

[GitHub] spark issue #18946: [SPARK-19471][SQL]AggregationIterator does not initializ...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18946 LGTM Thanks! Merging to master --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #18947: [SPARK-21721][SQL][Backport-2.1] Clear FileSystem delete...

2017-08-15 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/18947 LGTM Merging to 2.1 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #18907: [SPARK-18464][SQL][followup] support old table which doe...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18907 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80682/ Test PASSed. ---

[GitHub] spark issue #18907: [SPARK-18464][SQL][followup] support old table which doe...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18907 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18907: [SPARK-18464][SQL][followup] support old table which doe...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18907 **[Test build #80682 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80682/testReport)** for PR 18907 at commit [`0a18435`](https://github.com/apache/spark/commit/0

[GitHub] spark issue #18940: SPARK-21501 Change CacheLoader to limit entries based on...

2017-08-15 Thread redsanket
Github user redsanket commented on the issue: https://github.com/apache/spark/pull/18940 @dbolshak there were no unit tests for google cache implementation here before, I could add a simple test to check for cache behavior if it is necessary but ideally a scale test is necessary to un

[GitHub] spark pull request #18940: SPARK-21501 Change CacheLoader to limit entries b...

2017-08-15 Thread redsanket
Github user redsanket commented on a diff in the pull request: https://github.com/apache/spark/pull/18940#discussion_r133220047 --- Diff: common/network-shuffle/src/main/java/org/apache/spark/network/shuffle/ExternalShuffleBlockResolver.java --- @@ -104,15 +105,22 @@ public Extern

[GitHub] spark issue #18798: [SPARK-19634][ML] Multivariate summarizer - dataframes A...

2017-08-15 Thread thunterdb
Github user thunterdb commented on the issue: https://github.com/apache/spark/pull/18798 Thank you @yanboliang. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark issue #18950: [SPARK-20589][Core][Scheduler] Allow limiting task concu...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18950 **[Test build #80690 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80690/testReport)** for PR 18950 at commit [`d3f8162`](https://github.com/apache/spark/commit/d3

[GitHub] spark pull request #18950: [SPARK-20589][Core][Scheduler] Allow limiting tas...

2017-08-15 Thread dhruve
GitHub user dhruve opened a pull request: https://github.com/apache/spark/pull/18950 [SPARK-20589][Core][Scheduler] Allow limiting task concurrency per job group ## What changes were proposed in this pull request? This change allows the user to specify the maximum no. of tasks ru

[GitHub] spark issue #18786: [SPARK-21584][SQL][SparkR] Update R method for summary t...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18786 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18786: [SPARK-21584][SQL][SparkR] Update R method for summary t...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18786 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80687/ Test PASSed. ---

[GitHub] spark issue #18786: [SPARK-21584][SQL][SparkR] Update R method for summary t...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18786 **[Test build #80687 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80687/testReport)** for PR 18786 at commit [`9c9f0f6`](https://github.com/apache/spark/commit/9

[GitHub] spark issue #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17373 **[Test build #80689 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80689/testReport)** for PR 17373 at commit [`eedc647`](https://github.com/apache/spark/commit/ee

[GitHub] spark issue #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-15 Thread WeichenXu123
Github user WeichenXu123 commented on the issue: https://github.com/apache/spark/pull/17373 Jenkins, test this please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #18933: [WIP][SPARK-21722][SQL][PYTHON] Enable timezone-a...

2017-08-15 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18933#discussion_r133209362 --- Diff: python/pyspark/sql/tests.py --- @@ -2507,6 +2507,37 @@ def test_to_pandas(self): self.assertEquals(types[2], np.bool)

[GitHub] spark pull request #18930: [SPARK-21677][SQL] json_tuple throws NullPointExc...

2017-08-15 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/18930#discussion_r133202748 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/json/JsonSuite.scala --- @@ -2034,4 +2034,25 @@ class JsonSuite extends Qu

[GitHub] spark issue #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17373 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17373 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80684/ Test FAILed. ---

[GitHub] spark issue #17373: [SPARK-12664][ML] Expose probability in mlp model

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17373 **[Test build #80684 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80684/testReport)** for PR 17373 at commit [`eedc647`](https://github.com/apache/spark/commit/e

[GitHub] spark pull request #18930: [SPARK-21677][SQL] json_tuple throws NullPointExc...

2017-08-15 Thread jmchung
Github user jmchung commented on a diff in the pull request: https://github.com/apache/spark/pull/18930#discussion_r133200977 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -362,12 +362,12 @@ case class JsonTuple(childr

[GitHub] spark issue #18930: [SPARK-21677][SQL] json_tuple throws NullPointException ...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18930 **[Test build #80688 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80688/testReport)** for PR 18930 at commit [`ab16929`](https://github.com/apache/spark/commit/ab

[GitHub] spark issue #18942: [BACKPORT-2.1][SPARK-19372][SQL] Fix throwing a Java exc...

2017-08-15 Thread poplav
Github user poplav commented on the issue: https://github.com/apache/spark/pull/18942 @kiszk , I updated the PR to remove the `prunePartionsByFilter` bit. Please let me know now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub a

[GitHub] spark issue #18918: [SPARK-21707][SQL]Improvement a special case for non-det...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18918 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80686/ Test FAILed. ---

[GitHub] spark issue #18918: [SPARK-21707][SQL]Improvement a special case for non-det...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18918 **[Test build #80686 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80686/testReport)** for PR 18918 at commit [`df7ecaa`](https://github.com/apache/spark/commit/d

[GitHub] spark issue #18918: [SPARK-21707][SQL]Improvement a special case for non-det...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18918 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18918: [SPARK-21707][SQL]Improvement a special case for non-det...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18918 **[Test build #80686 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80686/testReport)** for PR 18918 at commit [`df7ecaa`](https://github.com/apache/spark/commit/df

[GitHub] spark issue #18786: [SPARK-21584][SQL][SparkR] Update R method for summary t...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18786 **[Test build #80687 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80687/testReport)** for PR 18786 at commit [`9c9f0f6`](https://github.com/apache/spark/commit/9c

[GitHub] spark issue #18918: [SPARK-21707][SQL]Improvement a special case for non-det...

2017-08-15 Thread heary-cao
Github user heary-cao commented on the issue: https://github.com/apache/spark/pull/18918 sorry, Rang not rand --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes s

[GitHub] spark pull request #18798: [SPARK-19634][ML] Multivariate summarizer - dataf...

2017-08-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18798#discussion_r133195248 --- Diff: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala --- @@ -0,0 +1,593 @@ +/* + * Licensed to the Apache Software Foundation (ASF

[GitHub] spark pull request #18488: [SPARK-21255][SQL][WIP] Fixed NPE when creating e...

2017-08-15 Thread mike0sv
Github user mike0sv commented on a diff in the pull request: https://github.com/apache/spark/pull/18488#discussion_r133194603 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/ExpressionInfo.java --- @@ -79,7 +79,7 @@ public ExpressionInfo(

[GitHub] spark issue #18488: [SPARK-21255][SQL][WIP] Fixed NPE when creating encoder ...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18488 **[Test build #80685 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80685/testReport)** for PR 18488 at commit [`fbdc599`](https://github.com/apache/spark/commit/fb

[GitHub] spark issue #18926: [SPARK-21712] [PySpark] Clarify type error for Column.su...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18926 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature e

[GitHub] spark issue #18926: [SPARK-21712] [PySpark] Clarify type error for Column.su...

2017-08-15 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18926 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/80683/ Test PASSed. ---

[GitHub] spark issue #18926: [SPARK-21712] [PySpark] Clarify type error for Column.su...

2017-08-15 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18926 **[Test build #80683 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/80683/testReport)** for PR 18926 at commit [`a7fea20`](https://github.com/apache/spark/commit/a

[GitHub] spark issue #18930: [SPARK-21677][SQL] json_tuple throws NullPointException ...

2017-08-15 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/18930 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the featur

[GitHub] spark pull request #18798: [SPARK-19634][ML] Multivariate summarizer - dataf...

2017-08-15 Thread WeichenXu123
Github user WeichenXu123 commented on a diff in the pull request: https://github.com/apache/spark/pull/18798#discussion_r133191237 --- Diff: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala --- @@ -0,0 +1,593 @@ +/* + * Licensed to the Apache Software Foundatio

[GitHub] spark pull request #18930: [SPARK-21677][SQL] json_tuple throws NullPointExc...

2017-08-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18930#discussion_r133190654 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -362,12 +362,12 @@ case class JsonTuple(childre

[GitHub] spark pull request #18930: [SPARK-21677][SQL] json_tuple throws NullPointExc...

2017-08-15 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/18930#discussion_r133190497 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/jsonExpressions.scala --- @@ -426,10 +426,11 @@ case class JsonTuple(childre

<    1   2   3   4   5   >