[GitHub] spark pull request #17458: [SPARK-20127][CORE] few warning have been fixed w...

2017-03-29 Thread dbolshak
Github user dbolshak commented on a diff in the pull request: https://github.com/apache/spark/pull/17458#discussion_r108678514 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/AllStagesPage.scala --- @@ -34,9 +34,9 @@ private[ui] class AllStagesPage(parent: StagesTab)

[GitHub] spark issue #12004: [SPARK-7481] [build] Add spark-cloud module to pull in o...

2017-03-29 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/12004 Any comments on the latest patch? Anyone? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #17468: [SPARK-20143][SQL] DataType.fromJson should throw an exc...

2017-03-29 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/17468 (BTW, I believe this does not make a conflict with PR 17406) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15049 **[Test build #75360 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75360/testReport)** for PR 15049 at commit

[GitHub] spark issue #14731: [SPARK-17159] [streaming]: optimise check for new files ...

2017-03-29 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/14731 Is there anything else I need to do here? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #17468: [SPARK-20143][SQL] DataType.fromJson should throw an exc...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17468 **[Test build #75357 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75357/testReport)** for PR 17468 at commit

[GitHub] spark issue #17458: [SPARK-20127][CORE] few warning have been fixed which In...

2017-03-29 Thread dbolshak
Github user dbolshak commented on the issue: https://github.com/apache/spark/pull/17458 Looks like all comments have been addressed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #17468: [SPARK-20143][SQL] DataType.fromJson should throw...

2017-03-29 Thread HyukjinKwon
GitHub user HyukjinKwon opened a pull request: https://github.com/apache/spark/pull/17468 [SPARK-20143][SQL] DataType.fromJson should throw an exception with better message ## What changes were proposed in this pull request? Currently, `DataType.fromJson` throws

[GitHub] spark issue #17417: [DOCS] Docs-only improvements

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17417 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75355/ Test PASSed. ---

[GitHub] spark issue #17417: [DOCS] Docs-only improvements

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17417 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15332: [SPARK-10364][SQL] Support Parquet logical type T...

2017-03-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15332#discussion_r108682129 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -362,7 +375,15 @@ private void

[GitHub] spark pull request #15332: [SPARK-10364][SQL] Support Parquet logical type T...

2017-03-29 Thread squito
Github user squito commented on a diff in the pull request: https://github.com/apache/spark/pull/15332#discussion_r108681155 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/DateTimeUtils.scala --- @@ -237,6 +238,24 @@ object DateTimeUtils {

[GitHub] spark pull request #17435: [SPARK-20098][PYSPARK] dataType's typeName fix

2017-03-29 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/17435#discussion_r108685381 --- Diff: python/pyspark/sql/types.py --- @@ -57,7 +57,25 @@ def __ne__(self, other): @classmethod def typeName(cls): -

[GitHub] spark issue #16781: [SPARK-12297][SQL] Hive compatibility for Parquet Timest...

2017-03-29 Thread squito
Github user squito commented on the issue: https://github.com/apache/spark/pull/16781 thanks @ueshin ... I am going to chat w/ some folks involved in that hive patch, that was not my understanding conceptually of their patch. I heard that there is a bug they need to fix so maybe its

[GitHub] spark issue #17417: [DOCS] Docs-only improvements

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17417 **[Test build #75355 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75355/testReport)** for PR 17417 at commit

[GitHub] spark pull request #17435: [SPARK-20098][PYSPARK] dataType's typeName fix

2017-03-29 Thread szalai1
Github user szalai1 commented on a diff in the pull request: https://github.com/apache/spark/pull/17435#discussion_r108683954 --- Diff: python/pyspark/sql/types.py --- @@ -57,7 +57,25 @@ def __ne__(self, other): @classmethod def typeName(cls): -

[GitHub] spark pull request #17458: [SPARK-20127][CORE] few warning have been fixed w...

2017-03-29 Thread dbolshak
Github user dbolshak commented on a diff in the pull request: https://github.com/apache/spark/pull/17458#discussion_r10860 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagePage.scala --- @@ -290,7 +290,7 @@ private[ui] class StagePage(parent: StagesTab) extends

[GitHub] spark issue #17364: [SPARK-20038] [SQL]: FileFormatWriter.ExecuteWriteTask.r...

2017-03-29 Thread steveloughran
Github user steveloughran commented on the issue: https://github.com/apache/spark/pull/17364 I don't have a time/plans to do the test here, as it's a fairly complex piece of test setup for what a review should show isn't doing anything other than guarantee the outcome pf

[GitHub] spark pull request #17458: [SPARK-20127][CORE] few warning have been fixed w...

2017-03-29 Thread dbolshak
Github user dbolshak commented on a diff in the pull request: https://github.com/apache/spark/pull/17458#discussion_r108677907 --- Diff: core/src/main/scala/org/apache/spark/ui/jobs/StagesTab.scala --- @@ -35,7 +35,7 @@ private[ui] class StagesTab(parent: SparkUI) extends

[GitHub] spark pull request #17435: [SPARK-20098][PYSPARK] dataType's typeName fix

2017-03-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/17435#discussion_r108691360 --- Diff: python/pyspark/sql/types.py --- @@ -57,7 +57,25 @@ def __ne__(self, other): @classmethod def typeName(cls): -

[GitHub] spark issue #16611: [SPARK-17967][SPARK-17878][SQL][PYTHON] Support for arra...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16611 **[Test build #75359 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75359/testReport)** for PR 16611 at commit

[GitHub] spark issue #16989: [WIP][SPARK-19659] Fetch big blocks to disk when shuffle...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16989 **[Test build #75358 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75358/testReport)** for PR 16989 at commit

[GitHub] spark issue #15332: [SPARK-10364][SQL] Support Parquet logical type TIMESTAM...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15332 **[Test build #75361 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75361/testReport)** for PR 15332 at commit

[GitHub] spark issue #16611: [SPARK-17967][SPARK-17878][SQL][PYTHON] Support for arra...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16611 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17388: [SPARK-20059][YARN] Use the correct classloader f...

2017-03-29 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17388 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17469: [SPARK-20132][Docs]

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17469 Can one of the admins verify this patch? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark issue #13932: [SPARK-15354] [CORE] Topology aware block replication st...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13932 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75356/ Test PASSed. ---

[GitHub] spark pull request #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-29 Thread felixcheung
Github user felixcheung commented on a diff in the pull request: https://github.com/apache/spark/pull/17170#discussion_r108735289 --- Diff: R/pkg/R/mllib_fpm.R --- @@ -99,7 +99,10 @@ setMethod("spark.fpGrowth", signature(data = "SparkDataFrame"), # Get frequent itemsets.

[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.

2017-03-29 Thread shaneknapp
Github user shaneknapp commented on the issue: https://github.com/apache/spark/pull/17355 lgtm++ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the

[GitHub] spark issue #16989: [WIP][SPARK-19659] Fetch big blocks to disk when shuffle...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16989 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75358/ Test FAILed. ---

[GitHub] spark issue #13932: [SPARK-15354] [CORE] Topology aware block replication st...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/13932 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #15332: [SPARK-10364][SQL] Support Parquet logical type T...

2017-03-29 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/15332#discussion_r108726983 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedColumnReader.java --- @@ -362,7 +375,15 @@ private void

[GitHub] spark issue #17468: [SPARK-20143][SQL] DataType.fromJson should throw an exc...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17468 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75357/ Test PASSed. ---

[GitHub] spark issue #17468: [SPARK-20143][SQL] DataType.fromJson should throw an exc...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17468 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #16611: [SPARK-17967][SPARK-17878][SQL][PYTHON] Support for arra...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16611 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75359/ Test PASSed. ---

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15049 **[Test build #75360 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75360/testReport)** for PR 15049 at commit

[GitHub] spark pull request #17469: [SPARK-20132][Docs]

2017-03-29 Thread map222
GitHub user map222 opened a pull request: https://github.com/apache/spark/pull/17469 [SPARK-20132][Docs] ## What changes were proposed in this pull request? Add docstrings to column.py for the Column functions `rlike`, `like`, `startswith`, and `endswith`. Pass these docstrings

[GitHub] spark pull request #17450: [SPARK-20121][SQL] simplify NullPropagation with ...

2017-03-29 Thread nsyca
Github user nsyca commented on a diff in the pull request: https://github.com/apache/spark/pull/17450#discussion_r108720512 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/expressions.scala --- @@ -384,33 +379,13 @@ case class NullPropagation(conf:

[GitHub] spark issue #16611: [SPARK-17967][SPARK-17878][SQL][PYTHON] Support for arra...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16611 **[Test build #75359 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75359/testReport)** for PR 16611 at commit

[GitHub] spark issue #13932: [SPARK-15354] [CORE] Topology aware block replication st...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/13932 **[Test build #75356 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75356/testReport)** for PR 13932 at commit

[GitHub] spark issue #16989: [WIP][SPARK-19659] Fetch big blocks to disk when shuffle...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/16989 **[Test build #75358 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75358/testReport)** for PR 16989 at commit

[GitHub] spark issue #16989: [WIP][SPARK-19659] Fetch big blocks to disk when shuffle...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/16989 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15049 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75360/ Test PASSed. ---

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15049 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17450: [SPARK-20121][SQL] simplify NullPropagation with ...

2017-03-29 Thread nsyca
Github user nsyca commented on a diff in the pull request: https://github.com/apache/spark/pull/17450#discussion_r108720888 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -1122,7 +1119,7 @@ case class

[GitHub] spark issue #17468: [SPARK-20143][SQL] DataType.fromJson should throw an exc...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17468 **[Test build #75357 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75357/testReport)** for PR 17468 at commit

[GitHub] spark issue #17388: [SPARK-20059][YARN] Use the correct classloader for HBas...

2017-03-29 Thread vanzin
Github user vanzin commented on the issue: https://github.com/apache/spark/pull/17388 Merging to master / 2.1. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes

[GitHub] spark pull request #17415: [SPARK-19408][SQL] filter estimation on two colum...

2017-03-29 Thread ron8hu
Github user ron8hu commented on a diff in the pull request: https://github.com/apache/spark/pull/17415#discussion_r108754614 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -515,8 +530,138 @@ case

[GitHub] spark issue #17470: [SPARK-20146][SQL] fix comment missing issue for thrift ...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17470 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17419: [SPARK-19634][ML] Multivariate summarizer - dataf...

2017-03-29 Thread thunterdb
Github user thunterdb commented on a diff in the pull request: https://github.com/apache/spark/pull/17419#discussion_r108743634 --- Diff: mllib/src/test/scala/org/apache/spark/ml/stat/SummarizerSuite.scala --- @@ -335,4 +335,65 @@ class SummarizerSuite extends SparkFunSuite with

[GitHub] spark issue #17379: [SPARK-20048][SQL] Cloning SessionState does not clone q...

2017-03-29 Thread kunalkhamar
Github user kunalkhamar commented on the issue: https://github.com/apache/spark/pull/17379 @hvanhovell it should be HiveSessionStateBuilder now --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not

[GitHub] spark pull request #17355: [SPARK-19955][PySpark] Jenkins Python Conda based...

2017-03-29 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17355 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17470: [SPARK-20146][SQL] fix comment missing issue for thrift ...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17470 **[Test build #75362 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75362/testReport)** for PR 17470 at commit

[GitHub] spark issue #17470: [SPARK-20146][SQL] fix comment missing issue for thrift ...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17470 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75362/ Test PASSed. ---

[GitHub] spark issue #17251: [SPARK-19910][SQL] `stack` should not reject NULL values...

2017-03-29 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17251 Retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17251: [SPARK-19910][SQL] `stack` should not reject NULL values...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17251 **[Test build #75364 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75364/testReport)** for PR 17251 at commit

[GitHub] spark issue #17445: [SPARK-20115] [CORE] Fix DAGScheduler to recompute all t...

2017-03-29 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/17445 Have you seen #17088? I just glanced at this quickly but I think this is a duplicate of that (SPARK-19753) --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #17450: [SPARK-20121][SQL] simplify NullPropagation with ...

2017-03-29 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17450#discussion_r108755645 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -1122,7 +1119,7 @@ case class

[GitHub] spark pull request #17471: [SPARK-3577] Report Spill size on disk for Unsafe...

2017-03-29 Thread sitalkedia
GitHub user sitalkedia opened a pull request: https://github.com/apache/spark/pull/17471 [SPARK-3577] Report Spill size on disk for UnsafeExternalSorter ## What changes were proposed in this pull request? Report Spill size on disk for UnsafeExternalSorter ## How

[GitHub] spark pull request #17419: [SPARK-19634][ML] Multivariate summarizer - dataf...

2017-03-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/17419#discussion_r108742186 --- Diff: mllib/src/test/scala/org/apache/spark/ml/stat/SummarizerSuite.scala --- @@ -335,4 +335,65 @@ class SummarizerSuite extends SparkFunSuite with

[GitHub] spark issue #17445: [SPARK-20115] [CORE] Fix DAGScheduler to recompute all t...

2017-03-29 Thread kayousterhout
Github user kayousterhout commented on the issue: https://github.com/apache/spark/pull/17445 Jenkins this is OK to test --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark pull request #17470: [SPARK-20146][SQL] fix comment missing issue for ...

2017-03-29 Thread bomeng
GitHub user bomeng opened a pull request: https://github.com/apache/spark/pull/17470 [SPARK-20146][SQL] fix comment missing issue for thrift server ## What changes were proposed in this pull request? The column comment was missing while constructing the Hive TableSchema.

[GitHub] spark pull request #17415: [SPARK-19408][SQL] filter estimation on two colum...

2017-03-29 Thread ron8hu
Github user ron8hu commented on a diff in the pull request: https://github.com/apache/spark/pull/17415#discussion_r108753830 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -515,8 +530,138 @@ case

[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.

2017-03-29 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/17355 Merged to master. Please do not backport. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17450: [SPARK-20121][SQL] simplify NullPropagation with ...

2017-03-29 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17450#discussion_r108756250 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -297,8 +297,8 @@ case class Lower(child:

[GitHub] spark issue #15332: [SPARK-10364][SQL] Support Parquet logical type TIMESTAM...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15332 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #15332: [SPARK-10364][SQL] Support Parquet logical type TIMESTAM...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15332 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75361/ Test PASSed. ---

[GitHub] spark issue #14527: [SPARK-16938][SQL] `drop/dropDuplicate` should handle th...

2017-03-29 Thread samelamin
Github user samelamin commented on the issue: https://github.com/apache/spark/pull/14527 @cloud-fan by adding a new API do you mean move the logic into a new method in the Dataset class? --- If your project is set up for it, you can reply to this email and have your reply appear on

[GitHub] spark pull request #17379: [SPARK-20048][SQL] Cloning SessionState does not ...

2017-03-29 Thread kunalkhamar
Github user kunalkhamar commented on a diff in the pull request: https://github.com/apache/spark/pull/17379#discussion_r108748031 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/internal/BaseSessionStateBuilder.scala --- @@ -134,6 +135,14 @@ abstract class

[GitHub] spark pull request #17415: [SPARK-19408][SQL] filter estimation on two colum...

2017-03-29 Thread ron8hu
Github user ron8hu commented on a diff in the pull request: https://github.com/apache/spark/pull/17415#discussion_r108751882 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -515,8 +530,138 @@ case

[GitHub] spark issue #17470: [SPARK-20146][SQL] fix comment missing issue for thrift ...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17470 **[Test build #75362 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75362/testReport)** for PR 17470 at commit

[GitHub] spark issue #17355: [SPARK-19955][PySpark] Jenkins Python Conda based test.

2017-03-29 Thread holdenk
Github user holdenk commented on the issue: https://github.com/apache/spark/pull/17355 Great, yay 2.6 deprecation adventures :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17424: [SPARK-20089] [SQL] [TEST] Added DESC FUNCTION and DESC ...

2017-03-29 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17424 @viirya That is not only for document generator. It can help us review/correct the output of `DESC FUNCTIONS EXTENDED`. You can find many inconsistent parts we should correct when reading

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-03-29 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r108744852 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -2828,4 +2839,16 @@ class Dataset[T] private[sql](

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-03-29 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r108744456 --- Diff: python/pyspark/sql/tests.py --- @@ -56,6 +56,15 @@ from pyspark.sql.utils import AnalysisException, ParseException,

[GitHub] spark pull request #15821: [SPARK-13534][PySpark] Using Apache Arrow to incr...

2017-03-29 Thread BryanCutler
Github user BryanCutler commented on a diff in the pull request: https://github.com/apache/spark/pull/15821#discussion_r108746678 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala --- @@ -2747,6 +2747,17 @@ class Dataset[T] private[sql]( } }

[GitHub] spark pull request #17419: [SPARK-19634][ML] Multivariate summarizer - dataf...

2017-03-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/17419#discussion_r108746608 --- Diff: mllib/src/test/scala/org/apache/spark/ml/stat/SummarizerSuite.scala --- @@ -335,4 +335,65 @@ class SummarizerSuite extends SparkFunSuite with

[GitHub] spark pull request #17415: [SPARK-19408][SQL] filter estimation on two colum...

2017-03-29 Thread ron8hu
Github user ron8hu commented on a diff in the pull request: https://github.com/apache/spark/pull/17415#discussion_r108752975 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/statsEstimation/FilterEstimation.scala --- @@ -515,8 +530,138 @@ case

[GitHub] spark issue #17471: [SPARK-3577] Report Spill size on disk for UnsafeExterna...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17471 **[Test build #75363 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75363/testReport)** for PR 17471 at commit

[GitHub] spark issue #15332: [SPARK-10364][SQL] Support Parquet logical type TIMESTAM...

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15332 **[Test build #75361 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75361/testReport)** for PR 15332 at commit

[GitHub] spark pull request #17450: [SPARK-20121][SQL] simplify NullPropagation with ...

2017-03-29 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17450#discussion_r108764164 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala --- @@ -1122,7 +1119,7 @@ case class

[GitHub] spark issue #17472: [SPARK-19999]: Fix for flakey tests due to java.nio.Bits...

2017-03-29 Thread samelamin
Github user samelamin commented on the issue: https://github.com/apache/spark/pull/17472 @gatorsmile I think my Intellij has some annoying auto indenting changes. Does spark have some project wide settings I can auto import? --- If your project is set up for it, you can reply to

[GitHub] spark issue #17471: [SPARK-3577] Report Spill size on disk for UnsafeExterna...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17471 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75363/ Test PASSed. ---

[GitHub] spark issue #17251: [SPARK-19910][SQL] `stack` should not reject NULL values...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17251 Merged build finished. Test PASSed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17251: [SPARK-19910][SQL] `stack` should not reject NULL values...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17251 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/75364/ Test PASSed. ---

[GitHub] spark issue #17251: [SPARK-19910][SQL] `stack` should not reject NULL values...

2017-03-29 Thread dongjoon-hyun
Github user dongjoon-hyun commented on the issue: https://github.com/apache/spark/pull/17251 Hi, @gatorsmile . Could you review this PR when you have sometime? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17472: [SPARK-19999]: Fix for flakey tests due to java.nio.Bits...

2017-03-29 Thread samelamin
Github user samelamin commented on the issue: https://github.com/apache/spark/pull/17472 ok @gatorsmile i set default indentation to 2, how does it look now? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17436: [SPARK-20101][SQL] Use OffHeapColumnVector when "spark.m...

2017-03-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17436 Merged build finished. Test FAILed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark pull request #17473: [SPARK-19088][SQL] Fix 2.10 build.

2017-03-29 Thread ueshin
GitHub user ueshin opened a pull request: https://github.com/apache/spark/pull/17473 [SPARK-19088][SQL] Fix 2.10 build. ## What changes were proposed in this pull request? Commit 6c70a38 broke the build for scala 2.10. The commit uses some reflections which are not

[GitHub] spark issue #17170: [SPARK-19825][R][ML] spark.ml R API for FPGrowth

2017-03-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17170 **[Test build #75366 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/75366/testReport)** for PR 17170 at commit

[GitHub] spark pull request #17472: [SPARK-19999]: Fix for flakey tests due to java.n...

2017-03-29 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/17472#discussion_r108797960 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/Platform.java --- @@ -46,18 +46,22 @@ private static final boolean unaligned;

[GitHub] spark issue #17449: [SPARK-20120][SQL] spark-sql support silent mode

2017-03-29 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17449 Thanks! Merging to master. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #17449: [SPARK-20120][SQL] spark-sql support silent mode

2017-03-29 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17449 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17472: [SPARK-19999]: Fix for flakey tests due to java.nio.Bits...

2017-03-29 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17472 Clear code comments can help code reading. : ) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #17419: [SPARK-19634][ML] Multivariate summarizer - dataf...

2017-03-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/17419#discussion_r108838518 --- Diff: mllib/src/main/scala/org/apache/spark/ml/stat/Summarizer.scala --- @@ -0,0 +1,746 @@ +/* + * Licensed to the Apache Software Foundation

[GitHub] spark issue #17251: [SPARK-19910][SQL] `stack` should not reject NULL values...

2017-03-29 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/17251 Will review it tonight. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark issue #17474: [Minor][SparkR]: Add run command comment in examples

2017-03-29 Thread felixcheung
Github user felixcheung commented on the issue: https://github.com/apache/spark/pull/17474 merged to master, thanks! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and

[GitHub] spark pull request #17474: [Minor][SparkR]: Add run command comment in examp...

2017-03-29 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/17474 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] spark issue #17476: [SPARK-20151][SQL] Account for partition pruning in scan...

2017-03-29 Thread rxin
Github user rxin commented on the issue: https://github.com/apache/spark/pull/17476 cc @ericl, @bogdanrdc, @adrian-ionescu, @cloud-fan --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this

[GitHub] spark pull request #17476: [SPARK-20151][SQL] Account for partition pruning ...

2017-03-29 Thread rxin
GitHub user rxin opened a pull request: https://github.com/apache/spark/pull/17476 [SPARK-20151][SQL] Account for partition pruning in scan metadataTime metrics ## What changes were proposed in this pull request? After SPARK-20136, we report metadata timing metrics in scan

  1   2   3   4   >