[GitHub] spark pull request #22314: [SPARK-25307][SQL] ArraySort function may return ...

2018-09-02 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/22314#discussion_r214582048 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1047,7 +1047,8 @@ trait

[GitHub] spark pull request #22314: [SPARK-25307][SQL] ArraySort function may return ...

2018-09-02 Thread dilipbiswal
Github user dilipbiswal commented on a diff in the pull request: https://github.com/apache/spark/pull/22314#discussion_r214581948 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1047,7 +1047,8 @@ trait

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21638: [SPARK-22357][CORE] SparkContext.binaryFiles ignore minP...

2018-09-02 Thread gatorsmile
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21638 @bomeng Could you submit a follow-up PR to add a test case? --- - To unsubscribe, e-mail:

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22319 **[Test build #95599 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95599/testReport)** for PR 22319 at commit

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95599/ Test FAILed. ---

[GitHub] spark pull request #22319: [SPARK-25044][SQL][followup] add back UserDefined...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22319#discussion_r214581734 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -129,3 +135,11 @@ case class UserDefinedFunction

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22319 **[Test build #95600 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95600/testReport)** for PR 22319 at commit

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #21638: [SPARK-22357][CORE] SparkContext.binaryFiles igno...

2018-09-02 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21638#discussion_r214581248 --- Diff: core/src/main/scala/org/apache/spark/input/PortableDataStream.scala --- @@ -47,7 +47,7 @@ private[spark] abstract class

[GitHub] spark pull request #22319: [SPARK-25044][SQL][followup] add back UserDefined...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22319#discussion_r214581204 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -129,3 +135,11 @@ case class UserDefinedFunction

[GitHub] spark pull request #21638: [SPARK-22357][CORE] SparkContext.binaryFiles igno...

2018-09-02 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/21638#discussion_r214581076 --- Diff: core/src/main/scala/org/apache/spark/input/PortableDataStream.scala --- @@ -47,7 +47,7 @@ private[spark] abstract class

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22319 **[Test build #95599 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95599/testReport)** for PR 22319 at commit

[GitHub] spark pull request #22319: [SPARK-25044][SQL][followup] add back UserDefined...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22319#discussion_r214577826 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -41,12 +41,18 @@ import

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95598/ Test FAILed. ---

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22319 **[Test build #95598 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95598/testReport)** for PR 22319 at commit

[GitHub] spark issue #22313: [SPARK-25306][SQL] Use cache to speed up `createFilter` ...

2018-09-02 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22313 Do you know why `createFilter` function has exponential time complexity? Let's make sure the algorithm is good before adding the cache. ---

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22319 **[Test build #95598 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95598/testReport)** for PR 22319 at commit

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22319 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22319: [SPARK-25044][SQL][followup] add back UserDefinedFunctio...

2018-09-02 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22319 cc @srowen @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22319: [SPARK-25044][SQL][followup] add back UserDefined...

2018-09-02 Thread cloud-fan
GitHub user cloud-fan opened a pull request: https://github.com/apache/spark/pull/22319 [SPARK-25044][SQL][followup] add back UserDefinedFunction.inputTypes ## What changes were proposed in this pull request? This is a followup of https://github.com/apache/spark/pull/22259

[GitHub] spark issue #22307: [SPARK-25301][SQL] When a view uses an UDF from a non de...

2018-09-02 Thread vinodkc
Github user vinodkc commented on the issue: https://github.com/apache/spark/pull/22307 @HyukjinKwon , even with this ```create function d100.udf100 as 'org.apache.hadoop.hive.ql.udf.generic.GenericUDFUpper'; ``` we can simulate this issue. I've updated PR description. ---

[GitHub] spark issue #22317: [SPARK-25310][SQL] ArraysOverlap may throw a Compilation...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22317 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22317: [SPARK-25310][SQL] ArraysOverlap may throw a Compilation...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22317 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/95594/ Test PASSed. ---

[GitHub] spark issue #22317: [SPARK-25310][SQL] ArraysOverlap may throw a Compilation...

2018-09-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22317 **[Test build #95594 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95594/testReport)** for PR 22317 at commit

[GitHub] spark pull request #22010: [SPARK-21436][CORE] Take advantage of known parti...

2018-09-02 Thread holdenk
Github user holdenk commented on a diff in the pull request: https://github.com/apache/spark/pull/22010#discussion_r214575455 --- Diff: core/src/main/scala/org/apache/spark/rdd/RDD.scala --- @@ -396,7 +396,16 @@ abstract class RDD[T: ClassTag]( * Return a new RDD containing

[GitHub] spark issue #22311: [SPARK-25305][SQL] Respect attribute name in CollapsePro...

2018-09-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22311 ok, thanks. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214573505 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala --- @@ -295,4 +295,17 @@ class DataFrameJoinSuite extends QueryTest with

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214573271 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala --- @@ -295,4 +295,17 @@ class DataFrameJoinSuite extends QueryTest with

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22318#discussion_r214573288 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameJoinSuite.scala --- @@ -295,4 +295,17 @@ class DataFrameJoinSuite extends QueryTest with

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22318 Could you describe more in the PR description?; what's the root cause of this issue? How did you solve this by this pr? --- - To

[GitHub] spark issue #22281: [SPARK-25280][SQL] Add support for USING syntax for Data...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22281 Would you guys mind if I ask to point out some concerns that I might better have to double check by myself? --- - To

[GitHub] spark issue #18447: [SPARK-21232][SQL][SparkR][PYSPARK] New built-in SQL fun...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/18447 ping @mmolimar to close --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #22234: [SPARK-25241][SQL] Configurable empty values when...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22234#discussion_r214572617 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVInferSchema.scala --- @@ -79,7 +79,8 @@ private[csv] object

[GitHub] spark pull request #22234: [SPARK-25241][SQL] Configurable empty values when...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22234#discussion_r214572625 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVDataSource.scala --- @@ -91,9 +91,10 @@ abstract class

[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/6#discussion_r214572336 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonUtils.scala --- @@ -32,29 +32,29 @@ object JacksonUtils { }

[GitHub] spark pull request #22226: [SPARK-25252][SQL] Support arrays of any types by...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/6#discussion_r214572178 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/json/JacksonGenerator.scala --- @@ -43,20 +42,22 @@ private[sql] class

[GitHub] spark issue #22311: [SPARK-25305][SQL] Respect attribute name in CollapsePro...

2018-09-02 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22311 > This behaivour depends on spark.sql.caseSensitive? No. It's writing not resolving a column, so Spark should be case-preserving. ---

[GitHub] spark pull request #22311: [SPARK-25305][SQL] Respect attribute name in Coll...

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22311#discussion_r214570600 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala --- @@ -515,8 +515,7 @@ object PushProjectionThroughUnion

[GitHub] spark issue #22311: [SPARK-25305][SQL] Respect attribute name in CollapsePro...

2018-09-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22311 This behaivour depends on `spark.sql.caseSensitive`? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-02 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/22112 Update, according to the discussion in https://github.com/apache/spark/pull/9214 , the current behavior of shuffle writing is: "first write wins". We can't simply change it to "last write wins",

[GitHub] spark issue #21273: [SPARK-17916][SQL] Fix empty string being parsed as null...

2018-09-02 Thread koertkuipers
Github user koertkuipers commented on the issue: https://github.com/apache/spark/pull/21273 it would provide a workaround i think, yes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #22063: [WIP][SPARK-25044][SQL] Address translation of LM...

2018-09-02 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/22063#discussion_r214567945 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -40,7 +41,7 @@ import

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22112 **[Test build #95597 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95597/testReport)** for PR 22112 at commit

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22112 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22112: [SPARK-23243][Core] Fix RDD.repartition() data correctne...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22112 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22308: [SPARK-25304][SQL][TEST] Fix HiveSparkSubmitSuite...

2018-09-02 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/spark/pull/22308 --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark issue #22308: [SPARK-25304][SQL][TEST] Fix HiveSparkSubmitSuite SPARK-...

2018-09-02 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/22308 Merged to master --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #21330: [SPARK-22234] Support distinct window functions

2018-09-02 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/21330 If this feature is interested, could you please help start the review @jiangxb1987 Thanks a lot. --- - To unsubscribe,

[GitHub] spark pull request #22316: [SPARK-25048][SQL] Pivoting by multiple columns i...

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22316#discussion_r214566503 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFramePivotSuite.scala --- @@ -308,4 +308,27 @@ class DataFramePivotSuite extends QueryTest with

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214566285 --- Diff: R/pkg/R/functions.R --- @@ -3410,13 +3410,15 @@ setMethod("collect_set", #' \dontrun{ #' head(select(df, split_string(df$Sex, "a")))

[GitHub] spark pull request #22316: [SPARK-25048][SQL] Pivoting by multiple columns i...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22316#discussion_r214566083 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/RelationalGroupedDataset.scala --- @@ -406,6 +407,14 @@ class RelationalGroupedDataset

[GitHub] spark issue #22302: [SPARK-21786][SQL][FOLLOWUP] Add compressionCodec test f...

2018-09-02 Thread fjh100456
Github user fjh100456 commented on the issue: https://github.com/apache/spark/pull/22302 @maropu I'd update the PR description, thank you! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #22314: [SPARK-25307][SQL] ArraySort function may return ...

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22314#discussion_r214565371 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1047,7 +1047,8 @@ trait ArraySortLike

[GitHub] spark pull request #22314: [SPARK-25307][SQL] ArraySort function may return ...

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22314#discussion_r214565311 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1047,7 +1047,8 @@ trait ArraySortLike

[GitHub] spark issue #22306: [SPARK-25300][CORE]Unified the configuration parameter `...

2018-09-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22306 **[Test build #95596 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95596/testReport)** for PR 22306 at commit

[GitHub] spark issue #22310: [Spark-25298][Build] Improve build definition for Scala ...

2018-09-02 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/22310 **[Test build #95595 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/95595/testReport)** for PR 22310 at commit

[GitHub] spark issue #22306: [SPARK-25300][CORE]Unified the configuration parameter `...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22306 Test PASSed. Refer to this link for build results (access rights to CI server needed):

[GitHub] spark issue #22306: [SPARK-25300][CORE]Unified the configuration parameter `...

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22306 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22310: [Spark-25298][Build] Improve build definition for Scala ...

2018-09-02 Thread sadhen
Github user sadhen commented on the issue: https://github.com/apache/spark/pull/22310 The problem of package hierarchy is fixed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #22315: [SPARK-25308][SQL] ArrayContains function may ret...

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22315#discussion_r214564677 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1464,17 +1464,35 @@ case class

[GitHub] spark pull request #22270: [SPARK-25267][SQL][TEST] Disable ConvertToLocalRe...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/22270#discussion_r214564426 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/DataFrameFunctionsSuite.scala --- @@ -85,12 +85,12 @@ class DataFrameFunctionsSuite extends

[GitHub] spark pull request #22315: [SPARK-25308][SQL] ArrayContains function may ret...

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22315#discussion_r214564353 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala --- @@ -1464,17 +1464,35 @@ case class

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22318 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22318 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #22310: [Spark-25298][Build] Improve build definition for Scala ...

2018-09-02 Thread sadhen
Github user sadhen commented on the issue: https://github.com/apache/spark/pull/22310 @srowen Sorry I should have explained why I made these changes. The follow steps failed to compile: ``` $ ./dev/change-scala-version.sh 2.12 $ ./build/sbt -Dscala-2.12

[GitHub] spark issue #22312: [SPARK-17916][SQL] Fix new behavior when quote is set an...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22312 #22234 was already open. Wouldn't it be able to workaround if it's configurable? --- - To unsubscribe, e-mail:

[GitHub] spark issue #22318: [SPARK-25150][SQL] Fix attribute deduplication in join

2018-09-02 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/22318 Can one of the admins verify this patch? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #21273: [SPARK-17916][SQL] Fix empty string being parsed as null...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21273 https://github.com/apache/spark/pull/22234 was already open. Wouldn't it be able to workaround if it's configurable? --- -

[GitHub] spark pull request #22318: [SPARK-25150][SQL] Fix attribute deduplication in...

2018-09-02 Thread peter-toth
GitHub user peter-toth opened a pull request: https://github.com/apache/spark/pull/22318 [SPARK-25150][SQL] Fix attribute deduplication in join ## What changes were proposed in this pull request? Fixes attribute deduplication in join conditions. ## How was this

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214563659 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala --- @@ -229,36 +229,58 @@ case class

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214563339 --- Diff: R/pkg/R/functions.R --- @@ -3410,13 +3410,15 @@ setMethod("collect_set", #' \dontrun{ #' head(select(df, split_string(df$Sex, "a")))

[GitHub] spark issue #22308: [SPARK-25304][SQL][TEST] Fix HiveSparkSubmitSuite SPARK-...

2018-09-02 Thread sadhen
Github user sadhen commented on the issue: https://github.com/apache/spark/pull/22308 @srowen The 2.12 jar is compiled and packaged from `Main.scala` and `MyCoolClass.scala`. Not a copy of 2.10 jar. Diff it, you will verify it. The steps to generate it: ``` mvn

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214562895 --- Diff: sql/core/src/test/resources/sql-tests/inputs/string-functions.sql --- @@ -46,4 +46,10 @@ FROM ( encode(string(id + 2), 'utf-8')

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214562760 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala --- @@ -229,36 +229,58 @@ case class

[GitHub] spark issue #22300: [SPARK-25296][SQL][TEST] Create ExplainSuite

2018-09-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22300 @kiszk ping, could you do that? https://github.com/apache/spark/pull/22300#issuecomment-417706754 --- - To unsubscribe, e-mail:

[GitHub] spark issue #22198: [SPARK-25121][SQL] Supports multi-part table names for b...

2018-09-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22198 ping --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #22204: [SPARK-25196][SQL] Analyze column statistics in cached q...

2018-09-02 Thread maropu
Github user maropu commented on the issue: https://github.com/apache/spark/pull/22204 @dongjoon-hyun could you check again? thanks! (btw, congrats, committer!) --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214562525 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/regexpExpressions.scala --- @@ -229,36 +229,58 @@ case class

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214562493 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -952,6 +952,11 @@ public static UTF8String

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214562429 --- Diff: common/unsafe/src/test/java/org/apache/spark/unsafe/types/UTF8StringSuite.java --- @@ -394,12 +394,14 @@ public void substringSQL() {

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214562374 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/types/UTF8String.java --- @@ -952,6 +952,11 @@ public static UTF8String concatWs(UTF8String

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214562388 --- Diff: R/pkg/tests/fulltests/test_sparkSQL.R --- @@ -1803,6 +1803,18 @@ test_that("string operators", { collect(select(df4,

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214562340 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2546,15 +2546,39 @@ object functions { def soundex(e: Column):

[GitHub] spark issue #22306: [SPARK-25300][CORE]Unified the configuration parameter `...

2018-09-02 Thread 10110346
Github user 10110346 commented on the issue: https://github.com/apache/spark/pull/22306 Thanks,I will apply them to test cases @kiszk --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214562121 --- Diff: python/pyspark/sql/functions.py --- @@ -1669,20 +1669,33 @@ def repeat(col, n): return

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214562034 --- Diff: python/pyspark/sql/functions.py --- @@ -1669,20 +1669,33 @@ def repeat(col, n): return

[GitHub] spark pull request #22218: [SPARK-25228][CORE]Add executor CPU time metric.

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22218#discussion_r214561873 --- Diff: core/src/main/scala/org/apache/spark/executor/ExecutorSource.scala --- @@ -73,6 +76,28 @@ class ExecutorSource(threadPool: ThreadPoolExecutor,

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214561691 --- Diff: sql/core/src/test/resources/sql-tests/inputs/string-functions.sql --- @@ -46,4 +46,10 @@ FROM ( encode(string(id + 2), 'utf-8')

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214561410 --- Diff: python/pyspark/sql/functions.py --- @@ -1669,20 +1669,33 @@ def repeat(col, n): return

[GitHub] spark pull request #22227: [SPARK-25202] [SQL] Implements split with limit s...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/7#discussion_r214561362 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/functions.scala --- @@ -2546,15 +2546,39 @@ object functions { def soundex(e: Column):

[GitHub] spark pull request #22218: [SPARK-25228][CORE]Add executor CPU time metric.

2018-09-02 Thread maropu
Github user maropu commented on a diff in the pull request: https://github.com/apache/spark/pull/22218#discussion_r214561269 --- Diff: core/src/main/scala/org/apache/spark/executor/ExecutorSource.scala --- @@ -73,6 +75,29 @@ class ExecutorSource(threadPool: ThreadPoolExecutor,

[GitHub] spark issue #22307: [SPARK-25301][SQL] When a view uses an UDF from a non de...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22307 The problem here looks some inconsistency between Hive and Spark - since Spark claims Hive compatibility, looks we should either explain the difference or fix it. ---

[GitHub] spark issue #22307: [SPARK-25301][SQL] When a view uses an UDF from a non de...

2018-09-02 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/22307 @vinodkc, do you have the JAR for `/usr/udf/masking.jar`? Want to reproduce and check. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #22063: [WIP][SPARK-25044][SQL] Address translation of LM...

2018-09-02 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22063#discussion_r214560630 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -40,7 +41,7 @@ import

[GitHub] spark pull request #22063: [WIP][SPARK-25044][SQL] Address translation of LM...

2018-09-02 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/22063#discussion_r214560519 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -40,7 +41,7 @@ import

[GitHub] spark pull request #22063: [WIP][SPARK-25044][SQL] Address translation of LM...

2018-09-02 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/22063#discussion_r214560313 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/expressions/UserDefinedFunction.scala --- @@ -40,7 +41,7 @@ import

  1   2   >