[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19601 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19601 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83179/ Test PASSed. ---

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19601 **[Test build #83179 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83179/testReport)** for PR 19601 at commit

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r147571324 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java --- @@ -19,28 +19,48 @@ import

[GitHub] spark pull request #19602: [SPARK-22384][SQL] Refine partition pruning when ...

2017-10-29 Thread gatorsmile
Github user gatorsmile commented on a diff in the pull request: https://github.com/apache/spark/pull/19602#discussion_r147572381 --- Diff: sql/hive/src/test/scala/org/apache/spark/sql/hive/client/HiveClientSuite.scala --- @@ -53,7 +52,7 @@ class HiveClientSuite(version: String)

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19601 @Ueshin @cloud-fan could you please review this? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r147573354 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java --- @@ -19,28 +19,48 @@ import

[GitHub] spark issue #19567: [SPARK-22291][SQL] Conversion error when transforming ar...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19567 **[Test build #83183 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83183/testReport)** for PR 19567 at commit

[GitHub] spark issue #19563: [SPARK-22284][SQL] Fix 64KB JVM bytecode limit problem i...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19563 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19567: [SPARK-22291][SQL] Conversion error when transforming ar...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19567 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19563: [SPARK-22284][SQL] Fix 64KB JVM bytecode limit problem i...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19563 **[Test build #83185 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83185/testReport)** for PR 19563 at commit

[GitHub] spark issue #19563: [SPARK-22284][SQL] Fix 64KB JVM bytecode limit problem i...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19563 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83185/ Test PASSed. ---

[GitHub] spark issue #19567: [SPARK-22291][SQL] Conversion error when transforming ar...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19567 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83183/ Test PASSed. ---

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15049 **[Test build #83186 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83186/testReport)** for PR 15049 at commit

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17899 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17899 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83193/ Test FAILed. ---

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17899 **[Test build #83193 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83193/testReport)** for PR 17899 at commit

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19601 I'd like to also improve the write path. I think the current way to cache array type is not efficient, arrow-like format which put all elements(including nested array) together is better for

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19601 both ways work, just pick the simpler one. I'm concerned about how to access the nested array, you can try both approaches and see which one can solve the problem easier. ---

[GitHub] spark issue #19592: [SPARK-22347][SQL][PySpark] Support optionally running P...

2017-10-29 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19592 ping @ueshin @BryanCutler @cloud-fan Would you mind to provide some insights? Should we add just a document for it or fix it in your opinions? Thanks. ---

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19601 Current `ColumnVector` uses primitive type array (e.g. `int[]` or `double[]`) based on data type of each column. On the other hand, cached data uses `byte[]` for all data type. Do we change

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17899 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83189/ Test FAILed. ---

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17899 **[Test build #83189 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83189/testReport)** for PR 17899 at commit

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19601 For now, this implementation has an limitation only to support non-nested array for ease of review. --- - To unsubscribe, e-mail:

[GitHub] spark issue #19567: [SPARK-22291][SQL] Conversion error when transforming ar...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19567 **[Test build #83195 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83195/testReport)** for PR 19567 at commit

[GitHub] spark issue #19603: [SPARK-22385][SQL] MapObjects should not access list ele...

2017-10-29 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19603 Good catch! LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2017-10-29 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/19602 @gatorsmile Thanks again for review this pr. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r147573017 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java --- @@ -19,28 +19,48 @@ import

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17899 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19567: [SPARK-22291][SQL] Conversion error when transforming ar...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19567 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r147573145 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/array/LongArray.java --- @@ -33,15 +32,13 @@ private static final long WIDTH = 8;

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r147573154 --- Diff: sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/UnsafeArrayData.java --- @@ -230,7 +233,10 @@ public UTF8String

[GitHub] spark issue #19563: [SPARK-22284][SQL] Fix 64KB JVM bytecode limit problem i...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19563 **[Test build #83185 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83185/testReport)** for PR 19563 at commit

[GitHub] spark issue #19553: [SPARK-22330][CORE] Linear containsKey operation for ser...

2017-10-29 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19553 @Tagar it's the same general type of issue, but not directly related nor exactly the same cause. --- - To unsubscribe, e-mail:

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r147576840 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java --- @@ -19,28 +19,48 @@ import

[GitHub] spark issue #19529: [SPARK-22308] Support alternative unit testing styles in...

2017-10-29 Thread srowen
Github user srowen commented on the issue: https://github.com/apache/spark/pull/19529 @nkronenfeld @gatorsmile I think this has been failing the master build (Maven only) for a few days:

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15049 **[Test build #83184 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83184/testReport)** for PR 15049 at commit

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15049 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83184/ Test PASSed. ---

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15049 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19567: [SPARK-22291][SQL] Conversion error when transfor...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19567#discussion_r147578399 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala --- @@ -456,8 +456,10 @@ object JdbcUtils extends

[GitHub] spark pull request #19567: [SPARK-22291][SQL] Conversion error when transfor...

2017-10-29 Thread jmchung
Github user jmchung commented on a diff in the pull request: https://github.com/apache/spark/pull/19567#discussion_r147579149 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala --- @@ -456,8 +456,10 @@ object JdbcUtils extends

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17899 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/17899 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83191/ Test FAILed. ---

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17899 **[Test build #83191 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83191/testReport)** for PR 17899 at commit

[GitHub] spark pull request #19603: [SPARK-22385][SQL] MapObjects should not access l...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19603#discussion_r147579950 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -591,18 +591,40 @@ case class MapObjects

[GitHub] spark issue #18251: [SPARK-17788][SPARK-21033][SQL] fix the potential OOM in...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18251 **[Test build #83187 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83187/testReport)** for PR 18251 at commit

[GitHub] spark issue #18251: [SPARK-17788][SPARK-21033][SQL] fix the potential OOM in...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18251 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83180/ Test FAILed. ---

[GitHub] spark issue #18251: [SPARK-17788][SPARK-21033][SQL] fix the potential OOM in...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18251 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18251: [SPARK-17788][SPARK-21033][SQL] fix the potential OOM in...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18251 **[Test build #83180 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83180/testReport)** for PR 18251 at commit

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19602 Merged build finished. Test FAILed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19602 **[Test build #83181 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83181/testReport)** for PR 19602 at commit

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19602 Test FAILed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83181/ Test FAILed. ---

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r147573035 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/OffHeapMemoryBlock.java --- @@ -0,0 +1,103 @@ +/* + * Licensed to the Apache

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/15049 **[Test build #83184 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83184/testReport)** for PR 15049 at commit

[GitHub] spark issue #19567: [SPARK-22291][SQL] Conversion error when transforming ar...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19567 **[Test build #83183 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83183/testReport)** for PR 19567 at commit

[GitHub] spark pull request #15049: [SPARK-17310][SQL] Add an option to disable recor...

2017-10-29 Thread HyukjinKwon
Github user HyukjinKwon commented on a diff in the pull request: https://github.com/apache/spark/pull/15049#discussion_r147573054 --- Diff: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilterSuite.scala --- @@ -555,6 +572,32 @@ class

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r147573040 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/HeapMemoryAllocator.java --- @@ -73,6 +73,12 @@ public MemoryBlock allocate(long size)

[GitHub] spark pull request #19563: [SPARK-22284][SQL] Fix 64KB JVM bytecode limit pr...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19563#discussion_r147573635 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/hash.scala --- @@ -389,9 +389,10 @@ abstract class HashExpression[E] extends

[GitHub] spark pull request #19563: [SPARK-22284][SQL] Fix 64KB JVM bytecode limit pr...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19563#discussion_r147573624 --- Diff: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/HashExpressionsSuite.scala --- @@ -639,6 +639,63 @@ class

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19602 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83182/ Test PASSed. ---

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19602 **[Test build #83182 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83182/testReport)** for PR 19602 at commit

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19602 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19601 My feeling is that, we should change the cache format of array type to make it compatible with `ColumnVector`, then we don't need conversion from cached data to columnar batch. ---

[GitHub] spark pull request #19601: [SPARK-22383][SQL] Generate code to directly get ...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19601#discussion_r147578474 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVector.java --- @@ -367,9 +551,13 @@ public Object get(int ordinal,

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17899 **[Test build #83189 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83189/testReport)** for PR 17899 at commit

[GitHub] spark issue #19567: [SPARK-22291][SQL] Conversion error when transforming ar...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19567 **[Test build #83190 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83190/testReport)** for PR 19567 at commit

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread kiszk
Github user kiszk commented on the issue: https://github.com/apache/spark/pull/19601 I agree with you that we need to improve the write path. It will be addressed after improving the frequently-executed read path, as you suggested before. It will be addressed by the following PR.

[GitHub] spark pull request #19567: [SPARK-22291][SQL] Conversion error when transfor...

2017-10-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19567#discussion_r147581064 --- Diff: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala --- @@ -134,11 +149,28 @@ class

[GitHub] spark pull request #19603: [SPARK-22385][SQL] MapObjects should not access l...

2017-10-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19603#discussion_r147581519 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -591,18 +591,40 @@ case class MapObjects

[GitHub] spark pull request #19603: [SPARK-22385][SQL] MapObjects should not access l...

2017-10-29 Thread srowen
Github user srowen commented on a diff in the pull request: https://github.com/apache/spark/pull/19603#discussion_r147581528 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -591,18 +591,40 @@ case class MapObjects

[GitHub] spark pull request #19603: [SPARK-22385][SQL] MapObjects should not access l...

2017-10-29 Thread viirya
Github user viirya commented on a diff in the pull request: https://github.com/apache/spark/pull/19603#discussion_r147581548 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -591,18 +591,40 @@ case class MapObjects

[GitHub] spark issue #19603: [SPARK-22385][SQL] MapObjects should not access list ele...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19603 **[Test build #83188 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83188/testReport)** for PR 19603 at commit

[GitHub] spark issue #19603: [SPARK-22385][SQL] MapObjects should not access list ele...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19603 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83188/ Test PASSed. ---

[GitHub] spark issue #19603: [SPARK-22385][SQL] MapObjects should not access list ele...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/19603 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19597: [SPARK-22375][TEST] Test script can fail if eggs are ins...

2017-10-29 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19597 @xynny, mind if I ask your JIRA id? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19567: [SPARK-22291][SQL] Conversion error when transforming ar...

2017-10-29 Thread jmchung
Github user jmchung commented on the issue: https://github.com/apache/spark/pull/19567 gentle ping @cloud-fan and @viirya, there are some feedbacks about the behavior of obj to string. ``` scala case StringType => (array: Object) =>

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r147573486 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/MemoryBlock.java --- @@ -17,47 +17,168 @@ package

[GitHub] spark issue #19600: Added more information to Imputer

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19600 **[Test build #3963 has finished](https://amplab.cs.berkeley.edu/jenkins/job/NewSparkPullRequestBuilder/3963/testReport)** for PR 19600 at commit

[GitHub] spark pull request #19553: [SPARK-22330][CORE] Linear containsKey operation ...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19553#discussion_r147576141 --- Diff: core/src/main/scala/org/apache/spark/api/java/JavaUtils.scala --- @@ -43,10 +43,15 @@ private[spark] object JavaUtils { override

[GitHub] spark issue #18251: [SPARK-17788][SPARK-21033][SQL] fix the potential OOM in...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/18251 **[Test build #83187 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83187/testReport)** for PR 18251 at commit

[GitHub] spark issue #19603: [SPARK-22385][SQL] MapObjects should not access list ele...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19603 **[Test build #83188 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83188/testReport)** for PR 19603 at commit

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15049 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark pull request #19601: [SPARK-22383][SQL] Generate code to directly get ...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19601#discussion_r147578600 --- Diff: sql/core/src/main/java/org/apache/spark/sql/execution/vectorized/ColumnVector.java --- @@ -367,9 +551,13 @@ public Object get(int ordinal,

[GitHub] spark issue #15049: [SPARK-17310][SQL] Add an option to disable record-level...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/15049 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83186/ Test PASSed. ---

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19601 So for primitive types, we encode and compress them to binary. When reading cached data, they are decoded to primitive array and can be put in `OnHeadColumnVector` directly. For

[GitHub] spark pull request #19603: [SPARK-22385][SQL] MapObjects should not access l...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19603#discussion_r147580091 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -591,18 +591,40 @@ case class MapObjects

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17899 **[Test build #83193 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83193/testReport)** for PR 17899 at commit

[GitHub] spark issue #19567: [SPARK-22291][SQL] Conversion error when transforming ar...

2017-10-29 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19567 LGTM too --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark pull request #19567: [SPARK-22291][SQL] Conversion error when transfor...

2017-10-29 Thread jmchung
Github user jmchung commented on a diff in the pull request: https://github.com/apache/spark/pull/19567#discussion_r147581349 --- Diff: external/docker-integration-tests/src/test/scala/org/apache/spark/sql/jdbc/PostgresIntegrationSuite.scala --- @@ -134,11 +149,28 @@ class

[GitHub] spark issue #18251: [SPARK-17788][SPARK-21033][SQL] fix the potential OOM in...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18251 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/83187/ Test PASSed. ---

[GitHub] spark issue #18251: [SPARK-17788][SPARK-21033][SQL] fix the potential OOM in...

2017-10-29 Thread AmplabJenkins
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/18251 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2017-10-29 Thread jinxing64
Github user jinxing64 commented on the issue: https://github.com/apache/spark/pull/19602 @gatorsmile Thanks a lot for your help :) >Can we just evaluate the right side CAST(2017 as STRING), since it is foldable? Do you mean to add a new rule ? -- cast the type

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19602 **[Test build #83182 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83182/testReport)** for PR 19602 at commit

[GitHub] spark issue #19597: [SPARK-22375][TEST] Test script can fail if eggs are ins...

2017-10-29 Thread HyukjinKwon
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/19597 Merged to master. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

[GitHub] spark issue #19602: [SPARK-22384][SQL] Refine partition pruning when attribu...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/19602 **[Test build #83181 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83181/testReport)** for PR 19602 at commit

[GitHub] spark pull request #19222: [SPARK-10399][CORE][SQL] Introduce multiple Memor...

2017-10-29 Thread kiszk
Github user kiszk commented on a diff in the pull request: https://github.com/apache/spark/pull/19222#discussion_r147578071 --- Diff: common/unsafe/src/main/java/org/apache/spark/unsafe/memory/UnsafeMemoryAllocator.java --- @@ -19,28 +19,48 @@ import

[GitHub] spark issue #19601: [SPARK-22383][SQL] Generate code to directly get value o...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/19601 Can we use `OffHeapColumnVector` for cached data? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For

[GitHub] spark pull request #19603: [SPARK-22385][SQL] MapObjects should not access l...

2017-10-29 Thread cloud-fan
Github user cloud-fan commented on a diff in the pull request: https://github.com/apache/spark/pull/19603#discussion_r147580056 --- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/objects/objects.scala --- @@ -591,18 +591,40 @@ case class MapObjects

[GitHub] spark issue #17899: [SPARK-20636] Add new optimization rule to transpose adj...

2017-10-29 Thread SparkQA
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/17899 **[Test build #83194 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/83194/testReport)** for PR 17899 at commit

[GitHub] spark issue #19567: [SPARK-22291][SQL] Conversion error when transforming ar...

2017-10-29 Thread viirya
Github user viirya commented on the issue: https://github.com/apache/spark/pull/19567 LGTM --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail:

  1   2   3   >