[GitHub] spark pull request #20483: [SPARK-23312][SQL] add a config to turn off vecto...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/spark/pull/20483


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20483: [SPARK-23312][SQL] add a config to turn off vecto...

[cloud-fan]

Github user cloud-fan commented on a diff in the pull request:

https://github.com/apache/spark/pull/20483#discussion_r165609895
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -141,6 +141,12 @@ object SQLConf {
   .booleanConf
   .createWithDefault(true)
 
+  val CACHE_VECTORIZED_READER_ENABLED =
+buildConf("spark.sql.inMemoryColumnarStorage.enableVectorizedReader")
+  .doc("Enables vectorized reader for columnar caching.")
+  .booleanConf
+  .createWithDefault(true)
--- End diff --

the parquet/orc vectorized reader conf is also public.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20483: [SPARK-23312][SQL] add a config to turn off vecto...

[viirya]

Github user viirya commented on a diff in the pull request:

https://github.com/apache/spark/pull/20483#discussion_r165581115
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -141,6 +141,12 @@ object SQLConf {
   .booleanConf
   .createWithDefault(true)
 
+  val CACHE_VECTORIZED_READER_ENABLED =
+buildConf("spark.sql.inMemoryColumnarStorage.enableVectorizedReader")
+  .doc("Enables vectorized reader for columnar caching.")
+  .booleanConf
+  .createWithDefault(true)
--- End diff --

`internal`?


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20483: [SPARK-23312][SQL] add a config to turn off vecto...

[dongjoon-hyun]

Github user dongjoon-hyun commented on a diff in the pull request:

https://github.com/apache/spark/pull/20483#discussion_r165569731
  
--- Diff: 
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -141,6 +141,12 @@ object SQLConf {
   .booleanConf
   .createWithDefault(true)
 
+  val CACHE_VECTORIZED_READER_ENABLED =
+buildConf("spark.sql.inMemoryColumnarStorage.enableVectorizedReader")
+  .doc("Enables vectorized reader for columnar caching.")
+  .booleanConf
+  .createWithDefault(true)
--- End diff --

To unblock 2.3, I think we need to disable this.


---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org

[GitHub] spark pull request #20483: [SPARK-23312][SQL] add a config to turn off vecto...

[cloud-fan]

GitHub user cloud-fan opened a pull request:

https://github.com/apache/spark/pull/20483

[SPARK-23312][SQL] add a config to turn off vectorized cache reader

## What changes were proposed in this pull request?

https://issues.apache.org/jira/browse/SPARK-23309 reported a performance 
regression about cached table in Spark 2.3. While the investigating is still 
going on, this PR adds a conf to turn off the vectorized cache reader, to 
unblock the 2.3 release.

## How was this patch tested?

a new test

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cloud-fan/spark cache

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/spark/pull/20483.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #20483


commit 376c85526f4ec2328f2d05a0816f632c7dd9379f
Author: Wenchen Fan 
Date:   2018-02-02T05:55:01Z

add a config to turn off vectorized cache reader




---

-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org