[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21296 > ... does this push down the column selection into the reading of data Yes, it does. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user koertkuipers commented on the issue: https://github.com/apache/spark/pull/21296 if i do not select a schema (and i use inferSchema), and i do a select for only a few column, does this push down the column selection into the reading of data (for schema inference and for the actual data read)? --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21296 https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/91045/testReport/org.apache.spark.sql.execution.datasources.csv/UnivocityParserSuite/_It_is_not_a_test_it_is_a_sbt_testing_SuiteSelector_/history/ --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user gatorsmile commented on the issue: https://github.com/apache/spark/pull/21296 This PR breaks the tests. Let me revert it. Please resubmit the PR with the fix. Thanks! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21296 thanks, merging to master! --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21296 I added the word `parser` to the feature name because as @HyukjinKwon wrote above we do pruning in type conversion already. This PR enables column pruning by CSV parser only. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21296 Just for clarification, I'm okay. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21296 LGTM except a minor comment about the config name --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user MaxGekk commented on the issue: https://github.com/apache/spark/pull/21296 @cloud-fan @HyukjinKwon Could you look at the PR, please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21296 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90797/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21296 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21296 **[Test build #90797 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90797/testReport)** for PR 21296 at commit [`6ff6d4f`](https://github.com/apache/spark/commit/6ff6d4fda9f7e8ee43d7aa04818204de4c49440b). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21296 **[Test build #90797 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90797/testReport)** for PR 21296 at commit [`6ff6d4f`](https://github.com/apache/spark/commit/6ff6d4fda9f7e8ee43d7aa04818204de4c49440b). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21296 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21296 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90752/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21296 **[Test build #90752 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90752/testReport)** for PR 21296 at commit [`f89eeb7`](https://github.com/apache/spark/commit/f89eeb7f7ba86888ad3f7994577a4d4ebbf09197). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21296 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90751/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21296 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21296 **[Test build #90751 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90751/testReport)** for PR 21296 at commit [`7dcfc7a`](https://github.com/apache/spark/commit/7dcfc7a7664fcd5311cb352f0ea7a24b3cc1c639). * This patch passes all tests. * This patch merges cleanly. * This patch adds the following public classes _(experimental)_: * ` case class WidenSetOperationTypes(conf: SQLConf) extends Rule[LogicalPlan] ` * ` case class FunctionArgumentConversion(conf: SQLConf) extends TypeCoercionRule ` * ` case class CaseWhenCoercion(conf: SQLConf) extends TypeCoercionRule ` * ` case class IfCoercion(conf: SQLConf) extends TypeCoercionRule ` * ` case class ImplicitTypeCasts(conf: SQLConf) extends TypeCoercionRule ` * `class ContinuousMemoryStream[A : Encoder](id: Int, sqlContext: SQLContext, numPartitions: Int = 2)` --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21296 **[Test build #90752 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90752/testReport)** for PR 21296 at commit [`f89eeb7`](https://github.com/apache/spark/commit/f89eeb7f7ba86888ad3f7994577a4d4ebbf09197). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21296 **[Test build #90751 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90751/testReport)** for PR 21296 at commit [`7dcfc7a`](https://github.com/apache/spark/commit/7dcfc7a7664fcd5311cb352f0ea7a24b3cc1c639). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21296 Merged build finished. Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user AmplabJenkins commented on the issue: https://github.com/apache/spark/pull/21296 Test PASSed. Refer to this link for build results (access rights to CI server needed): https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/90555/ Test PASSed. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21296 **[Test build #90555 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90555/testReport)** for PR 21296 at commit [`f90daa7`](https://github.com/apache/spark/commit/f90daa7ea33d119be978c27de10978c2d6281e25). * This patch passes all tests. * This patch merges cleanly. * This patch adds no public classes. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user SparkQA commented on the issue: https://github.com/apache/spark/pull/21296 **[Test build #90555 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/90555/testReport)** for PR 21296 at commit [`f90daa7`](https://github.com/apache/spark/commit/f90daa7ea33d119be978c27de10978c2d6281e25). --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user HyukjinKwon commented on the issue: https://github.com/apache/spark/pull/21296 Can we update the migration guide then? I want to see if the note makes sense. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
[GitHub] spark issue #21296: [SPARK-24244][SQL] Passing only required columns to the ...
Github user cloud-fan commented on the issue: https://github.com/apache/spark/pull/21296 I'm fine with breaking the CSV behavior of malformed input a little, as long as we documented it well in the migration guide. cc @gatorsmile --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
