[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray
[ https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242274#comment-17242274 ] Apache Spark commented on SPARK-33544: -- User 'HyukjinKwon' has created a pull request for this issue: https://github.com/apache/spark/pull/30570 > explode should not filter when used with CreateArray > > > Key: SPARK-33544 > URL: https://issues.apache.org/jira/browse/SPARK-33544 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.1.0 >Reporter: Thomas Graves >Assignee: Thomas Graves >Priority: Major > Fix For: 3.1.0 > > > https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to > insert a filter for not null and size > 0 when using inner explode/inline. > This is fine in most cases but the extra filter is not needed if the explode > is with a create array and not using Literals (it already handles LIterals). > When this happens you know that the values aren't null and it has a size. It > already handles the empty array. > for instance: > val df = someDF.selectExpr("number", "explode(array(word, col3))") > So in this case we shouldn't be inserting the extra Filter and that filter > can get pushed down into like a parquet reader as well. This is just causing > extra overhead. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray
[ https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238892#comment-17238892 ] Apache Spark commented on SPARK-33544: -- User 'tgravescs' has created a pull request for this issue: https://github.com/apache/spark/pull/30504 > explode should not filter when used with CreateArray > > > Key: SPARK-33544 > URL: https://issues.apache.org/jira/browse/SPARK-33544 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.1.0 >Reporter: Thomas Graves >Priority: Major > > https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to > insert a filter for not null and size > 0 when using inner explode/inline. > This is fine in most cases but the extra filter is not needed if the explode > is with a create array and not using Literals (it already handles LIterals). > When this happens you know that the values aren't null and it has a size. It > already handles the empty array. > for instance: > val df = someDF.selectExpr("number", "explode(array(word, col3))") > So in this case we shouldn't be inserting the extra Filter and that filter > can get pushed down into like a parquet reader as well. This is just causing > extra overhead. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray
[ https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238473#comment-17238473 ] L. C. Hsieh commented on SPARK-33544: - Thanks [~hyukjin.kwon]. Will help review if [~tgraves] create a patch. > explode should not filter when used with CreateArray > > > Key: SPARK-33544 > URL: https://issues.apache.org/jira/browse/SPARK-33544 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.1.0 >Reporter: Thomas Graves >Priority: Major > > https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to > insert a filter for not null and size > 0 when using inner explode/inline. > This is fine in most cases but the extra filter is not needed if the explode > is with a create array and not using Literals (it already handles LIterals). > When this happens you know that the values aren't null and it has a size. It > already handles the empty array. > for instance: > val df = someDF.selectExpr("number", "explode(array(word, col3))") > So in this case we shouldn't be inserting the extra Filter and that filter > can get pushed down into like a parquet reader as well. This is just causing > extra overhead. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray
[ https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238469#comment-17238469 ] Hyukjin Kwon commented on SPARK-33544: -- cc [~viirya] FYI > explode should not filter when used with CreateArray > > > Key: SPARK-33544 > URL: https://issues.apache.org/jira/browse/SPARK-33544 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.1.0 >Reporter: Thomas Graves >Priority: Major > > https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to > insert a filter for not null and size > 0 when using inner explode/inline. > This is fine in most cases but the extra filter is not needed if the explode > is with a create array and not using Literals (it already handles LIterals). > When this happens you know that the values aren't null and it has a size. It > already handles the empty array. > for instance: > val df = someDF.selectExpr("number", "explode(array(word, col3))") > So in this case we shouldn't be inserting the extra Filter and that filter > can get pushed down into like a parquet reader as well. This is just causing > extra overhead. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-33544) explode should not filter when used with CreateArray
[ https://issues.apache.org/jira/browse/SPARK-33544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17238371#comment-17238371 ] Thomas Graves commented on SPARK-33544: --- I'm working on a patch for this. > explode should not filter when used with CreateArray > > > Key: SPARK-33544 > URL: https://issues.apache.org/jira/browse/SPARK-33544 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 3.1.0 >Reporter: Thomas Graves >Priority: Major > > https://issues.apache.org/jira/browse/SPARK-32295 added in an optimization to > insert a filter for not null and size > 0 when using inner explode/inline. > This is fine in most cases but the extra filter is not needed if the explode > is with a create array and not using Literals (it already handles LIterals). > When this happens you know that the values aren't null and it has a size. It > already handles the empty array. > for instance: > val df = someDF.selectExpr("number", "explode(array(word, col3))") > So in this case we shouldn't be inserting the extra Filter and that filter > can get pushed down into like a parquet reader as well. This is just causing > extra overhead. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org