kbendick commented on issue #4997:
URL: https://github.com/apache/iceberg/issues/4997#issuecomment-1150618527
> @kbendick Hi thanks for the suggestion, I think we can have what you
mentioned as a workaround for now, however, we still want more for it to be
resolved as it is not easy for us to change every SQL query that fall into this
case for our customers.
>
> with that I will ping the reviewer of #3400 here: hey, @rdblue @flyrain
@karuppayya @szehon-ho @RussellSpitzer @aokolnychyi when you get a chance,
could you take a look? thanks.
I agree it's not a good solution for pure SQL users. I'm surprised that
disabling the broadcast join didn't help. The people you tagged would know more
than I would.
But I just ran a full test and using the syntax `select * from db.tbl WHERE
dt between '2022-06-06' - interval 1 days AND '2022-06-06';` the pushdown
worked entirely.
```
spark-sql> create table date_partition_pushdown_test(id bigint, dt date)
using iceberg partitioned by (dt);
spark-sql> insert into date_partition_pushdown_test VALUES(1, date
'2022-06-06'), (2, date '2022-06-05'), (3, date '2022-06-04');
spark-sql> explain extended select * from date_partition_pushdown_test where
dt between '2022-06-06' - interval 1 days AND '2022-06-06'
> ;
== Parsed Logical Plan ==
'Project [*]
+- 'Filter (('dt >= (2022-06-06 - INTERVAL '1' DAY)) AND ('dt <= 2022-06-06))
+- 'UnresolvedRelation [date_partition_pushdown_test], [], false
== Analyzed Logical Plan ==
id: bigint, dt: date
Project [id#6L, dt#7]
+- Filter ((dt#7 >= cast(cast(2022-06-06 - INTERVAL '1' DAY as string) as
date)) AND (dt#7 <= cast(2022-06-06 as date)))
+- SubqueryAlias demo.date_partition_pushdown_test
+- RelationV2[id#6L, dt#7] demo.date_partition_pushdown_test
== Optimized Logical Plan ==
Filter ((isnotnull(dt#7) AND (dt#7 >= 2022-06-05)) AND (dt#7 <= 2022-06-06))
+- RelationV2[id#6L, dt#7] demo.date_partition_pushdown_test
== Physical Plan ==
*(1) Project [id#6L, dt#7]
+- *(1) Filter ((isnotnull(dt#7) AND (dt#7 >= 2022-06-05)) AND (dt#7 <=
2022-06-06))
+- BatchScan[id#6L, dt#7] demo.date_partition_pushdown_test [filters=dt
IS NOT NULL, dt >= 19148, dt <= 19149] RuntimeFilters: []
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
