sgrebnov commented on PR #13267: URL: https://github.com/apache/datafusion/pull/13267#issuecomment-2460507776
> Would this be equivalent to disabling `optimize_projections`?
@findepi - the logic of `optimize_projections` is still working in this
case pushing down only required columns, for example
```sql
explain SELECT ss_item_sk, trip_distance_mi, total_amount
FROM store_sales
JOIN taxi_trips
ON store_sales.ss_quantity = taxi_trips.passenger_count
WHERE taxi_trips.total_amount=3.0 and ss_item_sk= 180
ORDER BY trip_distance_mi DESC LIMIT 10;
```
Without `optimize_projections`
```console
| logical_plan | Sort: taxi_trips.trip_distance_mi DESC NULLS FIRST,
fetch=10
|
| | Projection: store_sales.ss_item_sk,
taxi_trips.trip_distance_mi, taxi_trips.total_amount
|
| | Inner Join: CAST(store_sales.ss_quantity AS Int64) =
taxi_trips.passenger_count
|
| | Projection: store_sales.ss_item_sk,
store_sales.ss_quantity
|
| | BytesProcessedNode
|
| | Federated
|
| | Projection: store_sales.ss_sold_date_sk,
store_sales.ss_sold_time_sk, store_sales.ss_item_sk,
store_sales.ss_customer_sk, store_sales.ss_cdemo_sk, store_sales.ss_hdemo_sk,
store_sales.ss_addr_sk, store_sales.ss_store_sk, store_sales.ss_promo_sk,
store_sales.ss_ticket_number, store_sales.ss_quantity,
store_sales.ss_wholesale_cost, store_sales.ss_list_price,
store_sales.ss_sales_price, store_sales.ss_ext_discount_amt,
store_sales.ss_ext_sales_price, store_sales.ss_ext_wholesale_cost,
store_sales.ss_ext_list_price, store_sales.ss_ext_tax,
store_sales.ss_coupon_amt, store_sales.ss_net_paid,
store_sales.ss_net_paid_inc_tax, store_sales.ss_net_profit
|
| | Filter: CAST(store_sales.ss_item_sk AS Int64) =
Int64(180)
|
| | TableScan: store_sales,
partial_filters=[CAST(store_sales.ss_item_sk AS Int64) = Int64(180)]
|
| | Projection: taxi_trips.passenger_count,
taxi_trips.trip_distance_mi, taxi_trips.total_amount
|
| | BytesProcessedNode
|
| | Federated
|
| | Projection: taxi_trips.pickup_datetime,
taxi_trips.passenger_count, taxi_trips.trip_distance_mi,
taxi_trips.fare_amount, taxi_trips.tip_amount, taxi_trips.total_amount
|
| | TableScan: taxi_trips,
full_filters=[taxi_trips.total_amount = Float64(3)]
|
```
With `optimize_projections` and
`optimize_projections_preserve_existing_projections `
```console
| logical_plan | Sort: taxi_trips.trip_distance_mi DESC NULLS FIRST,
fetch=10
|
| | Projection: store_sales.ss_item_sk,
taxi_trips.trip_distance_mi, taxi_trips.total_amount
|
| | Inner Join: CAST(store_sales.ss_quantity AS Int64) =
taxi_trips.passenger_count
|
| | BytesProcessedNode
|
| | Federated
|
| | Projection: store_sales.ss_item_sk,
store_sales.ss_quantity
|
| | Filter: CAST(store_sales.ss_item_sk AS Int64) =
Int64(180)
|
| | TableScan: store_sales projection=[ss_item_sk,
ss_quantity], partial_filters=[CAST(store_sales.ss_item_sk AS Int64) =
Int64(180)]
|
| | BytesProcessedNode
|
| | Federated
|
| | Projection: taxi_trips.passenger_count,
taxi_trips.trip_distance_mi, taxi_trips.total_amount
|
| | TableScan: taxi_trips projection=[passenger_count,
trip_distance_mi, total_amount], full_filters=[taxi_trips.total_amount =
Float64(3)]
|
```
Before
```
Projection: store_sales.ss_sold_date_sk, store_sales.ss_sold_time_sk,
store_sales.ss_item_sk, store_sales.ss_customer_sk, store_sales.ss_cdemo_sk,
store_sales.ss_hdemo_sk, store_sales.ss_addr_sk, store_sales.ss_store_sk,
store_sales.ss_promo_sk, store_sales.ss_ticket_number, store_sales.ss_quantity,
store_sales.ss_wholesale_cost, store_sales.ss_list_price,
store_sales.ss_sales_price, store_sales.ss_ext_discount_amt,
store_sales.ss_ext_sales_price, store_sales.ss_ext_wholesale_cost,
store_sales.ss_ext_list_price, store_sales.ss_ext_tax,
store_sales.ss_coupon_amt, store_sales.ss_net_paid,
store_sales.ss_net_paid_inc_tax, store_sales.ss_net_profit
|
| | Filter: CAST(store_sales.ss_item_sk AS Int64) =
Int64(180)
|
| | TableScan: store_sales,
partial_filters=[CAST(store_sales.ss_item_sk AS Int64) = Int64(180)]
```
After:
```
| | Projection: store_sales.ss_item_sk,
store_sales.ss_quantity
|
| | Filter: CAST(store_sales.ss_item_sk AS Int64) =
Int64(180)
|
| | TableScan: store_sales projection=[ss_item_sk,
ss_quantity], partial_filters=[CAST(store_sales.ss_item_sk AS Int64) =
Int64(180)]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org
For additional commands, e-mail: github-h...@datafusion.apache.org
