yjshen commented on PR #2217:
URL:
https://github.com/apache/arrow-datafusion/pull/2217#issuecomment-1098090215
Another physical plan description optimization would be to limit each
physical plan's max line width. This would be especially useful for the
ParquetExec node:
```
ProjectionExec: expr=[ss_sold_date_sk@0 as ss_sold_date_sk, ss_customer_sk@1
as ss_customer_sk]
FilterExec: ss_sold_date_sk@0 IS NOT NULL
ParquetExec: limit=None,
partitions=[/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00000-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00023-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00012-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00001-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00024-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00030-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00008-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.
parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00056-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00010-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00078-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00028-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00016-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00017-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00002-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-bl
aze-extension/dev/tpcds_1g/store_sales/part-00021-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00027-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00032-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00031-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00034-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00014-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00005-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales
/part-00026-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00019-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00011-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00013-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00089-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00020-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00029-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00004-8ecba12f-a127-4bca-8d50-f9
516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00015-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00007-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00018-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00067-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00022-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00009-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00006-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users
/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00045-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00033-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00003-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00025-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00035-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00096-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00062-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/d
ev/tpcds_1g/store_sales/part-00068-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00066-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00072-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00077-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00086-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00061-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00063-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00052-8ec
ba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00082-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00075-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00087-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00059-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00070-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00053-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00050-8ecba12f-a127-4bca-8d50-f9516808f050-c000
.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00084-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00085-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00049-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00080-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00038-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00046-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00074-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/
spark-blaze-extension/dev/tpcds_1g/store_sales/part-00043-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00076-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00093-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00058-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00039-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00090-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00036-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/sto
re_sales/part-00083-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00051-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00079-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00092-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00055-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00081-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00088-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00071-8ecba12f-a127-4bca
-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00044-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00042-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00069-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00048-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00054-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00095-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00065-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet
,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00047-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00094-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00037-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00073-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00064-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00057-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00041-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-ext
ension/dev/tpcds_1g/store_sales/part-00060-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00091-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00040-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet,
/Users/shenyijie/oss/spark-blaze-extension/dev/tpcds_1g/store_sales/part-00097-8ecba12f-a127-4bca-8d50-f9516808f050-c000.snappy.parquet],
predicate=true, projection=[ss_sold_date_sk, ss_customer_sk]
```
It would be wonderful if I could reduce this to:
```
ParquetExec: limit=None, predicate=true, projection=[ss_sold_date_sk,
ss_customer_sk], partitions=[100 items ... ]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
