alamb commented on code in PR #26751: URL: https://github.com/apache/datafusion/pull/26751#discussion_r4391722698
########## dev/update_config_docs.sh: ########## @@ -190,14 +192,36 @@ does not apply to user defined data sources and only works when DataFusion has a If you know your data is small, you can set the `datafusion.execution.target_partitions` option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less -than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. +than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether. ```sql SET datafusion.execution.target_partitions = '1'; ``` [`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html +For point lookups, the following techniques also help: Review Comment: Specifically something like - Write Parquet files with smaller row groups and data pages than the default so that less data needs to be read and decoded for each lookup. ... Here is an example of how to write Parquet files with smaller row groups and data pages: ```sql COPY (SELECT * FROM t) TO 't.parquet' OPTIONS ( MAX_ROW_GROUP_SIZE 131072, DATA_PAGE_ROW_COUNT_LIMIT 2048 ); ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
