[GitHub] [arrow-datafusion] andygrove commented on issue #5999: Improve DataFusion scalability as more cores are added

Mon, 17 Apr 2023 07:27:24 -0700 


andygrove commented on issue #5999:
URL: 
https://github.com/apache/arrow-datafusion/issues/5999#issuecomment-1511468857

   Here are the results for just the table scan:
   
   ```sql
   select
       *
   from
        lineitem;
   ```
   
   Cores | DataFusion Python 22.0.0 | DuckDB | DuckDB x Times Faster
   -- | -- | -- | --
   1 | 14501.3 | 14751 | 0.98
   2 | 7562.5 | 8135.8 | 0.93
   4 | 4508.7 | 3618.6 | 1.25
   8 | 2363.2 | 1837.5 | 1.29
   16 | 2135.4 | 888 | 2.40
   32 | 2355 | 511.4 | 4.61
   
   This screenshot shows the DataFusion run on the left and the DuckDB run on 
the right, both using 32 cores.
   
   ![Screenshot from 2023-04-17 
08-22-12](https://user-images.githubusercontent.com/934084/232516149-6086d77d-4f71-49e8-9ad0-e7501982626c.png)
   
   Here is the directory listing of my lineitem directory:
   
   ```
   $ ls -l /mnt/bigdata/tpch/sf10-parquet/lineitem.parquet/
   total 2134532
   -rw-rw-r-- 1 andy andy 91100218 Feb  3 08:45 part-0.parquet
   -rw-rw-r-- 1 andy andy 90998179 Feb  3 08:46 part-10.parquet
   -rw-rw-r-- 1 andy andy 91113498 Feb  3 08:46 part-11.parquet
   -rw-rw-r-- 1 andy andy 90979631 Feb  3 08:46 part-12.parquet
   -rw-rw-r-- 1 andy andy 91042031 Feb  3 08:46 part-13.parquet
   -rw-rw-r-- 1 andy andy 91013004 Feb  3 08:46 part-14.parquet
   -rw-rw-r-- 1 andy andy 91102199 Feb  3 08:46 part-15.parquet
   -rw-rw-r-- 1 andy andy 91113429 Feb  3 08:46 part-16.parquet
   -rw-rw-r-- 1 andy andy 91057291 Feb  3 08:46 part-17.parquet
   -rw-rw-r-- 1 andy andy 90968073 Feb  3 08:46 part-18.parquet
   -rw-rw-r-- 1 andy andy 91085729 Feb  3 08:47 part-19.parquet
   -rw-rw-r-- 1 andy andy 91065849 Feb  3 08:45 part-1.parquet
   -rw-rw-r-- 1 andy andy 90977725 Feb  3 08:47 part-20.parquet
   -rw-rw-r-- 1 andy andy 91092444 Feb  3 08:47 part-21.parquet
   -rw-rw-r-- 1 andy andy 91114853 Feb  3 08:47 part-22.parquet
   -rw-rw-r-- 1 andy andy 91029929 Feb  3 08:47 part-23.parquet
   -rw-rw-r-- 1 andy andy 91139350 Feb  3 08:46 part-2.parquet
   -rw-rw-r-- 1 andy andy 91167180 Feb  3 08:46 part-3.parquet
   -rw-rw-r-- 1 andy andy 91083996 Feb  3 08:46 part-4.parquet
   -rw-rw-r-- 1 andy andy 91024700 Feb  3 08:46 part-5.parquet
   -rw-rw-r-- 1 andy andy 91022201 Feb  3 08:46 part-6.parquet
   -rw-rw-r-- 1 andy andy 91101885 Feb  3 08:46 part-7.parquet
   -rw-rw-r-- 1 andy andy 91198702 Feb  3 08:46 part-8.parquet
   -rw-rw-r-- 1 andy andy 91120030 Feb  3 08:46 part-9.parquet
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to