pitrou commented on PR #41761: URL: https://github.com/apache/arrow/pull/41761#issuecomment-2122894466
Benchmark results here. We see that performance is _O(num_columns * num_row_groups)_. ``` ------------------------------------------------------------------------------------------------------------- Benchmark Time CPU Iterations UserCounters... ------------------------------------------------------------------------------------------------------------- WriteMetadata/num_columns:1/num_row_groups:1 11493 ns 11491 ns 61331 file_size=459 items_per_second=87.0256k/s WriteMetadata/num_columns:1/num_row_groups:100 820026 ns 819815 ns 854 file_size=37.383k items_per_second=1.21979k/s WriteMetadata/num_columns:1/num_row_groups:1000 8024208 ns 8022519 ns 87 file_size=374.885k items_per_second=124.649/s WriteMetadata/num_columns:10/num_row_groups:1 98586 ns 98558 ns 7083 file_size=3.762k items_per_second=10.1463k/s WriteMetadata/num_columns:10/num_row_groups:100 7816090 ns 7814200 ns 89 file_size=358.835k items_per_second=127.972/s WriteMetadata/num_columns:10/num_row_groups:1000 79490918 ns 79462535 ns 8 file_size=3.614M items_per_second=12.5845/s WriteMetadata/num_columns:100/num_row_groups:1 932833 ns 932560 ns 759 file_size=37.352k items_per_second=1.07232k/s WriteMetadata/num_columns:100/num_row_groups:100 78799934 ns 78771226 ns 9 file_size=3.61693M items_per_second=12.695/s WriteMetadata/num_columns:100/num_row_groups:1000 857600506 ns 857330657 ns 1 file_size=36.2887M items_per_second=1.16641/s WriteMetadata/num_columns:1000/num_row_groups:1 9051274 ns 9049407 ns 77 file_size=376.655k items_per_second=110.504/s WriteMetadata/num_columns:1000/num_row_groups:100 827747343 ns 827468643 ns 1 file_size=36.4815M items_per_second=1.20851/s WriteMetadata/num_columns:10000/num_row_groups:1 95165920 ns 95125167 ns 7 file_size=3.82213M items_per_second=10.5125/s WriteMetadata/num_columns:10000/num_row_groups:100 8698273757 ns 8693696946 ns 1 file_size=369.089M items_per_second=0.115026/s ReadMetadata/num_columns:1/num_row_groups:1 3767 ns 3766 ns 185550 file_size=459 items_per_second=265.553k/s ReadMetadata/num_columns:1/num_row_groups:100 87250 ns 87235 ns 8004 file_size=37.383k items_per_second=11.4633k/s ReadMetadata/num_columns:1/num_row_groups:1000 831546 ns 831380 ns 842 file_size=374.885k items_per_second=1.20282k/s ReadMetadata/num_columns:10/num_row_groups:1 19477 ns 19474 ns 35220 file_size=3.762k items_per_second=51.3513k/s ReadMetadata/num_columns:10/num_row_groups:100 698405 ns 698268 ns 994 file_size=358.835k items_per_second=1.43211k/s ReadMetadata/num_columns:10/num_row_groups:1000 6841245 ns 6839685 ns 102 file_size=3.614M items_per_second=146.206/s ReadMetadata/num_columns:100/num_row_groups:1 174932 ns 174898 ns 3979 file_size=37.352k items_per_second=5.71763k/s ReadMetadata/num_columns:100/num_row_groups:100 6640500 ns 6638581 ns 105 file_size=3.61693M items_per_second=150.635/s ReadMetadata/num_columns:100/num_row_groups:1000 75471970 ns 75433100 ns 9 file_size=36.2887M items_per_second=13.2568/s ReadMetadata/num_columns:1000/num_row_groups:1 1671059 ns 1670522 ns 421 file_size=376.655k items_per_second=598.615/s ReadMetadata/num_columns:1000/num_row_groups:100 74756894 ns 74713295 ns 9 file_size=36.4815M items_per_second=13.3845/s ReadMetadata/num_columns:10000/num_row_groups:1 17139819 ns 17132091 ns 41 file_size=3.82213M items_per_second=58.37/s ReadMetadata/num_columns:10000/num_row_groups:100 774363229 ns 773118912 ns 1 file_size=369.089M items_per_second=1.29346/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
