REASY commented on issue #4973: URL: https://github.com/apache/arrow-rs/issues/4973#issuecomment-1774746598
If I replace default allocator with jemalloc, memory consumption is similar between two modes. The changes are in https://github.com/REASY/parquet-example-rs/pull/1 ``` ➜ parquet-example-rs git:(feature/experiment-with-jemalloc) ✗ date Mon Oct 23 05:03:35 PM +08 2023 ➜ parquet-example-rs git:(feature/experiment-with-jemalloc) ✗ cargo build --features=jemalloc --release && /usr/bin/time -pv target/release/parquet-example-rs --output-parquet-folder output --rows 4000 --statistics-mode page Finished release [optimized + debuginfo] target(s) in 0.03s Received args: AppArgs { output_parquet_folder: "output", rows: 4000, statistics_mode: Page } Processed 500 msgs with throughout 71.429 msg/s Processed 1000 msgs with throughout 66.667 msg/s Processed 1500 msgs with throughout 65.217 msg/s Processed 2000 msgs with throughout 66.667 msg/s Processed 2500 msgs with throughout 65.789 msg/s Processed 3000 msgs with throughout 65.217 msg/s Processed 3500 msgs with throughout 64.815 msg/s Processed 4000 msgs with throughout 64.516 msg/s Wrote 4000 Lidar Point Cloud to parquet in 62.136 seconds, average throughput 64.375 msg/s Command being timed: "target/release/parquet-example-rs --output-parquet-folder output --rows 4000 --statistics-mode page" User time (seconds): 52.13 System time (seconds): 8.61 Percent of CPU this job got: 97% Elapsed (wall clock) time (h:mm:ss or m:ss): 1:02.15 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 797584 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 390983 Voluntary context switches: 1854 Involuntary context switches: 1227 Swaps: 0 File system inputs: 0 File system outputs: 39064480 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 ➜ parquet-example-rs git:(feature/experiment-with-jemalloc) ✗ date Mon Oct 23 05:04:46 PM +08 2023 ➜ parquet-example-rs git:(feature/experiment-with-jemalloc) ✗ cargo build --features=jemalloc --release && /usr/bin/time -pv target/release/parquet-example-rs --output-parquet-folder output --rows 4000 --statistics-mode none Finished release [optimized + debuginfo] target(s) in 0.03s Received args: AppArgs { output_parquet_folder: "output", rows: 4000, statistics_mode: None } Processed 500 msgs with throughout 83.333 msg/s Processed 1000 msgs with throughout 83.333 msg/s Processed 1500 msgs with throughout 78.947 msg/s Processed 2000 msgs with throughout 76.923 msg/s Processed 2500 msgs with throughout 78.125 msg/s Processed 3000 msgs with throughout 78.947 msg/s Processed 3500 msgs with throughout 79.545 msg/s Processed 4000 msgs with throughout 78.431 msg/s Wrote 4000 Lidar Point Cloud to parquet in 51.196 seconds, average throughput 78.131 msg/s Command being timed: "target/release/parquet-example-rs --output-parquet-folder output --rows 4000 --statistics-mode none" User time (seconds): 40.25 System time (seconds): 8.94 Percent of CPU this job got: 96% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:51.22 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 807580 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 400793 Voluntary context switches: 2327 Involuntary context switches: 947 Swaps: 0 File system inputs: 0 File system outputs: 39063920 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
