REASY commented on issue #4973:
URL: https://github.com/apache/arrow-rs/issues/4973#issuecomment-1774310158
@tustvold I'm experiencing the same behavior when profile with heaptrack.
That's probably why I didn't see anything suspicious in DHAT as well.
But again, if I run it without profiler, I see the difference in RAM:
```
➜ parquet-example-rs git:(main) ✗ date
Mon Oct 23 09:48:31 AM +08 2023
➜ parquet-example-rs git:(main) ✗ cargo build --release && /usr/bin/time
-pv target/release/parquet-example-rs --output-parquet-folder output --rows
4000 --statistics-mode page
Finished release [optimized + debuginfo] target(s) in 0.03s
Received args: AppArgs { output_parquet_folder: "output", rows: 4000,
statistics_mode: Page }
Processed 500 msgs with throughout 71.429 msg/s
Processed 1000 msgs with throughout 66.667 msg/s
Processed 1500 msgs with throughout 65.217 msg/s
Processed 2000 msgs with throughout 64.516 msg/s
Processed 2500 msgs with throughout 64.103 msg/s
Processed 3000 msgs with throughout 62.500 msg/s
Processed 3500 msgs with throughout 63.636 msg/s
Processed 4000 msgs with throughout 62.500 msg/s
Wrote 4000 Lidar Point Cloud to parquet in 64.054 seconds, average
throughput 62.447 msg/s
Command being timed: "target/release/parquet-example-rs
--output-parquet-folder output --rows 4000 --statistics-mode page"
User time (seconds): 53.96
System time (seconds): 9.36
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 1:04.15
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 4322052
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1490997
Voluntary context switches: 2227
Involuntary context switches: 835
Swaps: 0
File system inputs: 0
File system outputs: 39064488
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
➜ parquet-example-rs git:(main) ✗ date
Mon Oct 23 09:49:57 AM +08 2023
➜ parquet-example-rs git:(main) ✗ cargo build --release && /usr/bin/time
-pv target/release/parquet-example-rs --output-parquet-folder output --rows
4000 --statistics-mode none
Finished release [optimized + debuginfo] target(s) in 0.03s
Received args: AppArgs { output_parquet_folder: "output", rows: 4000,
statistics_mode: None }
Processed 500 msgs with throughout 83.333 msg/s
Processed 1000 msgs with throughout 83.333 msg/s
Processed 1500 msgs with throughout 83.333 msg/s
Processed 2000 msgs with throughout 83.333 msg/s
Processed 2500 msgs with throughout 80.645 msg/s
Processed 3000 msgs with throughout 81.081 msg/s
Processed 3500 msgs with throughout 79.545 msg/s
Processed 4000 msgs with throughout 80.000 msg/s
Wrote 4000 Lidar Point Cloud to parquet in 50.245 seconds, average
throughput 79.610 msg/s
Command being timed: "target/release/parquet-example-rs
--output-parquet-folder output --rows 4000 --statistics-mode none"
User time (seconds): 40.76
System time (seconds): 8.81
Percent of CPU this job got: 98%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:50.26
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 744364
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 596603
Voluntary context switches: 1764
Involuntary context switches: 550
Swaps: 0
File system inputs: 0
File system outputs: 39063928
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
