Hattonuri commented on PR #39124: URL: https://github.com/apache/arrow/pull/39124#issuecomment-1846171509
Comparing to my pull request i have 5% speedup But I accidentally found that between this commit and https://github.com/apache/arrow/commit/2dcee3f82c6cf54b53a64729fd81840efa583244 There was a commit that increased the number of page faults by two times In this commit i have > Command being timed: "./parquet_playground" User time (seconds): 132.78 System time (seconds): 101.82 Percent of CPU this job got: 497% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:47.18 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 654652 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 56145278 Voluntary context switches: 69281 Involuntary context switches: 716 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 And before it was > Command being timed: "./parquet_playground" User time (seconds): 153.61 System time (seconds): 44.16 Percent of CPU this job got: 506% Elapsed (wall clock) time (h:mm:ss or m:ss): 0:39.06 Average shared text size (kbytes): 0 Average unshared data size (kbytes): 0 Average stack size (kbytes): 0 Average total size (kbytes): 0 Maximum resident set size (kbytes): 707356 Average resident set size (kbytes): 0 Major (requiring I/O) page faults: 0 Minor (reclaiming a frame) page faults: 27169121 Voluntary context switches: 46224 Involuntary context switches: 570 Swaps: 0 File system inputs: 0 File system outputs: 0 Socket messages sent: 0 Socket messages received: 0 Signals delivered: 0 Page size (bytes): 4096 Exit status: 0 As we can see page faults increased from 27169121 to 56145278. Also context switches increased by 1.5 times. In my "playground" i just create acero ScanNode + SinkNode and use MakeGeneratorReader calls to Next() Fragment readahead is 2, batch readahead is 1 and require_sequenced_output is enabled with default backpressure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
