[
https://issues.apache.org/jira/browse/ARROW-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424684#comment-17424684
]
David Li commented on ARROW-14224:
----------------------------------
Poking at Docker, it looks like we have -j4 being set somewhere, which could be
reduced. Interestingly it seems it's actually builder.cc that consumes most of
the memory (~4.5GB in Docker), and additionally takes a long time to compile
(it's one of the first jobs to start, yet is still chugging away once we reach
the kernels). The kernel source files only need ~800MB of memory each, but this
is just enough to trip the memory limit.
Building outside of docker, we see the same thing: this one file eats ~60% of
the available memory and additionally takes nearly 8 minutes to compile:
{noformat}
> /usr/bin/time -v ninja src/arrow/CMakeFiles/arrow_objlib.dir/builder.cc.o
>
>
> (dev)
[1/1] Building CXX object src/arrow/CMakeFiles/arrow_objlib.dir/builder.cc.o
Command being timed: "ninja
src/arrow/CMakeFiles/arrow_objlib.dir/builder.cc.o"
User time (seconds): 462.61
System time (seconds): 2.44
Percent of CPU this job got: 99%
Elapsed (wall clock) time (h:mm:ss or m:ss): 7:45.68
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 4219328
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 1476577
Voluntary context switches: 46
Involuntary context switches: 8006
Swaps: 0
File system inputs: 0
File system outputs: 688168
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0 {noformat}
I did change this file a little while ago while doing kernel work and I suppose
that's added up, so I'll try to reduce the memory usage again.
> [R] [CI] R sanitizer build failing
> ----------------------------------
>
> Key: ARROW-14224
> URL: https://issues.apache.org/jira/browse/ARROW-14224
> Project: Apache Arrow
> Issue Type: Bug
> Components: Continuous Integration, R
> Reporter: Jonathan Keane
> Priority: Major
>
> https://dev.azure.com/ursacomputing/crossbow/_build/results?buildId=12665&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=d9b15392-e4ce-5e4c-0c8c-b69645229181&l=3124
> which might be another out-of-memory error
> {code}
> Error: `docker-compose --file /home/vsts/work/1/s/arrow/docker-compose.yml
> run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=6.0.0.dev391 ubuntu-r-sanitizer`
> exited with a non-zero exit code -9, see the process log above.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
