[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-311858916 Two runs done with no segfaults. I'm declaring victory for now - we would have had a segfault by now. Again, please reopen this ticket if another segfault happens in Jenkins. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-311832165 One run done so far with no segfaults (though 4 other errors). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-311813243 Images are still uploading; once they have I'll push the button on Jenkins to run a few times and will share results in this ticket. If it's not fixed, I'll reopen. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-311736726 Lunch gave me some perspective. I'm proceeding with a 3-step approach: 1. Compile js-1.8.5 from source in all Docker images, with the `--disable-methodjit` configure option. This should remove the possibility for the current known segfault. 2. Watch for more segfaults in test runs & in the field. Address on a case-by-case basis. If the chronic problem resurfaces in our test suite, reopen this issue. 3. Because of the gcc6+ issues and this ticket, move couchjs to couchjs-chakracore as soon as possible (same external view server interface, just replacing SM 1.8.5) @janl what do you think about for #3 above, adding the daemon in 2.2, and making it the default in 3.0? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-311714266 So we've been talking this over on #couchdb-dev IRC. The working theory is: ``` jit + fortify source + extra stack protection causes segfault ``` On Ubuntu 12.04, our Docker images currently use Debian's package, which is what @nickva analysed above. I will change our 12.04 image to build SM from source using flags as recommended via https://paste.apache.org/2UId . I'm not going to put a lot of effort in here - Ubuntu 12.04 is officially EOL at this point. Also worth noting is that the equivalent Debian package has a `--disable-methodjit` (and wrapped in a `ifdef` for `platform==Debian` only), which should also bypass the bug as found. Our CentOS 6/7 images currently build SM from source but do *not* include the build options above. That is, The *package* builds (in couchdb-pkg) **do** use these options, though (derived from the official CentOS 7 js-devel package definition). None of this helps Ubuntu 14, though, which has shown multiple segfaults (including at least 2 recorded in JIRA), and should have the flags listed above as ideal. All of this mess, plus the problems with building SM1.8.5 with gcc6+, make me want to get a `couchjs-chakracore` out the door as soon as possible, maybe for 2.2. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-311714266 So we've been talking this over on #couchdb-dev IRC. The working theory is: ``` jit + fortify source + extra stack protection causes segfault ``` On Ubuntu 12.04, our Docker images currently use Debian's package, which is what @nickva analysed above. I will change our 12.04 image to build SM from source using flags as recommended via https://paste.apache.org/2UId . I'm not going to put a lot of effort in here - Ubuntu 12.04 is officially EOL at this point. Also worth noting is that the equivalent Debian package has a `--disable-methodjit` (and wrapped in a `ifdef` for `platform==Debian` only), which should also bypass the bug as found. Our CentOS 6/7 images currently build SM from source but do *not* include the build options above. The *package* builds (in couchdb-pkg) **do** use these options, though (derived from the official CentOS 7 js-devel package definition). I'm going to revise our Docker images to use the official package (on 7) or build then deploy the official package (on 6). None of this helps Ubuntu 14, though, which has shown multiple segfaults (including at least 2 recorded in JIRA), and should have the flags listed above as ideal. All of this mess, plus the problems with building SM1.8.5 with gcc6+, make me want to get a `couchjs-chakracore` out the door as soon as possible, maybe for 2.2. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-307992413 So to me this suggests it's not a strict memory leak that's failing. Your idea from IRC about capturing all the stdio for the couchjs process and using that to playback against the process to find the source of the problem is intriguing. I'd also appreciate a review of the compile-time settings for our js libs across platforms, compilers used, and whether anything there might be at fault. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-306934907 Based on @nickva 's suppositions, I attempted a low-memory, low-stack-size invocation of couch and couchjs respectively. For reference: ```sh $ docker run -it -m 256M --memory-swap 0 --memory-reservation 256M couchdbdev/centos-7-erlang-18.3 /bin/bash ``` Then, after pulling & building couch master inside the VM, editing `dev/run` to add `-S 65535` or `-S 16384` to the `javascript =` line (L268) Finally, running the test in a loop: ```sh $ for i in `seq 1 500`; do make suites=reduce.js javascript; done ``` No failures after 50 or so executions. Gonna let it run for a little longer, but I'm skeptical this will help reproduce the problem. :( This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-306934907 Based on @nickva 's suppositions, I attempted a low-memory, low-stack-size invocation of couch and couchjs respectively. For reference: ```sh $ docker run -it -m 128M --memory-swap 0 --memory-reservation 128M couchdbdev/centos-7-erlang-18.3 /bin/bash ``` Then, after pulling & building couch master inside the VM, editing `dev/run` to add `-S 65535` to the `javascript =` line (L268) Finally, running the test in a loop: ```sh $ for i in `seq 1 500`; do make suites=reduce.js javascript; done ``` No failures after 50 or so executions. Gonna let it run for a little longer, but I'm skeptical this will help reproduce the problem. :( This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-306934907 Based on @nickva 's suppositions, I attempted a low-memory, low-stack-size invocation of couch and couchjs respectively. For reference: ```sh $ docker run -it -m 128M --memory-swap 0 --memory-reservation 128M couchdbdev/centos-7-erlang-18.3 /bin/bash ``` Then, after pulling & building couch master inside the VM, editing `dev/run to add `-S 65535` to the `javascript =` line (L268) Finally, running the test in a loop: ```sh $ for i in `seq 1 500`; do make suites=reduce.js javascript; done ``` No failures after 50 or so executions. Gonna let it run for a little longer, but I'm skeptical this will help reproduce the problem. :( This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] wohali commented on issue #551: [Jenkins] couchjs segfaults
wohali commented on issue #551: [Jenkins] couchjs segfaults URL: https://github.com/apache/couchdb/issues/551#issuecomment-306934907 Based on @nickva 's suppositions, I attempted a low-memory, low-stack-size invocation of couch and couchjs respectively. For reference: ```sh $ docker run -it -m 128M --memory-swap 0 --memory-reservation 128M couchdbdev/centos-7-erlang-18.3 /bin/bash ``` Then, after pulling & building couch master inside the VM, editing `rel/overlay/etc/default.ini` to add `-S 65535` to the `javascript =` line. Finally, running the test in a loop: ```sh $ for i in `seq 1 500`; do make suites=reduce.js javascript; done ``` No failures after 50 or so executions. Gonna let it run for a little longer, but I'm skeptical this will help reproduce the problem. :( This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
