Re: Replica sets re-enabled for local provider
On Mon, Jun 9, 2014 at 10:04 AM, Andrew Wilkins andrew.wilk...@canonical.com wrote: On Sat, Jun 7, 2014 at 1:32 AM, Nate Finch nate.fi...@canonical.com wrote: That is awesome. I presume that means we can re-re-re-enable HA on local provider? At least Michael has all the code to do so now, so it should be quick this time. Yes, I think we should re-enable unconditional HA now. I'd like to get to the bottom of this, but I concur with Curtis that we should not be too concerned about supporting ephemeral disks. It's not something we'd ever recommend. I spent a little bit more time looking at this this afternoon. I had previously thought that the I/O speed wouldn't be the issue, since the ephemeral disk is faster than the EBS-backed one on those nodes. I remembered earlier that the replication oplog's size is, by default, a percentage (5%) of the available space on the disk that contains the db. With a cleanish install of 14.04 I have ~5.3G free on /, and ~380G free on /mnt. We don't preallocate the oplog files like we do for the journal, so mongo creates the files and takes its time about it. It seems that this is what is causing the problem. I set the oplog size to 1G (passing --oplogSize 1024 to mongod) and it all works. I've created https://bugs.launchpad.net/juju-core/+bug/1327940 Very nice work, Andrew. I'm not sure it ever would have occurred to me that the storage type would matter. On Fri, Jun 6, 2014 at 12:56 PM, Curtis Hovey-Canonical cur...@canonical.com wrote: On Fri, Jun 6, 2014 at 4:50 AM, Andrew Wilkins andrew.wilk...@canonical.com wrote: I've gotten a bit closer to figuring out what the deal is. Even on the CI machine, local provider works with HA enabled, but only if you use the EBS storage. If you use that machine's ephemeral disk (as your jenkins setup does), then mongo is not happy. The fact that it's running Precise has nothing to do with it. I can reproduce it on Trusty. I bow to your mad skillz. I symlinked /var/lib/mongodb to the EBS volume. We did this for jenkins and LXC. The tests pass. We have a new blessed revision. I think this is an edge case for the local provider. I would expect special support for ephemeral disk -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Replica sets re-enabled for local provider
On Sat, Jun 7, 2014 at 1:32 AM, Nate Finch nate.fi...@canonical.com wrote: That is awesome. I presume that means we can re-re-re-enable HA on local provider? At least Michael has all the code to do so now, so it should be quick this time. Yes, I think we should re-enable unconditional HA now. I'd like to get to the bottom of this, but I concur with Curtis that we should not be too concerned about supporting ephemeral disks. It's not something we'd ever recommend. Very nice work, Andrew. I'm not sure it ever would have occurred to me that the storage type would matter. On Fri, Jun 6, 2014 at 12:56 PM, Curtis Hovey-Canonical cur...@canonical.com wrote: On Fri, Jun 6, 2014 at 4:50 AM, Andrew Wilkins andrew.wilk...@canonical.com wrote: I've gotten a bit closer to figuring out what the deal is. Even on the CI machine, local provider works with HA enabled, but only if you use the EBS storage. If you use that machine's ephemeral disk (as your jenkins setup does), then mongo is not happy. The fact that it's running Precise has nothing to do with it. I can reproduce it on Trusty. I bow to your mad skillz. I symlinked /var/lib/mongodb to the EBS volume. We did this for jenkins and LXC. The tests pass. We have a new blessed revision. I think this is an edge case for the local provider. I would expect special support for ephemeral disk -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Replica sets re-enabled for local provider
On Fri, Jun 6, 2014 at 4:50 AM, Andrew Wilkins andrew.wilk...@canonical.com wrote: I've gotten a bit closer to figuring out what the deal is. Even on the CI machine, local provider works with HA enabled, but only if you use the EBS storage. If you use that machine's ephemeral disk (as your jenkins setup does), then mongo is not happy. The fact that it's running Precise has nothing to do with it. I can reproduce it on Trusty. I bow to your mad skillz. I symlinked /var/lib/mongodb to the EBS volume. We did this for jenkins and LXC. The tests pass. We have a new blessed revision. I think this is an edge case for the local provider. I would expect special support for ephemeral disk -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Replica sets re-enabled for local provider
That is awesome. I presume that means we can re-re-re-enable HA on local provider? At least Michael has all the code to do so now, so it should be quick this time. Very nice work, Andrew. I'm not sure it ever would have occurred to me that the storage type would matter. On Fri, Jun 6, 2014 at 12:56 PM, Curtis Hovey-Canonical cur...@canonical.com wrote: On Fri, Jun 6, 2014 at 4:50 AM, Andrew Wilkins andrew.wilk...@canonical.com wrote: I've gotten a bit closer to figuring out what the deal is. Even on the CI machine, local provider works with HA enabled, but only if you use the EBS storage. If you use that machine's ephemeral disk (as your jenkins setup does), then mongo is not happy. The fact that it's running Precise has nothing to do with it. I can reproduce it on Trusty. I bow to your mad skillz. I symlinked /var/lib/mongodb to the EBS volume. We did this for jenkins and LXC. The tests pass. We have a new blessed revision. I think this is an edge case for the local provider. I would expect special support for ephemeral disk -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Replica sets re-enabled for local provider
I think this is an edge case for the local provider. I would expect special support for ephemeral disk Oops, I mean is *would not* expect special support for ephemeral disk. -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
