Re: Proposal: making apt-get upgrade optional
On Mon, Jul 7, 2014 at 6:40 AM, Andrew Wilkins andrew.wilk...@canonical.com wrote: On Sat, Jul 5, 2014 at 4:05 AM, Antonio Rosales antonio.rosa...@canonical.com wrote: On Thu, Jul 3, 2014 at 7:41 PM, Andrew Wilkins andrew.wilk...@canonical.com wrote: On Fri, Jul 4, 2014 at 5:23 AM, Tim Penhey tim.pen...@canonical.com wrote: I do just want to make the point that we are not just an ubuntu only system any more, nor even linux only. I'd prefer if we kept away from terms like apt-get as it doesn't make sense for windows nor centos. While we could certainly treat those values differently on the other platforms, it definitely gives the feeling that we are *mainly* ubuntu and (hand wavey) some others. cloud-init generally supporst other Linux distros and I think Cloudbase has cloudbase-init for Windows. Any ideas for better names? upgrade-packages? Still kinda Linux, so alternatively upgrade-system. In cloud-init, it's package_upgrade, with apt_upgrade as an alias. +1 suggest to keep close to the cloud-init packaging as that is what Juju will be leveraging. Is there a straightforward mechanism to allow users and/or charm authors access to the cloud-init configuration? No, because we don't support all of cloud-config in bootstrap or manual provisioning. We convert cloud-config to a bash script. There's a TODO to have cloud-init support direct invocation (rather than via a metadata service), but I'm not sure this is going to happen. Doing so would require additional coordination across distributions, making portability more difficult. We could allow users to specify specific options, such as specifying additional packages to install. Can you elaborate on use cases? Custom disk partitioning/filesystem config, user script, and vendor scripts are immediate use cases that come up. Adding Scott and Ben to the cc list to see if they may have any additional use cases. Ben/Scott, Any suggestions on what cloud-init options/config should be made transparent at the Juju config level? Perhaps there is a cloud-init config option you thought would be beneficial to see at the Juju config level. -thanks, Antonio -thanks, Antonio Tim On 04/07/14 02:56, Matt Bruzek wrote: +1 to making these options configurable and having sane defaults. Thanks! - Matt Bruzek matthew.bru...@canonical.com mailto:matthew.bru...@canonical.com On Thu, Jul 3, 2014 at 9:50 AM, Antonio Rosales antonio.rosa...@canonical.com mailto:antonio.rosa...@canonical.com wrote: On Tue, Jul 1, 2014 at 7:19 PM, Andrew Wilkins andrew.wilk...@canonical.com mailto:andrew.wilk...@canonical.com wrote: On Wed, Jul 2, 2014 at 3:38 AM, Antonio Rosales antonio.rosa...@canonical.com mailto:antonio.rosa...@canonical.com wrote: Suggest we make an environments.yaml key value of say apt-get-update set to a boolean with the default being true. Existing charms are timing out[0] when apt-get update is turned off due to stale apt-get metadata. Users then can them make the choice, and we can make suggestions in the docs as to what this key value means and how it can improve performance especially in the developer scenario when the care more about fast iterative deploys. Thoughts? I'm not suggesting we turn off update, just upgrade. We add repos (cloud-tools, ppa), so we need to update for juju's dependencies anyway. I don't think my proposal will affect charms. Ah yes, sorry. However, I would still suggest upgrade and update be config parameter with the default being past behavior. On that note it would also be nice to have a utility for Juju to pass on additional user defined cloud-init config options. -thanks, Antonio [0] https://bugs.launchpad.net/juju-core/+bug/1336353 -thanks, Antonio On Tue, Jul 1, 2014 at 4:43 AM, Andrew Wilkins andrew.wilk...@canonical.com mailto:andrew.wilk...@canonical.com wrote: On Tue, Jul 1, 2014 at 5:45 PM, John Meinel j...@arbash-meinel.com mailto:j...@arbash-meinel.com wrote: I would just caution that we'd really prefer behavior to be consistent across platforms and clouds, and if we can work with Microsoft to make 'apt-get update' faster in their cloud everyone wins who uses Ubuntu there, not just us. I was meaning to disable it across all providers. It would be ideal to improve upgrades for all Ubuntu users, but from what I can tell it's a case of Azure's OS disks being a tad slow. If you start going up
Re: Proposal: making apt-get upgrade optional
On 11/07/14 02:47, Nate Finch wrote: Late to the party, but +1 for OS-neutral names. Keep in mind, there's no separate update/upgrade steps on Windows. There's no list of software that exists that needs to get updated on Windows, as that is done automatically. Luckily, it sounds like we want update to always happen on Ubuntu anyway, so we don't need to find a name for it. Not necessarily. We want to default it to always update, especially for cloud machines, but we want to be able to turn it off for the local provider when demoing as it will be much faster. Tim -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposal: making apt-get upgrade optional
On Sat, Jul 5, 2014 at 4:05 AM, Antonio Rosales antonio.rosa...@canonical.com wrote: On Thu, Jul 3, 2014 at 7:41 PM, Andrew Wilkins andrew.wilk...@canonical.com wrote: On Fri, Jul 4, 2014 at 5:23 AM, Tim Penhey tim.pen...@canonical.com wrote: I do just want to make the point that we are not just an ubuntu only system any more, nor even linux only. I'd prefer if we kept away from terms like apt-get as it doesn't make sense for windows nor centos. While we could certainly treat those values differently on the other platforms, it definitely gives the feeling that we are *mainly* ubuntu and (hand wavey) some others. cloud-init generally supporst other Linux distros and I think Cloudbase has cloudbase-init for Windows. Any ideas for better names? upgrade-packages? Still kinda Linux, so alternatively upgrade-system. In cloud-init, it's package_upgrade, with apt_upgrade as an alias. +1 suggest to keep close to the cloud-init packaging as that is what Juju will be leveraging. Is there a straightforward mechanism to allow users and/or charm authors access to the cloud-init configuration? No, because we don't support all of cloud-config in bootstrap or manual provisioning. We convert cloud-config to a bash script. There's a TODO to have cloud-init support direct invocation (rather than via a metadata service), but I'm not sure this is going to happen. Doing so would require additional coordination across distributions, making portability more difficult. We could allow users to specify specific options, such as specifying additional packages to install. Can you elaborate on use cases? -thanks, Antonio Tim On 04/07/14 02:56, Matt Bruzek wrote: +1 to making these options configurable and having sane defaults. Thanks! - Matt Bruzek matthew.bru...@canonical.com mailto:matthew.bru...@canonical.com On Thu, Jul 3, 2014 at 9:50 AM, Antonio Rosales antonio.rosa...@canonical.com mailto:antonio.rosa...@canonical.com wrote: On Tue, Jul 1, 2014 at 7:19 PM, Andrew Wilkins andrew.wilk...@canonical.com mailto: andrew.wilk...@canonical.com wrote: On Wed, Jul 2, 2014 at 3:38 AM, Antonio Rosales antonio.rosa...@canonical.com mailto:antonio.rosa...@canonical.com wrote: Suggest we make an environments.yaml key value of say apt-get-update set to a boolean with the default being true. Existing charms are timing out[0] when apt-get update is turned off due to stale apt-get metadata. Users then can them make the choice, and we can make suggestions in the docs as to what this key value means and how it can improve performance especially in the developer scenario when the care more about fast iterative deploys. Thoughts? I'm not suggesting we turn off update, just upgrade. We add repos (cloud-tools, ppa), so we need to update for juju's dependencies anyway. I don't think my proposal will affect charms. Ah yes, sorry. However, I would still suggest upgrade and update be config parameter with the default being past behavior. On that note it would also be nice to have a utility for Juju to pass on additional user defined cloud-init config options. -thanks, Antonio [0] https://bugs.launchpad.net/juju-core/+bug/1336353 -thanks, Antonio On Tue, Jul 1, 2014 at 4:43 AM, Andrew Wilkins andrew.wilk...@canonical.com mailto:andrew.wilk...@canonical.com wrote: On Tue, Jul 1, 2014 at 5:45 PM, John Meinel j...@arbash-meinel.com mailto:j...@arbash-meinel.com wrote: I would just caution that we'd really prefer behavior to be consistent across platforms and clouds, and if we can work with Microsoft to make 'apt-get update' faster in their cloud everyone wins who uses Ubuntu there, not just us. I was meaning to disable it across all providers. It would be ideal to improve upgrades for all Ubuntu users, but from what I can tell it's a case of Azure's OS disks being a tad slow. If you start going up the instance-type scale, then you do get more IOPS. I haven't measured how much of a difference it makes. Have we looked into why Upgrade is taking 3m+? Is it the time to download things, is it the time to install things? I've certainly heard things like disk ops is a bit poor on Azure (vs CPU is actually better than average). Given the variance of 6m+ to 3m20s with Eat my data, it would seem disk sync
Re: Proposal: making apt-get upgrade optional
On Thu, Jul 3, 2014 at 7:41 PM, Andrew Wilkins andrew.wilk...@canonical.com wrote: On Fri, Jul 4, 2014 at 5:23 AM, Tim Penhey tim.pen...@canonical.com wrote: I do just want to make the point that we are not just an ubuntu only system any more, nor even linux only. I'd prefer if we kept away from terms like apt-get as it doesn't make sense for windows nor centos. While we could certainly treat those values differently on the other platforms, it definitely gives the feeling that we are *mainly* ubuntu and (hand wavey) some others. cloud-init generally supporst other Linux distros and I think Cloudbase has cloudbase-init for Windows. Any ideas for better names? upgrade-packages? Still kinda Linux, so alternatively upgrade-system. In cloud-init, it's package_upgrade, with apt_upgrade as an alias. +1 suggest to keep close to the cloud-init packaging as that is what Juju will be leveraging. Is there a straightforward mechanism to allow users and/or charm authors access to the cloud-init configuration? -thanks, Antonio Tim On 04/07/14 02:56, Matt Bruzek wrote: +1 to making these options configurable and having sane defaults. Thanks! - Matt Bruzek matthew.bru...@canonical.com mailto:matthew.bru...@canonical.com On Thu, Jul 3, 2014 at 9:50 AM, Antonio Rosales antonio.rosa...@canonical.com mailto:antonio.rosa...@canonical.com wrote: On Tue, Jul 1, 2014 at 7:19 PM, Andrew Wilkins andrew.wilk...@canonical.com mailto:andrew.wilk...@canonical.com wrote: On Wed, Jul 2, 2014 at 3:38 AM, Antonio Rosales antonio.rosa...@canonical.com mailto:antonio.rosa...@canonical.com wrote: Suggest we make an environments.yaml key value of say apt-get-update set to a boolean with the default being true. Existing charms are timing out[0] when apt-get update is turned off due to stale apt-get metadata. Users then can them make the choice, and we can make suggestions in the docs as to what this key value means and how it can improve performance especially in the developer scenario when the care more about fast iterative deploys. Thoughts? I'm not suggesting we turn off update, just upgrade. We add repos (cloud-tools, ppa), so we need to update for juju's dependencies anyway. I don't think my proposal will affect charms. Ah yes, sorry. However, I would still suggest upgrade and update be config parameter with the default being past behavior. On that note it would also be nice to have a utility for Juju to pass on additional user defined cloud-init config options. -thanks, Antonio [0] https://bugs.launchpad.net/juju-core/+bug/1336353 -thanks, Antonio On Tue, Jul 1, 2014 at 4:43 AM, Andrew Wilkins andrew.wilk...@canonical.com mailto:andrew.wilk...@canonical.com wrote: On Tue, Jul 1, 2014 at 5:45 PM, John Meinel j...@arbash-meinel.com mailto:j...@arbash-meinel.com wrote: I would just caution that we'd really prefer behavior to be consistent across platforms and clouds, and if we can work with Microsoft to make 'apt-get update' faster in their cloud everyone wins who uses Ubuntu there, not just us. I was meaning to disable it across all providers. It would be ideal to improve upgrades for all Ubuntu users, but from what I can tell it's a case of Azure's OS disks being a tad slow. If you start going up the instance-type scale, then you do get more IOPS. I haven't measured how much of a difference it makes. Have we looked into why Upgrade is taking 3m+? Is it the time to download things, is it the time to install things? I've certainly heard things like disk ops is a bit poor on Azure (vs CPU is actually better than average). Given the variance of 6m+ to 3m20s with Eat my data, it would seem disk sync performance is at least a factor here. I just looked, and it is mostly not network related (I assume mostly I/O bound). On ec2 an upgrade fetches all the bits in 0s; on Azure it's taking 5s. Given I believe apt-get update is also disabled for local (it is run on the initial template, and then not run for the other instances copied from that), there is certainly precedence. I think a big concern is that we would probably still want to do apt-get update for security related updates. Though perhaps that is all of the updates we are applying anyway... If I read the aws.json file
Re: Proposal: making apt-get upgrade optional
I do just want to make the point that we are not just an ubuntu only system any more, nor even linux only. I'd prefer if we kept away from terms like apt-get as it doesn't make sense for windows nor centos. While we could certainly treat those values differently on the other platforms, it definitely gives the feeling that we are *mainly* ubuntu and (hand wavey) some others. Any ideas for better names? Tim On 04/07/14 02:56, Matt Bruzek wrote: +1 to making these options configurable and having sane defaults. Thanks! - Matt Bruzek matthew.bru...@canonical.com mailto:matthew.bru...@canonical.com On Thu, Jul 3, 2014 at 9:50 AM, Antonio Rosales antonio.rosa...@canonical.com mailto:antonio.rosa...@canonical.com wrote: On Tue, Jul 1, 2014 at 7:19 PM, Andrew Wilkins andrew.wilk...@canonical.com mailto:andrew.wilk...@canonical.com wrote: On Wed, Jul 2, 2014 at 3:38 AM, Antonio Rosales antonio.rosa...@canonical.com mailto:antonio.rosa...@canonical.com wrote: Suggest we make an environments.yaml key value of say apt-get-update set to a boolean with the default being true. Existing charms are timing out[0] when apt-get update is turned off due to stale apt-get metadata. Users then can them make the choice, and we can make suggestions in the docs as to what this key value means and how it can improve performance especially in the developer scenario when the care more about fast iterative deploys. Thoughts? I'm not suggesting we turn off update, just upgrade. We add repos (cloud-tools, ppa), so we need to update for juju's dependencies anyway. I don't think my proposal will affect charms. Ah yes, sorry. However, I would still suggest upgrade and update be config parameter with the default being past behavior. On that note it would also be nice to have a utility for Juju to pass on additional user defined cloud-init config options. -thanks, Antonio [0] https://bugs.launchpad.net/juju-core/+bug/1336353 -thanks, Antonio On Tue, Jul 1, 2014 at 4:43 AM, Andrew Wilkins andrew.wilk...@canonical.com mailto:andrew.wilk...@canonical.com wrote: On Tue, Jul 1, 2014 at 5:45 PM, John Meinel j...@arbash-meinel.com mailto:j...@arbash-meinel.com wrote: I would just caution that we'd really prefer behavior to be consistent across platforms and clouds, and if we can work with Microsoft to make 'apt-get update' faster in their cloud everyone wins who uses Ubuntu there, not just us. I was meaning to disable it across all providers. It would be ideal to improve upgrades for all Ubuntu users, but from what I can tell it's a case of Azure's OS disks being a tad slow. If you start going up the instance-type scale, then you do get more IOPS. I haven't measured how much of a difference it makes. Have we looked into why Upgrade is taking 3m+? Is it the time to download things, is it the time to install things? I've certainly heard things like disk ops is a bit poor on Azure (vs CPU is actually better than average). Given the variance of 6m+ to 3m20s with Eat my data, it would seem disk sync performance is at least a factor here. I just looked, and it is mostly not network related (I assume mostly I/O bound). On ec2 an upgrade fetches all the bits in 0s; on Azure it's taking 5s. Given I believe apt-get update is also disabled for local (it is run on the initial template, and then not run for the other instances copied from that), there is certainly precedence. I think a big concern is that we would probably still want to do apt-get update for security related updates. Though perhaps that is all of the updates we are applying anyway... If I read the aws.json file correctly, I see only 8 releases of the 'precise' image. 6 of 'trusty' and 32 total dates of released items. And some of the trusty releases are 2014-01-22.1 which means it is likely to be beta releases. Anyway, that means that they are actually averaging an update only 1/month, which is a fairly big window of updates to apply by the end of month (I would imagine). And while that does mean it takes longer to boot, it also means you would be open to more security holes without it. My contention is that if we don't *keep* it updated, we may as well just leave it to the user. When you create an instance in ec2 or Azure
Re: Proposal: making apt-get upgrade optional
On Tue, Jul 1, 2014 at 9:19 PM, Andrew Wilkins andrew.wilk...@canonical.com wrote: On Wed, Jul 2, 2014 at 3:38 AM, Antonio Rosales antonio.rosa...@canonical.com wrote: Suggest we make an environments.yaml key value of say apt-get-update set to a boolean with the default being true. Existing charms are timing out[0] when apt-get update is turned off due to stale apt-get metadata. Users then can them make the choice, and we can make suggestions in the docs as to what this key value means and how it can improve performance especially in the developer scenario when the care more about fast iterative deploys. Thoughts? I'm not suggesting we turn off update, just upgrade. We add repos (cloud-tools, ppa), so we need to update for juju's dependencies anyway. I don't think my proposal will affect charms. [0] https://bugs.launchpad.net/juju-core/+bug/1336353 \ ^ ^ this bug implies juju is not running update or not running it at the correct time time. This might be because provisioning for a charm is different than provisioning a state-server? -- Curtis Hovey Canonical Cloud Development and Operations http://launchpad.net/~sinzui -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposal: making apt-get upgrade optional
On 03/07/14 03:27, John Meinel wrote: local provider is special. It generates a template image, and runs apt-get update once there, and then clones and doesn't ever run it again. Also, the template isn't destroyed by destroy-environment which means if you tried out the local provider 3 months ago, you'll still have a really old template today. Tim had intended to do more work here (provide a way to refresh the template), but ran out of time to work on local provider stuff. There's currently work in progress (as a background task, so it will get done eventually) to complete the work needed to allow the template to be refreshed. Note this will also effect LXC deployments, as other people requested us to use the template + clone on all clouds. There's a config setting to turn this feature off (it's on by default). lxc-clone: false -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposal: making apt-get upgrade optional
I would just caution that we'd really prefer behavior to be consistent across platforms and clouds, and if we can work with Microsoft to make 'apt-get update' faster in their cloud everyone wins who uses Ubuntu there, not just us. Have we looked into why Upgrade is taking 3m+? Is it the time to download things, is it the time to install things? I've certainly heard things like disk ops is a bit poor on Azure (vs CPU is actually better than average). Given the variance of 6m+ to 3m20s with Eat my data, it would seem disk sync performance is at least a factor here. Given I believe apt-get update is also disabled for local (it is run on the initial template, and then not run for the other instances copied from that), there is certainly precedence. I think a big concern is that we would probably still want to do apt-get update for security related updates. Though perhaps that is all of the updates we are applying anyway... If I read the aws.json file correctly, I see only 8 releases of the 'precise' image. 6 of 'trusty' and 32 total dates of released items. And some of the trusty releases are 2014-01-22.1 which means it is likely to be beta releases. Anyway, that means that they are actually averaging an update only 1/month, which is a fairly big window of updates to apply by the end of month (I would imagine). And while that does mean it takes longer to boot, it also means you would be open to more security holes without it. John =:- On Mon, Jun 30, 2014 at 5:05 PM, Andrew Wilkins andrew.wilk...@canonical.com wrote: Hi folks, I've been debugging a bootstrap bug [0] that was caused by ssh timing out (and the client not noticing), which was caused by apt-get upgrade taking an awfully long time (6 minutes on Azure). [0] https://bugs.launchpad.net/juju-core/+bug/1316185 I just filed https://bugs.launchpad.net/juju-core/+bug/1335822, and did a quick and dirty hack that brought the upgrade down to 3 minutes on Azure. I don't know the variance, so I can't be sure that it's all due to eatmydata, but smoser's results are similar. Even with eatmydata, a full bootstrap on Azure just took me 10 minutes. That's roughly broken down into: - apt-get update: 20s - apt-get upgrade: 3m20s - apt-get install various: 10s - Download tools (from shared Azure storage account): 5s - jujud bootstrap: 1m50s We could bring the 10m down to 6m40s. Still not brilliant, but considerably better IMO. I propose that we remove the apt-get upgrade altogether. Cloud images are regularly updated and tested, and I think we should be able to rely on that alone. If users want something more up-to-date, they can use the daily images which are not tested as a whole, but are composed of SRUs, which is effectively what users get today. Cheers, Andrew -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposal: making apt-get upgrade optional
On Tue, Jul 1, 2014 at 5:45 PM, John Meinel j...@arbash-meinel.com wrote: I would just caution that we'd really prefer behavior to be consistent across platforms and clouds, and if we can work with Microsoft to make 'apt-get update' faster in their cloud everyone wins who uses Ubuntu there, not just us. I was meaning to disable it across all providers. It would be ideal to improve upgrades for all Ubuntu users, but from what I can tell it's a case of Azure's OS disks being a tad slow. If you start going up the instance-type scale, then you do get more IOPS. I haven't measured how much of a difference it makes. Have we looked into why Upgrade is taking 3m+? Is it the time to download things, is it the time to install things? I've certainly heard things like disk ops is a bit poor on Azure (vs CPU is actually better than average). Given the variance of 6m+ to 3m20s with Eat my data, it would seem disk sync performance is at least a factor here. I just looked, and it is mostly not network related (I assume mostly I/O bound). On ec2 an upgrade fetches all the bits in 0s; on Azure it's taking 5s. Given I believe apt-get update is also disabled for local (it is run on the initial template, and then not run for the other instances copied from that), there is certainly precedence. I think a big concern is that we would probably still want to do apt-get update for security related updates. Though perhaps that is all of the updates we are applying anyway... If I read the aws.json file correctly, I see only 8 releases of the 'precise' image. 6 of 'trusty' and 32 total dates of released items. And some of the trusty releases are 2014-01-22.1 which means it is likely to be beta releases. Anyway, that means that they are actually averaging an update only 1/month, which is a fairly big window of updates to apply by the end of month (I would imagine). And while that does mean it takes longer to boot, it also means you would be open to more security holes without it. My contention is that if we don't *keep* it updated, we may as well just leave it to the user. When you create an instance in ec2 or Azure or whatever, it doesn't come fully up-to-date. You get the released image, and then you can update it if you want to. John =:- On Mon, Jun 30, 2014 at 5:05 PM, Andrew Wilkins andrew.wilk...@canonical.com wrote: Hi folks, I've been debugging a bootstrap bug [0] that was caused by ssh timing out (and the client not noticing), which was caused by apt-get upgrade taking an awfully long time (6 minutes on Azure). [0] https://bugs.launchpad.net/juju-core/+bug/1316185 I just filed https://bugs.launchpad.net/juju-core/+bug/1335822, and did a quick and dirty hack that brought the upgrade down to 3 minutes on Azure. I don't know the variance, so I can't be sure that it's all due to eatmydata, but smoser's results are similar. Even with eatmydata, a full bootstrap on Azure just took me 10 minutes. That's roughly broken down into: - apt-get update: 20s - apt-get upgrade: 3m20s - apt-get install various: 10s - Download tools (from shared Azure storage account): 5s - jujud bootstrap: 1m50s We could bring the 10m down to 6m40s. Still not brilliant, but considerably better IMO. I propose that we remove the apt-get upgrade altogether. Cloud images are regularly updated and tested, and I think we should be able to rely on that alone. If users want something more up-to-date, they can use the daily images which are not tested as a whole, but are composed of SRUs, which is effectively what users get today. Cheers, Andrew -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposal: making apt-get upgrade optional
Suggest we make an environments.yaml key value of say apt-get-update set to a boolean with the default being true. Existing charms are timing out[0] when apt-get update is turned off due to stale apt-get metadata. Users then can them make the choice, and we can make suggestions in the docs as to what this key value means and how it can improve performance especially in the developer scenario when the care more about fast iterative deploys. Thoughts? [0] https://bugs.launchpad.net/juju-core/+bug/1336353 -thanks, Antonio On Tue, Jul 1, 2014 at 4:43 AM, Andrew Wilkins andrew.wilk...@canonical.com wrote: On Tue, Jul 1, 2014 at 5:45 PM, John Meinel j...@arbash-meinel.com wrote: I would just caution that we'd really prefer behavior to be consistent across platforms and clouds, and if we can work with Microsoft to make 'apt-get update' faster in their cloud everyone wins who uses Ubuntu there, not just us. I was meaning to disable it across all providers. It would be ideal to improve upgrades for all Ubuntu users, but from what I can tell it's a case of Azure's OS disks being a tad slow. If you start going up the instance-type scale, then you do get more IOPS. I haven't measured how much of a difference it makes. Have we looked into why Upgrade is taking 3m+? Is it the time to download things, is it the time to install things? I've certainly heard things like disk ops is a bit poor on Azure (vs CPU is actually better than average). Given the variance of 6m+ to 3m20s with Eat my data, it would seem disk sync performance is at least a factor here. I just looked, and it is mostly not network related (I assume mostly I/O bound). On ec2 an upgrade fetches all the bits in 0s; on Azure it's taking 5s. Given I believe apt-get update is also disabled for local (it is run on the initial template, and then not run for the other instances copied from that), there is certainly precedence. I think a big concern is that we would probably still want to do apt-get update for security related updates. Though perhaps that is all of the updates we are applying anyway... If I read the aws.json file correctly, I see only 8 releases of the 'precise' image. 6 of 'trusty' and 32 total dates of released items. And some of the trusty releases are 2014-01-22.1 which means it is likely to be beta releases. Anyway, that means that they are actually averaging an update only 1/month, which is a fairly big window of updates to apply by the end of month (I would imagine). And while that does mean it takes longer to boot, it also means you would be open to more security holes without it. My contention is that if we don't *keep* it updated, we may as well just leave it to the user. When you create an instance in ec2 or Azure or whatever, it doesn't come fully up-to-date. You get the released image, and then you can update it if you want to. John =:- On Mon, Jun 30, 2014 at 5:05 PM, Andrew Wilkins andrew.wilk...@canonical.com wrote: Hi folks, I've been debugging a bootstrap bug [0] that was caused by ssh timing out (and the client not noticing), which was caused by apt-get upgrade taking an awfully long time (6 minutes on Azure). [0] https://bugs.launchpad.net/juju-core/+bug/1316185 I just filed https://bugs.launchpad.net/juju-core/+bug/1335822, and did a quick and dirty hack that brought the upgrade down to 3 minutes on Azure. I don't know the variance, so I can't be sure that it's all due to eatmydata, but smoser's results are similar. Even with eatmydata, a full bootstrap on Azure just took me 10 minutes. That's roughly broken down into: - apt-get update: 20s - apt-get upgrade: 3m20s - apt-get install various: 10s - Download tools (from shared Azure storage account): 5s - jujud bootstrap: 1m50s We could bring the 10m down to 6m40s. Still not brilliant, but considerably better IMO. I propose that we remove the apt-get upgrade altogether. Cloud images are regularly updated and tested, and I think we should be able to rely on that alone. If users want something more up-to-date, they can use the daily images which are not tested as a whole, but are composed of SRUs, which is effectively what users get today. Cheers, Andrew -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Antonio Rosales Juju Ecosystem Canonical -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposal: making apt-get upgrade optional
Hello Andrew, I ran into a problem when Juju was no longer calling apt-get update. I filed bug: https://bugs.launchpad.net/juju-core/+bug/1336353 The problem appears to be the apt metadata was out of date and the charms that call apt-get install ... in the install hook fail. We also saw this problem on a different system where the cloud image was out of date and the install hook failed because no apt-get update was run. Not calling apt-get update will break a large number of existing charms and needs to be fixed. It looks like we have 2 problems: 1) Juju should provide a way to turn on or off the apt-get update. I would suggest an environment variable to enable or disable apt-get update. 2) Juju does not manage the local cloud image found in /var/cache/lxc/cloud-series/ directory. The image I was using was older (05/28/2014) and that is why the apt-get meta data was out of date. Not quite sure how to solve this, someone suggested a cron job to download the lastest cloud image in the background. Someone else mentioned this is a security issue if the stale image does not have a patch and is not updated we could have an issue. This is not the experience we want to give to Juju users. As developers we should be able to turn off apt-get update for our own development purposes, but leave it on so existing charms work. I am willing to help resolve these two problems. Please reach out to mbruzek on irc or email me directly. Thanks, - Matt Bruzek matthew.bru...@canonical.com On Mon, Jun 30, 2014 at 8:05 AM, Andrew Wilkins andrew.wilk...@canonical.com wrote: Hi folks, I've been debugging a bootstrap bug [0] that was caused by ssh timing out (and the client not noticing), which was caused by apt-get upgrade taking an awfully long time (6 minutes on Azure). [0] https://bugs.launchpad.net/juju-core/+bug/1316185 I just filed https://bugs.launchpad.net/juju-core/+bug/1335822, and did a quick and dirty hack that brought the upgrade down to 3 minutes on Azure. I don't know the variance, so I can't be sure that it's all due to eatmydata, but smoser's results are similar. Even with eatmydata, a full bootstrap on Azure just took me 10 minutes. That's roughly broken down into: - apt-get update: 20s - apt-get upgrade: 3m20s - apt-get install various: 10s - Download tools (from shared Azure storage account): 5s - jujud bootstrap: 1m50s We could bring the 10m down to 6m40s. Still not brilliant, but considerably better IMO. I propose that we remove the apt-get upgrade altogether. Cloud images are regularly updated and tested, and I think we should be able to rely on that alone. If users want something more up-to-date, they can use the daily images which are not tested as a whole, but are composed of SRUs, which is effectively what users get today. Cheers, Andrew -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposal: making apt-get upgrade optional
Why not use the auto update feature that is in Ubuntu. I use it and recoeve emails about the upgrades Sent from Samsung Mobile Original message From: Antonio Rosales antonio.rosa...@canonical.com Date: 01/07/2014 21:38 (GMT+01:00) To: Andrew Wilkins andrew.wilk...@canonical.com Cc: juju-dev@lists.ubuntu.com Subject: Re: Proposal: making apt-get upgrade optional Suggest we make an environments.yaml key value of say apt-get-update set to a boolean with the default being true. Existing charms are timing out[0] when apt-get update is turned off due to stale apt-get metadata. Users then can them make the choice, and we can make suggestions in the docs as to what this key value means and how it can improve performance especially in the developer scenario when the care more about fast iterative deploys. Thoughts? [0] https://bugs.launchpad.net/juju-core/+bug/1336353 -thanks, Antonio On Tue, Jul 1, 2014 at 4:43 AM, Andrew Wilkins andrew.wilk...@canonical.com wrote: On Tue, Jul 1, 2014 at 5:45 PM, John Meinel j...@arbash-meinel.com wrote: I would just caution that we'd really prefer behavior to be consistent across platforms and clouds, and if we can work with Microsoft to make 'apt-get update' faster in their cloud everyone wins who uses Ubuntu there, not just us. I was meaning to disable it across all providers. It would be ideal to improve upgrades for all Ubuntu users, but from what I can tell it's a case of Azure's OS disks being a tad slow. If you start going up the instance-type scale, then you do get more IOPS. I haven't measured how much of a difference it makes. Have we looked into why Upgrade is taking 3m+? Is it the time to download things, is it the time to install things? I've certainly heard things like disk ops is a bit poor on Azure (vs CPU is actually better than average). Given the variance of 6m+ to 3m20s with Eat my data, it would seem disk sync performance is at least a factor here. I just looked, and it is mostly not network related (I assume mostly I/O bound). On ec2 an upgrade fetches all the bits in 0s; on Azure it's taking 5s. Given I believe apt-get update is also disabled for local (it is run on the initial template, and then not run for the other instances copied from that), there is certainly precedence. I think a big concern is that we would probably still want to do apt-get update for security related updates. Though perhaps that is all of the updates we are applying anyway... If I read the aws.json file correctly, I see only 8 releases of the 'precise' image. 6 of 'trusty' and 32 total dates of released items. And some of the trusty releases are 2014-01-22.1 which means it is likely to be beta releases. Anyway, that means that they are actually averaging an update only 1/month, which is a fairly big window of updates to apply by the end of month (I would imagine). And while that does mean it takes longer to boot, it also means you would be open to more security holes without it. My contention is that if we don't *keep* it updated, we may as well just leave it to the user. When you create an instance in ec2 or Azure or whatever, it doesn't come fully up-to-date. You get the released image, and then you can update it if you want to. John =:- On Mon, Jun 30, 2014 at 5:05 PM, Andrew Wilkins andrew.wilk...@canonical.com wrote: Hi folks, I've been debugging a bootstrap bug [0] that was caused by ssh timing out (and the client not noticing), which was caused by apt-get upgrade taking an awfully long time (6 minutes on Azure). [0] https://bugs.launchpad.net/juju-core/+bug/1316185 I just filed https://bugs.launchpad.net/juju-core/+bug/1335822, and did a quick and dirty hack that brought the upgrade down to 3 minutes on Azure. I don't know the variance, so I can't be sure that it's all due to eatmydata, but smoser's results are similar. Even with eatmydata, a full bootstrap on Azure just took me 10 minutes. That's roughly broken down into: - apt-get update: 20s - apt-get upgrade: 3m20s - apt-get install various: 10s - Download tools (from shared Azure storage account): 5s - jujud bootstrap: 1m50s We could bring the 10m down to 6m40s. Still not brilliant, but considerably better IMO. I propose that we remove the apt-get upgrade altogether. Cloud images are regularly updated and tested, and I think we should be able to rely on that alone. If users want something more up-to-date, they can use the daily images which are not tested as a whole, but are composed of SRUs, which is effectively what users get today. Cheers, Andrew -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Antonio Rosales Juju Ecosystem Canonical
Re: Proposal: making apt-get upgrade optional
On Tue, Jul 1, 2014 at 1:53 PM, Matt Bruzek matthew.bru...@canonical.com wrote: Hello Andrew, I ran into a problem when Juju was no longer calling apt-get update. I filed bug: https://bugs.launchpad.net/juju-core/+bug/1336353 Agreed -- I've fixed this problem multiple times in charms by making the first step apt-get upgrade. Which always seemed a bit wasteful to me. :) It happens more on the local provider since those images are copied from templates which are not rebuilt until you remove them (do lxc-ls --fancy to see them). So, the templates package cache goes out of date, and your cloned machine also goes out of date. -- David Britton david.brit...@canonical.com -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposal: making apt-get upgrade optional
I actually don't see a problem with removing apt-get upgrade, but what apt-get update? It's only 20s user time according to the original post. For stale cloud images, local provider and manual, it's just a no brained. Marco On Jul 1, 2014 4:04 PM, David Britton david.brit...@canonical.com wrote: On Tue, Jul 1, 2014 at 1:53 PM, Matt Bruzek matthew.bru...@canonical.com wrote: Hello Andrew, I ran into a problem when Juju was no longer calling apt-get update. I filed bug: https://bugs.launchpad.net/juju-core/+bug/1336353 Agreed -- I've fixed this problem multiple times in charms by making the first step apt-get upgrade. Which always seemed a bit wasteful to me. :) It happens more on the local provider since those images are copied from templates which are not rebuilt until you remove them (do lxc-ls --fancy to see them). So, the templates package cache goes out of date, and your cloned machine also goes out of date. -- David Britton david.brit...@canonical.com -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposal: making apt-get upgrade optional
With respect to the local provider, I had always intended there to be a command in the local provider plugin (which is currently empty) that updates the template image. Tim On 02/07/14 08:22, Marco Ceppi wrote: I actually don't see a problem with removing apt-get upgrade, but what apt-get update? It's only 20s user time according to the original post. For stale cloud images, local provider and manual, it's just a no brained. Marco On Jul 1, 2014 4:04 PM, David Britton david.brit...@canonical.com mailto:david.brit...@canonical.com wrote: On Tue, Jul 1, 2014 at 1:53 PM, Matt Bruzek matthew.bru...@canonical.com mailto:matthew.bru...@canonical.com wrote: Hello Andrew, I ran into a problem when Juju was no longer calling apt-get update. I filed bug: https://bugs.launchpad.net/juju-core/+bug/1336353 Agreed -- I've fixed this problem multiple times in charms by making the first step apt-get upgrade. Which always seemed a bit wasteful to me. :) It happens more on the local provider since those images are copied from templates which are not rebuilt until you remove them (do lxc-ls --fancy to see them). So, the templates package cache goes out of date, and your cloned machine also goes out of date. -- David Britton david.brit...@canonical.com mailto:david.brit...@canonical.com -- Juju-dev mailing list Juju-dev@lists.ubuntu.com mailto:Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
Re: Proposal: making apt-get upgrade optional
On Wed, Jul 2, 2014 at 3:38 AM, Antonio Rosales antonio.rosa...@canonical.com wrote: Suggest we make an environments.yaml key value of say apt-get-update set to a boolean with the default being true. Existing charms are timing out[0] when apt-get update is turned off due to stale apt-get metadata. Users then can them make the choice, and we can make suggestions in the docs as to what this key value means and how it can improve performance especially in the developer scenario when the care more about fast iterative deploys. Thoughts? I'm not suggesting we turn off update, just upgrade. We add repos (cloud-tools, ppa), so we need to update for juju's dependencies anyway. I don't think my proposal will affect charms. [0] https://bugs.launchpad.net/juju-core/+bug/1336353 -thanks, Antonio On Tue, Jul 1, 2014 at 4:43 AM, Andrew Wilkins andrew.wilk...@canonical.com wrote: On Tue, Jul 1, 2014 at 5:45 PM, John Meinel j...@arbash-meinel.com wrote: I would just caution that we'd really prefer behavior to be consistent across platforms and clouds, and if we can work with Microsoft to make 'apt-get update' faster in their cloud everyone wins who uses Ubuntu there, not just us. I was meaning to disable it across all providers. It would be ideal to improve upgrades for all Ubuntu users, but from what I can tell it's a case of Azure's OS disks being a tad slow. If you start going up the instance-type scale, then you do get more IOPS. I haven't measured how much of a difference it makes. Have we looked into why Upgrade is taking 3m+? Is it the time to download things, is it the time to install things? I've certainly heard things like disk ops is a bit poor on Azure (vs CPU is actually better than average). Given the variance of 6m+ to 3m20s with Eat my data, it would seem disk sync performance is at least a factor here. I just looked, and it is mostly not network related (I assume mostly I/O bound). On ec2 an upgrade fetches all the bits in 0s; on Azure it's taking 5s. Given I believe apt-get update is also disabled for local (it is run on the initial template, and then not run for the other instances copied from that), there is certainly precedence. I think a big concern is that we would probably still want to do apt-get update for security related updates. Though perhaps that is all of the updates we are applying anyway... If I read the aws.json file correctly, I see only 8 releases of the 'precise' image. 6 of 'trusty' and 32 total dates of released items. And some of the trusty releases are 2014-01-22.1 which means it is likely to be beta releases. Anyway, that means that they are actually averaging an update only 1/month, which is a fairly big window of updates to apply by the end of month (I would imagine). And while that does mean it takes longer to boot, it also means you would be open to more security holes without it. My contention is that if we don't *keep* it updated, we may as well just leave it to the user. When you create an instance in ec2 or Azure or whatever, it doesn't come fully up-to-date. You get the released image, and then you can update it if you want to. John =:- On Mon, Jun 30, 2014 at 5:05 PM, Andrew Wilkins andrew.wilk...@canonical.com wrote: Hi folks, I've been debugging a bootstrap bug [0] that was caused by ssh timing out (and the client not noticing), which was caused by apt-get upgrade taking an awfully long time (6 minutes on Azure). [0] https://bugs.launchpad.net/juju-core/+bug/1316185 I just filed https://bugs.launchpad.net/juju-core/+bug/1335822, and did a quick and dirty hack that brought the upgrade down to 3 minutes on Azure. I don't know the variance, so I can't be sure that it's all due to eatmydata, but smoser's results are similar. Even with eatmydata, a full bootstrap on Azure just took me 10 minutes. That's roughly broken down into: - apt-get update: 20s - apt-get upgrade: 3m20s - apt-get install various: 10s - Download tools (from shared Azure storage account): 5s - jujud bootstrap: 1m50s We could bring the 10m down to 6m40s. Still not brilliant, but considerably better IMO. I propose that we remove the apt-get upgrade altogether. Cloud images are regularly updated and tested, and I think we should be able to rely on that alone. If users want something more up-to-date, they can use the daily images which are not tested as a whole, but are composed of SRUs, which is effectively what users get today. Cheers, Andrew -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev -- Antonio Rosales Juju
