FYI, I've filed a bug for the issue below: https://bugs.launchpad.net/juju-core/+bug/1609343
And there's a live tested fix up for review: https://github.com/juju/juju/pull/5922 James, I'd appreciate if you can give the fix a try (let me know by mail or in #juju-dev on FreeNode IRC, if you need some help how to do that)! Cheers, Dimiter On 08/03/2016 01:17 PM, Dimiter Naydenov wrote: > Hey James, > > Thanks for the detailed information! I think you did find a bug, let me > try to explain. > > First, it looks like your vpc-ff069a98 meets the minimum connectivity > requirements of Juju (i.e. it's deemed suitable to host a controller), > so passing --config vpc-id-force=true at bootstrap shouldn't be > necessary (if it is, I'd like to know more about how your VPC's subnets > and route tables look like, if possible!). > > The bug I think you've discovered is related to how Juju does automatic > distribution of instances across AZ for redundancy. Since EC2 subnets > can only be part of one AZ, and when using a non-default VPC the AWS API > requires you to pass SubnetID, not just AvailZone to RunInstances used > to start an instance. Security groups are VPC-specific as well. > > The essence of the issue I think is demonstrated here: > > 2016-08-02 16:57:13 INFO juju.provider.ec2 environ.go:516 selected > random subnet "subnet-f30d3fd9" from all matching in zone "us-east-1a": > [subnet-9b013ab1 subnet-f30d3fd9 subnet-fb380ad1] > > I strongly suspect those other 2 subnets: subnet-9b013ab1, > subnet-fb380ad1 in AZ us-east-1a are part of a different VPC, but are > still picked up "randomly" during StartInstance in the EC2 provider. > > In other words Juju is not filtering subnets in a given AZ by the > user-specified vpc-id when deciding which one to pick. > > I looked at the code and it looks easy to fix this, fortunately. > If I give you a patch to apply on top of the tip of juju/juju master > branch, would you be willing to give it a try to see if it works? > > Thanks for your patience! I'll file a bug about what you've discovered > and start working on a fix for beta14! > > Cheers, > Dimiter > > On 08/02/2016 08:33 PM, James Beedy wrote: >> Ok, here we go .... hopefully this will provide a better overhead view. >> >> `juju bootstrap creativedrive aws --credential creativedrive --config >> vpc-id=vpc-ff069a98 --config vpc-id-force='true' --upload-tools --debug >> --config logging-config='<root>=TRACE'` <- http://paste.ubuntu.com/21914113/ >> >> cat machine-0.log <- http://paste.ubuntu.com/21914847/ >> >> >> ### debug add-model >> >> `juju add-model consul --credential creativedrive --config >> vpc-id=vpc-ff069a98 --config vpc-id-force='true' --debug` <- >> http://paste.ubuntu.com/21915182/ >> >> ### juju status on new model >> >> juju status --format yaml <- http://paste.ubuntu.com/21915512/ >> >> ### debug add space and subnet >> >> `juju add-space common-infrastructure --debug` <- >> http://paste.ubuntu.com/21915888/ >> >> #### add a subnet in my vpc to the newly created space >> `juju add-subnet subnet-9b013ab1 common-infrastructure us-east-1a >> --debug` <- http://paste.ubuntu.com/21916055/ >> >> ### list spaces and subnets >> >> `juju subnets && juju spaces` <- http://paste.ubuntu.com/21916258/ >> >> >> ### deploy something to the model >> `juju deploy ubuntu --debug` <- http://paste.ubuntu.com/21916540/ >> >> >> ### deploy something to a network space >> >> `juju deploy ubuntu ubuntu-spaces --constraints >> spaces=common-infrastructure --debug` <- http://paste.ubuntu.com/21916804/ >> >> >> ### juju status now shows a success for the machine deployed to the >> model w/out a space constraint, and and error for the instance deployed >> to the space. >> >> `juju status --format yaml` <- http://paste.ubuntu.com/21917006/ >> >> >> No matter what, I can't seem to get anything deployed to a "space".... >> >> So strange ... possibly I have stumbled upon a bug? >> >> >> Thanks again for your insight here. >> >> >> >> On Tue, Aug 2, 2016 at 9:51 AM, James Beedy <[email protected] >> <mailto:[email protected]>> wrote: >> >> To my utter dismay, setting the correct config 'vpc-id-force' gave >> me the same result.... >> >> Let me scrub and collect my machine-0.log for you. >> >> >> >> >> On Tue, Aug 2, 2016 at 9:36 AM, James Beedy <[email protected] >> <mailto:[email protected]>> wrote: >> >> Dimiter, >> >> Thanks for the insight. >> / >> / >> /Can you please also paste the full logs (scrubbed of secrets) >> of `juju >> bootstrap ... --debug` (with the vpc-id etc., but please also >> include >> `--config logging-config='<root>=TRACE'`), and machine-0.log from >> /var/log/juju on the bootstrap node, once completed? That will help >> figuring out the issue. >> / >> `juju bootstrap creativedrive aws --credential creativedrive >> --config vpc-id=vpc-ff069a98 --config force-vpc-id='true' >> --config loggin-config='<root>=TRACE' --upload-tools --debug` <- >> http://paste.ubuntu.com/21908548/ >> >> machine-0.log shows "2016-08-02 16:16:16 TRACE juju.apiserver >> request_notifier.go:127 -> [2] machine-0 >> >> {"request-id":53,"response":{"config":{"access-key":"","agent-version":"2.0-beta13","authorized-keys":"juju-client-key\nssh-rsa >> ssh-rsa >> >> juju-system-key\n","automatically-retry-hooks":true,"default-series":"","development":false,"disable-network-management":false,"firewall-mode":"instance","force-vpc-id":true,"ignore-machine-addresses":false,"logging-config":"\u003croot\u003e=TRACE;unit=DEBUG","name":"controller","proxy-ssh":false,"region":"us-east-1","secret-key":"/E","ssl-hostname-verification":true,"storage-default-block-source":"ebs","test-mode":false,"type":"ec2","uuid":"259be235-a255-416d-8bbf-75e128d05794","vpc-id":"vpc-ff069a98","vpc-id-force":false}}}" >> >> >> Just realizing now, I have been specifying 'vpc-force-id', not >> 'vpc-id-force' (grrrr). >> >> I would expect to see this resolved when I apply the correct >> config. I'll report back shortly. >> >> Thanks for your time! >> >> /From what I can understand, you're trying to bootstrap on a >> non-default, >> possibly private VPC (accessed via its internal address over a VPN >> connection maybe?), and then add a model with the same VPC and >> credentials. >> >> / >> ^ Exactly/ >> / >> / >> If OTOH, the VPC used for add-model is different, the >> machines there won't be able to talk to the controller's VPC >> unless it >> has a public address (cross VPC communication currently relies >> on having >> that, fancier setups with VPN gateways is not yet supported)./ >> >> ^ >> >> The error in status implies 2 separate VPCs are used (or a VPC and >> EC2-Classic - i.e. no VPC) for the controller and hosted model. >> >> Cheers, >> Dimiter >> >> >> >> >> > > > > -- Dimiter Naydenov <[email protected]> Juju Core Sapphire team <http://juju.ubuntu.com>
signature.asc
Description: OpenPGP digital signature
-- Juju-dev mailing list [email protected] Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
