From: Waldemar Kozaczuk <jwkozac...@gmail.com> Committer: Waldemar Kozaczuk <jwkozac...@gmail.com> Branch: master
aws: add new script to easily deploy to and run OSv on EC2 This patch adds new script deploy_to_aws.sh which streamlines the process of uploading OSv image (usr.img) to AWS as a snapshot, creating AMI out of it and finally instantiating simple stack with a single EC2 instance. To run the scripts you need to have AWS cli installed and configured to work with your account. In addition, you also need to clone the snapshot tool https://github.com/awslabs/flexible-snapshot-proxy and adjust deploy_to_aws.sh accordingly to point to it on your local filesystem. The workflow is this: 1. Build your desired OSv image, for example (use fs_size_mb to limit the image size): ./scripts/build image=golang-pie-httpserver,httpserver-monitoring-api fs_size_mb=72 2. Run the image locally or use imgedit.py to set/change desired boot command line: ./scripts/imgedit.py setargs build/release/usr.img "<cmdline>" 3. Run deploy_to_aws.sh to upload the image to AWS and create the stack: ./scripts/deploy_to_aws.sh <name> Behind the scenes, deploy_to_aws.sh converts usr.img to usr.raw, then uploads usr.raw to AWS as a snapshot using flexible-snapshot-proxy tool and then creates AMI out of it (this patch adjusts ec2-make-ami.py to support creating AMI our of pre-created snapshost). Finally, it uses aws cli to create a stack with single EC2 instance based on the new AMI. Before you use deploy_to_aws.sh make sure to fill in the values of your account VPC ID and subnet ID in scripts/aws/instance-parameters.json. Also change instance type to the one you desire (the t2-* instances use Xen and t3, t4, etc use Nitro). The new process is much simpler and faster. You no longer need to run ec2-make-ami.py on some tool EC2 instance to build OSv ami which also takes long time. Instead you can build and deploy your OSv image locally under 30 seconds time. Signed-off-by: Waldemar Kozaczuk <jwkozac...@gmail.com> --- diff --git a/scripts/aws/instance-parameters.json b/scripts/aws/instance-parameters.json --- a/scripts/aws/instance-parameters.json +++ b/scripts/aws/instance-parameters.json @@ -0,0 +1,22 @@ +[ + { + "ParameterKey": "pVpcId", + "ParameterValue": "vpc-????????" + }, + { + "ParameterKey": "pSubnetId", + "ParameterValue": "subnet-????????" + }, + { + "ParameterKey": "pInstanceName", + "ParameterValue": "INSTANCE_NAME" + }, + { + "ParameterKey": "pInstanceType", + "ParameterValue": "t2.nano" + }, + { + "ParameterKey": "pImageId", + "ParameterValue": "AMI_ID" + } +] diff --git a/scripts/aws/instance.yaml b/scripts/aws/instance.yaml --- a/scripts/aws/instance.yaml +++ b/scripts/aws/instance.yaml @@ -0,0 +1,70 @@ +AWSTemplateFormatVersion: '2010-09-09' +Description: Create EC2 Instance +Parameters: + pVpcId: + Description: ID of the VPC + Type: AWS::EC2::VPC::Id + Default: '-' + pSubnetId: + Description: Subnet ID + Type: AWS::EC2::Subnet::Id + Default: '-' + pInstanceName: + Description: Instance Name + Type: String + Default: '-' + pInstanceType: + Description: Size of the Instance + Type: String + AllowedValues: + - t2.nano + - t2.micro + - t2.small + - t3.nano + - t3.micro + - t3.small + Default: t2.nano + pImageId: + Description: AMI for the instances + Type: AWS::EC2::Image::Id + Default: '-' +Resources: + InstanceSecurityGroup: + Type: AWS::EC2::SecurityGroup + Properties: + GroupDescription: Allow external addresses to access to management console + SecurityGroupIngress: + - IpProtocol: tcp + FromPort: '9000' + ToPort: '9000' + CidrIp: '0.0.0.0/0' + - IpProtocol: tcp + FromPort: '8000' + ToPort: '8000' + CidrIp: '0.0.0.0/0' + VpcId: + Ref: pVpcId + Instance: + Type: AWS::EC2::Instance + Properties: + ImageId: + Ref: pImageId + InstanceType: + Ref: pInstanceType + SecurityGroupIds: + - Ref: InstanceSecurityGroup + SubnetId: + Ref: pSubnetId + Tags: + - Key: Name + Value: + Ref: pInstanceName +Outputs: + PublicDnsName: + Value: + Fn::GetAtt: + - Instance + - PublicDnsName + + InstanceID: + Value: !Ref Instance diff --git a/scripts/deploy_to_aws.sh b/scripts/deploy_to_aws.sh --- a/scripts/deploy_to_aws.sh +++ b/scripts/deploy_to_aws.sh @@ -0,0 +1,24 @@ +#!/bin/bash +# + +NAME=$1 + +qemu-img convert -O raw build/release/usr.img build/release/usr.raw +echo "Converted to raw image" + +snapshot_id=$(python3 ~/projects/flexible-snapshot-proxy/src/main.py upload build/release/usr.raw | tail -n 1) +echo "Created snapshot: $snapshot_id" + +ami_id=$(./scripts/ec2-make-ami.py -n "$NAME" -s "$snapshot_id" | grep '^ami' | tail -n 1) +echo "Created AMI: $ami_id" + +cat scripts/aws/instance-parameters.json | sed -e "s/INSTANCE_NAME/$NAME/" | sed -e "s/AMI_ID/$ami_id/" > /tmp/instance-parameters.json +aws cloudformation create-stack \ + --stack-name $NAME \ + --template-body file://./scripts/aws/instance.yaml \ + --capabilities CAPABILITY_IAM \ + --parameters file:///tmp/instance-parameters.json + +#To clean +#aws ec2 deregister-image --image-id <id> +#aws ec2 delete-snapshot --snapshot-id <id> diff --git a/scripts/ec2-make-ami.py b/scripts/ec2-make-ami.py --- a/scripts/ec2-make-ami.py +++ b/scripts/ec2-make-ami.py @@ -1,12 +1,13 @@ -#!/usr/bin/python2 +#!/usr/bin/python3 import boto3, argparse, urllib, time, json, subprocess, os.path import argparse +from urllib.request import urlopen class Metadata(object): base = 'http://169.254.169.254/latest/meta-data/' def _get(self, what): - return urllib.urlopen(Metadata.base + what).read() + return urlopen(Metadata.base + what).read() def instance_id(self): return self._get('instance-id') def availability_zone(self): @@ -71,16 +72,24 @@ def make_snapshot(input): print('Snapshot {} created\n'.format(snap)) return snap.id -def make_ami_from_snapshot(name,snapshot_id): - metadata = Metadata() - print('Connecting') - ec2 = boto3.resource('ec2',region_name=metadata.region()) - print('STEP 7: Registering image from {}'.format(snapshot_id)) # aws ec2 register-image +def make_ami_from_snapshot(name,snapshot_id,on_ec2): + print('Connecting to make AMI from snapshot') + if on_ec2: + metadata = Metadata() + ec2 = boto3.resource('ec2',region_name=metadata.region()) + else: + ec2 = boto3.resource('ec2') + print('STEP 7: Registering image {} from {}'.format(name, snapshot_id)) # aws ec2 register-image time_point = time.time() + + snapshot = ec2.Snapshot(snapshot_id) + snapshot.wait_until_completed() + ami = ec2.register_image(Name=name, Architecture='x86_64', RootDeviceName='xvda', VirtualizationType='hvm', + EnaSupport=True, BlockDeviceMappings=[ { 'DeviceName' : 'xvda', @@ -92,6 +101,7 @@ def make_ami_from_snapshot(name,snapshot_id): ]) print('STEP 7: Took {} seconds to create ami'.format(time.time() - time_point,ami)) print('ami {} created\n'.format(ami)) + print('{}\n'.format(ami.id)) return ami if __name__ == "__main__": @@ -101,7 +111,13 @@ def make_ami_from_snapshot(name,snapshot_id): help="ami name to be created") parser.add_argument("-i", "--input", action="store", default="build/release.x64/usr.img", help="path to the image on local filesysten") + parser.add_argument("-s", "--snapshot", action="store", + help="snapshot ID") args = parser.parse_args() - snapshot_id = make_snapshot(args.input) - make_ami_from_snapshot(args.name,snapshot_id) + if args.snapshot == None: + snapshot_id = make_snapshot(args.input) + else: + snapshot_id = args.snapshot + + make_ami_from_snapshot(args.name, snapshot_id, args.snapshot == None) -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to osv-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/00000000000035b5a0060b638ef1%40google.com.
