ashb commented on pull request #14531: URL: https://github.com/apache/airflow/pull/14531#issuecomment-796635976
> Later on ( next few weekends I guess) I will take a stab on #14656 -> I think introducing two types of self-hosted runners will help us much further in driving the costs significantly down and we will be able to upgrade the self-hosted to 8CPU /64 and get the full benefit of 15 minutes test execution (or we can decide to do it even now). But it hurts seing 8CPU 64 GB machine used to build an image or wait for images being built. Yeah I agree. Rough steps we will need to do to have two "pools" of runners: 1. Get ASF infra to adjust the labels on the existing runners we have configured. Currently they just have `self-hosted` label -- we should ask them to set all 100 to additionally have a `large` label or something 1. Update https://github.com/apache/airflow-ci-infra/blob/master/scripts/runner-supervisor.py to pull credentials from a different path in AWS SSM. Right now it looks for creds in `/runners/apache/airflow` -- i.e. `/runners/apache/airflow/1`...`/runners/apache/airflow/100` We would probably have to change that to look at `/runners/apache/airflow/{size}/{idx}` 1. Open a ticket with Infra, and ask them to create a new runner registration token. Use that token and the https://github.com/apache/airflow-ci-infra/blob/master/scripts/store-agent-creds.py script to store those creds in SSM. Script will need updating to change/add labels and store in a different location too. 1. Create a second ASG with smaller instances, also configuring/changing the userdata so the instance starts in the right "pool". 1. The Lambda function will need updating to know which ASG to scale at based on the job name. I _think_ we might have to just duplicate the job ids and sizes in to the runner, as we don't get much info in the webhook payload (I've included a sample one below in the details block) There may be a better approach here. I'm more than happy to help or do all of the above myself if you'd like, just laying out the steps that come to mind. <details> <summary>Webhook payload</summary> <pre><code>{ "action": "created", "check_run": { "id": 2085234698, "node_id": "MDg6Q2hlY2tSdW4yMDg1MjM0Njk4", "head_sha": "9070ba280253e918c2582f52c2b77a07c311198f", "external_id": "ff9e9115-6980-59bb-9ae7-308d64bdf685", "url": "https://api.github.com/repos/apache/airflow/check-runs/2085234698";, "html_url": "https://github.com/apache/airflow/runs/2085234698";, "details_url": "https://github.com/apache/airflow/runs/2085234698";, "status": "queued", "conclusion": null, "started_at": "2021-03-11T10:09:49Z", "completed_at": null, "output": { "title": null, "summary": null, "text": null, "annotations_count": 0, "annotations_url": "https://api.github.com/repos/apache/airflow/check-runs/2085234698/annotations"; }, "name": "K8s 3.7 v1.16.9 image", "check_suite": { "id": 2233450802, "node_id": "MDEwOkNoZWNrU3VpdGUyMjMzNDUwODAy", "head_branch": "master", "head_sha": "9070ba280253e918c2582f52c2b77a07c311198f", "status": "queued", "conclusion": null, "url": "https://api.github.com/repos/apache/airflow/check-suites/2233450802";, "before": "943baff6701f9f8591090bf76219571d7f5e2cc5", "after": "9070ba280253e918c2582f52c2b77a07c311198f", "pull_requests": [ { "url": "https://api.github.com/repos/rpatil524/airflow/pulls/80";, "id": 581521861, "number": 80, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "fff344443f764a0a5c9f5ba7dfbfa1e5e1115348", "repo": { "id": 336576614, "url": "https://api.github.com/repos/rpatil524/airflow";, "name": "airflow" } } }, { "url": "https://api.github.com/repos/yufeixin/airflow/pulls/305";, "id": 581371796, "number": 305, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "f9cc775adc0dff49b0e288d8f2745bf097017321", "repo": { "id": 320438482, "url": "https://api.github.com/repos/yufeixin/airflow";, "name": "airflow" } } }, { "url": "https://api.github.com/repos/ddelange/airflow/pulls/6";, "id": 571967064, "number": 6, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "f9c9e9c38f444a39987478f3d1a262db909de8c4", "repo": { "id": 219798743, "url": "https://api.github.com/repos/ddelange/airflow";, "name": "airflow" } } }, { "url": "https://api.github.com/repos/weiplanet/airflow/pulls/1";, "id": 562187084, "number": 1, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "2345cd1f0337f4f06a2f2595a1a59c30dc4a1535", "repo": { "id": 303579760, "url": "https://api.github.com/repos/weiplanet/airflow";, "name": "airflow" } } }, { "url": "https://api.github.com/repos/TestingCodeReview/airflow/pulls/1";, "id": 503500807, "number": 1, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "0f02e45b7e127da3539752607ab02347d9fcd733", "repo": { "id": 169064440, "url": "https://api.github.com/repos/TestingCodeReview/airflow";, "name": "airflow" } } }, { "url": "https://api.github.com/repos/mehrdad-shokri/incubator-airflow/pulls/2";, "id": 470767538, "number": 2, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "5739ba28a7d8634ba3702d5fa39e5ef663f5bf7c", "repo": { "id": 110538040, "url": "https://api.github.com/repos/mehrdad-shokri/incubator-airflow";, "name": "incubator-airflow" } } } ], "app": { "id": 15368, "slug": "github-actions", "node_id": "MDM6QXBwMTUzNjg=", "owner": { "login": "github", "id": 9919, "node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4";, "gravatar_id": "", "url": "https://api.github.com/users/github";, "html_url": "https://github.com/github";, "followers_url": "https://api.github.com/users/github/followers";, "following_url": "https://api.github.com/users/github/following{/other_user}";, "gists_url": "https://api.github.com/users/github/gists{/gist_id}";, "starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}";, "subscriptions_url": "https://api.github.com/users/github/subscriptions";, "organizations_url": "https://api.github.com/users/github/orgs";, "repos_url": "https://api.github.com/users/github/repos";, "events_url": "https://api.github.com/users/github/events{/privacy}";, "received_events_url": "https://api.github.com/users/github/received_events";, "type": "Organization", "site_admin": false }, "name": "GitHub Actions", "description": "Automate your workflow from idea to production", "external_url": "https://help.github.com/en/actions";, "html_url": "https://github.com/apps/github-actions";, "created_at": "2018-07-30T09:30:17Z", "updated_at": "2019-12-10T19:04:12Z", "permissions": { "actions": "write", "checks": "write", "contents": "write", "deployments": "write", "issues": "write", "metadata": "read", "organization_packages": "write", "packages": "write", "pages": "write", "pull_requests": "write", "repository_hooks": "write", "repository_projects": "write", "security_events": "write", "statuses": "write", "vulnerability_alerts": "read" }, "events": [ "check_run", "check_suite", "create", "delete", "deployment", "deployment_status", "fork", "gollum", "issues", "issue_comment", "label", "milestone", "page_build", "project", "project_card", "project_column", "public", "pull_request", "pull_request_review", "pull_request_review_comment", "push", "registry_package", "release", "repository", "repository_dispatch", "status", "watch", "workflow_dispatch", "workflow_run" ] }, "created_at": "2021-03-11T09:44:47Z", "updated_at": "2021-03-11T10:09:45Z" }, "app": { "id": 15368, "slug": "github-actions", "node_id": "MDM6QXBwMTUzNjg=", "owner": { "login": "github", "id": 9919, "node_id": "MDEyOk9yZ2FuaXphdGlvbjk5MTk=", "avatar_url": "https://avatars.githubusercontent.com/u/9919?v=4";, "gravatar_id": "", "url": "https://api.github.com/users/github";, "html_url": "https://github.com/github";, "followers_url": "https://api.github.com/users/github/followers";, "following_url": "https://api.github.com/users/github/following{/other_user}";, "gists_url": "https://api.github.com/users/github/gists{/gist_id}";, "starred_url": "https://api.github.com/users/github/starred{/owner}{/repo}";, "subscriptions_url": "https://api.github.com/users/github/subscriptions";, "organizations_url": "https://api.github.com/users/github/orgs";, "repos_url": "https://api.github.com/users/github/repos";, "events_url": "https://api.github.com/users/github/events{/privacy}";, "received_events_url": "https://api.github.com/users/github/received_events";, "type": "Organization", "site_admin": false }, "name": "GitHub Actions", "description": "Automate your workflow from idea to production", "external_url": "https://help.github.com/en/actions";, "html_url": "https://github.com/apps/github-actions";, "created_at": "2018-07-30T09:30:17Z", "updated_at": "2019-12-10T19:04:12Z", "permissions": { "actions": "write", "checks": "write", "contents": "write", "deployments": "write", "issues": "write", "metadata": "read", "organization_packages": "write", "packages": "write", "pages": "write", "pull_requests": "write", "repository_hooks": "write", "repository_projects": "write", "security_events": "write", "statuses": "write", "vulnerability_alerts": "read" }, "events": [ "check_run", "check_suite", "create", "delete", "deployment", "deployment_status", "fork", "gollum", "issues", "issue_comment", "label", "milestone", "page_build", "project", "project_card", "project_column", "public", "pull_request", "pull_request_review", "pull_request_review_comment", "push", "registry_package", "release", "repository", "repository_dispatch", "status", "watch", "workflow_dispatch", "workflow_run" ] }, "pull_requests": [ { "url": "https://api.github.com/repos/rpatil524/airflow/pulls/80";, "id": 581521861, "number": 80, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "fff344443f764a0a5c9f5ba7dfbfa1e5e1115348", "repo": { "id": 336576614, "url": "https://api.github.com/repos/rpatil524/airflow";, "name": "airflow" } } }, { "url": "https://api.github.com/repos/yufeixin/airflow/pulls/305";, "id": 581371796, "number": 305, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "f9cc775adc0dff49b0e288d8f2745bf097017321", "repo": { "id": 320438482, "url": "https://api.github.com/repos/yufeixin/airflow";, "name": "airflow" } } }, { "url": "https://api.github.com/repos/ddelange/airflow/pulls/6";, "id": 571967064, "number": 6, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "f9c9e9c38f444a39987478f3d1a262db909de8c4", "repo": { "id": 219798743, "url": "https://api.github.com/repos/ddelange/airflow";, "name": "airflow" } } }, { "url": "https://api.github.com/repos/weiplanet/airflow/pulls/1";, "id": 562187084, "number": 1, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "2345cd1f0337f4f06a2f2595a1a59c30dc4a1535", "repo": { "id": 303579760, "url": "https://api.github.com/repos/weiplanet/airflow";, "name": "airflow" } } }, { "url": "https://api.github.com/repos/TestingCodeReview/airflow/pulls/1";, "id": 503500807, "number": 1, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "0f02e45b7e127da3539752607ab02347d9fcd733", "repo": { "id": 169064440, "url": "https://api.github.com/repos/TestingCodeReview/airflow";, "name": "airflow" } } }, { "url": "https://api.github.com/repos/mehrdad-shokri/incubator-airflow/pulls/2";, "id": 470767538, "number": 2, "head": { "ref": "master", "sha": "9070ba280253e918c2582f52c2b77a07c311198f", "repo": { "id": 33884891, "url": "https://api.github.com/repos/apache/airflow";, "name": "airflow" } }, "base": { "ref": "master", "sha": "5739ba28a7d8634ba3702d5fa39e5ef663f5bf7c", "repo": { "id": 110538040, "url": "https://api.github.com/repos/mehrdad-shokri/incubator-airflow";, "name": "incubator-airflow" } } } ] }, "repository": { "id": 33884891, "node_id": "MDEwOlJlcG9zaXRvcnkzMzg4NDg5MQ==", "name": "airflow", "full_name": "apache/airflow", "private": false, "owner": { "login": "apache", "id": 47359, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ3MzU5", "avatar_url": "https://avatars.githubusercontent.com/u/47359?v=4";, "gravatar_id": "", "url": "https://api.github.com/users/apache";, "html_url": "https://github.com/apache";, "followers_url": "https://api.github.com/users/apache/followers";, "following_url": "https://api.github.com/users/apache/following{/other_user}";, "gists_url": "https://api.github.com/users/apache/gists{/gist_id}";, "starred_url": "https://api.github.com/users/apache/starred{/owner}{/repo}";, "subscriptions_url": "https://api.github.com/users/apache/subscriptions";, "organizations_url": "https://api.github.com/users/apache/orgs";, "repos_url": "https://api.github.com/users/apache/repos";, "events_url": "https://api.github.com/users/apache/events{/privacy}";, "received_events_url": "https://api.github.com/users/apache/received_events";, "type": "Organization", "site_admin": false }, "html_url": "https://github.com/apache/airflow";, "description": "Apache Airflow - A platform to programmatically author, schedule, and monitor workflows", "fork": false, "url": "https://api.github.com/repos/apache/airflow";, "forks_url": "https://api.github.com/repos/apache/airflow/forks";, "keys_url": "https://api.github.com/repos/apache/airflow/keys{/key_id}";, "collaborators_url": "https://api.github.com/repos/apache/airflow/collaborators{/collaborator}";, "teams_url": "https://api.github.com/repos/apache/airflow/teams";, "hooks_url": "https://api.github.com/repos/apache/airflow/hooks";, "issue_events_url": "https://api.github.com/repos/apache/airflow/issues/events{/number}";, "events_url": "https://api.github.com/repos/apache/airflow/events";, "assignees_url": "https://api.github.com/repos/apache/airflow/assignees{/user}";, "branches_url": "https://api.github.com/repos/apache/airflow/branches{/branch}";, "tags_url": "https://api.github.com/repos/apache/airflow/tags";, "blobs_url": "https://api.github.com/repos/apache/airflow/git/blobs{/sha}";, "git_tags_url": "https://api.github.com/repos/apache/airflow/git/tags{/sha}";, "git_refs_url": "https://api.github.com/repos/apache/airflow/git/refs{/sha}";, "trees_url": "https://api.github.com/repos/apache/airflow/git/trees{/sha}";, "statuses_url": "https://api.github.com/repos/apache/airflow/statuses/{sha}";, "languages_url": "https://api.github.com/repos/apache/airflow/languages";, "stargazers_url": "https://api.github.com/repos/apache/airflow/stargazers";, "contributors_url": "https://api.github.com/repos/apache/airflow/contributors";, "subscribers_url": "https://api.github.com/repos/apache/airflow/subscribers";, "subscription_url": "https://api.github.com/repos/apache/airflow/subscription";, "commits_url": "https://api.github.com/repos/apache/airflow/commits{/sha}";, "git_commits_url": "https://api.github.com/repos/apache/airflow/git/commits{/sha}";, "comments_url": "https://api.github.com/repos/apache/airflow/comments{/number}";, "issue_comment_url": "https://api.github.com/repos/apache/airflow/issues/comments{/number}";, "contents_url": "https://api.github.com/repos/apache/airflow/contents/{+path}";, "compare_url": "https://api.github.com/repos/apache/airflow/compare/{base}...{head}";, "merges_url": "https://api.github.com/repos/apache/airflow/merges";, "archive_url": "https://api.github.com/repos/apache/airflow/{archive_format}{/ref}";, "downloads_url": "https://api.github.com/repos/apache/airflow/downloads";, "issues_url": "https://api.github.com/repos/apache/airflow/issues{/number}";, "pulls_url": "https://api.github.com/repos/apache/airflow/pulls{/number}";, "milestones_url": "https://api.github.com/repos/apache/airflow/milestones{/number}";, "notifications_url": "https://api.github.com/repos/apache/airflow/notifications{?since,all,participating}";, "labels_url": "https://api.github.com/repos/apache/airflow/labels{/name}";, "releases_url": "https://api.github.com/repos/apache/airflow/releases{/id}";, "deployments_url": "https://api.github.com/repos/apache/airflow/deployments";, "created_at": "2015-04-13T18:04:58Z", "updated_at": "2021-03-11T09:44:52Z", "pushed_at": "2021-03-11T09:58:35Z", "git_url": "git://github.com/apache/airflow.git", "ssh_url": "[email protected]:apache/airflow.git", "clone_url": "https://github.com/apache/airflow.git";, "svn_url": "https://github.com/apache/airflow";, "homepage": "https://airflow.apache.org/";, "size": 111044, "stargazers_count": 20723, "watchers_count": 20723, "language": "Python", "has_issues": true, "has_projects": true, "has_downloads": true, "has_wiki": false, "has_pages": false, "forks_count": 8134, "mirror_url": null, "archived": false, "disabled": false, "open_issues_count": 987, "license": { "key": "apache-2.0", "name": "Apache License 2.0", "spdx_id": "Apache-2.0", "url": "https://api.github.com/licenses/apache-2.0";, "node_id": "MDc6TGljZW5zZTI=" }, "forks": 8134, "open_issues": 987, "watchers": 20723, "default_branch": "master" }, "organization": { "login": "apache", "id": 47359, "node_id": "MDEyOk9yZ2FuaXphdGlvbjQ3MzU5", "url": "https://api.github.com/orgs/apache";, "repos_url": "https://api.github.com/orgs/apache/repos";, "events_url": "https://api.github.com/orgs/apache/events";, "hooks_url": "https://api.github.com/orgs/apache/hooks";, "issues_url": "https://api.github.com/orgs/apache/issues";, "members_url": "https://api.github.com/orgs/apache/members{/member}";, "public_members_url": "https://api.github.com/orgs/apache/public_members{/member}";, "avatar_url": "https://avatars.githubusercontent.com/u/47359?v=4";, "description": "" }, "sender": { "login": "ashb", "id": 34150, "node_id": "MDQ6VXNlcjM0MTUw", "avatar_url": "https://avatars.githubusercontent.com/u/34150?v=4";, "gravatar_id": "", "url": "https://api.github.com/users/ashb";, "html_url": "https://github.com/ashb";, "followers_url": "https://api.github.com/users/ashb/followers";, "following_url": "https://api.github.com/users/ashb/following{/other_user}";, "gists_url": "https://api.github.com/users/ashb/gists{/gist_id}";, "starred_url": "https://api.github.com/users/ashb/starred{/owner}{/repo}";, "subscriptions_url": "https://api.github.com/users/ashb/subscriptions";, "organizations_url": "https://api.github.com/users/ashb/orgs";, "repos_url": "https://api.github.com/users/ashb/repos";, "events_url": "https://api.github.com/users/ashb/events{/privacy}";, "received_events_url": "https://api.github.com/users/ashb/received_events";, "type": "User", "site_admin": false }, "installation": { "id": 14172953, "node_id": "MDIzOkludGVncmF0aW9uSW5zdGFsbGF0aW9uMTQxNzI5NTM=" } }</code></pre> </details> ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
