hussein-awala commented on PR #59:
URL: https://github.com/apache/airflow-ci-infra/pull/59#issuecomment-1676427829
@ashb I made some changes in this PR:
- I recreated the whole stack to delete the NAT gateway and move the EKS
cluster node to a public subnet
- I removed all the files used in the dockerfile, and now I'm extending the
official docker image (ubunto 22.04 instead of 20.04)
But there are a few more to discussM
> Our current runners are running on any of the following instance types --
which ever is the cheapest spot price
r6a.2xlarge
r5.2xlarge
r5ad.2xlarge
r5n.2xlarge
r5b.2xlarge
r5a.2xlarge
r6i.2xlarge
All these instances are AMD, one of the goals was migrating completely to
AMR (Graviton), according to your list, we can use r6g, WDYT? I created a new
var to easily define the instance types:
```hcl
variable "runners_node_types" {
description = "Node type for the runners"
type = list(string)
default = ["t4g"]
}
```
However, I wonder if the usage of different types with different cpu/ram
ratios is a good idea, because when we update a job and it starts to reach the
memory limit, we will not be able to detect the problem easily, because
sometimes we will run it on a node with a bigger ram. Personally I don't like
mixing the node types, but if there is a benefit for the price and the
availability of spot instances, I'm ok with that.
> And we mount things as tmpfs using ~50% of the RAM (I thought):
- [tmpfs, /var/lib/docker, tmpfs, "defaults,noatime,size=85%"]
- [tmpfs, /home/runner/actions-runner/_work, tmpfs, "defaults,noatime"]
Not ready yet, I will check the current runners and try to do the same thing
The runner consider the version `v2.304.0-airflow8` as deprecated and it
update it by default:
```
√ Connected to GitHub
Current runner version: '2.304.0-airflow8'
2023-08-13 15:07:22Z: Listening for Jobs
Runner update in progress, do not shutdown runner.
Downloading 2.307.1 runner
Waiting for current job finish running.
Generate and execute update script.
Runner will exit shortly for update, should be back online within 10 seconds.
Runner update process finished.
Runner listener exit because of updating, re-launch runner after successful
update
Update finished successfully.
Restarting runner...
√ Connected to GitHub
Current runner version: '2.307.1'
2023-08-13 15:07:35Z: Listening for Jobs
2023-08-13 15:07:36Z: Running job: Build info
```
And when I added a conf to disable auto update, the container entered in a
crush loop:
```
√ Connected to GitHub
Current runner version: '2.304.0-airflow8'
2023-08-13 14:15:41Z: Listening for Jobs
An error occurred: Runner version v2.304.0-airflow8 is deprecated and cannot
receive messages.
Runner listener exit with retryable error, re-launch runner in 5 seconds.
Restarting runner...
```
Is there a reason to not upgrade the Airflow runner to 2.307.1 or it's just
because of a lack of free time? If it's related to your time, I can try it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
