Hey Airflow devs,
Following up from the conversation a couple of weeks ago, let's get
together to sort through the issue triage process. I believe that the group
who had volunteered to help standardize the issue labelling and triage
process was:
- Elad Kalif @eladkal
- Asif Saif Uddin @auvipy
-
Really hard to say now. But I did some - rather generic - calculations
https://cloud.google.com/products/calculator#id=abb18f23-0ea5-495e-a1fc-9cca1953096b
and is some 400 USD /month. But I think when we connect it with free tier
from GA, it could be half that I think.
J.
On Tue, Oct 13, 2020 at
What are the estimated yearly costs?
On Tue, Oct 13, 2020 at 9:17 AM Jarek Potiuk
wrote:
> Yep, we can do it: *docker build --cpu-shares=100 --memory=1024m *
>
> On Tue, Oct 13, 2020 at 6:15 PM Jarek Potiuk
> wrote:
>
>> Plus the "workflow_runs" (image building) for all PRs can also be done in
I'm proud to announce the availability of Apache Airlow 2.0.0.alpha1 for
testing!
First the caveat: this is an alpha release. Do not run it in production, it
might not be without serious problems, and in the extreme case you may have to
reset your database between this and the beta or release
I do expect some small teething problems again, but I hope the big one is
over and I will try to address those problems if they arise. Apologies for
that - this was rather difficult to test on "Apache Organization" scale. We
are also talking about adding some github custom runners, because we
There is a bad news and a good news :).
* The bad one is that the change did not go well with its original scope.
It turned out that many small jobs are not a good idea when you have 180
slots in a queue and a number (growing) of Apache projects and yours are
competing for those. Seems that our
Yep, we can do it: *docker build --cpu-shares=100 --memory=1024m *
On Tue, Oct 13, 2020 at 6:15 PM Jarek Potiuk
wrote:
> Plus the "workflow_runs" (image building) for all PRs can also be done in
> the self-hosted workers. They are safe as they are using master scripts
> (the only potentially
re: security conerns, this is a case where we could require committer
approval before running full tests (though leaves the risk that a PR is
approved for testing and then the user adds something concerning after).
via Newton Mail
Plus the "workflow_runs" (image building) for all PRs can also be done in
the self-hosted workers. They are safe as they are using master scripts
(the only potentially dangerous part in them is that someone could do some
"mining" as "malicious" Docker image building step, This is the only part
I think this part is easy:
* First of all - It is similar to GA - someone could have used all the 180
workers of Apache by submitting PRs to various projects. So we just need a
limited worker queue. All those can run as workers in GKE and it should be
easy to manage (we could have auto-scaling
Yep. Now we just need credits :)
On Tue, Oct 13, 2020 at 5:30 PM Kaxil Naik wrote:
> That's ace, we should go ahead with self-hosted runners then.
>
> On Tue, Oct 13, 2020 at 4:06 PM Ash Berlin-Taylor wrote:
>
>> Confirmed, we *can* do it - Arrow has done it already
>>
That's ace, we should go ahead with self-hosted runners then.
On Tue, Oct 13, 2020 at 4:06 PM Ash Berlin-Taylor wrote:
> Confirmed, we *can* do it - Arrow has done it already
> https://issues.apache.org/jira/browse/INFRA-19875
>
> But lets have a think on how to not be a bot net :)
>
> On Oct
Confirmed, we can do it - Arrow has done it already
https://issues.apache.org/jira/browse/INFRA-19875
But lets have a think on how to not be a bot net :)
On Oct 13 2020, at 3:59 pm, Ash Berlin-Taylor wrote:
> I've spoken to a few members of ASF Infra directly, and they are just
> confirming
I've spoken to a few members of ASF Infra directly, and they are just
confirming but they are okay with the idea of us adding self hosted runners to
our repo, and also okay that we can manage those nodes ourselves. Should get
final confirmation today.
I wanted to double check that we could use
This is also a slight problem as mentioned in the build@ thread:
https://lists.apache.org/thread.html/r1708881f52adbdae722afb8fea16b23325b739b254b60890e72375e1%40%3Cbuilds.apache.org%3E
- managing hosting runners has to be done through infrastructure and they
are not really responsive recently (I
I've thought about private/self-hosted runners, and I think long term that's
the way to go to alievate our CI bottlenecks.
There's a bit of work we need to do around security of builds - as mentioned
here
Hello Aizhamal, Everyone,
We've had some problems recently with concurrency for Github Actions and
suggested solution for now is to use self-hosted runners (This is suggested
by GitHub Support)
I made some comments in the issue here:
https://github.com/apache/airflow/issues/11496
And also
17 matches
Mail list logo
