@Grant @Gerard: WRT Static workers: I think that the downside of a slightly
longer start-up time is significantly less important than the massive
upshot in scalability offered by having non-static workers. This also opens
up really interesting opportunities (i.e. being able to identify how many
res
p.s. it looks like git-sync has received an "official" release since the
last time I looked at it: https://github.com/kubernetes/git-sync
On Thu, Jul 13, 2017 at 8:18 AM Jeremiah Lowin wrote:
> Hi Gerard (and anyone else for whom this might be helpful),
>
> We've run Airflow on GCP for a few yea
Hi Gerard (and anyone else for whom this might be helpful),
We've run Airflow on GCP for a few years. The structure has changed over
time but at the moment we use the following basic outline:
1. Build a container that includes all Airflow and DAG dependencies and
push it to Google container regis
It would be really good if you'd share experiences on how to run this on
kubernetes and ECS.
I'm not aware of a good guide on how to run this on either for example, but
it's a very useful and
quick setup to start with, especially combining that with deployment
manager and cloudformation (probably).
for what it's worth we've been running airflow on ECS for a few years
already.
On Wed, Jul 12, 2017 at 12:21 PM, Grant Nicholas <
grantnicholas2...@u.northwestern.edu> wrote:
> Is having a static set of workers necessary? Launching a job on Kubernetes
> from a cached docker image takes a few seco
Is having a static set of workers necessary? Launching a job on Kubernetes from
a cached docker image takes a few seconds max. I think this is an acceptable
delay for a batch processing system like airflow.
Additionally, if you dynamically launch workers you can start dynamically
launching *an
t;
>>> > > > - About rescaling and starting jobs: it doesn't come for free as
>>> you
>>> > > > mention. I think it's a great idea to be able to scale out on busy
>>> > > > intervals (we intend to just use cycle scaling her
gt; >
>> > > > Gerard
>> > > >
>> > > >
>> > > > On Wed, Jul 5, 2017 at 8:43 PM, Daniel Imberman <
>> > > daniel.imber...@gmail.com
>> > > > >
>> > > > wrote:
>> > > >
>&g
github.com/bloomberg/airflow/blob/29694ae9903c4dad3f18fb8eb767c4
> > > > > 922dbef2e8/dimberman-KubernetesExecutorProposal-050717-1423-36.pdf
> > > > >
> > > > > Daniel
> > > > >
> > > > > On Wed, Jul 5, 2017 at 11:36 AM Dan
> > > > Daniel
> > > >
> > > > On Wed, Jul 5, 2017 at 11:36 AM Daniel Imberman <
> > > daniel.imber...@gmail.com
> > > > >
> > > > wrote:
> > > >
> > > > > Hi Amit,
> > > > >
> > >
e:
> > >
> > > > Hi Amit,
> > > >
> > > > For now the design doc is included as an attachment to the original
> > > email.
> > > > Once I am able to get permission to edit the wiki I would like add it
> > > there
> > > > but for no
uld like add it
> > there
> > > but for now I figured that this would get the ball rolling.
> > >
> > >
> > > Daniel
> > >
> > >
> > > On Wed, Jul 5, 2017 at 11:33 AM Amit Kulkarni wrote:
> > >
> > >> Hi Dani
;>
> >>
> >> Amit Kulkarni
> >> Site Reliability Engineer
> >> Mobile: (716)-352-3270 <(716)%20352-3270>
> >>
> >> Payments partner to the platform economy
> >>
> >> On Wed, Jul 5, 2017 at 11:25 AM, Daniel Imberman <
>
gt;>
>> Payments partner to the platform economy
>>
>> On Wed, Jul 5, 2017 at 11:25 AM, Daniel Imberman <
>> daniel.imber...@gmail.com>
>> wrote:
>>
>> > Hello Airflow community!
>> >
>> > My name is Daniel Imberman, and I hav
...@gmail.com>
> wrote:
>
> > Hello Airflow community!
> >
> > My name is Daniel Imberman, and I have been working on behalf of
> Bloomberg
> > LP to create an airflow kubernetes executor/operator. We wanted to allow
> > for maximum throughput/scalability, w
een working on behalf of Bloomberg
> LP to create an airflow kubernetes executor/operator. We wanted to allow
> for maximum throughput/scalability, while keeping a lot of the kubernetes
> details abstracted away from the users. Below I have a link to the WIP PR
> and the PDF of the initi
Hello Airflow community!
My name is Daniel Imberman, and I have been working on behalf of Bloomberg
LP to create an airflow kubernetes executor/operator. We wanted to allow
for maximum throughput/scalability, while keeping a lot of the kubernetes
details abstracted away from the users. Below I
17 matches
Mail list logo
