[jira] [Commented] (AIRFLOW-2975) Add support for Amazon cloudwatch (computing power at will)
[ https://issues.apache.org/jira/browse/AIRFLOW-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596223#comment-16596223 ] Ash Berlin-Taylor commented on AIRFLOW-2975: It depends on your workload - if you are mostly network bound (i.e. lots of HTTP requests, or EMR calls etc) you could go much higher, but 1 or 2 slots per core is a good starting point, yes. > Add support for Amazon cloudwatch (computing power at will) > --- > > Key: AIRFLOW-2975 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2975 > Project: Apache Airflow > Issue Type: New Feature >Affects Versions: 1.10.0 >Reporter: jack >Priority: Major > Fix For: 2.0.0, 1.10.1 > > > Some of have one machine that runs airfow… > While we can scale up the executor to have many resources over different > servers this is consider to be expensive. > There is another solution.. using Amazon Cloudwatch: > [https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/] > > This enable the user to create and close EC2 machines on specific intervals > for specific tasks. > Basically if I have 50 DAGS to run on 1PM-3PM and few dags to run on other > hours there is no point in paying for 2nd server 24/7. > This could be an enhancement to one of the executors that know to work with > more than one server. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2975) Add support for Amazon cloudwatch (computing power at will)
[ https://issues.apache.org/jira/browse/AIRFLOW-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596216#comment-16596216 ] jack commented on AIRFLOW-2975: --- [~ashb] How can I optimize the number of workers? 1 per core? > Add support for Amazon cloudwatch (computing power at will) > --- > > Key: AIRFLOW-2975 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2975 > Project: Apache Airflow > Issue Type: New Feature >Affects Versions: 1.10.0 >Reporter: jack >Priority: Major > Fix For: 2.0.0, 1.10.1 > > > Some of have one machine that runs airfow… > While we can scale up the executor to have many resources over different > servers this is consider to be expensive. > There is another solution.. using Amazon Cloudwatch: > [https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/] > > This enable the user to create and close EC2 machines on specific intervals > for specific tasks. > Basically if I have 50 DAGS to run on 1PM-3PM and few dags to run on other > hours there is no point in paying for 2nd server 24/7. > This could be an enhancement to one of the executors that know to work with > more than one server. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (AIRFLOW-2975) Add support for Amazon cloudwatch (computing power at will)
[ https://issues.apache.org/jira/browse/AIRFLOW-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596093#comment-16596093 ] Ash Berlin-Taylor commented on AIRFLOW-2975: Airflow already has support for "queues" on the workers - https://airflow.readthedocs.io/en/stable/concepts.html#queues so you can set a queue on your tasks, and only have the "temporal" instance listen to this queue. Or you could just use the default queue and have the normal worker share in the load, and just provide more processing at the 1-3pm period. > Add support for Amazon cloudwatch (computing power at will) > --- > > Key: AIRFLOW-2975 > URL: https://issues.apache.org/jira/browse/AIRFLOW-2975 > Project: Apache Airflow > Issue Type: New Feature >Affects Versions: 1.10.0 >Reporter: jack >Priority: Major > Fix For: 2.0.0, 1.10.1 > > > Some of have one machine that runs airfow… > While we can scale up the executor to have many resources over different > servers this is consider to be expensive. > There is another solution.. using Amazon Cloudwatch: > [https://aws.amazon.com/premiumsupport/knowledge-center/start-stop-lambda-cloudwatch/] > > This enable the user to create and close EC2 machines on specific intervals > for specific tasks. > Basically if I have 50 DAGS to run on 1PM-3PM and few dags to run on other > hours there is no point in paying for 2nd server 24/7. > This could be an enhancement to one of the executors that know to work with > more than one server. > > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
