Re: Delay between tasks executions issue

Thanks Raman, Understood. We have around 500 DAGs. What value do you suggest for max_threads? On Tue, Sep 11, 2018, 5:44 PM ramandu...@gmail.com wrote: > Hi Chandu, > How many dag files are there on the scheduler. As per my understanding > scheduler processes each file to trigger any dag/task

Delay between tasks executions issue

Hi All, We are seeing there is a delay between triggering the next task. Please find the attached screenshot. First and the last task are very small. So, you can see the tiny object. [image: image.png] Do you suggest any airflow config can resolve this problem? Thanks, Chandu

Re: Delay between tasks executions issue

Hi Chandu, How many dag files are there on the scheduler. As per my understanding scheduler processes each file to trigger any dag/task run. It spawns number of processes equivalent to "max_threads" count to parallelize file processing. So you can try by increasing airflow config's max_threads

Re: Delay between tasks executions issue

We use max_threads = number of scheduler cores. On 2018/09/11 09:49:53, Chandu Kavar wrote: > Thanks Raman, > > Understood. > > We have around 500 DAGs. What value do you suggest for max_threads? > > On Tue, Sep 11, 2018, 5:44 PM ramandu...@gmail.com > wrote: > > > Hi Chandu, > > How many

Re: [VOTE][RESULT] Replace with Gitter with Slack?

are there any instructions on how to join the slack workspace? On Mon, Sep 10, 2018 at 9:57 PM Sid Anand wrote: > Thanks for the context. > > So, I created the slack workspace and it has ~ 100 people on it now and it > pretty active. > > What's pretty cool is that some folks created regional

Re: Delay between tasks executions issue

Thanks a lot. On Tue, Sep 11, 2018, 6:25 PM ramandu...@gmail.com wrote: > We use max_threads = number of scheduler cores. > > On 2018/09/11 09:49:53, Chandu Kavar wrote: > > Thanks Raman, > > > > Understood. > > > > We have around 500 DAGs. What value do you suggest for max_threads? > > > > On

Re: [VOTE][RESULT] Replace with Gitter with Slack?

Here you go Andrew -- This link should work for the next 27 days: https://join.slack.com/t/apache-airflow/shared_invite/enQtNDMxMTU3MzM1NDg4LTVkMTg3MGU5MzhhMDQ1NDNjNjgyNTQ3MjQzNjk1YmVmZWFiYWY4NjlhNDQ5MWQyNzZlYzE3NTIyMDBiYWQxNjM On Tue, Sep 11, 2018 at 4:31 AM Andrew Harmon wrote: > are there

Re: Salesforce Hook/Operator

The version in airflow-plugins/salesforce_plugin is borrowed from the one in airflow.contrib, so I would prefer the official one. If you have time to test, feel free to send us a PR that upgrades to the latest stable version (looks like some changes are on master but not v1-10-stable) from

Re: Cold-case PRs

Looks cool Max. Filed an Infra ticket: https://issues.apache.org/jira/browse/INFRA-17005 Created a Jira to add a policy file: .github/stale.yml https://issues.apache.org/jira/browse/AIRFLOW-3040 & raised a PR : https://github.com/apache/incubator-airflow/pull/3883 We can manage the Jira

Re: Salesforce Hook/Operator

I used airflow-plugins/salesforce_plugin as a base for my work. I disregarded the hook in the repo since it didn't work and used the OOTB hook