Re: Airflow Meetup in India

2018-08-21 Thread Sai Phanindhra
I can speak/attend if its in bangalore On Tue 21 Aug, 2018, 12:09 Sumit Maheshwari, wrote: > Hi Raman, > > Folks from Qubole certainly join/talk if the meetup is happening in > Bangalore. However, sometime back I did a poll about the same and got only > 7 responses, so have dropped the idea of

Re: Airflow Meetup in India

2018-08-21 Thread ramandumcs
Thanks Sumit, We work for Adobe and will try to have it in bangalore based on the participation. Would it be possible for you to share the poll link. Thanks, Raman Gupta On 2018/08/21 06:38:37, Sumit Maheshwari wrote: > Hi Raman, > > Folks from Qubole certainly join/talk if the meetup is

Re: Airflow Meetup in India

2018-08-21 Thread Sumit Maheshwari
Hi Raman, Folks from Qubole certainly join/talk if the meetup is happening in Bangalore. However, sometime back I did a poll about the same and got only 7 responses, so have dropped the idea of conducting. Let us know if you get enough traction this time. Thanks, Sumit On Tue, Aug 21, 2018 at

Re: Airflow Meetup in India

2018-08-21 Thread Sumit Maheshwari
Sure, here it is https://goo.gl/forms/2NF6HtUwpOOTvi373 On Tue, Aug 21, 2018 at 1:56 PM ramandu...@gmail.com wrote: > Thanks Sumit, > We work for Adobe and will try to have it in bangalore based on the > participation. > Would it be possible for you to share the poll link. > > Thanks, > Raman

Re: Airflow Meetup in India

2018-08-21 Thread Sai Phanindhra
Forms says preferred month is may, june or july. Is the form outdated? On Tue 21 Aug, 2018, 17:43 Sumit Maheshwari, wrote: > Sure, here it is https://goo.gl/forms/2NF6HtUwpOOTvi373 > > On Tue, Aug 21, 2018 at 1:56 PM ramandu...@gmail.com > > wrote: > > > Thanks Sumit, > > We work for Adobe and

Re: Airflow Meetup in India

2018-08-21 Thread Sumit Maheshwari
Yes, the poll was conducted around March end. On Tue, Aug 21, 2018 at 5:49 PM Sai Phanindhra wrote: > Forms says preferred month is may, june or july. Is the form outdated? > > On Tue 21 Aug, 2018, 17:43 Sumit Maheshwari, > wrote: > > > Sure, here it is https://goo.gl/forms/2NF6HtUwpOOTvi373

Re: Task is Stuck in Up_For_Retry

2018-08-21 Thread vardanguptacse
Is there any possibility that on call of function _check_and_change_state_before_execution at https://github.com/apache/incubator-airflow/blob/v1-9-stable/airflow/jobs.py#L2500, this method is not actually being called

Re: Airflow Meetup in India

2018-08-21 Thread Vaibhav Kumar
Does the meet up need some minimum number of responses ?? If so, I'm also ready to attend it Sent from Yahoo Mail on Android On Tue, 21 Aug 2018 at 18:01, Sumit Maheshwari wrote: Yes, the poll was conducted around March end. On Tue, Aug 21, 2018 at 5:49 PM Sai Phanindhra wrote: >

Re: Airflow Meetup in India

2018-08-21 Thread Sai Phanindhra
Sumit, can you update the form? It still shows past months. Add future months where we can plan a airflow meetup. On Tue 21 Aug, 2018, 18:01 Sumit Maheshwari, wrote: > Yes, the poll was conducted around March end. > > > > On Tue, Aug 21, 2018 at 5:49 PM Sai Phanindhra > wrote: > > > Forms says

Re: updating dags best practice

2018-08-21 Thread Aleksander Sumowski
We're currently using CeleryExecutor ... On 21/08/2018, 15:12, "Driesprong, Fokko" wrote: Hi Aleksander, What kind of executor are you using? This is really important when choosing a strategy. For most of the Airflow deployments that I've worked with simply uses git to deploy

Re: Task is Stuck in Up_For_Retry

2018-08-21 Thread ramandumcs
Hi All, As per http://docs.sqlalchemy.org/en/latest/core/connections.html link db engine is not portable across process boundaries "For a multiple-process application that uses the os.fork system call, or for example the Python multiprocessing module, it’s usually required that a separate

Re: [RESULT][VOTE] Release Airflow 1.10.0

2018-08-21 Thread Maxime Beauchemin
I can, what's your PyPI ID? Max On Mon, Aug 20, 2018 at 2:11 PM Driesprong, Fokko wrote: > Thanks Bolke! > > I've just pushed the artifacts to Apache Dist: > > https://dist.apache.org/repos/dist/release/incubator/airflow/1.10.0-incubating/ > > I don't have any access to pypi, this means that

Re: Task is Stuck in Up_For_Retry

2018-08-21 Thread raman gupta
One possibility is the unavailability of session while calling self.task_instance._check_and_change_state_before_execution function. (Session is provided via @provide_session decorator) On Tue, Aug 21, 2018 at 7:09 PM vardangupta...@gmail.com < vardangupta...@gmail.com> wrote: > Is there any

Re: Airflow Meetup in India

2018-08-21 Thread Sumit Maheshwari
@Vaibhav, I guess if there are atleast 20 or plus people are interested, then only it makes sense to conduct a meetup. @Sai, have updated the form. On Tue, Aug 21, 2018 at 7:31 PM Sai Phanindhra wrote: > Sumit, can you update the form? It still shows past months. Add future > months where we

Re: Regarding airflow 1.10

2018-08-21 Thread Driesprong, Fokko
Hi Hemanth, Thanks for the question. There is the KubernetesExecutor which just uses a container with Airflow to schedule a task on the kubernetes cluster. This container will then execute a single task, and then the Executor checks the exitcode of the container. There is also the

Re: updating dags best practice

2018-08-21 Thread Ry Walker
We follow the strategy to re-build Docker with each deployment, because sometimes DAG changes can include new or updated dependencies. This Docker-first approach makes local development nice too (run same Docker images on local machine during debug as production). This isnt painful for us because

updating dags best practice

2018-08-21 Thread Dev Aleksander
Hi all, in the place I'm currently at we're building and redeploying a new set of containers with the latest code every time we want to update a DAG. That doesn't feel like the fastest way. Anyone can share their approach? Thanks, Aleksander

Regarding airflow 1.10

2018-08-21 Thread Musunuru, Hemanth
Hi Team, Does Kubernetes executor in 1.10 create new pod for every task ? (Which means won’t there be any active workers running except for new Tasks) We are trying to containerize our existing to airflow environment, your reply Will help us to understand it more. Thanks in Advance . Thanks

Re: updating dags best practice

2018-08-21 Thread Driesprong, Fokko
Hi Aleksander, What kind of executor are you using? This is really important when choosing a strategy. For most of the Airflow deployments that I've worked with simply uses git to deploy the master branch. This is with a LocalExecutor. Cheers, Fokko Op di 21 aug. 2018 om 15:25 schreef Dev

Re: updating dags best practice

2018-08-21 Thread Dev Aleksander
We're currently using CeleryExecutor On Tue, Aug 21, 2018 at 3:11 PM, Driesprong, Fokko wrote: > Hi Aleksander, > > What kind of executor are you using? This is really important when choosing > a strategy. For most of the Airflow deployments that I've worked with > simply uses git to deploy the