Re: About the project support in Airflow

Hi Feng, Thanks for your information, indeed I have noticed this work also. But if I am understanding correctly, it is focus on the permission (edit/read etc.) with the DAG itself. “project concept” is some kind of “Group” but it is more meaningful than the “Tag”, so if we don’t want to

Re: About the project support in Airflow

Hi Song, Just noted that we are also working on dag-level access on top of RBAC(AIRFLOW-2267) which should provide dag-level acl functionality. The WIP pr could be found at https://github.com/apache/incubator-airflow/pull/3197 On Wed, Apr 25, 2018 at 7:42 PM, 刘松(Cycle++开发组)

Re: About the project support in Airflow

+1 Sent from a device with less than stellar autocorrect > On Apr 25, 2018, at 12:04 PM, James Meickle wrote: > > Another reason you would want separated infrastructure is that there are a > lot of ways to exhaust Airflow resources or otherwise cause contention - >

Re: About the project support in Airflow

Something else that might be relevant for your multi-user use case is the new RBAC support that Joy Gao added. https://github.com/apache/incubator-airflow/pull/3015 *Taylor Edmiston* Blog | Stack Overflow CV | LinkedIn

Re: About the project support in Airflow

Another reason you would want separated infrastructure is that there are a lot of ways to exhaust Airflow resources or otherwise cause contention - like having too many sensors or sub-DAGs using up all available tasks. Doesn't seem like a great idea to push for having different teams with

Re: Task get stuck in up_for_retry state

What are the logs from the scheduler and the specific scheduler subprocess for that DAG? On Mon, Apr 23, 2018 at 9:25 AM, ramandu...@gmail.com wrote: > Hi , > I am providing the "retries" and "retry_delay" as an argument to one of my > operator in the DAG. But the

Re: Airflow - YARN as an executor?

As long as that code is serializable (through pickle, cloudpickle or any other Python code serializaers ), the answer should be yes. Thanks. -- Ruslan Dautkhanov On Wed, Apr 25, 2018 at 9:54 AM, Taylor Edmiston wrote: > Is it possible for the (hypothetical) Airflow

Re: Airflow - YARN as an executor?

Is it possible for the (hypothetical) Airflow SparkExecutor to handle general execution of any operator (i.e., run non-Spark code)? *Taylor Edmiston* Blog | Stack Overflow CV | LinkedIn |

Re: Airflow - YARN as an executor?

I used "Executor" as an Airflow term, not meant spark executor ... Like Spark would be one of Executors in here https://github.com/apache/incubator-airflow/tree/master/airflow/executors or in here https://github.com/apache/incubator-airflow/tree/master/airflow/contrib/executors Thanks. --

Re: Airflow - YARN as an executor?

Im a bit lost on the spark executor to be honest. To my knowledge the spark driver creates spark executors which run spark code. In other words in can’t arbitrarily run generic code. Or can it? B. Verstuurd vanaf mijn iPad > Op 25 apr. 2018 om 17:11 heeft Ruslan Dautkhanov