Re: WebHdfsSensor doesn't support HDFS HA

Manu - This is the relevant code I was referencing before: https://github.com/apache/incubator-airflow/blob/master/airflow/hooks/webhdfs_hook.py#L54-L71 So multiple connections for a given conn_id is already built into some hooks, but we need a way to set this from CLI. I'll be creating a JIRA sh

Re: WebHdfsSensor doesn't support HDFS HA

Thanks Xiaodong, that works like a charm. Manu On Thu, Aug 30, 2018 at 11:34 AM Deng Xiaodong wrote: > Hi Manu, > > You can set up multiple connections with the same conn_id and different > host, rather than setting in one single connection. > > > XD > > On Thu, Aug 30, 2018 at 11:17 Manu Zhang

Re: WebHdfsSensor doesn't support HDFS HA

Hi Manu, You can set up multiple connections with the same conn_id and different host, rather than setting in one single connection. XD On Thu, Aug 30, 2018 at 11:17 Manu Zhang wrote: > Hi Ben, > > How do you set multiple connections through Web UI (from Connections item > of Admin pull-down

Re: WebHdfsSensor doesn't support HDFS HA

Hi Ben, How do you set multiple connections through Web UI (from Connections item of Admin pull-down list) ? I'm tried setting a comma-separated list to a conn_id but that doesn't work. Thanks, Manu On Wed, Aug 29, 2018 at 11:31 PM Ben Laird wrote: > Hi Manu, > > We have the same use case as

Re: WebHdfsSensor doesn't support HDFS HA

Hi Manu, We have the same use case as you, a primary and backup namenode. If I understand your issue correctly, the WebHDFSSensor code checks an iterable of Airflow connections to the namenode to find one that is active. However, my issue (which I've emailed this list about) was that you cannot s

Re: WebHdfsSensor doesn't support HDFS HA

Hi Manu, Thanks for raising this question. There is a PR for moving to hdfs3. There is code in the existing codebase, which support HA