Subject: [DISCUSS] New Community Provider: Apache HBase

Hi Airflow community,

I'd like to propose adding a new community provider for Apache HBase.
The provider would enter at the incubation stage with state not-ready
in provider.yaml until the first release. It targets Airflow 2.11+
and Python 3.11+

Integration overview
--------------------
Apache HBase (https://hbase.apache.org/) is an ASF top-level project,
a distributed, scalable, big data store for random, realtime read/write access. I work at Arenadata, where HBase is part of our data platform and we have been running this integration in our Airflow fork

There is currently no HBase provider in the Apache Airflow ecosystem.
This provider communicates with HBase via the Thrift2 API and the HBase
CLI. It introduces an HBase connection type, HBaseThriftHook,
HBaseCLIHook, operators, and sensors for typical HBase workflows.
The only additional Python dependency is thrift. The Thrift2 bindings
are vendored as hbase_thrift2_generated/ cause there is no standalone PyPI
package for them and the HBase Thrift2 API is stable

System tests
------------
Unit tests mock both the Thrift2 and CLI interfaces and run without a
live cluster. The Apache Airflow CI environment does not include an
HBase service. We run integration tests against the apache/hbase Docker
image in our fork but they require a dedicated Breeze integration which
is out of scope for this PR. We plan to submit that as a follow-up.
We are not publishing a public system-test dashboard at this stage

Proposed stewards
-----------------
- Vyacheslav Chaplygin, @slavachy, [email protected]
- Mikhail Zaplatin, @ZMichael3012, [email protected]
- Aleksey Gladkih, @gaxeliy, [email protected]
- Dmitry Bezlutskiy, @Dimitrionian, [email protected]

Sponsoring committer(s)
-----------------------
- Amogh Desai, @amoghrajesh

Working implementation
----------------------
https://github.com/apache/airflow/pull/77029

Incubation commitment
---------------------
We commit to maintaining the provider and responding to issues in a
timely manner, meeting the incubation health metrics within 6 months,
and participating in quarterly governance updates

Best regards,
Dmitry Bezlutskiy

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to