[
https://issues.apache.org/jira/browse/AIRFLOW-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daniel Ferguson updated AIRFLOW-6786:
-------------------------------------
Description:
Add the KafkaProducerHook.
Add the KafkaConsumerHook.
Add the KafkaSensor which listens to messages with a specific topic.
Related Issue:
#1311 (Pre-dates Jira Migration)
Reminder to contributors:
You must add an Apache License header to all new files
Please squash your commits when possible and follow the 7 rules of good Git
commits
I am new to the community, I am not sure the files are at the right place or
missing anything.
The sensor could be used as the first node of a dag where the second node can
be a TriggerDagRunOperator. The messages are polled in a batch and the dag runs
are dynamically generated.
Thanks!
Note, as per denied PR [#1415|https://github.com/apache/airflow/pull/1415], it
is important to mention these integrations are not suitable for
low-latency/high-throughput/streaming. For reference, [#1415
(comment)|https://github.com/apache/airflow/pull/1415#issuecomment-484429806].
Co-authored-by: Dan Ferguson
[[email protected]|mailto:[email protected]]
Co-authored-by: YuanfΞi Zhu
was:
Add the KafkaProducerHook.
Add the KafkaConsumerHook.
Add the KafkaSensor which listens to messages with a specific topic.
Related Issue:
#1311 (Pre-dates Jira Migration)
Reminder to contributors:
You must add an Apache License header to all new files
Please squash your commits when possible and follow the 7 rules of good Git
commits
I am new to the community, I am not sure the files are at the right place or
missing anything.
The sensor could be used as the first node of a dag where the second node can
be a TriggerDagRunOperator. The messages are polled in a batch and the dag runs
are dynamically generated.
Thanks!
Note, as per denied PR [#1415|https://github.com/apache/airflow/pull/1415], it
is important to mention these integrations are not suitable for
low-latency/high-throughput/streaming. For reference, [#1415
(comment)|https://github.com/apache/airflow/pull/1415#issuecomment-484429806].
Co-authored-by: Dan Ferguson
[[email protected]|mailto:[email protected]]
Co-authored-by: Hanan Shteingart
Co-authored-by: YuanfΞi Zhu
> Adding KafkaConsumerHook, KafkaProducerHook, and KafkaSensor
> ------------------------------------------------------------
>
> Key: AIRFLOW-6786
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6786
> Project: Apache Airflow
> Issue Type: New Feature
> Components: contrib, hooks
> Affects Versions: 1.10.9
> Reporter: Daniel Ferguson
> Assignee: Daniel Ferguson
> Priority: Minor
>
> Add the KafkaProducerHook.
> Add the KafkaConsumerHook.
> Add the KafkaSensor which listens to messages with a specific topic.
> Related Issue:
> #1311 (Pre-dates Jira Migration)
> Reminder to contributors:
> You must add an Apache License header to all new files
> Please squash your commits when possible and follow the 7 rules of good Git
> commits
> I am new to the community, I am not sure the files are at the right place or
> missing anything.
> The sensor could be used as the first node of a dag where the second node can
> be a TriggerDagRunOperator. The messages are polled in a batch and the dag
> runs are dynamically generated.
> Thanks!
> Note, as per denied PR [#1415|https://github.com/apache/airflow/pull/1415],
> it is important to mention these integrations are not suitable for
> low-latency/high-throughput/streaming. For reference, [#1415
> (comment)|https://github.com/apache/airflow/pull/1415#issuecomment-484429806].
> Co-authored-by: Dan Ferguson
> [[email protected]|mailto:[email protected]]
> Co-authored-by: YuanfΞi Zhu
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
