[
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-18976:
------------------------------------
Status: Patch Available (was: Open)
> Add ability to setup Druid Kafka Ingestion from Hive
> ----------------------------------------------------
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
> Reporter: Nishant Bangarwa
> Assignee: Nishant Bangarwa
> Priority: Major
> Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and
> druid can start ingesting events from kafka.
> {code:java}
>
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language
> string, `user` string, added int, deleted int, delta int)
> STORED BY
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that
> extends existing DruidStorageHandler and adds the additional functionality
> for Streaming.
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster
> + Single Node Kafka Broker. The broker can be populated with a test topic
> that has some predefined data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
