[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive
[
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-18976:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Nishant!
> Add ability to setup Druid Kafka Ingestion from Hive
>
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-18976.03.patch, HIVE-18976.04.patch,
> HIVE-18976.05.patch, HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and
> druid can start ingesting events from kafka.
> {code:java}
>
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language
> string, `user` string, added int, deleted int, delta int)
> STORED BY
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that
> extends existing DruidStorageHandler and adds the additional functionality
> for Streaming.
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster
> + Single Node Kafka Broker. The broker can be populated with a test topic
> that has some predefined data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive
[
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-18976:
Attachment: HIVE-18976.05.patch
> Add ability to setup Druid Kafka Ingestion from Hive
>
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.03.patch, HIVE-18976.04.patch,
> HIVE-18976.05.patch, HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and
> druid can start ingesting events from kafka.
> {code:java}
>
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language
> string, `user` string, added int, deleted int, delta int)
> STORED BY
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that
> extends existing DruidStorageHandler and adds the additional functionality
> for Streaming.
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster
> + Single Node Kafka Broker. The broker can be populated with a test topic
> that has some predefined data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive
[
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-18976:
Attachment: HIVE-18976.04.patch
> Add ability to setup Druid Kafka Ingestion from Hive
>
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.03.patch, HIVE-18976.04.patch,
> HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and
> druid can start ingesting events from kafka.
> {code:java}
>
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language
> string, `user` string, added int, deleted int, delta int)
> STORED BY
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that
> extends existing DruidStorageHandler and adds the additional functionality
> for Streaming.
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster
> + Single Node Kafka Broker. The broker can be populated with a test topic
> that has some predefined data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive
[
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-18976:
Attachment: HIVE-18976.03.patch
> Add ability to setup Druid Kafka Ingestion from Hive
>
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.03.patch, HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and
> druid can start ingesting events from kafka.
> {code:java}
>
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language
> string, `user` string, added int, deleted int, delta int)
> STORED BY
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that
> extends existing DruidStorageHandler and adds the additional functionality
> for Streaming.
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster
> + Single Node Kafka Broker. The broker can be populated with a test topic
> that has some predefined data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive
[
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-18976:
Attachment: HIVE-18976.patch
> Add ability to setup Druid Kafka Ingestion from Hive
>
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and
> druid can start ingesting events from kafka.
> {code:java}
>
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language
> string, `user` string, added int, deleted int, delta int)
> STORED BY
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that
> extends existing DruidStorageHandler and adds the additional functionality
> for Streaming.
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster
> + Single Node Kafka Broker. The broker can be populated with a test topic
> that has some predefined data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive
[
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-18976:
Attachment: (was: HIVE-18976.patch)
> Add ability to setup Druid Kafka Ingestion from Hive
>
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and
> druid can start ingesting events from kafka.
> {code:java}
>
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language
> string, `user` string, added int, deleted int, delta int)
> STORED BY
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that
> extends existing DruidStorageHandler and adds the additional functionality
> for Streaming.
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster
> + Single Node Kafka Broker. The broker can be populated with a test topic
> that has some predefined data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive
[
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-18976:
Attachment: HIVE-18976.patch
> Add ability to setup Druid Kafka Ingestion from Hive
>
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and
> druid can start ingesting events from kafka.
> {code:java}
>
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language
> string, `user` string, added int, deleted int, delta int)
> STORED BY
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that
> extends existing DruidStorageHandler and adds the additional functionality
> for Streaming.
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster
> + Single Node Kafka Broker. The broker can be populated with a test topic
> that has some predefined data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18976) Add ability to setup Druid Kafka Ingestion from Hive
[
https://issues.apache.org/jira/browse/HIVE-18976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nishant Bangarwa updated HIVE-18976:
Status: Patch Available (was: Open)
> Add ability to setup Druid Kafka Ingestion from Hive
>
>
> Key: HIVE-18976
> URL: https://issues.apache.org/jira/browse/HIVE-18976
> Project: Hive
> Issue Type: Bug
> Components: Druid integration
>Reporter: Nishant Bangarwa
>Assignee: Nishant Bangarwa
>Priority: Major
> Attachments: HIVE-18976.patch
>
>
> Add Ability to setup druid kafka Ingestion using Hive CREATE TABLE statement
> e.g. Below query can submit a kafka supervisor spec to the druid overlord and
> druid can start ingesting events from kafka.
> {code:java}
>
> CREATE TABLE druid_kafka_test(`__time` timestamp, page string, language
> string, `user` string, added int, deleted int, delta int)
> STORED BY
> 'org.apache.hadoop.hive.druid.DruidKafkaStreamingStorageHandler'
> TBLPROPERTIES (
> "druid.segment.granularity" = "HOUR",
> "druid.query.granularity" = "MINUTE",
> "kafka.bootstrap.servers" = "localhost:9092",
> "kafka.topic" = "test-topic",
> "druid.kafka.ingest.useEarliestOffset" = "true"
> );
> {code}
> Design - This can be done via a DruidKafkaStreamingStorageHandler that
> extends existing DruidStorageHandler and adds the additional functionality
> for Streaming.
> Testing - Add a DruidKafkaMiniCluster which will consist of DruidMiniCluster
> + Single Node Kafka Broker. The broker can be populated with a test topic
> that has some predefined data.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
