Thanks Anil
Can you please share more details of the approach here for querying streaming data? Are we going to create table mapped to Kafka Topic here and pull data from topic at regular intervals? Are we going to extract event from message and insert as row to table. Data pushed to Kafka topic may be unstructured. Are we planning to provide some mechanism through which user can define meta-data of message pushed to Kafka Topic? Regards Chetan -----Original Message----- From: AnilKumar B [mailto:[email protected]] Sent: Wednesday, August 30, 2017 12:19 AM To: [email protected] Subject: Re: Querying Streaming Data using Drill Hi Padma & Chetan, Just wanted to update on https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_DRILL-2D4779&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=XdafK486-x96ZJoTcDF35EpcYc2j9nO0sbpt27_VGCg&m=_sbH_Ztnc2gGMatSIcqTEJ5Oy1A4Q6mDnQPvVOCqwhg&s=RhoBrti0LOnAXP0jWjDD4C1VH1keATistN8CMoAq8cQ&e= We have developed and tested kafka integration and currently it's working for JSON messages. And currently we are working on test cases and Avro support. We are planning to present this on Sept 18th Drill Developer's day. Due to multiple reasons, this feature delayed from long time, but we are almost there. Repo: https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_akumarb2010_incubator-2Ddrill_tree_master_contrib_storage-2Dkafka&d=DwIBaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=XdafK486-x96ZJoTcDF35EpcYc2j9nO0sbpt27_VGCg&m=_sbH_Ztnc2gGMatSIcqTEJ5Oy1A4Q6mDnQPvVOCqwhg&s=13f8RiuuC2F7dMIIuzBERWl6jIjmZuMu-X9SnCVxNP4&e= Thanks & Regards, B Anil Kumar. On Tue, Aug 29, 2017 at 10:26 AM, Chetan Kothari <HYPERLINK "mailto:[email protected]"[email protected]> wrote: > Thanks Padma for quick response. > > > > This will be very critical feature to support in Drill as user will > look for > > single SQL Engine which supports querying both batch and streaming data. > > > > Any inputs on when support for querying streaming data will be supported? > > > > Regards > > Chetan > > > > -----Original Message----- > From: Padma Penumarthy [mailto:[email protected]] > Sent: Tuesday, August 29, 2017 10:53 PM > To: HYPERLINK "mailto:[email protected]"[email protected] > Subject: Re: Querying Streaming Data using Drill > > > > Currently, we do not have support for these storage plugins. > > I see an open JIRA for Kafka, not sure how much progress was made (as > last update was a while back). > > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__issues. > apache.org_jira_browse_DRILL-2D4779&d=DwIFAg&c= > RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=XdafK486- > x96ZJoTcDF35EpcYc2j9nO0sbpt27_VGCg&m=bHLm9AOBdeCey8wjxCKXHxHTwlpzXu > JF3dhrqs-qX-0&s=UCGsbhOa9QLDAmUEfJYiq_cOfGg5dUi1RdcPB7YTMMQ&e= > > > > Thanks, > > Padma > > > > > > On Aug 29, 2017, at 10:14 AM, Chetan Kothari <HYPERLINK "mailto: > HYPERLINK > "mailto:[email protected]%3cmailto:[email protected]"[email protected]%3cmailto:[email protected]"; > HYPERLINK > "mailto:[email protected]%3cmailto:[email protected]"[email protected]<mailto:[email protected]>> > wrote: > > > > Is there any support for querying streaming data using Drill? > > > > Presto provides out-of-box Kafka and Amazon Kinesis Connectors for > querying streaming data. > > > > > > > > Regards > > > > Chetan > > > > >
