Hi Tug, It's Parquet data on HDFS and the data to HDFS is constantly written by spark while consuming from Kafka.
Is polling a common technique for say real time analytics dashboard ? More importantly if I poll does Drill due the scan every time? if the answer is no, how does it know which is the new data? since the data is written HDFS constantly as a stream (The query can be the same however the new data will be appended or updated to HDFS in parquet format as a stream). Thanks! On Thu, Nov 9, 2017 at 4:47 AM, Tugdual Grall <[email protected]> wrote: > Hello, > > > Today Drill cannot do continuous/streaming query, so as you mentioned you > will have to use a polling technique. > > > Just out of curiosity, Which data source are you planning to use ? > > Regards > Tug > > > > > On Thu 9 Nov 2017 at 04:31, kant kodali <[email protected]> wrote: > > > Hi All, > > > > I am new to Apache Drill. I am wondering if Apache Drill can perform > > Streaming Queries? For example, I have a constant stream of data in 24 > hour > > period and I would like to get updates as soon as I receive them. > > > > Do I need to have a polling thread that issues a Drill query every > second? > > > > Thanks! > > >
