Hi, I have a table which is around 250GB and I want to run certain group by queries on this. I know using Drill daemons on SQL server wont imporve anything as we will have a single Drill daemon.
I am planning to create parquet file for single day's data and upload to HDFS. Then use drill to query it. However, the issue is that some of the data keeps on changing and thus I will need to find a way to change the hdfs file. What is best way to keep this data updated? So suppose i create the hdfs file across say 10 different server and then suppose every server keeps around 25GB of data. Ram is around 50GB on these server. Now users will run different query against this table. I assume that the data will be in memory for every query once it is loaded first time or will drill read the hdfs file everytime? Thanks On Wed, Mar 23, 2016 at 4:49 AM, Tomer Shiran <[email protected]> wrote: > There are two separate questions here: > > 1. Where to run the Drill daemons (drillbits) > 2. What data to query with Drill. For example, you could run queries > directly against SQL Server tables and potentially join SQL Server > tables > with JSON/CSV/Parquet files on Hadoop in a single query. Or you could > CTAS > data from SQL Server into HDFS. > > Can you share more details on what you're trying to accomplish here so that > we can be more helpful? > > Thanks, > Tomer > > On Tue, Mar 22, 2016 at 3:47 PM, Darshan Singh <[email protected]> > wrote: > > > Hi Drill Users, > > > > I have a MS SQL server and I would like to use the drill. One option is > > that i query ms sql server and store the data in drill tables on hdfs > and > > then run queries against this. > > > > However, I would like to know will it be useful if I start drill on each > of > > the servers where MS SQL server is installed. Will it give a boost? I > doubt > > that this will be useful as ultimately it will rely on MS SQL Server for > > performance. > > > > Just want to know what are different options? > > > > Thanks > > > > > > -- > Tomer Shiran > CEO and Co-Founder, Dremio >
