Dear Drill Community, I am working of evaluating Drill as an OLAP query engine for a new open source product in the education domain. Just to give you some background: the solution we are building is a reporting tool for the student assessments outcomes. When a student takes an assessment test, it gets scored and then the outcome of the scoring along with the assessment details is loaded into the reporting warehouse. We are building that warehouse and the reporting tool on top of it. As part of the reporting tool we need to report on individual tests, as well as the aggregated data. The solution has to have an OLTP data store for the individual test reporting, and we are considering Drill for the analytical reports. As you could see our data is pretty well structured and the volume is not ‘big’ data, but is estimates at about 150+ million assessments outcomes. I see that Drill performs the best with Parquet file store. And I have two questions: 1. What is the best way to create Parquet files as the data flow in. Please consider that we need to support data update and deletes. Also the volume of data flow may vary. I see that some people capture data for a day and then convert them into Parquet. Is this the way to go? I am concerned that: a. we may end up with a lot of small size Parquet files b. for delete/updates we will need to delete the existing files with the changed records and recreate them. How to handle the system being up while we are doing this change? 2. Would it be better for us instead of creating Parquet files, point Drill to our OLTP data store? We know that OLTP data store by itself does not perform the queries we need within the desired speed (under 10 sec.). Would Drill be able to imrpove performance in this case?
Thank you in advance for your help! Alla Gorina Principal Developer [http://fairwaytech.com/wp-content/uploads/2015/11/fairwaydark.png] Fairway Technologies Inc. | Our People, Your Solution.® [http://www.fairwaytech.com/media/cell-icon.gif] (858) 774-1067(mobile) fairwaytech.com<http://www.fairwaytech.com/> | [http://www.fairwaytech.com/wp-content/uploads/2015/11/linkedin_email.gif] <http://www.linkedin.com/company/fairway-technologies> [http://www.fairwaytech.com/wp-content/uploads/2015/11/twitter_email.gif] <http://twitter.com/FairwayTech> [http://fairwaytech.com/wp-content/uploads/2015/11/facebook_email.gif] <http://www.facebook.com/pages/Fairway-Technologies/127195970658682>
