Re: Drill performance tuning parquet

2017-08-01 Thread Saurabh Mahapatra
d profit ... > > > > > -Original Message- > From: Kunal Khatua [mailto:kkha...@mapr.com] > Sent: Friday, July 28, 2017 12:51 PM > To: user@drill.apache.org > Subject: RE: Drill performance tuning parquet > > I also forgot to mention... within the drill-over

Re: Drill performance tuning parquet

2017-07-31 Thread Padma Penumarthy
July 31, 2017 5:23 AM To: user@drill.apache.org Subject: RE: Drill performance tuning parquet When is it right to add nodes vs adding CPUs? Since my installation is on AWS, adding CPUs is relatively easy. When does it make sense to add nodes instead of CPUs? Dan Holmes | Revenue Analytics

RE: Drill performance tuning parquet

2017-07-31 Thread Dan Holmes
Khatua [mailto:kkha...@mapr.com] Sent: Friday, July 28, 2017 12:51 PM To: user@drill.apache.org Subject: RE: Drill performance tuning parquet I also forgot to mention... within the drill-override.conf, is a parameter you'd need to set to constrain the async parquet reader's scan pool siz

RE: Drill performance tuning parquet

2017-07-28 Thread Kunal Khatua
hat the metrics in query profile reveal. -Original Message- From: Jinfeng Ni [mailto:j...@apache.org] Sent: Friday, July 28, 2017 7:41 AM To: user Subject: Re: Drill performance tuning parquet The number you posted seems to show that the query elapse time is highly impacted by the

Re: Drill performance tuning parquet

2017-07-28 Thread Jinfeng Ni
t; > It is also possible that since you're running on AWS, the compute and > storage layers and not as tightly coupled as Athena is with their own S3, > which would make sense since they need an incentive for users to try Athena > on their AWS infrastructure. :) > > Happy Drillin

RE: Drill performance tuning parquet

2017-07-28 Thread Dan Holmes
hatua [mailto:kkha...@mapr.com] Sent: Friday, July 28, 2017 2:38 AM To: user@drill.apache.org Subject: RE: Drill performance tuning parquet Look at the query profile's (in the UI) "operator profiles - overview" section. The % Query Time is a good indicator of which operator co

RE: Drill performance tuning parquet

2017-07-27 Thread Kunal Khatua
e for users to try Athena on their AWS infrastructure. :) Happy Drilling! -Original Message- From: Dan Holmes [mailto:dhol...@revenueanalytics.com] Sent: Thursday, July 27, 2017 6:23 PM To: user@drill.apache.org Subject: RE: Drill performance tuning parquet Let's pretend there is on

RE: Drill performance tuning parquet

2017-07-27 Thread Dan Holmes
; This is where you'll find more about this: > https://drill.apache.org/docs/performance-tuning/ > > ~ Kunal > > -Original Message- > From: Dan Holmes [mailto:dhol...@revenueanalytics.com] > Sent: Thursday, July 27, 2017 1:06 PM > To: user@drill.apache.org > Subject: RE: Dr

Re: Drill performance tuning parquet

2017-07-27 Thread Saurabh Mahapatra
ut this: > https://drill.apache.org/docs/performance-tuning/ > > ~ Kunal > > -Original Message- > From: Dan Holmes [mailto:dhol...@revenueanalytics.com] > Sent: Thursday, July 27, 2017 1:06 PM > To: user@drill.apache.org > Subject: RE: Drill performance tuning parqu

RE: Drill performance tuning parquet

2017-07-27 Thread Kunal Khatua
Kunal -Original Message- From: Dan Holmes [mailto:dhol...@revenueanalytics.com] Sent: Thursday, July 27, 2017 1:06 PM To: user@drill.apache.org Subject: RE: Drill performance tuning parquet I did not partition the data when I created the parquet files (CTAS without a PARITION BY) H

RE: Drill performance tuning parquet

2017-07-27 Thread Dan Holmes
I did not partition the data when I created the parquet files (CTAS without a PARITION BY) Here is the file list. Thank you. [dholmes@ip-10-20-49-40 sales_p]$ ll total 1021372 -rw-rw-r-- 1 dholmes dholmes 393443418 Jul 27 19:05 1_0_0.parquet -rw-rw-r-- 1 dholmes dholmes 321665234 Jul 27 19:06