> On March 24, 2015, 6:22 a.m., Jaideep dhok wrote: > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/priority/DurationBasedQueryPriorityDecider.java, > > line 146 > > <https://reviews.apache.org/r/32393/diff/1/?file=902982#file902982line146> > > > > These are default partition costs for update periods - > > MONTHLY = 15 > > DAILY = 0.75 > > HOURLY = 0.04 (Assuming we change it to 1.0f/24) > > > > So what we are saying is that a monthly partition is 20 times as > > expensive as a daily partition, and 375 times as expensive as an hourly > > partition. Do you think this will hold for vast majority of cases? Our > > defaults should be tuned accordingly.
bq. So what we are saying is that a monthly partition is 20 times as expensive as a daily partition, and 375 times as expensive as an hourly partition. Do you think this will hold for vast majority of cases? Our defaults should be tuned accordingly. For monthly : monthly (=20*daily) < day*30; monthly( = 375*hourly) < 720*hourly For daily : daily (0.75) < 24 * hourly (= 0.96 = 24* 0.04) The default values have been tuned according to existing system which is already tuned. At anypoint, we have the configuration exposed to tune more. > On March 24, 2015, 6:22 a.m., Jaideep dhok wrote: > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/priority/DurationBasedQueryPriorityDecider.java, > > line 121 > > <https://reviews.apache.org/r/32393/diff/1/?file=902982#file902982line121> > > > > 1. How are we setting cost for dim tables? > > 2. This cost computation needs more explanation. > > What is the reasoning behind this formula? Why do we have a table > > weight AND a partition weight? I think table weights could be aggregated > > from the partitions themselves. >> How are we setting cost for dim tables? Only fact table partitions are considered right now for priority calculation. >> What is the reasoning behind this formula? Why do we have a table weight AND >> a partition weight? I think table weights could be aggregated from the >> partitions themselves. Table weights are there to differentiate a single partition in raw table is not same as single partition in aggregated table. > On March 24, 2015, 6:22 a.m., Jaideep dhok wrote: > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java, > > line 501 > > <https://reviews.apache.org/r/32393/diff/1/?file=902980#file902980line501> > > > > What is the priority set to in this case? Will be usually NORMAL - the default in underlying system. Updated doc accordingly > On March 24, 2015, 6:22 a.m., Jaideep dhok wrote: > > lens-cube/src/main/java/org/apache/lens/driver/cube/RewriterPlan.java, line > > 101 > > <https://reviews.apache.org/r/32393/diff/1/?file=902978#file902978line101> > > > > Do we still need to do this check after rewriter as completed? Won't > > rewriter make sure that all tables are present? yes. Tablewieghts is a map in DriverQueryPlan which will get updated only here. The check is to avoid the lookup if same table is accessed in more than one cube query. - Amareshwari ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32393/#review77526 ----------------------------------------------------------- On March 23, 2015, 1:05 p.m., Amareshwari Sriramadasu wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/32393/ > ----------------------------------------------------------- > > (Updated March 23, 2015, 1:05 p.m.) > > > Review request for lens, Jaideep dhok and Rajat Khandelwal. > > > Bugs: LENS-323 > https://issues.apache.org/jira/browse/LENS-323 > > > Repository: lens > > > Description > ------- > > WIP patch for early comments. > > Changes include: > > - Generating rewriter plan with cube queries involved > - Populating FactPartitions queried in RewriterPlan > - priority getting set only for cube queries; native tables queries are left > with defaults > - TestHiveDriver.testPriority updated > > Still pending > - Fix other test failures > - Add unit tests for rewriter plan construction > - Clear transient state in QueryContext after launch. > > Can be included or done in a followup > - Constructing driver plan from RewriterPlan, instead of parsing output. > > > Diffs > ----- > > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateDim.java > 3ac207e > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateFact.java > 66c3b25 > lens-cube/src/main/java/org/apache/lens/cube/parse/CandidateTable.java > 5eaf7ec > lens-cube/src/main/java/org/apache/lens/cube/parse/CubeQueryContext.java > df5a212 > > lens-cube/src/main/java/org/apache/lens/cube/parse/StorageTableResolver.java > c29d88f > lens-cube/src/main/java/org/apache/lens/driver/cube/RewriteUtil.java > 8ed4070 > lens-cube/src/main/java/org/apache/lens/driver/cube/RewriterPlan.java > PRE-CREATION > lens-driver-hive/pom.xml 3b8e73a > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveDriver.java > 9e3c723 > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/HiveQueryPlan.java > 64a6bba > > lens-driver-hive/src/main/java/org/apache/lens/driver/hive/priority/DurationBasedQueryPriorityDecider.java > 4f471cb > > lens-driver-hive/src/test/java/org/apache/lens/driver/hive/TestHiveDriver.java > 8a5839b > lens-driver-hive/src/test/resources/priority_tests.data 230df95 > > lens-server-api/src/main/java/org/apache/lens/server/api/driver/DriverQueryPlan.java > 38ebc55 > > lens-server-api/src/main/java/org/apache/lens/server/api/query/DriverSelectorQueryContext.java > fef834f > > lens-server-api/src/test/java/org/apache/lens/server/api/driver/MockDriver.java > 67c73b8 > > Diff: https://reviews.apache.org/r/32393/diff/ > > > Testing > ------- > > > Thanks, > > Amareshwari Sriramadasu > >
