btw, that small table 'application' has only one partition right now, 20k rows.
On Mon, Jun 15, 2009 at 1:59 PM, Min Zhou <[email protected]> wrote: > failed with null pointer exception. > hive>select /*+ MAPJOIN(a) */ a.url_pattern, w.url from (select > x.url_pattern from application x where x.dt = '20090609') a join web_log w > where w.logdate='20090611' and w.url rlike a.url_pattern; > FAILED: Unknown exception : null > > > $cat /tmp/hive/hive.log | tail... > > 2009-06-15 13:57:02,933 ERROR ql.Driver (SessionState.java:printError(279)) > - FAILED: Unknown exception : null > java.lang.NullPointerException > at > org.apache.hadoop.hive.ql.parse.QBMetaData.getTableForAlias(QBMetaData.java:76) > at > org.apache.hadoop.hive.ql.parse.PartitionPruner.getTableColumnDesc(PartitionPruner.java:284) > at > org.apache.hadoop.hive.ql.parse.PartitionPruner.genExprNodeDesc(PartitionPruner.java:217) > at > org.apache.hadoop.hive.ql.parse.PartitionPruner.genExprNodeDesc(PartitionPruner.java:231) > at > org.apache.hadoop.hive.ql.parse.PartitionPruner.genExprNodeDesc(PartitionPruner.java:231) > at > org.apache.hadoop.hive.ql.parse.PartitionPruner.genExprNodeDesc(PartitionPruner.java:231) > at > org.apache.hadoop.hive.ql.parse.PartitionPruner.addExpression(PartitionPruner.java:377) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPartitionPruners(SemanticAnalyzer.java:608) > at > org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:3785) > at > org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:76) > at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:177) > at org.apache.hadoop.hive.ql.Driver.run(Driver.java:209) > at > org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:176) > at > org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:216) > at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:309) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:165) > at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65) > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79) > at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68) > > > On Mon, Jun 15, 2009 at 1:52 PM, Namit Jain <[email protected]> wrote: > >> The problem seems to be in partition pruning – The small table >> ‘application’ is partitioned – and probably, there are 20k rows in the >> partition >> 20090609. >> >> Due to a bug, the pruning is not happening, and all partitions of >> ‘application’ are being loaded – which may be too much for map-join to >> handle. >> This is a serious bug, but for now can you put in a subquery and try - >> >> select /*+ MAPJOIN(a) */ a.url_pattern, w.url from (select x.url_pattern >> from application x where x.dt = ‘20090609’) a join web_log w where >> w.logdate='20090611' and w.url rlike a.url_pattern; >> >> >> Please file a JIRA for the above. >> >> >> >> >> On 6/14/09 10:20 PM, "Min Zhou" <[email protected]> wrote: >> >> hive> explain select /*+ MAPJOIN(a) */ a.url_pattern, w.url from >> application a join web_log w where w.logdate='20090611' and w.url rlike >> a.url_pattern and a.dt='20090609'; >> OK >> ABSTRACT SYNTAX TREE: >> (TOK_QUERY (TOK_FROM (TOK_JOIN (TOK_TABREF application a) (TOK_TABREF >> web_log w))) (TOK_INSERT (TOK_DESTINATION (TOK_DIR TOK_TMP_FILE)) >> (TOK_SELECT (TOK_HINTLIST (TOK_HINT TOK_MAPJOIN (TOK_HINTARGLIST a))) >> (TOK_SELEXPR (. (TOK_TABLE_OR_COL a) url_pattern)) (TOK_SELEXPR (. >> (TOK_TABLE_OR_COL w) url))) (TOK_WHERE (and (and (= (. (TOK_TABLE_OR_COL w) >> logdate) '20090611') (rlike (. (TOK_TABLE_OR_COL w) url) (. >> (TOK_TABLE_OR_COL a) url_pattern))) (= (. (TOK_TABLE_OR_COL a) dt) >> '20090609'))))) >> >> STAGE DEPENDENCIES: >> Stage-1 is a root stage >> Stage-2 depends on stages: Stage-1 >> Stage-0 is a root stage >> >> STAGE PLANS: >> Stage: Stage-1 >> Map Reduce >> Alias -> Map Operator Tree: >> w >> Select Operator >> expressions: >> expr: url >> type: string >> expr: logdate >> type: string >> Common Join Operator >> condition map: >> Inner Join 0 to 1 >> condition expressions: >> 0 {0} {1} >> 1 {0} {1} >> keys: >> 0 >> 1 >> Position of Big Table: 1 >> File Output Operator >> compressed: false >> GlobalTableId: 0 >> table: >> input format: >> org.apache.hadoop.mapred.SequenceFileInputFormat >> output format: >> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat >> Local Work: >> Map Reduce Local Work >> Alias -> Map Local Tables: >> a >> Fetch Operator >> limit: -1 >> Alias -> Map Local Operator Tree: >> a >> Select Operator >> expressions: >> expr: url_pattern >> type: string >> expr: dt >> type: string >> Common Join Operator >> condition map: >> Inner Join 0 to 1 >> condition expressions: >> 0 {0} {1} >> 1 {0} {1} >> keys: >> 0 >> 1 >> Position of Big Table: 1 >> File Output Operator >> compressed: false >> GlobalTableId: 0 >> table: >> input format: >> org.apache.hadoop.mapred.SequenceFileInputFormat >> output format: >> org.apache.hadoop.hive.ql.io.HiveSequenceFileOutputFormat >> >> Stage: Stage-2 >> Map Reduce >> Alias -> Map Operator Tree: >> hdfs://hdpnn.cm3:9000/group/taobao/hive/hive-tmp/220575636/10004 >> Select Operator >> Filter Operator >> predicate: >> expr: (((3 = '20090611') and (2 regexp 0)) and (1 = >> '20090609')) >> type: boolean >> Select Operator >> expressions: >> expr: 0 >> type: string >> expr: 2 >> type: string >> File Output Operator >> compressed: true >> GlobalTableId: 0 >> table: >> input format: >> org.apache.hadoop.mapred.TextInputFormat >> output format: >> org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat >> >> Stage: Stage-0 >> Fetch Operator >> limit: -1 >> >> On Mon, Jun 15, 2009 at 1:14 PM, Namit Jain <[email protected]> wrote: >> >> I was looking at the code – and there may be a bug in cartesian product >> codepath for map-join. >> >> Can you do a explain plan and send it ? >> >> >> >> >> >> >> On 6/14/09 10:06 PM, "Min Zhou" <[email protected]> wrote: >> >> >> 1. tried setting hive.mapjoin.cache.numrows to be 100, failed with the >> same exception. >> 2. Actually, we used to do the same thing by loading small tables into >> memory of each map node in normal map-reduce with the same cluster, where >> same heap size is guranteed between running hive map-side join and our >> map-reduce job. OOM exceptions never happened in that only 1MB would be >> spent to load those 20k pieces of records while mapred.child.java.opts was >> set to be -Xmx200m. >> >> here is the schema of our small table: >> > describe application; >> transaction_id string >> subclass_id string >> class_id string >> memo string >> url_alias string >> url_pattern string >> dt string (daily partitioned) >> >> Thanks, >> Min >> On Mon, Jun 15, 2009 at 12:51 PM, Namit Jain <[email protected]> wrote: >> >> 1. Can you reduce the number of cached rows and try ? >> >> 2. Were you using default memory settings of the mapper ? If yes, can can >> increase it and try ? >> >> It would be useful to try both of them independently – it would give a >> good idea of memory consumption of JDBM also. >> >> >> Can you send the exact schema/data of the small table if possible ? You >> can file a jira and load it there if it not a security issue. >> >> Thanks, >> -namit >> >> >> >> On 6/14/09 9:23 PM, "Min Zhou" <[email protected]> wrote: >> >> 20k >> >> >> >> >> >> > > > -- > My research interests are distributed systems, parallel computing and > bytecode based virtual machine. > > My profile: > http://www.linkedin.com/in/coderplay > My blog: > http://coderplay.javaeye.com > -- My research interests are distributed systems, parallel computing and bytecode based virtual machine. My profile: http://www.linkedin.com/in/coderplay My blog: http://coderplay.javaeye.com
