[jira] Updated: (HIVE-1539) Concurrent metastore threading problem
[ https://issues.apache.org/jira/browse/HIVE-1539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bennie Schut updated HIVE-1539: --- Attachment: HIVE-1539.patch 2.2.0-m2 can now be found on maven so the previous patch can be discarded and you can simple update the ivy/libraries/properties file to use datanucleus 2.2.0-m2. since -m2 is a milestone release I'm not sure if we would want to include this or wait for 2.2.0 release. Concurrent metastore threading problem --- Key: HIVE-1539 URL: https://issues.apache.org/jira/browse/HIVE-1539 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Reporter: Bennie Schut Assignee: Bennie Schut Attachments: ClassLoaderResolver.patch, HIVE-1539.patch, thread_dump_hanging.txt When running hive as a service and running a high number of queries concurrently I end up with multiple threads running at 100% cpu without any progress. Looking at these threads I notice this thread(484e): at org.apache.hadoop.hive.metastore.ObjectStore.getMTable(ObjectStore.java:598) But on a different thread(63a2): at org.apache.hadoop.hive.metastore.model.MStorageDescriptor.jdoReplaceField(MStorageDescriptor.java) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1776) parallel execution and auto-local mode combine to place plan file in wrong file system
[ https://issues.apache.org/jira/browse/HIVE-1776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1776: - Resolution: Fixed Fix Version/s: 0.7.0 Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed. Thanks Joy parallel execution and auto-local mode combine to place plan file in wrong file system -- Key: HIVE-1776 URL: https://issues.apache.org/jira/browse/HIVE-1776 Project: Hive Issue Type: Bug Reporter: Joydeep Sen Sarma Assignee: Joydeep Sen Sarma Fix For: 0.7.0 Attachments: HIVE-1776.1.patch, HIVE-1776.2.patch A query (that i can't reproduce verbatim) submits a job to a MR cluster with a plan file that is resident on the local file system. This job obviously fails. This seems to result from an interaction between the parallel execution (which is trying to run one local and one remote job at the same time). Turning off either the parallel execution mode or the auto-local mode seems to fix the problem. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1588) Allowing Hive's metastore when it runs as a thrift service to run as a pure metadata service and not perform any filesystem operations
[ https://issues.apache.org/jira/browse/HIVE-1588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sushanth Sowmyan resolved HIVE-1588. Resolution: Invalid Per discussion in the September Hive Dev Meetup, we'd decided to not follow the approach of having the client do filesystem operations, and introduce authentication/authorization and have the metastore server do the operations as the client user in HIVE-1476. Thus, this jira is now not needed and can be closed. Allowing Hive's metastore when it runs as a thrift service to run as a pure metadata service and not perform any filesystem operations -- Key: HIVE-1588 URL: https://issues.apache.org/jira/browse/HIVE-1588 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sushanth Sowmyan To integrate with Secure Hadoop, filesystem accesses would require an auth token, which has an expiry. Thus, if we were running hive with a thrift metastore server, the server process would need to frequently renew tokens. One potential way of dealing with that is to have the metastore server be a pure metadata service, and not perform any filesystem operations, which is taking the approach HIVE-1476 takes a step further, extending it for reads as well as writes. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1781) outputs not populated for dynamic partitions at compile time
[ https://issues.apache.org/jira/browse/HIVE-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931091#action_12931091 ] Namit Jain commented on HIVE-1781: -- In the patch, I remove the incomplete entries before the post execution hooks, so that post-execute hooks don't have to change. outputs not populated for dynamic partitions at compile time Key: HIVE-1781 URL: https://issues.apache.org/jira/browse/HIVE-1781 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain OSTHOOK: query: create table tstsrcpart like srcpart POSTHOOK: type: CREATETABLE POSTHOOK: Output: defa...@tstsrcpart PREHOOK: query: from srcpart insert overwrite table tstsrcpart partition (ds, hr) select key, value, ds, hr where ds = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=12 POSTHOOK: query: from srcpart As is evident from above, the outputs are not populated at all at compile time. This may create a problem for many components that depend on outputs: locking, authorization etc. However, the exact set of outputs may be needed for some other components (for example. the internal deployment in Facebook has a replication hook which is used for replication which needs the exact set of outputs). It may be a good idea to extend WriteEntity to include a flag which indicates whether the output is complete or not, and then the hook can look at that flag if needed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1781) outputs not populated for dynamic partitions at compile time
[ https://issues.apache.org/jira/browse/HIVE-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain reassigned HIVE-1781: Assignee: Namit Jain outputs not populated for dynamic partitions at compile time Key: HIVE-1781 URL: https://issues.apache.org/jira/browse/HIVE-1781 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain OSTHOOK: query: create table tstsrcpart like srcpart POSTHOOK: type: CREATETABLE POSTHOOK: Output: defa...@tstsrcpart PREHOOK: query: from srcpart insert overwrite table tstsrcpart partition (ds, hr) select key, value, ds, hr where ds = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=12 POSTHOOK: query: from srcpart As is evident from above, the outputs are not populated at all at compile time. This may create a problem for many components that depend on outputs: locking, authorization etc. However, the exact set of outputs may be needed for some other components (for example. the internal deployment in Facebook has a replication hook which is used for replication which needs the exact set of outputs). It may be a good idea to extend WriteEntity to include a flag which indicates whether the output is complete or not, and then the hook can look at that flag if needed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1781) outputs not populated for dynamic partitions at compile time
[ https://issues.apache.org/jira/browse/HIVE-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1781: - Attachment: hive.1781.1.patch outputs not populated for dynamic partitions at compile time Key: HIVE-1781 URL: https://issues.apache.org/jira/browse/HIVE-1781 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1781.1.patch OSTHOOK: query: create table tstsrcpart like srcpart POSTHOOK: type: CREATETABLE POSTHOOK: Output: defa...@tstsrcpart PREHOOK: query: from srcpart insert overwrite table tstsrcpart partition (ds, hr) select key, value, ds, hr where ds = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=12 POSTHOOK: query: from srcpart As is evident from above, the outputs are not populated at all at compile time. This may create a problem for many components that depend on outputs: locking, authorization etc. However, the exact set of outputs may be needed for some other components (for example. the internal deployment in Facebook has a replication hook which is used for replication which needs the exact set of outputs). It may be a good idea to extend WriteEntity to include a flag which indicates whether the output is complete or not, and then the hook can look at that flag if needed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1781) outputs not populated for dynamic partitions at compile time
[ https://issues.apache.org/jira/browse/HIVE-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1781: - Status: Patch Available (was: Open) outputs not populated for dynamic partitions at compile time Key: HIVE-1781 URL: https://issues.apache.org/jira/browse/HIVE-1781 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1781.1.patch OSTHOOK: query: create table tstsrcpart like srcpart POSTHOOK: type: CREATETABLE POSTHOOK: Output: defa...@tstsrcpart PREHOOK: query: from srcpart insert overwrite table tstsrcpart partition (ds, hr) select key, value, ds, hr where ds = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=12 POSTHOOK: query: from srcpart As is evident from above, the outputs are not populated at all at compile time. This may create a problem for many components that depend on outputs: locking, authorization etc. However, the exact set of outputs may be needed for some other components (for example. the internal deployment in Facebook has a replication hook which is used for replication which needs the exact set of outputs). It may be a good idea to extend WriteEntity to include a flag which indicates whether the output is complete or not, and then the hook can look at that flag if needed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
No matching method for ROUND(1.0)
Hi, Using trunk build, I see: hive select ROUND(1.0) from pokes limit 1; FAILED: Error in semantic analysis: line 1:7 Wrong Arguments ROUND: No matching method for class org.apache.hadoop.hive.ql.udf.UDFRound with (). Possible choices: _FUNC_(double) _FUNC_(double, int)
[jira] Created: (HIVE-1782) add more debug information for hive locking
add more debug information for hive locking --- Key: HIVE-1782 URL: https://issues.apache.org/jira/browse/HIVE-1782 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain If a hive lock cannot be acquired, currently we dont print the object on which the lock failed. It would be useful to have that information. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1781) outputs not populated for dynamic partitions at compile time
[ https://issues.apache.org/jira/browse/HIVE-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931115#action_12931115 ] He Yongqiang commented on HIVE-1781: For a query like select key, value from src where key 10, the outputs used to be a temp file, now the outputs are empty. Is this good. Will the outputs be null if do a insert overwrite [local] directory select key, value from src? outputs not populated for dynamic partitions at compile time Key: HIVE-1781 URL: https://issues.apache.org/jira/browse/HIVE-1781 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1781.1.patch OSTHOOK: query: create table tstsrcpart like srcpart POSTHOOK: type: CREATETABLE POSTHOOK: Output: defa...@tstsrcpart PREHOOK: query: from srcpart insert overwrite table tstsrcpart partition (ds, hr) select key, value, ds, hr where ds = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=12 POSTHOOK: query: from srcpart As is evident from above, the outputs are not populated at all at compile time. This may create a problem for many components that depend on outputs: locking, authorization etc. However, the exact set of outputs may be needed for some other components (for example. the internal deployment in Facebook has a replication hook which is used for replication which needs the exact set of outputs). It may be a good idea to extend WriteEntity to include a flag which indicates whether the output is complete or not, and then the hook can look at that flag if needed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1712) Migrating metadata from derby to mysql thrown NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931132#action_12931132 ] Paul Yang commented on HIVE-1712: - Committed. Thanks Jake! We'll add you as a contributor so we can assign this issue to you. Migrating metadata from derby to mysql thrown NullPointerException -- Key: HIVE-1712 URL: https://issues.apache.org/jira/browse/HIVE-1712 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.5.0, 0.6.0 Reporter: Jake Farrell Fix For: 0.7.0 Attachments: hive-1712.patch, hive-1712_rebase.patch Exported derby data to csv, loaded data into mysql and ran hive query which worked in derby and got the following exception 2010-10-16 08:57:29,080 INFO metastore.ObjectStore (ObjectStore.java:setConf(106)) - Initialized ObjectStore 2010-10-16 08:57:29,552 INFO metastore.HiveMetaStore (HiveMetaStore.java:logStartFunction(171)) - 0: get_table : db=default tbl=testimport 2010-10-16 08:57:30,140 ERROR metadata.Hive (Hive.java:getTable(395)) - java.lang.NullPointerException at java.util.Hashtable.put(Hashtable.java:394) at java.util.Hashtable.putAll(Hashtable.java:466) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getSchema(MetaStoreUtils.java:520) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getSchema(MetaStoreUtils.java:489) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:381) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:333) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:683) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5200) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275) at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1712) Migrating metadata from derby to mysql thrown NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Yang updated HIVE-1712: Resolution: Fixed Status: Resolved (was: Patch Available) Migrating metadata from derby to mysql thrown NullPointerException -- Key: HIVE-1712 URL: https://issues.apache.org/jira/browse/HIVE-1712 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.5.0, 0.6.0 Reporter: Jake Farrell Fix For: 0.7.0 Attachments: hive-1712.patch, hive-1712_rebase.patch Exported derby data to csv, loaded data into mysql and ran hive query which worked in derby and got the following exception 2010-10-16 08:57:29,080 INFO metastore.ObjectStore (ObjectStore.java:setConf(106)) - Initialized ObjectStore 2010-10-16 08:57:29,552 INFO metastore.HiveMetaStore (HiveMetaStore.java:logStartFunction(171)) - 0: get_table : db=default tbl=testimport 2010-10-16 08:57:30,140 ERROR metadata.Hive (Hive.java:getTable(395)) - java.lang.NullPointerException at java.util.Hashtable.put(Hashtable.java:394) at java.util.Hashtable.putAll(Hashtable.java:466) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getSchema(MetaStoreUtils.java:520) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getSchema(MetaStoreUtils.java:489) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:381) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:333) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:683) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5200) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275) at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HIVE-1712) Migrating metadata from derby to mysql thrown NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-1712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Yang reassigned HIVE-1712: --- Assignee: Jake Farrell Migrating metadata from derby to mysql thrown NullPointerException -- Key: HIVE-1712 URL: https://issues.apache.org/jira/browse/HIVE-1712 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.5.0, 0.6.0 Reporter: Jake Farrell Assignee: Jake Farrell Fix For: 0.7.0 Attachments: hive-1712.patch, hive-1712_rebase.patch Exported derby data to csv, loaded data into mysql and ran hive query which worked in derby and got the following exception 2010-10-16 08:57:29,080 INFO metastore.ObjectStore (ObjectStore.java:setConf(106)) - Initialized ObjectStore 2010-10-16 08:57:29,552 INFO metastore.HiveMetaStore (HiveMetaStore.java:logStartFunction(171)) - 0: get_table : db=default tbl=testimport 2010-10-16 08:57:30,140 ERROR metadata.Hive (Hive.java:getTable(395)) - java.lang.NullPointerException at java.util.Hashtable.put(Hashtable.java:394) at java.util.Hashtable.putAll(Hashtable.java:466) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getSchema(MetaStoreUtils.java:520) at org.apache.hadoop.hive.metastore.MetaStoreUtils.getSchema(MetaStoreUtils.java:489) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:381) at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:333) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getMetaData(SemanticAnalyzer.java:683) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:5200) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:105) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:275) at org.apache.hadoop.hive.ql.Driver.runCommand(Driver.java:320) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:312) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:123) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:181) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:287) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931149#action_12931149 ] John Sichi commented on HIVE-1497: -- @Russell: I meant pluralize it unconditionally (i.e. just edit the schema in your patch). support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, HIVE-1497.6.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1781) outputs not populated for dynamic partitions at compile time
[ https://issues.apache.org/jira/browse/HIVE-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931162#action_12931162 ] He Yongqiang commented on HIVE-1781: +1 running tests. outputs not populated for dynamic partitions at compile time Key: HIVE-1781 URL: https://issues.apache.org/jira/browse/HIVE-1781 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1781.1.patch OSTHOOK: query: create table tstsrcpart like srcpart POSTHOOK: type: CREATETABLE POSTHOOK: Output: defa...@tstsrcpart PREHOOK: query: from srcpart insert overwrite table tstsrcpart partition (ds, hr) select key, value, ds, hr where ds = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=12 POSTHOOK: query: from srcpart As is evident from above, the outputs are not populated at all at compile time. This may create a problem for many components that depend on outputs: locking, authorization etc. However, the exact set of outputs may be needed for some other components (for example. the internal deployment in Facebook has a replication hook which is used for replication which needs the exact set of outputs). It may be a good idea to extend WriteEntity to include a flag which indicates whether the output is complete or not, and then the hook can look at that flag if needed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1752) Avoid UnionStructObjectInspector for partition columns when necessary
[ https://issues.apache.org/jira/browse/HIVE-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-1752: -- Description: Once HIVE-1750 and HIVE-1538 are finished, union struct for partition columns and normal columns is not needed anymore. We can save some costs by not having the struct. (was: Once HIVE-1750 is finished, union struct for partition columns and normal columns is not needed anymore. We can save some costs by not having the struct.) Avoid UnionStructObjectInspector for partition columns when necessary - Key: HIVE-1752 URL: https://issues.apache.org/jira/browse/HIVE-1752 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Siying Dong Once HIVE-1750 and HIVE-1538 are finished, union struct for partition columns and normal columns is not needed anymore. We can save some costs by not having the struct. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1783) CommonJoinOperator optimize the case that 1:1 join
CommonJoinOperator optimize the case that 1:1 join -- Key: HIVE-1783 URL: https://issues.apache.org/jira/browse/HIVE-1783 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Siying Dong Priority: Minor CommonJoinOperator.genObject() is expensive. It does a recursive and keeps lots of states because it has to: 1. handle null cases for outer joins 2. handle the case of duplicated keys from one join party We can do a minor optimization to detect a 1:1 join (which is quite common) before calling CommonJoinOperator.genObject() and forward columns in a simple for-loop if we are sure neither of 1 or 2 will happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1783) CommonJoinOperator optimize the case of 1:1 join
[ https://issues.apache.org/jira/browse/HIVE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-1783: -- Summary: CommonJoinOperator optimize the case of 1:1 join (was: CommonJoinOperator optimize the case that 1:1 join) CommonJoinOperator optimize the case of 1:1 join Key: HIVE-1783 URL: https://issues.apache.org/jira/browse/HIVE-1783 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Siying Dong Priority: Minor CommonJoinOperator.genObject() is expensive. It does a recursive and keeps lots of states because it has to: 1. handle null cases for outer joins 2. handle the case of duplicated keys from one join party We can do a minor optimization to detect a 1:1 join (which is quite common) before calling CommonJoinOperator.genObject() and forward columns in a simple for-loop if we are sure neither of 1 or 2 will happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1783) CommonJoinOperator optimize the case of 1:1 join
[ https://issues.apache.org/jira/browse/HIVE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-1783: -- Attachment: HIVE-1783.1.patch CommonJoinOperator optimize the case of 1:1 join Key: HIVE-1783 URL: https://issues.apache.org/jira/browse/HIVE-1783 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-1783.1.patch CommonJoinOperator.genObject() is expensive. It does a recursive and keeps lots of states because it has to: 1. handle null cases for outer joins 2. handle the case of duplicated keys from one join party We can do a minor optimization to detect a 1:1 join (which is quite common) before calling CommonJoinOperator.genObject() and forward columns in a simple for-loop if we are sure neither of 1 or 2 will happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1783) CommonJoinOperator optimize the case of 1:1 join
[ https://issues.apache.org/jira/browse/HIVE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siying Dong updated HIVE-1783: -- Status: Patch Available (was: Open) CommonJoinOperator optimize the case of 1:1 join Key: HIVE-1783 URL: https://issues.apache.org/jira/browse/HIVE-1783 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-1783.1.patch CommonJoinOperator.genObject() is expensive. It does a recursive and keeps lots of states because it has to: 1. handle null cases for outer joins 2. handle the case of duplicated keys from one join party We can do a minor optimization to detect a 1:1 join (which is quite common) before calling CommonJoinOperator.genObject() and forward columns in a simple for-loop if we are sure neither of 1 or 2 will happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1784) Ctrl+c should kill currently running query, but not exit the CLI
Ctrl+c should kill currently running query, but not exit the CLI Key: HIVE-1784 URL: https://issues.apache.org/jira/browse/HIVE-1784 Project: Hive Issue Type: Improvement Affects Versions: 0.7.0 Reporter: Paul Yang When a query is running and Ctrl+C is pressed, the query is killed and the CLI is exited. Instead, Ctrl+c should kill the query but return the user to the Hive prompt. This will make it easier to modify and re-submit the query. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1784) Ctrl+c should kill currently running query, but not exit the CLI
[ https://issues.apache.org/jira/browse/HIVE-1784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Yang updated HIVE-1784: Component/s: CLI Priority: Minor (was: Major) Ctrl+c should kill currently running query, but not exit the CLI Key: HIVE-1784 URL: https://issues.apache.org/jira/browse/HIVE-1784 Project: Hive Issue Type: Improvement Components: CLI Affects Versions: 0.7.0 Reporter: Paul Yang Priority: Minor When a query is running and Ctrl+C is pressed, the query is killed and the CLI is exited. Instead, Ctrl+c should kill the query but return the user to the Hive prompt. This will make it easier to modify and re-submit the query. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1526) Hive should depend on a release version of Thrift
[ https://issues.apache.org/jira/browse/HIVE-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1526: - Status: Open (was: Patch Available) Hive should depend on a release version of Thrift - Key: HIVE-1526 URL: https://issues.apache.org/jira/browse/HIVE-1526 Project: Hive Issue Type: Task Components: Build Infrastructure, Clients Reporter: Carl Steinbach Assignee: Todd Lipcon Fix For: 0.7.0 Attachments: HIVE-1526.2.patch.txt, hive-1526.txt, libfb303.jar, libthrift.jar Hive should depend on a release version of Thrift, and ideally it should use Ivy to resolve this dependency. The Thrift folks are working on adding Thrift artifacts to a maven repository here: https://issues.apache.org/jira/browse/THRIFT-363 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Delayed email updates from JIRA
Just a note that (at least for facebook.com email addresses), email notifications from JIRA have become very slow over the past few days (I'm pretty sure there must be a lot queued up for me based on what I've seen while browsing JIRA). I don't know what the cause is, but just letting everyone know in case you've posted a comment and not gotten any response. JVS
[jira] Updated: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Melick updated HIVE-1497: - Attachment: HIVE-1497.7.patch Pluralized col_names in the FORMATTED column header support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, HIVE-1497.6.patch, HIVE-1497.7.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1747) Make case-insensitive substring search as fast as case-sensitive variant
[
https://issues.apache.org/jira/browse/HIVE-1747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931227#action_12931227
]
John Sichi commented on HIVE-1747:
--
Also: while at ApacheCon, I heard that the Pig team was using this as a faster
replacement for Java regexp:
http://www.brics.dk/automaton/
We might want to do the same. See PIG-965 for more details.
Make case-insensitive substring search as fast as case-sensitive variant
Key: HIVE-1747
URL: https://issues.apache.org/jira/browse/HIVE-1747
Project: Hive
Issue Type: Improvement
Components: Query Processor
Affects Versions: 0.7.0
Reporter: John Sichi
{noformat}
WHERE str_col like '%XYZ%'
{noformat}
is very fast because the pattern matching code is special-cased. However,
{noformat}
WHERE str_col rlike '(?i).*XYZ.*'
{noformat}
is very slow because it goes through the generic Java pattern matching.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1782) add more debug information for hive locking
[ https://issues.apache.org/jira/browse/HIVE-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1782: - Status: Patch Available (was: Open) add more debug information for hive locking --- Key: HIVE-1782 URL: https://issues.apache.org/jira/browse/HIVE-1782 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1782.1.patch If a hive lock cannot be acquired, currently we dont print the object on which the lock failed. It would be useful to have that information. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1781) outputs not populated for dynamic partitions at compile time
[ https://issues.apache.org/jira/browse/HIVE-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1781: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed! Thanks Namit! outputs not populated for dynamic partitions at compile time Key: HIVE-1781 URL: https://issues.apache.org/jira/browse/HIVE-1781 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1781.1.patch OSTHOOK: query: create table tstsrcpart like srcpart POSTHOOK: type: CREATETABLE POSTHOOK: Output: defa...@tstsrcpart PREHOOK: query: from srcpart insert overwrite table tstsrcpart partition (ds, hr) select key, value, ds, hr where ds = '2008-04-08' PREHOOK: type: QUERY PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=11 PREHOOK: Input: defa...@srcpart@ds=2008-04-08/hr=12 POSTHOOK: query: from srcpart As is evident from above, the outputs are not populated at all at compile time. This may create a problem for many components that depend on outputs: locking, authorization etc. However, the exact set of outputs may be needed for some other components (for example. the internal deployment in Facebook has a replication hook which is used for replication which needs the exact set of outputs). It may be a good idea to extend WriteEntity to include a flag which indicates whether the output is complete or not, and then the hook can look at that flag if needed -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1337) Support HAVING clause
[ https://issues.apache.org/jira/browse/HIVE-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931281#action_12931281 ] John Sichi commented on HIVE-1337: -- Testing JIRA email notifications. Support HAVING clause - Key: HIVE-1337 URL: https://issues.apache.org/jira/browse/HIVE-1337 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Carl Steinbach Assignee: Carl Steinbach Support the HAVING clause in HiveQL. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1586) prune operators for empty inputs
[ https://issues.apache.org/jira/browse/HIVE-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931283#action_12931283 ] John Sichi commented on HIVE-1586: -- Testing email notifications. prune operators for empty inputs Key: HIVE-1586 URL: https://issues.apache.org/jira/browse/HIVE-1586 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Given this: create view v as select * from ( select *,'T1' as m from pokes union all select *,'T2' as m from pokes2) u; select * from v where m='T2'; When we optimize the query, we can prune away all inputs to the union other than the one which references pokes2. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931287#action_12931287 ] John Sichi commented on HIVE-1785: -- Can we use reflection to make this change backwards-compatible? change Pre/Post Query Hooks to take in 1 parameter: HookContext --- Key: HIVE-1785 URL: https://issues.apache.org/jira/browse/HIVE-1785 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Liyin Tang This way, it would be possible to add new parameters to the hooks without changing the existing hooks. This will be a incompatible change, and all the hooks need to change to the new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1337) Support HAVING clause
[ https://issues.apache.org/jira/browse/HIVE-1337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931290#action_12931290 ] John Sichi commented on HIVE-1337: -- Testing again. Support HAVING clause - Key: HIVE-1337 URL: https://issues.apache.org/jira/browse/HIVE-1337 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Carl Steinbach Assignee: Carl Steinbach Support the HAVING clause in HiveQL. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1785) change Pre/Post Query Hooks to take in 1 parameter: HookContext
[ https://issues.apache.org/jira/browse/HIVE-1785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-1785: - Hadoop Flags: (was: [Incompatible change]) change Pre/Post Query Hooks to take in 1 parameter: HookContext --- Key: HIVE-1785 URL: https://issues.apache.org/jira/browse/HIVE-1785 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Liyin Tang This way, it would be possible to add new parameters to the hooks without changing the existing hooks. This will be a incompatible change, and all the hooks need to change to the new API -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: Review Request: HIVE-78: Authorization infrastructure for Hive
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55/ --- (Updated 2010-11-11 17:18:49.328105) Review request for hive. Changes --- HIVE-78.2.nothrift.patch (this time actually without generated code) Summary --- Review for patch without generated thrift code (note that the index is still split over three pages due to size). This addresses bug HIVE-78. https://issues.apache.org/jira/browse/HIVE-78 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/if/hive_metastore.thrift 1033775 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityColumn.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityDB.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityRoleEntity.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityTablePartition.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityUser.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MSecurityUserRoleMap.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/model/package.jdo 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/WriteEntity.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRProcContext.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1033775 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1031914 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1031676 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1030336 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GrantDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/GrantRevokeRoleDDL.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveQueryReadWrite.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrincipalDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PrivilegeObjectDesc.java PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RevokeDesc.java PRE-CREATION
[jira] Commented: (HIVE-1586) prune operators for empty inputs
[ https://issues.apache.org/jira/browse/HIVE-1586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931299#action_12931299 ] Greg Dingle commented on HIVE-1586: --- I got this. prune operators for empty inputs Key: HIVE-1586 URL: https://issues.apache.org/jira/browse/HIVE-1586 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: John Sichi Fix For: 0.7.0 Given this: create view v as select * from ( select *,'T1' as m from pokes union all select *,'T2' as m from pokes2) u; select * from v where m='T2'; When we optimize the query, we can prune away all inputs to the union other than the one which references pokes2. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (HIVE-1787) optimize the code path when there are no outer joins
optimize the code path when there are no outer joins Key: HIVE-1787 URL: https://issues.apache.org/jira/browse/HIVE-1787 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Siying Dong Currently, outer joins and joins are handled in the same manner - a special case for no outer joins would be useful -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1783) CommonJoinOperator optimize the case of 1:1 join
[ https://issues.apache.org/jira/browse/HIVE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931306#action_12931306 ] He Yongqiang commented on HIVE-1783: namit, can you hold this for Liyin's patch? CommonJoinOperator optimize the case of 1:1 join Key: HIVE-1783 URL: https://issues.apache.org/jira/browse/HIVE-1783 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-1783.1.patch CommonJoinOperator.genObject() is expensive. It does a recursive and keeps lots of states because it has to: 1. handle null cases for outer joins 2. handle the case of duplicated keys from one join party We can do a minor optimization to detect a 1:1 join (which is quite common) before calling CommonJoinOperator.genObject() and forward columns in a simple for-loop if we are sure neither of 1 or 2 will happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Sichi updated HIVE-1497: - Status: Open (was: Patch Available) For idx_compound, the col_names should be key,value1 not just key. support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, HIVE-1497.6.patch, HIVE-1497.7.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1783) CommonJoinOperator optimize the case of 1:1 join
[ https://issues.apache.org/jira/browse/HIVE-1783?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931311#action_12931311 ] Namit Jain commented on HIVE-1783: -- sure - let me know when you are done CommonJoinOperator optimize the case of 1:1 join Key: HIVE-1783 URL: https://issues.apache.org/jira/browse/HIVE-1783 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-1783.1.patch CommonJoinOperator.genObject() is expensive. It does a recursive and keeps lots of states because it has to: 1. handle null cases for outer joins 2. handle the case of duplicated keys from one join party We can do a minor optimization to detect a 1:1 join (which is quite common) before calling CommonJoinOperator.genObject() and forward columns in a simple for-loop if we are sure neither of 1 or 2 will happen. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-1754) Remove JDBM component from Map Join
[ https://issues.apache.org/jira/browse/HIVE-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931318#action_12931318 ] He Yongqiang commented on HIVE-1754: running tests. Remove JDBM component from Map Join --- Key: HIVE-1754 URL: https://issues.apache.org/jira/browse/HIVE-1754 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0, 0.7.0 Reporter: Liyin Tang Assignee: Liyin Tang Fix For: 0.7.0 Attachments: Hive-1754.patch, Hive-1754_2.patch, Hive-1754_3.patch, hive-1754_4.patch, hive-1754_5.patch, hive-1754_7.patch, hive-1754_9.patch Right now, JDBM is the major performance bottleneck of performance. With the growth of the small table, the PUT and GET operation will take most of execution time. Map Join is designed to load the data of small table into memory. If the data is too large to hold in memory, then there is no need to use the map join strategy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1782) add more debug information for hive locking
[ https://issues.apache.org/jira/browse/HIVE-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang updated HIVE-1782: --- Resolution: Fixed Status: Resolved (was: Patch Available) Committed! Thanks Namit! add more debug information for hive locking --- Key: HIVE-1782 URL: https://issues.apache.org/jira/browse/HIVE-1782 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.1782.1.patch If a hive lock cannot be acquired, currently we dont print the object on which the lock failed. It would be useful to have that information. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HIVE-1754) Remove JDBM component from Map Join
[ https://issues.apache.org/jira/browse/HIVE-1754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] He Yongqiang resolved HIVE-1754. Resolution: Fixed Committed! Thanks Liyin! Remove JDBM component from Map Join --- Key: HIVE-1754 URL: https://issues.apache.org/jira/browse/HIVE-1754 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0, 0.7.0 Reporter: Liyin Tang Assignee: Liyin Tang Fix For: 0.7.0 Attachments: Hive-1754.patch, Hive-1754_2.patch, Hive-1754_3.patch, hive-1754_4.patch, hive-1754_5.patch, hive-1754_7.patch, hive-1754_9.patch Right now, JDBM is the major performance bottleneck of performance. With the growth of the small table, the PUT and GET operation will take most of execution time. Map Join is designed to load the data of small table into memory. If the data is too large to hold in memory, then there is no need to use the map join strategy. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-1497) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES
[ https://issues.apache.org/jira/browse/HIVE-1497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Russell Melick updated HIVE-1497: - Status: Patch Available (was: Open) support COMMENT clause on CREATE INDEX, and add new command for SHOW INDEXES Key: HIVE-1497 URL: https://issues.apache.org/jira/browse/HIVE-1497 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.7.0 Reporter: John Sichi Assignee: Russell Melick Fix For: 0.7.0 Attachments: HIVE-1497.4.patch, HIVE-1497.5.patch, HIVE-1497.6.patch, HIVE-1497.7.patch, HIVE-1497.8.patch, hive-1497.p1.patch, hive-1497.p2.patch, hive-1497.p3.patch We need to work out the syntax for SHOW/DESCRIBE, taking partitioning into account. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
