[jira] [Commented] (HIVE-4209) Cache evaluation result of deterministic expression and reuse it
[ https://issues.apache.org/jira/browse/HIVE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645175#comment-13645175 ] Namit Jain commented on HIVE-4209: -- Thanks [~navis] Looks good. Can you commit it if tests pass ? +1 Cache evaluation result of deterministic expression and reuse it Key: HIVE-4209 URL: https://issues.apache.org/jira/browse/HIVE-4209 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4209.6.patch.txt, HIVE-4209.D9585.1.patch, HIVE-4209.D9585.2.patch, HIVE-4209.D9585.3.patch, HIVE-4209.D9585.4.patch, HIVE-4209.D9585.5.patch For example, {noformat} select key from src where key + 1 100 AND key + 1 200 limit 3; {noformat} key + 1 need not to be evaluated twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4440) SMB Operator spills to disk like it's 1999
[ https://issues.apache.org/jira/browse/HIVE-4440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13645178#comment-13645178 ] Namit Jain commented on HIVE-4440: -- I really like the title of the jira. Changing the parameter name is backward incompatible. Can you support both the current parameter and the proposed parameter for now ? Document it clearly, and say that the current parameter hive.mapjoin.bucket.cache.size will not be supported for this from 0.13 or something like that. SMB Operator spills to disk like it's 1999 -- Key: HIVE-4440 URL: https://issues.apache.org/jira/browse/HIVE-4440 Project: Hive Issue Type: Bug Reporter: Gunther Hagleitner Assignee: Gunther Hagleitner Attachments: HIVE-4440.1.patch I was recently looking into some performance issue with a query that used SMB join and was running really slow. Turns out that the SMB join by default caches only 100 values per key before spilling to disk. That seems overly conservative to me. Changing the parameter resulted in a ~5x speedup - quite significant. The parameter is: hive.mapjoin.bucket.cache.size Which right now is only used the SMB Operator as far as I can tell. The parameter was introduced originally (3 yrs ago) for the map join operator (looks like pre-SMB) and set to 100 to avoid OOM. That seems to have been in a different context though where you had to avoid running out of memory with the cached hash table in the same process, I think. Two things I'd like to propose: a) Rename it to what it does: hive.smbjoin.cache.rows b) Set it to something less restrictive: 1 If you string together a 5 table smb join with a map join and a map-side group by aggregation you might still run out of memory, but the renamed parameter should be easier to find and reduce. For most queries, I would think that 1 is still a reasonable number to cache (On the reduce side we use 25000 for shuffle joins). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4409) Prevent incompatible column type changes
[ https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642699#comment-13642699 ] Namit Jain commented on HIVE-4409: -- my bad, will do Prevent incompatible column type changes Key: HIVE-4409 URL: https://issues.apache.org/jira/browse/HIVE-4409 Project: Hive Issue Type: Improvement Components: CLI, Metastore Affects Versions: 0.10.0 Reporter: Dilip Joseph Assignee: Dilip Joseph Priority: Minor Fix For: 0.12.0 Attachments: hive.4409.1.patch, HIVE-4409.D10539.1.patch, HIVE-4409.D10539.2.patch If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, b1) FROM src LIMIT 1; SELECT * from test_table123 WHERE ds=foo1; SET hive.metastore.disallow.invalid.col.type.changes=true; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds=foo1; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
Namit Jain created HIVE-4424: Summary: MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409 Key: HIVE-4424 URL: https://issues.apache.org/jira/browse/HIVE-4424 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.12.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
[ https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13642700#comment-13642700 ] Namit Jain commented on HIVE-4424: -- Deleted metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java.orig MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409 --- Key: HIVE-4424 URL: https://issues.apache.org/jira/browse/HIVE-4424 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.12.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
[ https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4424: - Status: Patch Available (was: Open) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409 --- Key: HIVE-4424 URL: https://issues.apache.org/jira/browse/HIVE-4424 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.12.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4424) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409
[ https://issues.apache.org/jira/browse/HIVE-4424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4424: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) MetaStoreUtils.java.orig checked in mistakenly by HIVE-4409 --- Key: HIVE-4424 URL: https://issues.apache.org/jira/browse/HIVE-4424 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Fix For: 0.12.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4409) Prevent incompatible column type changes
[ https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4409: - Status: Open (was: Patch Available) comments Prevent incompatible column type changes Key: HIVE-4409 URL: https://issues.apache.org/jira/browse/HIVE-4409 Project: Hive Issue Type: Improvement Components: CLI, Metastore Affects Versions: 0.10.0 Reporter: Dilip Joseph Assignee: Dilip Joseph Priority: Minor Attachments: HIVE-4409.D10539.1.patch If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, b1) FROM src LIMIT 1; SELECT * from test_table123 WHERE ds=foo1; SET hive.metastore.disallow.invalid.col.type.changes=true; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds=foo1; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3952) merge map-job followed by map-reduce job
[ https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3952: - Resolution: Fixed Fix Version/s: 0.12.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Vinod merge map-job followed by map-reduce job Key: HIVE-3952 URL: https://issues.apache.org/jira/browse/HIVE-3952 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Vinod Kumar Vavilapalli Fix For: 0.12.0 Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, HIVE-3952-20130424.txt Consider the query like: select count(*) FROM ( select idOne, idTwo, value FROM bigTable JOIN smallTableOne on (bigTable.idOne = smallTableOne.idOne) ) firstjoin JOIN smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo); where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and hive.auto.convert.join.noconditionaltask is set to true. The joins are collapsed into mapjoins, and it leads to a map-only job (for the map-joins) followed by a map-reduce job (for the group by). Ideally, the map-only job should be merged with the following map-reduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4005) Column truncation
[ https://issues.apache.org/jira/browse/HIVE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4005: - Resolution: Fixed Fix Version/s: 0.12.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Kevin Column truncation - Key: HIVE-4005 URL: https://issues.apache.org/jira/browse/HIVE-4005 Project: Hive Issue Type: New Feature Components: CLI Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Fix For: 0.12.0 Attachments: HIVE-4005.1.patch.txt, HIVE-4005.2.patch.txt, HIVE-4005.3.patch.txt, HIVE-4005.4.patch.txt, HIVE-4005.5.patch.txt, HIVE-4005.6.patch.txt, HIVE-4005.6.patch.txt, HIVE-4005.7.patch.txt Column truncation allows users to remove data for columns that are no longer useful. This is done by removing the data for the column and setting the length of the column data and related lengths to 0 in the RC file header. RC file was fixed to recognize columns with lengths of zero to be empty and are treated as if the column doesn't exist in the data, a null is returned for every value of that column in every row. This is the same thing that happens when more columns are selected than exist in the file. A new command was added to the CLI TRUNCATE TABLE ... PARTITION ... COLUMNS ... This launches a map only job where each mapper rewrites a single file without the unnecessary column data and the adjusted headers. It does not uncompress/deserialize the data so it is much faster than rewriting the data with NULLs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4409) Prevent incompatible column type changes
[ https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641552#comment-13641552 ] Namit Jain commented on HIVE-4409: -- +1 Prevent incompatible column type changes Key: HIVE-4409 URL: https://issues.apache.org/jira/browse/HIVE-4409 Project: Hive Issue Type: Improvement Components: CLI, Metastore Affects Versions: 0.10.0 Reporter: Dilip Joseph Assignee: Dilip Joseph Priority: Minor Attachments: HIVE-4409.D10539.1.patch, HIVE-4409.D10539.2.patch If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, b1) FROM src LIMIT 1; SELECT * from test_table123 WHERE ds=foo1; SET hive.metastore.disallow.invalid.col.type.changes=true; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds=foo1; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4209) Cache evaluation result of deterministic expression and reuse it
[ https://issues.apache.org/jira/browse/HIVE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641556#comment-13641556 ] Namit Jain commented on HIVE-4209: -- Can you refresh one more time - sorry, the parallel test framework is not applying the patch cleanly. ? conf/hive-default.xml.template.orig M conf/hive-default.xml.template ? common/src/java/org/apache/hadoop/hive/conf/HiveConf.java.orig M common/src/java/org/apache/hadoop/hive/conf/HiveConf.java M ql/src/java/org/apache/hadoop/hive/ql/exec/JoinUtil.java M ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorFactory.java M ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluator.java A +ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorHead.java M ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeFieldEvaluator.java M ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java M ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeNullEvaluator.java A +ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeEvaluatorRef.java M ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeConstantEvaluator.java M ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeColumnEvaluator.java M ql/src/java/org/apache/hadoop/hive/ql/exec/ExprNodeGenericFuncEvaluator.java M ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java M ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDF.java Can you rebase the patch so that there no +, the patch does not apply cleanly in that case. Cache evaluation result of deterministic expression and reuse it Key: HIVE-4209 URL: https://issues.apache.org/jira/browse/HIVE-4209 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4209.D9585.1.patch, HIVE-4209.D9585.2.patch, HIVE-4209.D9585.3.patch, HIVE-4209.D9585.4.patch, HIVE-4209.D9585.5.patch For example, {noformat} select key from src where key + 1 100 AND key + 1 200 limit 3; {noformat} key + 1 need not to be evaluated twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4209) Cache evaluation result of deterministic expression and reuse it
[ https://issues.apache.org/jira/browse/HIVE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4209: - Status: Open (was: Patch Available) Cache evaluation result of deterministic expression and reuse it Key: HIVE-4209 URL: https://issues.apache.org/jira/browse/HIVE-4209 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4209.D9585.1.patch, HIVE-4209.D9585.2.patch, HIVE-4209.D9585.3.patch, HIVE-4209.D9585.4.patch, HIVE-4209.D9585.5.patch For example, {noformat} select key from src where key + 1 100 AND key + 1 200 limit 3; {noformat} key + 1 need not to be evaluated twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3781) not all meta events call metastore event listener
[ https://issues.apache.org/jira/browse/HIVE-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641560#comment-13641560 ] Namit Jain commented on HIVE-3781: -- This patch is also not applying cleanly. ~/hive/hive2$ svn stat M metastore/src/test/org/apache/hadoop/hive/metastore/DummyListener.java M metastore/src/test/org/apache/hadoop/hive/metastore/TestMetaStoreEventListener.java A + metastore/src/java/org/apache/hadoop/hive/metastore/events/AlterIndexEvent.java A + metastore/src/java/org/apache/hadoop/hive/metastore/events/PreDropIndexEvent.java A + metastore/src/java/org/apache/hadoop/hive/metastore/events/AddIndexEvent.java A + metastore/src/java/org/apache/hadoop/hive/metastore/events/PreAlterIndexEvent.java A + metastore/src/java/org/apache/hadoop/hive/metastore/events/PreAddIndexEvent.java M metastore/src/java/org/apache/hadoop/hive/metastore/events/PreEventContext.java A + metastore/src/java/org/apache/hadoop/hive/metastore/events/DropIndexEvent.java M metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java M metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreEventListener.java M ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java M ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java M ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java ? ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java.orig M ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java not all meta events call metastore event listener - Key: HIVE-3781 URL: https://issues.apache.org/jira/browse/HIVE-3781 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Reporter: Sudhanshu Arora Assignee: Navis Attachments: hive.3781.3.patch, hive.3781.4.patch, HIVE-3781.D7731.1.patch, HIVE-3781.D7731.2.patch, HIVE-3781.D7731.3.patch, HIVE-3781.D7731.4.patch An event listener must be called for any DDL activity. For example, create_index, drop_index today does not call metaevent listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3781) not all meta events call metastore event listener
[ https://issues.apache.org/jira/browse/HIVE-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3781: - Status: Open (was: Patch Available) not all meta events call metastore event listener - Key: HIVE-3781 URL: https://issues.apache.org/jira/browse/HIVE-3781 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Reporter: Sudhanshu Arora Assignee: Navis Attachments: hive.3781.3.patch, hive.3781.4.patch, HIVE-3781.D7731.1.patch, HIVE-3781.D7731.2.patch, HIVE-3781.D7731.3.patch, HIVE-3781.D7731.4.patch An event listener must be called for any DDL activity. For example, create_index, drop_index today does not call metaevent listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4409) Prevent incompatible column type changes
[ https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4409: - Attachment: hive.4409.1.patch Prevent incompatible column type changes Key: HIVE-4409 URL: https://issues.apache.org/jira/browse/HIVE-4409 Project: Hive Issue Type: Improvement Components: CLI, Metastore Affects Versions: 0.10.0 Reporter: Dilip Joseph Assignee: Dilip Joseph Priority: Minor Attachments: hive.4409.1.patch, HIVE-4409.D10539.1.patch, HIVE-4409.D10539.2.patch If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, b1) FROM src LIMIT 1; SELECT * from test_table123 WHERE ds=foo1; SET hive.metastore.disallow.invalid.col.type.changes=true; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds=foo1; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641594#comment-13641594 ] Namit Jain commented on HIVE-4095: -- The parallel test failed - refreshing/running again Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, hive.4095.svn.thrift.patch, hive.4095.svn.thrift.patch.refresh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4095: - Attachment: hive.4095.refresh.patch Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, hive.4095.refresh.patch, hive.4095.svn.thrift.patch, hive.4095.svn.thrift.patch.refresh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641603#comment-13641603 ] Namit Jain commented on HIVE-4068: -- +1 [~navis], can you commit it if tests pass Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch, HIVE-4068.D8859.5.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4095: - Status: Patch Available (was: Open) Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, hive.4095.refresh.patch, hive.4095.svn.thrift.patch, hive.4095.svn.thrift.patch.refresh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4095: - Resolution: Fixed Fix Version/s: 0.12.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Dheeraj Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Fix For: 0.12.0 Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, hive.4095.refresh.patch, hive.4095.svn.thrift.patch, hive.4095.svn.thrift.patch.refresh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4340) ORC should provide raw data size
[ https://issues.apache.org/jira/browse/HIVE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4340: - Attachment: hive.4340.1.patch ORC should provide raw data size Key: HIVE-4340 URL: https://issues.apache.org/jira/browse/HIVE-4340 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-4340.1.patch.txt, HIVE-4340.2.patch.txt ORC's SerDe currently does nothing, and hence does not calculate a raw data size. WriterImpl, however, has enough information to provide one. WriterImpl should compute a raw data size for each row, aggregate them per stripe and record it in the strip information, as RC currently does in its key header, and allow the FileSinkOperator access to the size per row. FileSinkOperator should be able to get the raw data size from either the SerDe or the RecordWriter when the RecordWriter can provide it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4340) ORC should provide raw data size
[ https://issues.apache.org/jira/browse/HIVE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4340: - Attachment: (was: hive.4340.1.patch) ORC should provide raw data size Key: HIVE-4340 URL: https://issues.apache.org/jira/browse/HIVE-4340 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-4340.1.patch.txt, HIVE-4340.2.patch.txt ORC's SerDe currently does nothing, and hence does not calculate a raw data size. WriterImpl, however, has enough information to provide one. WriterImpl should compute a raw data size for each row, aggregate them per stripe and record it in the strip information, as RC currently does in its key header, and allow the FileSinkOperator access to the size per row. FileSinkOperator should be able to get the raw data size from either the SerDe or the RecordWriter when the RecordWriter can provide it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4340) ORC should provide raw data size
[ https://issues.apache.org/jira/browse/HIVE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641758#comment-13641758 ] Namit Jain commented on HIVE-4340: -- Compilation is failing: [javac] /Users/njain/hive/hive_commit3/ql/src/java/org/apache/hadoop/hive/ql/io/orc/WriterImpl.java:986: abstract method write(java.lang.Object) in org.apache.hadoop.hive.ql.io.orc.WriterImpl.TreeWriter cannot be accessed directly [javac] super.write(obj); [javac]^ ORC should provide raw data size Key: HIVE-4340 URL: https://issues.apache.org/jira/browse/HIVE-4340 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-4340.1.patch.txt, HIVE-4340.2.patch.txt ORC's SerDe currently does nothing, and hence does not calculate a raw data size. WriterImpl, however, has enough information to provide one. WriterImpl should compute a raw data size for each row, aggregate them per stripe and record it in the strip information, as RC currently does in its key header, and allow the FileSinkOperator access to the size per row. FileSinkOperator should be able to get the raw data size from either the SerDe or the RecordWriter when the RecordWriter can provide it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4409) Prevent incompatible column type changes
[ https://issues.apache.org/jira/browse/HIVE-4409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4409: - Resolution: Fixed Fix Version/s: 0.12.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Dilip Prevent incompatible column type changes Key: HIVE-4409 URL: https://issues.apache.org/jira/browse/HIVE-4409 Project: Hive Issue Type: Improvement Components: CLI, Metastore Affects Versions: 0.10.0 Reporter: Dilip Joseph Assignee: Dilip Joseph Priority: Minor Fix For: 0.12.0 Attachments: hive.4409.1.patch, HIVE-4409.D10539.1.patch, HIVE-4409.D10539.2.patch If a user changes the type of an existing column of a partitioned table to an incompatible type, subsequent accesses of old partitions will result in a ClassCastException (see example below). We should prevent the user from making incompatible type changes. This feature will be controlled by a new config parameter. Example: CREATE TABLE test_table123 (a INT, b MAPSTRING, STRING) PARTITIONED BY (ds STRING) STORED AS SEQUENCEFILE; INSERT OVERWRITE TABLE test_table123 PARTITION(ds=foo1) SELECT 1, MAP(a1, b1) FROM src LIMIT 1; SELECT * from test_table123 WHERE ds=foo1; SET hive.metastore.disallow.invalid.col.type.changes=true; ALTER TABLE test_table123 REPLACE COLUMNS (a INT, b STRING); SELECT * from test_table123 WHERE ds=foo1; The last SELECT fails with the following exception: Failed with exception java.io.IOException:java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector java.io.IOException: java.lang.ClassCastException: org.apache.hadoop.hive.serde2.lazy.objectinspector.LazyMapObjectInspector cannot be cast to org.apache.hadoop.hive.serde2.objectinspector.PrimitiveObjectInspector at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:544) at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:488) at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:136) at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1406) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:271) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348) at org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:790) at org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:124) at org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_class_cast(TestCliDriver.java:108) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date
[ https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4300: - Resolution: Fixed Fix Version/s: 0.12.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Roshan ant thriftif generated code that is checkedin is not up-to-date Key: HIVE-4300 URL: https://issues.apache.org/jira/browse/HIVE-4300 Project: Hive Issue Type: Bug Components: Thrift API Affects Versions: 0.10.0 Reporter: Roshan Naik Assignee: Roshan Naik Fix For: 0.12.0 Attachments: HIVE-4300.2.patch, HIVE-4300.patch running 'ant thriftif -Dthrift.home=/usr/local' on a freshly checkedout trunk should be a no-op as per [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode] However this is not the case. Some of files seem to be have been relocated or the classes in them are now in a different file. Below is the git status showing the state after the command is run: # On branch trunk # Changes not staged for commit: # (use git add/rm file... to update what will be committed) # (use git checkout -- file... to discard changes in working directory) # # modified: build.properties # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore_constants.php # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php # modified: metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php # deleted: service/src/gen/thrift/gen-php/hive_service/hive_service_types.php # modified: service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote # modified: service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote # # Untracked files: # (use git add file... to include in what will be committed) # # serde/src/gen/thrift/gen-cpp/complex_constants.cpp # serde/src/gen/thrift/gen-cpp/complex_constants.h # serde/src/gen/thrift/gen-cpp/complex_types.cpp # serde/src/gen/thrift/gen-cpp/complex_types.h # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp # serde/src/gen/thrift/gen-cpp
[jira] [Created] (HIVE-4411) negative test local_mapred_error_cache.q is failing trunk
Namit Jain created HIVE-4411: Summary: negative test local_mapred_error_cache.q is failing trunk Key: HIVE-4411 URL: https://issues.apache.org/jira/browse/HIVE-4411 Project: Hive Issue Type: Bug Reporter: Namit Jain -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4095: - Attachment: hive.4095.svn.thrift.patch Attaching the svn patch after generating thrift files, since the parallel testing tool does not work with a git patch. Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, hive.4095.svn.thrift.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4095: - Attachment: hive.4095.svn.thrift.patch.refresh Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt, hive.4095.svn.thrift.patch, hive.4095.svn.thrift.patch.refresh -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4340) ORC should provide raw data size
[ https://issues.apache.org/jira/browse/HIVE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640180#comment-13640180 ] Namit Jain commented on HIVE-4340: -- +1 ORC should provide raw data size Key: HIVE-4340 URL: https://issues.apache.org/jira/browse/HIVE-4340 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-4340.1.patch.txt ORC's SerDe currently does nothing, and hence does not calculate a raw data size. WriterImpl, however, has enough information to provide one. WriterImpl should compute a raw data size for each row, aggregate them per stripe and record it in the strip information, as RC currently does in its key header, and allow the FileSinkOperator access to the size per row. FileSinkOperator should be able to get the raw data size from either the SerDe or the RecordWriter when the RecordWriter can provide it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4324) ORC Turn off dictionary encoding when number of distinct keys is greater than threshold
[ https://issues.apache.org/jira/browse/HIVE-4324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4324: - Status: Open (was: Patch Available) can you address Owen's comments ? ORC Turn off dictionary encoding when number of distinct keys is greater than threshold --- Key: HIVE-4324 URL: https://issues.apache.org/jira/browse/HIVE-4324 Project: Hive Issue Type: Sub-task Components: Serializers/Deserializers Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-4324.1.patch.txt Add a configurable threshold so that if the number of distinct values in a string column is greater than that fraction of non-null values, dictionary encoding is turned off. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4005) Column truncation
[ https://issues.apache.org/jira/browse/HIVE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640186#comment-13640186 ] Namit Jain commented on HIVE-4005: -- Sorry, this one slipped under the radar. Can you refresh one more time ? Column truncation - Key: HIVE-4005 URL: https://issues.apache.org/jira/browse/HIVE-4005 Project: Hive Issue Type: New Feature Components: CLI Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-4005.1.patch.txt, HIVE-4005.2.patch.txt, HIVE-4005.3.patch.txt, HIVE-4005.4.patch.txt, HIVE-4005.5.patch.txt Column truncation allows users to remove data for columns that are no longer useful. This is done by removing the data for the column and setting the length of the column data and related lengths to 0 in the RC file header. RC file was fixed to recognize columns with lengths of zero to be empty and are treated as if the column doesn't exist in the data, a null is returned for every value of that column in every row. This is the same thing that happens when more columns are selected than exist in the file. A new command was added to the CLI TRUNCATE TABLE ... PARTITION ... COLUMNS ... This launches a map only job where each mapper rewrites a single file without the unnecessary column data and the adjusted headers. It does not uncompress/deserialize the data so it is much faster than rewriting the data with NULLs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4005) Column truncation
[ https://issues.apache.org/jira/browse/HIVE-4005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4005: - Status: Open (was: Patch Available) Column truncation - Key: HIVE-4005 URL: https://issues.apache.org/jira/browse/HIVE-4005 Project: Hive Issue Type: New Feature Components: CLI Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-4005.1.patch.txt, HIVE-4005.2.patch.txt, HIVE-4005.3.patch.txt, HIVE-4005.4.patch.txt, HIVE-4005.5.patch.txt Column truncation allows users to remove data for columns that are no longer useful. This is done by removing the data for the column and setting the length of the column data and related lengths to 0 in the RC file header. RC file was fixed to recognize columns with lengths of zero to be empty and are treated as if the column doesn't exist in the data, a null is returned for every value of that column in every row. This is the same thing that happens when more columns are selected than exist in the file. A new command was added to the CLI TRUNCATE TABLE ... PARTITION ... COLUMNS ... This launches a map only job where each mapper rewrites a single file without the unnecessary column data and the adjusted headers. It does not uncompress/deserialize the data so it is much faster than rewriting the data with NULLs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4375) Single sourced multi insert consists of native and non-native table mixed throws NPE
[ https://issues.apache.org/jira/browse/HIVE-4375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4375: - Status: Open (was: Patch Available) minor comments Single sourced multi insert consists of native and non-native table mixed throws NPE Key: HIVE-4375 URL: https://issues.apache.org/jira/browse/HIVE-4375 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4375.D10329.1.patch CREATE TABLE src_x1(key string, value string); CREATE TABLE src_x2(key string, value string) STORED BY 'org.apache.hadoop.hive.hbase.HBaseStorageHandler' WITH SERDEPROPERTIES (hbase.columns.mapping = :key,cf:string); explain from src a insert overwrite table src_x1 select key,value where a.key 0 AND a.key 50 insert overwrite table src_x2 select key,value where a.key 50 AND a.key 100; throws, {noformat} java.lang.NullPointerException at org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.addStatsTask(GenMRFileSink1.java:236) at org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(GenMRFileSink1.java:126) at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:89) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:87) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:55) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(GenMapRedWalker.java:67) at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:101) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genMapRedTasks(SemanticAnalyzer.java:8354) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8759) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:279) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:433) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4340) ORC should provide raw data size
[ https://issues.apache.org/jira/browse/HIVE-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4340: - Status: Open (was: Patch Available) The patch is not applying cleanly. Can you refresh ? ORC should provide raw data size Key: HIVE-4340 URL: https://issues.apache.org/jira/browse/HIVE-4340 Project: Hive Issue Type: Improvement Components: Serializers/Deserializers Affects Versions: 0.11.0 Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-4340.1.patch.txt ORC's SerDe currently does nothing, and hence does not calculate a raw data size. WriterImpl, however, has enough information to provide one. WriterImpl should compute a raw data size for each row, aggregate them per stripe and record it in the strip information, as RC currently does in its key header, and allow the FileSinkOperator access to the size per row. FileSinkOperator should be able to get the raw data size from either the SerDe or the RecordWriter when the RecordWriter can provide it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4209) Cache evaluation result of deterministic expression and reuse it
[ https://issues.apache.org/jira/browse/HIVE-4209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4209: - Status: Open (was: Patch Available) comments Cache evaluation result of deterministic expression and reuse it Key: HIVE-4209 URL: https://issues.apache.org/jira/browse/HIVE-4209 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4209.D9585.1.patch, HIVE-4209.D9585.2.patch, HIVE-4209.D9585.3.patch, HIVE-4209.D9585.4.patch For example, {noformat} select key from src where key + 1 100 AND key + 1 200 limit 3; {noformat} key + 1 need not to be evaluated twice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3781) not all meta events call metastore event listener
[ https://issues.apache.org/jira/browse/HIVE-3781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3781: - Status: Open (was: Patch Available) The code changes look good, but the patch is not applying cleanly. Can you refresh ? not all meta events call metastore event listener - Key: HIVE-3781 URL: https://issues.apache.org/jira/browse/HIVE-3781 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.9.0 Reporter: Sudhanshu Arora Assignee: Navis Attachments: hive.3781.3.patch, hive.3781.4.patch, HIVE-3781.D7731.1.patch, HIVE-3781.D7731.2.patch, HIVE-3781.D7731.3.patch An event listener must be called for any DDL activity. For example, create_index, drop_index today does not call metaevent listener. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4068) Size of aggregation buffer which uses non-primitive type is not estimated correctly
[ https://issues.apache.org/jira/browse/HIVE-4068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4068: - Status: Open (was: Patch Available) comments Size of aggregation buffer which uses non-primitive type is not estimated correctly --- Key: HIVE-4068 URL: https://issues.apache.org/jira/browse/HIVE-4068 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Attachments: HIVE-4068.D8859.1.patch, HIVE-4068.D8859.2.patch, HIVE-4068.D8859.3.patch, HIVE-4068.D8859.4.patch Currently, hive assumes an aggregation buffer which holds a map is occupying just 256 byte (fixed). If it's bigger than that in real, OutOfMemoryError can be thrown (especially for 1k buffer). workaround : set hive.map.aggr.hash.percentmemory=smaller value than default(0.5) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3562) Some limit can be pushed down to map stage
[ https://issues.apache.org/jira/browse/HIVE-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3562: - Status: Open (was: Patch Available) can you refresh ? Some limit can be pushed down to map stage -- Key: HIVE-3562 URL: https://issues.apache.org/jira/browse/HIVE-3562 Project: Hive Issue Type: Bug Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-3562.D5967.1.patch, HIVE-3562.D5967.2.patch, HIVE-3562.D5967.3.patch, HIVE-3562.D5967.4.patch Queries with limit clause (with reasonable number), for example {noformat} select * from src order by key limit 10; {noformat} makes operator tree, TS-SEL-RS-EXT-LIMIT-FS But LIMIT can be partially calculated in RS, reducing size of shuffling. TS-SEL-RS(TOP-N)-EXT-LIMIT-FS -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4181) Star argument without table alias for UDTF is not working
[ https://issues.apache.org/jira/browse/HIVE-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4181: - Resolution: Fixed Fix Version/s: 0.12.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Navis Star argument without table alias for UDTF is not working - Key: HIVE-4181 URL: https://issues.apache.org/jira/browse/HIVE-4181 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Navis Assignee: Navis Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4181.D9453.1.patch select explode(*) as x from src; thorws {noformat} java.lang.NullPointerException at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genUDTFPlan(SemanticAnalyzer.java:5344) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2526) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:2284) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:7079) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:7050) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:7792) at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:8462) at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:259) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:433) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:337) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902) at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413) at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:756) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:186) {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3952) merge map-job followed by map-reduce job
[ https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3952: - Attachment: hive.3952.1.patch merge map-job followed by map-reduce job Key: HIVE-3952 URL: https://issues.apache.org/jira/browse/HIVE-3952 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Vinod Kumar Vavilapalli Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, HIVE-3952-20130424.txt Consider the query like: select count(*) FROM ( select idOne, idTwo, value FROM bigTable JOIN smallTableOne on (bigTable.idOne = smallTableOne.idOne) ) firstjoin JOIN smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo); where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and hive.auto.convert.join.noconditionaltask is set to true. The joins are collapsed into mapjoins, and it leads to a map-only job (for the map-joins) followed by a map-reduce job (for the group by). Ideally, the map-only job should be merged with the following map-reduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3952) merge map-job followed by map-reduce job
[ https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641374#comment-13641374 ] Namit Jain commented on HIVE-3952: -- merge conflicts again - attached latest patch. merge map-job followed by map-reduce job Key: HIVE-3952 URL: https://issues.apache.org/jira/browse/HIVE-3952 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Vinod Kumar Vavilapalli Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, HIVE-3952-20130424.txt Consider the query like: select count(*) FROM ( select idOne, idTwo, value FROM bigTable JOIN smallTableOne on (bigTable.idOne = smallTableOne.idOne) ) firstjoin JOIN smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo); where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and hive.auto.convert.join.noconditionaltask is set to true. The joins are collapsed into mapjoins, and it leads to a map-only job (for the map-joins) followed by a map-reduce job (for the group by). Ideally, the map-only job should be merged with the following map-reduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3952) merge map-job followed by map-reduce job
[ https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641375#comment-13641375 ] Namit Jain commented on HIVE-3952: -- https://reviews.facebook.net/D10533 just for the record merge map-job followed by map-reduce job Key: HIVE-3952 URL: https://issues.apache.org/jira/browse/HIVE-3952 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Vinod Kumar Vavilapalli Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, HIVE-3952-20130424.txt Consider the query like: select count(*) FROM ( select idOne, idTwo, value FROM bigTable JOIN smallTableOne on (bigTable.idOne = smallTableOne.idOne) ) firstjoin JOIN smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo); where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and hive.auto.convert.join.noconditionaltask is set to true. The joins are collapsed into mapjoins, and it leads to a map-only job (for the map-joins) followed by a map-reduce job (for the group by). Ideally, the map-only job should be merged with the following map-reduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3952) merge map-job followed by map-reduce job
[ https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13641383#comment-13641383 ] Namit Jain commented on HIVE-3952: -- +1 merge map-job followed by map-reduce job Key: HIVE-3952 URL: https://issues.apache.org/jira/browse/HIVE-3952 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Vinod Kumar Vavilapalli Attachments: hive.3952.1.patch, HIVE-3952-20130226.txt, HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt, HIVE-3952-20130424.txt Consider the query like: select count(*) FROM ( select idOne, idTwo, value FROM bigTable JOIN smallTableOne on (bigTable.idOne = smallTableOne.idOne) ) firstjoin JOIN smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo); where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and hive.auto.convert.join.noconditionaltask is set to true. The joins are collapsed into mapjoins, and it leads to a map-only job (for the map-joins) followed by a map-reduce job (for the group by). Ideally, the map-only job should be merged with the following map-reduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4018) MapJoin failing with Distributed Cache error
[ https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638926#comment-13638926 ] Namit Jain commented on HIVE-4018: -- will do MapJoin failing with Distributed Cache error Key: HIVE-4018 URL: https://issues.apache.org/jira/browse/HIVE-4018 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.11.0 Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.11.0 Attachments: HIVE-4018.patch, hive.4018.test.2.patch, HIVE-4018-test.patch When I'm a running a star join query after HIVE-3784, it is failing with following error: 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error 2013-02-13 08:36:04,585 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278) at org.apache.hadoop.mapred.Child.main(Child.java:260) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4018) MapJoin failing with Distributed Cache error
[ https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638931#comment-13638931 ] Namit Jain commented on HIVE-4018: -- +1 MapJoin failing with Distributed Cache error Key: HIVE-4018 URL: https://issues.apache.org/jira/browse/HIVE-4018 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.11.0 Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.11.0 Attachments: HIVE-4018.patch, hive.4018.test.2.patch, HIVE-4018-test.patch When I'm a running a star join query after HIVE-3784, it is failing with following error: 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error 2013-02-13 08:36:04,585 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278) at org.apache.hadoop.mapred.Child.main(Child.java:260) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4018) MapJoin failing with Distributed Cache error
[ https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638934#comment-13638934 ] Namit Jain commented on HIVE-4018: -- [~amareshwari], can you load the latest patch (as an attachment) ? MapJoin failing with Distributed Cache error Key: HIVE-4018 URL: https://issues.apache.org/jira/browse/HIVE-4018 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.11.0 Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.11.0 Attachments: HIVE-4018.patch, hive.4018.test.2.patch, HIVE-4018-test.patch When I'm a running a star join query after HIVE-3784, it is failing with following error: 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error 2013-02-13 08:36:04,585 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278) at org.apache.hadoop.mapred.Child.main(Child.java:260) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4018) MapJoin failing with Distributed Cache error
[ https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4018: - Status: Open (was: Patch Available) MapJoin failing with Distributed Cache error Key: HIVE-4018 URL: https://issues.apache.org/jira/browse/HIVE-4018 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.11.0 Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.11.0 Attachments: HIVE-4018-2.txt, HIVE-4018.patch, hive.4018.test.2.patch, HIVE-4018-test.patch When I'm a running a star join query after HIVE-3784, it is failing with following error: 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error 2013-02-13 08:36:04,585 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278) at org.apache.hadoop.mapred.Child.main(Child.java:260) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638973#comment-13638973 ] Namit Jain commented on HIVE-3891: -- comments physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.14.patch, hive.3891.15.patch, hive.3891.16.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4018) MapJoin failing with Distributed Cache error
[ https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4018: - Fix Version/s: (was: 0.11.0) 0.12.0 Hadoop Flags: Reviewed Status: Patch Available (was: Open) Committed. Thanks Amareshwari MapJoin failing with Distributed Cache error Key: HIVE-4018 URL: https://issues.apache.org/jira/browse/HIVE-4018 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.11.0 Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.12.0 Attachments: HIVE-4018-2.txt, HIVE-4018.patch, hive.4018.test.2.patch, HIVE-4018-test.patch When I'm a running a star join query after HIVE-3784, it is failing with following error: 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error 2013-02-13 08:36:04,585 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278) at org.apache.hadoop.mapred.Child.main(Child.java:260) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events
[ https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13639284#comment-13639284 ] Namit Jain commented on HIVE-4393: -- +1 Running tests. Make the deleteData flag accessable from DropTable/Partition events --- Key: HIVE-4393 URL: https://issues.apache.org/jira/browse/HIVE-4393 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Morgan Phillips Assignee: Morgan Phillips Priority: Minor Attachments: HIVE-4393.D10443.1.patch, HIVE-4393.D10443.2.patch On occasion, due to some error during a drop, information is removed from the metastore but data, which should have been removed, remains intact on the DFS. In order to log such events via PreEvent and Event listeners a new method 'getDeleteData' should be added to (Pre)DropPartitionEvent and (Pre)DropTableEvent which returns the deleteData flag's value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events
[ https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4393: - Status: Patch Available (was: Open) Make the deleteData flag accessable from DropTable/Partition events --- Key: HIVE-4393 URL: https://issues.apache.org/jira/browse/HIVE-4393 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Morgan Phillips Assignee: Morgan Phillips Priority: Minor Attachments: HIVE-4393.D10443.1.patch, HIVE-4393.D10443.2.patch On occasion, due to some error during a drop, information is removed from the metastore but data, which should have been removed, remains intact on the DFS. In order to log such events via PreEvent and Event listeners a new method 'getDeleteData' should be added to (Pre)DropPartitionEvent and (Pre)DropTableEvent which returns the deleteData flag's value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events
[ https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4393: - Resolution: Fixed Fix Version/s: 0.12.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Morgan Make the deleteData flag accessable from DropTable/Partition events --- Key: HIVE-4393 URL: https://issues.apache.org/jira/browse/HIVE-4393 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Morgan Phillips Assignee: Morgan Phillips Priority: Minor Fix For: 0.12.0 Attachments: HIVE-4393.D10443.1.patch, HIVE-4393.D10443.2.patch On occasion, due to some error during a drop, information is removed from the metastore but data, which should have been removed, remains intact on the DFS. In order to log such events via PreEvent and Event listeners a new method 'getDeleteData' should be added to (Pre)DropPartitionEvent and (Pre)DropTableEvent which returns the deleteData flag's value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4018) MapJoin failing with Distributed Cache error
[ https://issues.apache.org/jira/browse/HIVE-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4018: - Resolution: Fixed Status: Resolved (was: Patch Available) MapJoin failing with Distributed Cache error Key: HIVE-4018 URL: https://issues.apache.org/jira/browse/HIVE-4018 Project: Hive Issue Type: Bug Components: SQL Affects Versions: 0.11.0 Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Fix For: 0.12.0 Attachments: HIVE-4018-2.txt, HIVE-4018.patch, hive.4018.test.2.patch, HIVE-4018-test.patch When I'm a running a star join query after HIVE-3784, it is failing with following error: 2013-02-13 08:36:04,584 ERROR org.apache.hadoop.hive.ql.exec.MapJoinOperator: Load Distributed Cache Error 2013-02-13 08:36:04,585 FATAL ExecMapper: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.EOFException at org.apache.hadoop.hive.ql.exec.MapJoinOperator.loadHashTable(MapJoinOperator.java:189) at org.apache.hadoop.hive.ql.exec.MapJoinOperator.cleanUpInputFileChangedOp(MapJoinOperator.java:203) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1421) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1425) at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:614) at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:144) at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:391) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:325) at org.apache.hadoop.mapred.Child$4.run(Child.java:266) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:416) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278) at org.apache.hadoop.mapred.Child.main(Child.java:260) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date
[ https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640052#comment-13640052 ] Namit Jain commented on HIVE-4300: -- sure, running tests ant thriftif generated code that is checkedin is not up-to-date Key: HIVE-4300 URL: https://issues.apache.org/jira/browse/HIVE-4300 Project: Hive Issue Type: Bug Components: Thrift API Affects Versions: 0.10.0 Reporter: Roshan Naik Assignee: Roshan Naik Attachments: HIVE-4300.2.patch, HIVE-4300.patch running 'ant thriftif -Dthrift.home=/usr/local' on a freshly checkedout trunk should be a no-op as per [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode] However this is not the case. Some of files seem to be have been relocated or the classes in them are now in a different file. Below is the git status showing the state after the command is run: # On branch trunk # Changes not staged for commit: # (use git add/rm file... to update what will be committed) # (use git checkout -- file... to discard changes in working directory) # # modified: build.properties # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore_constants.php # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php # modified: metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php # deleted: service/src/gen/thrift/gen-php/hive_service/hive_service_types.php # modified: service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote # modified: service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote # # Untracked files: # (use git add file... to include in what will be committed) # # serde/src/gen/thrift/gen-cpp/complex_constants.cpp # serde/src/gen/thrift/gen-cpp/complex_constants.h # serde/src/gen/thrift/gen-cpp/complex_types.cpp # serde/src/gen/thrift/gen-cpp/complex_types.h # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp # serde/src/gen/thrift/gen-cpp/megastruct_constants.h # serde/src/gen/thrift/gen-cpp/megastruct_types.cpp # serde/src/gen/thrift/gen-cpp
[jira] [Commented] (HIVE-4371) some issue with merging join trees
[ https://issues.apache.org/jira/browse/HIVE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637782#comment-13637782 ] Namit Jain commented on HIVE-4371: -- Looks good. Running tests some issue with merging join trees -- Key: HIVE-4371 URL: https://issues.apache.org/jira/browse/HIVE-4371 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Namit Jain Assignee: Navis Attachments: HIVE-4371.D10323.1.patch, HIVE-4371.D10323.2.patch [~navis], I would really appreciate if you can take a look. I am attaching a testcase, for which in the optimizer the join context left aliases and right aliases do not look correct. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4396) Create a efficient method to create a deep copy of a mapredwork
Namit Jain created HIVE-4396: Summary: Create a efficient method to create a deep copy of a mapredwork Key: HIVE-4396 URL: https://issues.apache.org/jira/browse/HIVE-4396 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain In many optimization steps, we create a copy of the query plan and then change some parts of it. Currently, the only reliable way to do so is to serialize and deserialize it, which is fairly expensive. It would be very useful to have a cheaper method to do the same. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3891: - Attachment: hive.3891.15.patch physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.14.patch, hive.3891.15.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events
[ https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637839#comment-13637839 ] Namit Jain commented on HIVE-4393: -- [~mrrrgn], when a patch is ready for review, please mark 'Submit Patch'. Make the deleteData flag accessable from DropTable/Partition events --- Key: HIVE-4393 URL: https://issues.apache.org/jira/browse/HIVE-4393 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Morgan Phillips Assignee: Morgan Phillips Priority: Minor On occasion, due to some error during a drop, information is removed from the metastore but data, which should have been removed, remains intact on the DFS. In order to log such events via PreEvent and Event listeners a new method 'getDeleteData' should be added to (Pre)DropPartitionEvent and (Pre)DropTableEvent which returns the deleteData flag's value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3952) merge map-job followed by map-reduce job
[ https://issues.apache.org/jira/browse/HIVE-3952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3952: - Status: Open (was: Patch Available) merge map-job followed by map-reduce job Key: HIVE-3952 URL: https://issues.apache.org/jira/browse/HIVE-3952 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Namit Jain Assignee: Vinod Kumar Vavilapalli Attachments: HIVE-3952-20130226.txt, HIVE-3952-20130227.1.txt, HIVE-3952-20130301.txt, HIVE-3952-20130421.txt Consider the query like: select count(*) FROM ( select idOne, idTwo, value FROM bigTable JOIN smallTableOne on (bigTable.idOne = smallTableOne.idOne) ) firstjoin JOIN smallTableTwo on (firstjoin.idTwo = smallTableTwo.idTwo); where smallTableOne and smallTableTwo are smaller than hive.auto.convert.join.noconditionaltask.size and hive.auto.convert.join.noconditionaltask is set to true. The joins are collapsed into mapjoins, and it leads to a map-only job (for the map-joins) followed by a map-reduce job (for the group by). Ideally, the map-only job should be merged with the following map-reduce job. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4371) some issue with merging join trees
[ https://issues.apache.org/jira/browse/HIVE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4371: - Resolution: Fixed Fix Version/s: 0.12.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Navis some issue with merging join trees -- Key: HIVE-4371 URL: https://issues.apache.org/jira/browse/HIVE-4371 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Namit Jain Assignee: Navis Fix For: 0.12.0 Attachments: HIVE-4371.D10323.1.patch, HIVE-4371.D10323.2.patch [~navis], I would really appreciate if you can take a look. I am attaching a testcase, for which in the optimizer the join context left aliases and right aliases do not look correct. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4106) SMB joins fail in multi-way joins
[ https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4106: - Resolution: Fixed Fix Version/s: 0.12.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Fixed as part of HIVE-4371 SMB joins fail in multi-way joins - Key: HIVE-4106 URL: https://issues.apache.org/jira/browse/HIVE-4106 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Namit Jain Priority: Blocker Fix For: 0.12.0 Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, hive.4106.2.patch, HIVE-4106.patch I see array out of bounds exception in case of multi way smb joins. This is related to changes that went in as part of HIVE-3403. This issue has been discussed in HIVE-3891. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3891: - Attachment: hive.3891.16.patch physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.14.patch, hive.3891.15.patch, hive.3891.16.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637908#comment-13637908 ] Namit Jain commented on HIVE-3891: -- Refreshed, addressed comments, ran tests. Can you take a look again ? cc [~ashutoshc] physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.14.patch, hive.3891.15.patch, hive.3891.16.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4377) Add more comment to https://reviews.facebook.net/D1209 (HIVE-2340)
[ https://issues.apache.org/jira/browse/HIVE-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4377: - Status: Open (was: Patch Available) Add more comment to https://reviews.facebook.net/D1209 (HIVE-2340) -- Key: HIVE-4377 URL: https://issues.apache.org/jira/browse/HIVE-4377 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Gang Tim Liu Assignee: Navis Attachments: HIVE-4377.D10377.1.patch thanks a lot for addressing optimization in HIVE-2340. Awesome! Since we are developing at a very fast pace, it would be really useful to think about maintainability and testing of the large codebase. Highlights which are applicable for D1209: 1. Javadoc for all public/private functions, except for setters/getters. For any complex function, clear examples (input/output) would really help. 2. Specially, for query optimizations, it might be a good idea to have a simple working query at the top, and the expected changes. For e.g.. The operator tree for that query at each step, or a detailed explanation at the top. 3. If possible, the test name (.q file) where the function is being invoked, or the query which would potentially test that scenario, if it is a query processor change. 4. Comments in each test (.q file) that should include the jira number, what is it trying to test. Assumptions about each query. 5. Reduce the output for each test whenever query is outputting more than 10 results, it should have a reason. Otherwise, each query result should be bounded by 10 rows. thanks a lot -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events
[ https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638746#comment-13638746 ] Namit Jain commented on HIVE-4393: -- Can you address my earlier comments on https://reviews.facebook.net/D10425 ? Make the deleteData flag accessable from DropTable/Partition events --- Key: HIVE-4393 URL: https://issues.apache.org/jira/browse/HIVE-4393 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Morgan Phillips Assignee: Morgan Phillips Priority: Minor Attachments: HIVE-4393.D10443.1.patch On occasion, due to some error during a drop, information is removed from the metastore but data, which should have been removed, remains intact on the DFS. In order to log such events via PreEvent and Event listeners a new method 'getDeleteData' should be added to (Pre)DropPartitionEvent and (Pre)DropTableEvent which returns the deleteData flag's value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4393) Make the deleteData flag accessable from DropTable/Partition events
[ https://issues.apache.org/jira/browse/HIVE-4393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4393: - Status: Open (was: Patch Available) Make the deleteData flag accessable from DropTable/Partition events --- Key: HIVE-4393 URL: https://issues.apache.org/jira/browse/HIVE-4393 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Morgan Phillips Assignee: Morgan Phillips Priority: Minor Attachments: HIVE-4393.D10443.1.patch On occasion, due to some error during a drop, information is removed from the metastore but data, which should have been removed, remains intact on the DFS. In order to log such events via PreEvent and Event listeners a new method 'getDeleteData' should be added to (Pre)DropPartitionEvent and (Pre)DropTableEvent which returns the deleteData flag's value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date
[ https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638747#comment-13638747 ] Namit Jain commented on HIVE-4300: -- [~roshan_naik], did you do anything different ? or just ant thriftif -Dthrift.home=... on a mac. ant thriftif generated code that is checkedin is not up-to-date Key: HIVE-4300 URL: https://issues.apache.org/jira/browse/HIVE-4300 Project: Hive Issue Type: Bug Components: Thrift API Affects Versions: 0.10.0 Reporter: Roshan Naik Assignee: Roshan Naik Attachments: HIVE-4300.2.patch, HIVE-4300.patch running 'ant thriftif -Dthrift.home=/usr/local' on a freshly checkedout trunk should be a no-op as per [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode] However this is not the case. Some of files seem to be have been relocated or the classes in them are now in a different file. Below is the git status showing the state after the command is run: # On branch trunk # Changes not staged for commit: # (use git add/rm file... to update what will be committed) # (use git checkout -- file... to discard changes in working directory) # # modified: build.properties # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore_constants.php # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php # modified: metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php # deleted: service/src/gen/thrift/gen-php/hive_service/hive_service_types.php # modified: service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote # modified: service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote # # Untracked files: # (use git add file... to include in what will be committed) # # serde/src/gen/thrift/gen-cpp/complex_constants.cpp # serde/src/gen/thrift/gen-cpp/complex_constants.h # serde/src/gen/thrift/gen-cpp/complex_types.cpp # serde/src/gen/thrift/gen-cpp/complex_types.h # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp # serde/src/gen/thrift/gen-cpp/megastruct_constants.h # serde/src
[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date
[ https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638755#comment-13638755 ] Namit Jain commented on HIVE-4300: -- ok, I will compare the patch with my patch - it should be same, and then merge it. ant thriftif generated code that is checkedin is not up-to-date Key: HIVE-4300 URL: https://issues.apache.org/jira/browse/HIVE-4300 Project: Hive Issue Type: Bug Components: Thrift API Affects Versions: 0.10.0 Reporter: Roshan Naik Assignee: Roshan Naik Attachments: HIVE-4300.2.patch, HIVE-4300.patch running 'ant thriftif -Dthrift.home=/usr/local' on a freshly checkedout trunk should be a no-op as per [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode] However this is not the case. Some of files seem to be have been relocated or the classes in them are now in a different file. Below is the git status showing the state after the command is run: # On branch trunk # Changes not staged for commit: # (use git add/rm file... to update what will be committed) # (use git checkout -- file... to discard changes in working directory) # # modified: build.properties # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore_constants.php # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php # modified: metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php # deleted: service/src/gen/thrift/gen-php/hive_service/hive_service_types.php # modified: service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote # modified: service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote # # Untracked files: # (use git add file... to include in what will be committed) # # serde/src/gen/thrift/gen-cpp/complex_constants.cpp # serde/src/gen/thrift/gen-cpp/complex_constants.h # serde/src/gen/thrift/gen-cpp/complex_types.cpp # serde/src/gen/thrift/gen-cpp/complex_types.h # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp # serde/src/gen/thrift/gen-cpp/megastruct_constants.h # serde/src/gen/thrift/gen
[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date
[ https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13638764#comment-13638764 ] Namit Jain commented on HIVE-4300: -- The results are same. +1 ant thriftif generated code that is checkedin is not up-to-date Key: HIVE-4300 URL: https://issues.apache.org/jira/browse/HIVE-4300 Project: Hive Issue Type: Bug Components: Thrift API Affects Versions: 0.10.0 Reporter: Roshan Naik Assignee: Roshan Naik Attachments: HIVE-4300.2.patch, HIVE-4300.patch running 'ant thriftif -Dthrift.home=/usr/local' on a freshly checkedout trunk should be a no-op as per [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode] However this is not the case. Some of files seem to be have been relocated or the classes in them are now in a different file. Below is the git status showing the state after the command is run: # On branch trunk # Changes not staged for commit: # (use git add/rm file... to update what will be committed) # (use git checkout -- file... to discard changes in working directory) # # modified: build.properties # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore_constants.php # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php # modified: metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php # deleted: service/src/gen/thrift/gen-php/hive_service/hive_service_types.php # modified: service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote # modified: service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote # # Untracked files: # (use git add file... to include in what will be committed) # # serde/src/gen/thrift/gen-cpp/complex_constants.cpp # serde/src/gen/thrift/gen-cpp/complex_constants.h # serde/src/gen/thrift/gen-cpp/complex_types.cpp # serde/src/gen/thrift/gen-cpp/complex_types.h # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp # serde/src/gen/thrift/gen-cpp/megastruct_constants.h # serde/src/gen/thrift/gen-cpp/megastruct_types.cpp # serde/src/gen/thrift/gen
[jira] [Commented] (HIVE-4106) SMB joins fail in multi-way joins
[ https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637524#comment-13637524 ] Namit Jain commented on HIVE-4106: -- [~ashutoshc], confirmed that the test still failed after HIVE-4371 SMB joins fail in multi-way joins - Key: HIVE-4106 URL: https://issues.apache.org/jira/browse/HIVE-4106 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Namit Jain Priority: Blocker Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, hive.4106.2.patch, HIVE-4106.patch I see array out of bounds exception in case of multi way smb joins. This is related to changes that went in as part of HIVE-3403. This issue has been discussed in HIVE-3891. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4342) NPE for query involving UNION ALL with nested JOIN and UNION ALL
[ https://issues.apache.org/jira/browse/HIVE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637537#comment-13637537 ] Namit Jain commented on HIVE-4342: -- +1 NPE for query involving UNION ALL with nested JOIN and UNION ALL Key: HIVE-4342 URL: https://issues.apache.org/jira/browse/HIVE-4342 Project: Hive Issue Type: Bug Components: Logging, Metastore, Query Processor Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Assignee: Navis Priority: Critical Attachments: HIVE-4342.D10407.1.patch, HiveCommands.txt, Query.txt, sourceData1.txt, sourceData2.txt UNION ALL query with JOIN in first part and another UNION ALL in second part gives NPE. bq. JOIN UNION ALL bq. UNION ALL Attachments: 1. HiveCommands.txt : command script to setup schema for query under consideration. 2. sourceData1.txt and sourceData2.txt : required for above command script. 3. Query.txt : Exact query which produces NPE. NOTE: you will need to update path to sourceData1.txt and sourceData2.txt in the HiveCommands.txt to suit your environment. Attached files contain the schema and exact query which fails on Hive 0.9. It is worthwhile to note that the same query executes successfully on Hive 0.7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4342) NPE for query involving UNION ALL with nested JOIN and UNION ALL
[ https://issues.apache.org/jira/browse/HIVE-4342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4342: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Navis NPE for query involving UNION ALL with nested JOIN and UNION ALL Key: HIVE-4342 URL: https://issues.apache.org/jira/browse/HIVE-4342 Project: Hive Issue Type: Bug Components: Logging, Metastore, Query Processor Affects Versions: 0.9.0 Environment: Red Hat Linux VM with Hive 0.9 and Hadoop 2.0 Reporter: Mihir Kulkarni Assignee: Navis Priority: Critical Attachments: HIVE-4342.D10407.1.patch, HiveCommands.txt, Query.txt, sourceData1.txt, sourceData2.txt UNION ALL query with JOIN in first part and another UNION ALL in second part gives NPE. bq. JOIN UNION ALL bq. UNION ALL Attachments: 1. HiveCommands.txt : command script to setup schema for query under consideration. 2. sourceData1.txt and sourceData2.txt : required for above command script. 3. Query.txt : Exact query which produces NPE. NOTE: you will need to update path to sourceData1.txt and sourceData2.txt in the HiveCommands.txt to suit your environment. Attached files contain the schema and exact query which fails on Hive 0.9. It is worthwhile to note that the same query executes successfully on Hive 0.7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HIVE-4389) thrift files are re-generated by compiling
Namit Jain created HIVE-4389: Summary: thrift files are re-generated by compiling Key: HIVE-4389 URL: https://issues.apache.org/jira/browse/HIVE-4389 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain I am not sure what is going on, but there seems to be a bunch of thrift changes if I perform ant thriftif. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4389) thrift files are re-generated by compiling
[ https://issues.apache.org/jira/browse/HIVE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637609#comment-13637609 ] Namit Jain commented on HIVE-4389: -- https://reviews.facebook.net/D10413 thrift files are re-generated by compiling -- Key: HIVE-4389 URL: https://issues.apache.org/jira/browse/HIVE-4389 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.4389.1.patch I am not sure what is going on, but there seems to be a bunch of thrift changes if I perform ant thriftif. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4389) thrift files are re-generated by compiling
[ https://issues.apache.org/jira/browse/HIVE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4389: - Attachment: hive.4389.1.patch thrift files are re-generated by compiling -- Key: HIVE-4389 URL: https://issues.apache.org/jira/browse/HIVE-4389 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.4389.1.patch I am not sure what is going on, but there seems to be a bunch of thrift changes if I perform ant thriftif. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date
[ https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637766#comment-13637766 ] Namit Jain commented on HIVE-4300: -- [~roshan_naik], I ran all the tests for HIVE-4389 (all of them ran fine other than leadlag.q, which is also failing on trunk, for which I have filed a jira There, I had just performed: ant thriftif -Dthrift.home=/usr/local on my mac -- no local changes. ant thriftif generated code that is checkedin is not up-to-date Key: HIVE-4300 URL: https://issues.apache.org/jira/browse/HIVE-4300 Project: Hive Issue Type: Bug Components: Thrift API Affects Versions: 0.10.0 Reporter: Roshan Naik Assignee: Roshan Naik Attachments: HIVE-4300.2.patch, HIVE-4300.patch running 'ant thriftif -Dthrift.home=/usr/local' on a freshly checkedout trunk should be a no-op as per [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode] However this is not the case. Some of files seem to be have been relocated or the classes in them are now in a different file. Below is the git status showing the state after the command is run: # On branch trunk # Changes not staged for commit: # (use git add/rm file... to update what will be committed) # (use git checkout -- file... to discard changes in working directory) # # modified: build.properties # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore_constants.php # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php # modified: metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php # deleted: service/src/gen/thrift/gen-php/hive_service/hive_service_types.php # modified: service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote # modified: service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote # # Untracked files: # (use git add file... to include in what will be committed) # # serde/src/gen/thrift/gen-cpp/complex_constants.cpp # serde/src/gen/thrift/gen-cpp/complex_constants.h # serde/src/gen/thrift/gen-cpp/complex_types.cpp # serde/src/gen/thrift/gen
[jira] [Created] (HIVE-4394) test leadlag.q fails
Namit Jain created HIVE-4394: Summary: test leadlag.q fails Key: HIVE-4394 URL: https://issues.apache.org/jira/browse/HIVE-4394 Project: Hive Issue Type: Bug Reporter: Namit Jain ant test -Dtestcase=TestCliDriver -Dqfile=leadlag.q fails. cc [~rhbutani] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date
[ https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637766#comment-13637766 ] Namit Jain edited comment on HIVE-4300 at 4/22/13 4:32 AM: --- [~roshan_naik], I ran all the tests for HIVE-4389 (all of them ran fine other than leadlag.q, which is also failing on trunk, for which I have filed a jira HIVE-4394. There, I had just performed: ant thriftif -Dthrift.home=/usr/local on my mac -- no local changes. was (Author: namit): [~roshan_naik], I ran all the tests for HIVE-4389 (all of them ran fine other than leadlag.q, which is also failing on trunk, for which I have filed a jira There, I had just performed: ant thriftif -Dthrift.home=/usr/local on my mac -- no local changes. ant thriftif generated code that is checkedin is not up-to-date Key: HIVE-4300 URL: https://issues.apache.org/jira/browse/HIVE-4300 Project: Hive Issue Type: Bug Components: Thrift API Affects Versions: 0.10.0 Reporter: Roshan Naik Assignee: Roshan Naik Attachments: HIVE-4300.2.patch, HIVE-4300.patch running 'ant thriftif -Dthrift.home=/usr/local' on a freshly checkedout trunk should be a no-op as per [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode] However this is not the case. Some of files seem to be have been relocated or the classes in them are now in a different file. Below is the git status showing the state after the command is run: # On branch trunk # Changes not staged for commit: # (use git add/rm file... to update what will be committed) # (use git checkout -- file... to discard changes in working directory) # # modified: build.properties # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore_constants.php # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php # modified: metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php # deleted: service/src/gen/thrift/gen-php/hive_service/hive_service_types.php # modified: service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote
[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date
[ https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637767#comment-13637767 ] Namit Jain commented on HIVE-4300: -- Did you do anything differently ? ant thriftif generated code that is checkedin is not up-to-date Key: HIVE-4300 URL: https://issues.apache.org/jira/browse/HIVE-4300 Project: Hive Issue Type: Bug Components: Thrift API Affects Versions: 0.10.0 Reporter: Roshan Naik Assignee: Roshan Naik Attachments: HIVE-4300.2.patch, HIVE-4300.patch running 'ant thriftif -Dthrift.home=/usr/local' on a freshly checkedout trunk should be a no-op as per [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode] However this is not the case. Some of files seem to be have been relocated or the classes in them are now in a different file. Below is the git status showing the state after the command is run: # On branch trunk # Changes not staged for commit: # (use git add/rm file... to update what will be committed) # (use git checkout -- file... to discard changes in working directory) # # modified: build.properties # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore_constants.php # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php # modified: metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MegaStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/MiniStruct.java # deleted:serde/src/gen/thrift/gen-php/serde/serde_constants.php # deleted:serde/src/gen/thrift/gen-php/serde/serde_types.php # deleted:service/src/gen/thrift/gen-php/hive_service/ThriftHive.php # deleted: service/src/gen/thrift/gen-php/hive_service/hive_service_types.php # modified: service/src/gen/thrift/gen-py/TCLIService/TCLIService-remote # modified: service/src/gen/thrift/gen-py/hive_service/ThriftHive-remote # # Untracked files: # (use git add file... to include in what will be committed) # # serde/src/gen/thrift/gen-cpp/complex_constants.cpp # serde/src/gen/thrift/gen-cpp/complex_constants.h # serde/src/gen/thrift/gen-cpp/complex_types.cpp # serde/src/gen/thrift/gen-cpp/complex_types.h # serde/src/gen/thrift/gen-cpp/megastruct_constants.cpp # serde/src/gen/thrift/gen-cpp/megastruct_constants.h # serde/src/gen/thrift/gen-cpp/megastruct_types.cpp # serde/src/gen
[jira] [Commented] (HIVE-4389) thrift files are re-generated by compiling
[ https://issues.apache.org/jira/browse/HIVE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637768#comment-13637768 ] Namit Jain commented on HIVE-4389: -- [~ashutoshc], the tests already ran fine for me. I updated HIVE-4300. thrift files are re-generated by compiling -- Key: HIVE-4389 URL: https://issues.apache.org/jira/browse/HIVE-4389 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.4389.1.patch I am not sure what is going on, but there seems to be a bunch of thrift changes if I perform ant thriftif. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4389) thrift files are re-generated by compiling
[ https://issues.apache.org/jira/browse/HIVE-4389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637769#comment-13637769 ] Namit Jain commented on HIVE-4389: -- For some reason, the patch on HIVE-4300 did not apply cleanly for me. Should we just commit this instead ? It should be the same change thrift files are re-generated by compiling -- Key: HIVE-4389 URL: https://issues.apache.org/jira/browse/HIVE-4389 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.4389.1.patch I am not sure what is going on, but there seems to be a bunch of thrift changes if I perform ant thriftif. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4300) ant thriftif generated code that is checkedin is not up-to-date
[ https://issues.apache.org/jira/browse/HIVE-4300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637770#comment-13637770 ] Namit Jain commented on HIVE-4300: -- I got fewer changes than yours. For eg: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java In each of the above files, only change is inside comments. word 'optional' changed to 'required' none of the above files changed. ant thriftif generated code that is checkedin is not up-to-date Key: HIVE-4300 URL: https://issues.apache.org/jira/browse/HIVE-4300 Project: Hive Issue Type: Bug Components: Thrift API Affects Versions: 0.10.0 Reporter: Roshan Naik Assignee: Roshan Naik Attachments: HIVE-4300.2.patch, HIVE-4300.patch running 'ant thriftif -Dthrift.home=/usr/local' on a freshly checkedout trunk should be a no-op as per [instructions|https://cwiki.apache.org/Hive/howtocontribute.html#HowToContribute-GeneratingThriftCode] However this is not the case. Some of files seem to be have been relocated or the classes in them are now in a different file. Below is the git status showing the state after the command is run: # On branch trunk # Changes not staged for commit: # (use git add/rm file... to update what will be committed) # (use git checkout -- file... to discard changes in working directory) # # modified: build.properties # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Database.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/EnvironmentContext.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Index.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Partition.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/PrincipalPrivilegeSet.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Schema.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SerDeInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/SkewedInfo.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/StorageDescriptor.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/Table.java # modified: metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java # deleted:metastore/src/gen/thrift/gen-php/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_constants.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore/hive_metastore_types.php # deleted: metastore/src/gen/thrift/gen-php/hive_metastore_constants.php # deleted:metastore/src/gen/thrift/gen-php/hive_metastore_types.php # modified: metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote # deleted:ql/src/gen/thrift/gen-php/queryplan/queryplan_types.php # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/InnerStruct.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde/test/ThriftTestObj.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/Complex.java # modified: serde/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/serde2/thrift/test/IntString.java # modified: serde/src/gen/thrift/gen
[jira] [Commented] (HIVE-4106) SMB joins fail in multi-way joins
[ https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13637201#comment-13637201 ] Namit Jain commented on HIVE-4106: -- I think it would still fail - I havent't tried it, but am nearly sure. Basically, it assumes the left aliases and right aliases in a certain way, and that assumption has been changed by the join merge now. It only matters in this peculiar case - when the table being joined 'd' does not show up in the join condition -- should not be a common case. SMB joins fail in multi-way joins - Key: HIVE-4106 URL: https://issues.apache.org/jira/browse/HIVE-4106 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Namit Jain Priority: Blocker Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, hive.4106.2.patch, HIVE-4106.patch I see array out of bounds exception in case of multi way smb joins. This is related to changes that went in as part of HIVE-3403. This issue has been discussed in HIVE-3891. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4310) optimize count(distinct) with hive.map.groupby.sorted
[ https://issues.apache.org/jira/browse/HIVE-4310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4310: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed. Thanks Gang optimize count(distinct) with hive.map.groupby.sorted - Key: HIVE-4310 URL: https://issues.apache.org/jira/browse/HIVE-4310 Project: Hive Issue Type: Improvement Reporter: Namit Jain Assignee: Namit Jain Attachments: hive.4310.1.patch, hive.4310.1.patch-nohcat, hive.4310.2.patch-nohcat, hive.4310.3.patch-nohcat, hive.4310.4.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4095: - Status: Open (was: Patch Available) Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3509) Exclusive locks are not acquired when using dynamic partitions
[ https://issues.apache.org/jira/browse/HIVE-3509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13636297#comment-13636297 ] Namit Jain commented on HIVE-3509: -- comments Exclusive locks are not acquired when using dynamic partitions -- Key: HIVE-3509 URL: https://issues.apache.org/jira/browse/HIVE-3509 Project: Hive Issue Type: Bug Components: Locking Affects Versions: 0.9.0 Reporter: Matt Martin Assignee: Matt Martin Attachments: HIVE-3509.1.patch.txt, HIVE-3509.D10065.1.patch, HIVE-3509.D10065.2.patch, HIVE-3509.D10065.3.patch, HIVE-3509.D10065.4.patch If locking is enabled, the acquireReadWriteLocks() method in org.apache.hadoop.hive.ql.Driver iterates through all of the input and output entities of the query plan and attempts to acquire the appropriate locks. In general, it should acquire SHARED locks for all of the input entities and exclusive locks for all of the output entities (see the Hive wiki page on [locking|https://cwiki.apache.org/confluence/display/Hive/Locking] for more detailed information). When the query involves dynamic partitions, the situation is a little more subtle. As the Hive wiki notes (see previous link): {quote} in some cases, the list of objects may not be known - for eg. in case of dynamic partitions, the list of partitions being modified is not known at compile time - so, the list is generated conservatively. Since the number of partitions may not be known, an exclusive lock is taken on the table, or the prefix that is known. {quote} After [HIVE-1781|https://issues.apache.org/jira/browse/HIVE-1781], the observed behavior is no longer consistent with the behavior described above. [HIVE-1781|https://issues.apache.org/jira/browse/HIVE-1781] appears to have altered the logic so that SHARED locks are acquired instead of EXCLUSIVE locks whenever the query involves dynamic partitions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4371) some issue with merging join trees
[ https://issues.apache.org/jira/browse/HIVE-4371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13634999#comment-13634999 ] Namit Jain commented on HIVE-4371: -- I am not sure about the last test case. Why is left Alias (es) and right Alias (es) not correct for that ? some issue with merging join trees -- Key: HIVE-4371 URL: https://issues.apache.org/jira/browse/HIVE-4371 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Namit Jain Assignee: Navis Attachments: HIVE-4371.D10323.1.patch [~navis], I would really appreciate if you can take a look. I am attaching a testcase, for which in the optimizer the join context left aliases and right aliases do not look correct. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3891: - Attachment: hive.3891.14.patch physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.14.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635005#comment-13635005 ] Namit Jain commented on HIVE-3891: -- [~ashutoshc], all the tests passed. Since this was accepted sometime back, can you take a look again ? physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.14.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635108#comment-13635108 ] Namit Jain commented on HIVE-4095: -- more comments Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Attachments: HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
hi
Hi, Since we are developing at a very fast pace, it would be really useful to think about maintainability and testing of the large codebase. Historically, we have not focussed on a few things, and they might soon bite us. I wanted to propose the following for all checkins: 1. Javadoc for all public/private functions, except for setters/getters. For any complex function, clear examples (input/output) would really help. 2. Convention for variable/function names – do we have any ? 3. If possible, the test name (.q file) where the function is being invoked, or the query which would potentially test that scenario, if it is a query processor change. 4. Specially, for query optimizations, it might be a good idea to have a simple working query at the top, and the expected changes. For e.g.. The operator tree for that query at each step, or a detailed explanation at the top. 5. Comments in each test (.q file)– that should include the jira number, what is it trying to test. Assumptions about each query. 6. Reduce the output for each test – whenever query is outputting more than 10 results, it should have a reason. Otherwise, each query result should be bounded by 10 rows. In general, focussing on a lot of comments in the code will go a long way for everyone to follow along. Thanks, -namit
Re: hi
Agreed. On 4/18/13 9:19 PM, Jarek Jarcec Cecho jar...@apache.org wrote: Hi Namit, I like your proposal very much and I would take it a bit further: 1. ... For any complex function, clear examples (input/output) would really help. I'm concerned that examples in the code (comments) might very quickly become obsolete as it can very easily happen that someone will change the code without changing the example. What about using for this purpose normal unit tests? Developers will still be able to see the expected input/output, but in addition we will have automatic way how to detect (possibly incompatible) changes. Please note that I'm not suggesting to abandon the *.q file tests, just to also include unit tests for complex methods. Jarcec On Thu, Apr 18, 2013 at 12:31:10PM +, Namit Jain wrote: Hi, Since we are developing at a very fast pace, it would be really useful to think about maintainability and testing of the large codebase. Historically, we have not focussed on a few things, and they might soon bite us. I wanted to propose the following for all checkins: 1. Javadoc for all public/private functions, except for setters/getters. For any complex function, clear examples (input/output) would really help. 2. Convention for variable/function names do we have any ? 3. If possible, the test name (.q file) where the function is being invoked, or the query which would potentially test that scenario, if it is a query processor change. 4. Specially, for query optimizations, it might be a good idea to have a simple working query at the top, and the expected changes. For e.g.. The operator tree for that query at each step, or a detailed explanation at the top. 5. Comments in each test (.q file) that should include the jira number, what is it trying to test. Assumptions about each query. 6. Reduce the output for each test whenever query is outputting more than 10 results, it should have a reason. Otherwise, each query result should be bounded by 10 rows. In general, focussing on a lot of comments in the code will go a long way for everyone to follow along. Thanks, -namit
Re: hi
Having said that, it might be difficult to write unit tests for operator trees. Might take more time initially - so, making it a constraint might slow us down. On 4/18/13 9:41 PM, Brock Noland br...@cloudera.com wrote: Hi, I like the proposal as well! On Thu, Apr 18, 2013 at 10:49 AM, Jarek Jarcec Cecho jar...@apache.orgwrote: Hi Namit, I like your proposal very much and I would take it a bit further: 1. ... For any complex function, clear examples (input/output) would really help. I'm concerned that examples in the code (comments) might very quickly become obsolete as it can very easily happen that someone will change the code without changing the example. What about using for this purpose normal unit tests? Developers will still be able to see the expected input/output, but in addition we will have automatic way how to detect (possibly incompatible) changes. Please note that I'm not suggesting to abandon the *.q file tests, just to also include unit tests for complex methods. I'd be interested in including more unit tests as well. I like the existing q file test framework but when working on code I find unit tests which can complete in less than a second or allows for faster iterations than waiting 30 or so seconds for a q-file test to complete. Brock
[jira] [Commented] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13635424#comment-13635424 ] Namit Jain commented on HIVE-4095: -- +1 Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Attachments: HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4095) Add exchange partition in Hive
[ https://issues.apache.org/jira/browse/HIVE-4095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4095: - Attachment: hive.4095.1.patch Add exchange partition in Hive -- Key: HIVE-4095 URL: https://issues.apache.org/jira/browse/HIVE-4095 Project: Hive Issue Type: New Feature Components: Query Processor Reporter: Namit Jain Assignee: Dheeraj Kumar Singh Attachments: hive.4095.1.patch, HIVE-4095.D10155.1.patch, HIVE-4095.D10155.2.patch, HIVE-4095.D10347.1.patch, HIVE-4095.part11.patch.txt, HIVE-4095.part12.patch.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3891) physical optimizer changes for auto sort-merge join
[ https://issues.apache.org/jira/browse/HIVE-3891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-3891: - Attachment: hive.3891.13.patch physical optimizer changes for auto sort-merge join --- Key: HIVE-3891 URL: https://issues.apache.org/jira/browse/HIVE-3891 Project: Hive Issue Type: Bug Reporter: Namit Jain Assignee: Namit Jain Attachments: auto_sortmerge_join_1.q, auto_sortmerge_join_1.q.out, hive.3891.10.patch, hive.3891.11.patch, hive.3891.12.patch, hive.3891.13.patch, hive.3891.1.patch, hive.3891.2.patch, hive.3891.3.patch, hive.3891.4.patch, hive.3891.5.patch, hive.3891.6.patch, hive.3891.7.patch, HIVE-3891_8.patch, hive.3891.9.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HIVE-4106) SMB joins fail in multi-way joins
[ https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633835#comment-13633835 ] Namit Jain edited comment on HIVE-4106 at 4/17/13 6:54 AM: --- [~ashutoshc], can you take a look ? was (Author: namit): [~ashutoshc], can you taje a look ? SMB joins fail in multi-way joins - Key: HIVE-4106 URL: https://issues.apache.org/jira/browse/HIVE-4106 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Namit Jain Priority: Blocker Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, hive.4106.2.patch, HIVE-4106.patch I see array out of bounds exception in case of multi way smb joins. This is related to changes that went in as part of HIVE-3403. This issue has been discussed in HIVE-3891. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-4106) SMB joins fail in multi-way joins
[ https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Namit Jain updated HIVE-4106: - Assignee: Namit Jain (was: Vikram Dixit K) Status: Patch Available (was: Open) SMB joins fail in multi-way joins - Key: HIVE-4106 URL: https://issues.apache.org/jira/browse/HIVE-4106 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Namit Jain Priority: Blocker Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, hive.4106.2.patch, HIVE-4106.patch I see array out of bounds exception in case of multi way smb joins. This is related to changes that went in as part of HIVE-3403. This issue has been discussed in HIVE-3891. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-4106) SMB joins fail in multi-way joins
[ https://issues.apache.org/jira/browse/HIVE-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13633835#comment-13633835 ] Namit Jain commented on HIVE-4106: -- [~ashutoshc], can you taje a look ? SMB joins fail in multi-way joins - Key: HIVE-4106 URL: https://issues.apache.org/jira/browse/HIVE-4106 Project: Hive Issue Type: Bug Components: Query Processor Affects Versions: 0.11.0 Reporter: Vikram Dixit K Assignee: Namit Jain Priority: Blocker Attachments: auto_sortmerge_join_12.q, hive.4106.1.patch, hive.4106.2.patch, HIVE-4106.patch I see array out of bounds exception in case of multi way smb joins. This is related to changes that went in as part of HIVE-3403. This issue has been discussed in HIVE-3891. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira