[jira] [Commented] (HIVE-2246) Dedupe tables' column schemas from partitions in the metastore db
[ https://issues.apache.org/jira/browse/HIVE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080216#comment-13080216 ] jirapos...@reviews.apache.org commented on HIVE-2246: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1183/ --- (Updated 2011-08-05 20:48:05.144312) Review request for hive, Ning Zhang and Paul Yang. Changes --- -On alter table, only change the column descriptor if the columns have changed. -Fix desc table partition... to use the partition's column schema, not the table's Summary --- This patch tries to make minimal changes to the API while keeping migration short and somewhat easy to revert. The new schema can be described as follows: - CDS is a table corresponding to Column Descriptor objects. Currently, it only stores a CD_ID. - COLUMNS_V2 is a table corresponding to MFieldSchema objects, or columns. A Column Descriptor holds a list of columns. COLUMNS_V2 has a foreign key to the CD_ID to which it belongs. - SDS was modified to reference a Column Descriptor. So SDS now has a foreign key to a CD_ID which describes its columns. During migration, we create Column Descriptors for tables in a straightforward manner: their columns are now just wrapped inside a column descriptor. The SDS of partitions use their parent table's column descriptor, since currently a partition and its table share the same list of columns. When altering or adding a partition, give it it's parent table's column descriptor IF the columns they describe are the same. Otherwise, create a new column descriptor for its columns. When adding or altering a table, create a new column descriptor every time. Whenever you drop a storage descriptor (e.g, when dropping tables or partitions), check to see if the related column descriptor has any other references in the table. That is, check to see if any other storage descriptors point to that column descriptor. If none do, then delete that column descriptor. This check is in place so we don't have unreferenced column descriptors and columns hanging around after schema evolution for tables. This addresses bug HIVE-2246. https://issues.apache.org/jira/browse/HIVE-2246 Diffs (updated) - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1153927 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1153927 trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java 1153927 trunk/metastore/src/model/package.jdo 1153927 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1153927 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java 1153927 Diff: https://reviews.apache.org/r/1183/diff Testing --- Passes facebook's regression testing and all existing test cases. In one instance, before migration, the overhead involved with storage descriptors and columns was ~11 GB. After migration, the overhead was ~1.5 GB. Thanks, Sohan Dedupe tables' column schemas from partitions in the metastore db - Key: HIVE-2246 URL: https://issues.apache.org/jira/browse/HIVE-2246 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2246.2.patch, HIVE-2246.3.patch Note: this patch proposes a schema change, and is therefore incompatible with the current metastore. We can re-organize the JDO models to reduce space usage to keep the metastore scalable for the future. Currently, partitions are the fastest growing objects in the metastore, and the metastore keeps a separate copy of the columns list for each partition. We can normalize the metastore db by decoupling Columns from Storage Descriptors and not storing duplicate lists of the columns for each partition. An idea is to create an additional level of indirection with a Column Descriptor that has a list of columns. A table has a reference to its latest Column Descriptor (note: a table may have more than one Column Descriptor in the case of schema evolution). Partitions and Indexes can reference the same Column Descriptors as their parent table. Currently, the COLUMNS table in the metastore has roughly (number of partitions + number of tables) * (average number of columns pertable) rows. We can reduce this to (number of tables) * (average number of columns per table) rows, while incurring a small cost proportional to the number of tables to store the Column Descriptors. Please see the latest review board for additional
[jira] [Commented] (HIVE-2334) DESCRIBE TABLE causes NPE when hive.cli.print.header=true
[ https://issues.apache.org/jira/browse/HIVE-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080217#comment-13080217 ] jirapos...@reviews.apache.org commented on HIVE-2334: - bq. On 2011-08-05 20:38:11, Carl Steinbach wrote: bq. cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java, line 235 bq. https://reviews.apache.org/r/1300/diff/1/?file=30859#file30859line235 bq. bq. Might want to consider using StringTokenizer or StreamTokenizer here. This is how it was in the original code. All of this can actually be done quite a bit better. I'm happy to switch to tokenizer; the patch is a bit schizophrenic about refactoring/improving. I didn't change this since it's not directly related to what I was trying to test. bq. On 2011-08-05 20:38:11, Carl Steinbach wrote: bq. cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java, line 37 bq. https://reviews.apache.org/r/1300/diff/1/?file=30860#file30860line37 bq. bq. There's already some very limited test coverage for the hive.cli.print.header feature in print_header.q. Why not extend this testcase instead of adding a new unit test? Because this is an actual unit test against a specific regression, which has value separate from the print_header.q integration test. I can add additional content to print_header.q, since this test is easier to identify what's gone wrong and runs in about 0.2 seconds, this one seems more useful. bq. On 2011-08-05 20:38:11, Carl Steinbach wrote: bq. ivy/libraries.properties, line 47 bq. https://reviews.apache.org/r/1300/diff/1/?file=30861#file30861line47 bq. bq. We need to manage this dependency with Ivy. The Hive build currently runs against hadoop-0.20.1, which does not include mockito-all-1.8.2.jar I'm sorry; I don't understand. This is being brought in by Ivy? As part of HIVE-2171, I had mentioned we need to make sure testing related jars don't get included during binary/package, but that should be done in a different JIRA. bq. On 2011-08-05 20:38:11, Carl Steinbach wrote: bq. cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java, line 1 bq. https://reviews.apache.org/r/1300/diff/1/?file=30860#file30860line1 bq. bq. cli/build.xml overrides the ant test target with a no-op, so this test is actually not getting run. I'll update cli/build.xml to not be a no-op, unless there's some reason to? - Jakob --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1300/#review1308 --- On 2011-08-05 01:22:01, Jakob Homan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1300/ bq. --- bq. bq. (Updated 2011-08-05 01:22:01) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. Commands that don't return a schema cause NPE when print headers is on. bq. bq. bq. This addresses bug HIVE-2334. bq. https://issues.apache.org/jira/browse/HIVE-2334 bq. bq. bq. Diffs bq. - bq. bq.cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 9fa7bc6 bq.cli/src/test/org/apache/hadoop/hive/cli/TestCliDriverMethods.java PRE-CREATION bq.ivy/libraries.properties af856bd bq. bq. Diff: https://reviews.apache.org/r/1300/diff bq. bq. bq. Testing bq. --- bq. bq. New unit tests (both positive and negative) and verification on manual cluster. bq. bq. bq. Thanks, bq. bq. Jakob bq. bq. DESCRIBE TABLE causes NPE when hive.cli.print.header=true - Key: HIVE-2334 URL: https://issues.apache.org/jira/browse/HIVE-2334 Project: Hive Issue Type: Bug Components: CLI Affects Versions: 0.7.1 Reporter: Carl Steinbach Assignee: Jakob Homan Attachments: h2334.patch -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2246) Dedupe tables' column schemas from partitions in the metastore db
[ https://issues.apache.org/jira/browse/HIVE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080218#comment-13080218 ] jirapos...@reviews.apache.org commented on HIVE-2246: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1183/ --- (Updated 2011-08-05 20:49:19.127572) Review request for hive, Ning Zhang and Paul Yang. Changes --- -Forgot to add a few files. NOTE: this is only a temporary diff; I need to add derby and postgres migration scripts. Summary --- This patch tries to make minimal changes to the API while keeping migration short and somewhat easy to revert. The new schema can be described as follows: - CDS is a table corresponding to Column Descriptor objects. Currently, it only stores a CD_ID. - COLUMNS_V2 is a table corresponding to MFieldSchema objects, or columns. A Column Descriptor holds a list of columns. COLUMNS_V2 has a foreign key to the CD_ID to which it belongs. - SDS was modified to reference a Column Descriptor. So SDS now has a foreign key to a CD_ID which describes its columns. During migration, we create Column Descriptors for tables in a straightforward manner: their columns are now just wrapped inside a column descriptor. The SDS of partitions use their parent table's column descriptor, since currently a partition and its table share the same list of columns. When altering or adding a partition, give it it's parent table's column descriptor IF the columns they describe are the same. Otherwise, create a new column descriptor for its columns. When adding or altering a table, create a new column descriptor every time. Whenever you drop a storage descriptor (e.g, when dropping tables or partitions), check to see if the related column descriptor has any other references in the table. That is, check to see if any other storage descriptors point to that column descriptor. If none do, then delete that column descriptor. This check is in place so we don't have unreferenced column descriptors and columns hanging around after schema evolution for tables. This addresses bug HIVE-2246. https://issues.apache.org/jira/browse/HIVE-2246 Diffs (updated) - trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1153927 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1153927 trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MColumnDescriptor.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java 1153927 trunk/metastore/src/model/package.jdo 1153927 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1153927 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java 1153927 Diff: https://reviews.apache.org/r/1183/diff Testing --- Passes facebook's regression testing and all existing test cases. In one instance, before migration, the overhead involved with storage descriptors and columns was ~11 GB. After migration, the overhead was ~1.5 GB. Thanks, Sohan Dedupe tables' column schemas from partitions in the metastore db - Key: HIVE-2246 URL: https://issues.apache.org/jira/browse/HIVE-2246 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2246.2.patch, HIVE-2246.3.patch Note: this patch proposes a schema change, and is therefore incompatible with the current metastore. We can re-organize the JDO models to reduce space usage to keep the metastore scalable for the future. Currently, partitions are the fastest growing objects in the metastore, and the metastore keeps a separate copy of the columns list for each partition. We can normalize the metastore db by decoupling Columns from Storage Descriptors and not storing duplicate lists of the columns for each partition. An idea is to create an additional level of indirection with a Column Descriptor that has a list of columns. A table has a reference to its latest Column Descriptor (note: a table may have more than one Column Descriptor in the case of schema evolution). Partitions and Indexes can reference the same Column Descriptors as their parent table. Currently, the COLUMNS table in the metastore has roughly (number of partitions + number of tables) * (average number of columns pertable) rows. We can reduce this to (number of tables) * (average number of columns per table) rows, while incurring a
[jira] [Commented] (HIVE-1694) Accelerate GROUP BY execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080235#comment-13080235 ] jirapos...@reviews.apache.org commented on HIVE-1694: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1194/#review1303 --- ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java https://reviews.apache.org/r/1194/#comment2955 Can't you just look up AGGREGATES in the map? ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java https://reviews.apache.org/r/1194/#comment2953 Add a helper method to avoid duplicating the code in the else block below. ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java https://reviews.apache.org/r/1194/#comment2954 Can't you just look up AGGREGATES in the map? ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java https://reviews.apache.org/r/1194/#comment2956 See recent changes in corresponding CompactIndexHandler code for HIVEOPTINDEXFILTER; need the same here (or better, factor out common code here and elsewhere). On a related note, you may be able to use the same technique instead of isQueryInsertToTable; this would be preferable since it's nice to be able to use the index rewrite in cases where it's a normal INSERT table with index being used for GROUP BY on SELECT from some other table. ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java https://reviews.apache.org/r/1194/#comment2957 @params here don't match actual params ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java https://reviews.apache.org/r/1194/#comment2958 Shouldn't this be the same as COUNT(*)? ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx.q https://reviews.apache.org/r/1194/#comment2980 Besides EXPLAIN, you should include a few queries against a non-empty table verifying that you get the correct results both with and without the optimization applied. Remember to include an ORDER BY for test determinism. ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx.q https://reviews.apache.org/r/1194/#comment2978 Isn't this set redundant? - John On 2011-08-03 10:31:42, Prajakta Kalmegh wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1194/ bq. --- bq. bq. (Updated 2011-08-03 10:31:42) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. This patch has defined a new AggregateIndexHandler which is used to optimize the query plan for groupby queries. bq. bq. bq. This addresses bug HIVE-1694. bq. https://issues.apache.org/jira/browse/HIVE-1694 bq. bq. bq. Diffs bq. - bq. bq.common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b46976f bq.ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndex.java 591c9ff bq.ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a57f9cf bq.ql/src/java/org/apache/hadoop/hive/ql/optimizer/IndexUtils.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 590d69a bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 8295687 bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java 699519b bq.ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx.q PRE-CREATION bq.ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1194/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Prajakta bq. bq. Accelerate GROUP BY execution using indexes --- Key: HIVE-1694 URL:
[jira] [Commented] (HIVE-2272) add TIMESTAMP data type
[ https://issues.apache.org/jira/browse/HIVE-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080273#comment-13080273 ] jirapos...@reviews.apache.org commented on HIVE-2272: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1135/#review1277 --- trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java https://reviews.apache.org/r/1135/#comment2918 I think we should put default timezone to be UTC and we internally in facebook have PST as default. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java https://reviews.apache.org/r/1135/#comment2922 Only loading it in client side is sufficient? - Siying On 2011-07-28 21:59:38, Franklin Hu wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1135/ bq. --- bq. bq. (Updated 2011-07-28 21:59:38) bq. bq. bq. Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong. bq. bq. bq. Summary bq. --- bq. bq. Adds TIMESTAMP type to serde2 with both string (LazySimple) and binary (LazyBinary) serialization. bq. Supports SQL style jdbc timestamps of the format with nanosecond precision bq. -MM-DD HH:MM:SS[.fff...] bq. bq. bq. This addresses bug HIVE-2272. bq. https://issues.apache.org/jira/browse/HIVE-2272 bq. bq. bq. Diffs bq. - bq. bq.trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1152003 bq.trunk/conf/hive-default.xml 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDate.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateAdd.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateDiff.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateSub.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDayOfMonth.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHour.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMinute.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMonth.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSecond.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFWeekOfYear.java 1152003 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFYear.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovarianceSample.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStd.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStdSample.java 1152003 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 1152003 bq.
[jira] [Commented] (HIVE-2246) Dedupe tables' column schemas from partitions in the metastore db
[ https://issues.apache.org/jira/browse/HIVE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080327#comment-13080327 ] jirapos...@reviews.apache.org commented on HIVE-2246: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1183/ --- (Updated 2011-08-06 01:40:49.118616) Review request for hive, Ning Zhang and Paul Yang. Changes --- -made listStorageDescriptors.. into one transaction -renamed dropStorageDescriptorCleanly to make it's functionality clearer -indents typo Summary --- This patch tries to make minimal changes to the API while keeping migration short and somewhat easy to revert. The new schema can be described as follows: - CDS is a table corresponding to Column Descriptor objects. Currently, it only stores a CD_ID. - COLUMNS_V2 is a table corresponding to MFieldSchema objects, or columns. A Column Descriptor holds a list of columns. COLUMNS_V2 has a foreign key to the CD_ID to which it belongs. - SDS was modified to reference a Column Descriptor. So SDS now has a foreign key to a CD_ID which describes its columns. During migration, we create Column Descriptors for tables in a straightforward manner: their columns are now just wrapped inside a column descriptor. The SDS of partitions use their parent table's column descriptor, since currently a partition and its table share the same list of columns. When altering or adding a partition, give it it's parent table's column descriptor IF the columns they describe are the same. Otherwise, create a new column descriptor for its columns. When adding or altering a table, create a new column descriptor every time. Whenever you drop a storage descriptor (e.g, when dropping tables or partitions), check to see if the related column descriptor has any other references in the table. That is, check to see if any other storage descriptors point to that column descriptor. If none do, then delete that column descriptor. This check is in place so we don't have unreferenced column descriptors and columns hanging around after schema evolution for tables. This addresses bug HIVE-2246. https://issues.apache.org/jira/browse/HIVE-2246 Diffs (updated) - trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1153927 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1153927 trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MColumnDescriptor.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java 1153927 trunk/metastore/src/model/package.jdo 1153927 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1153927 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java 1153927 Diff: https://reviews.apache.org/r/1183/diff Testing --- Passes facebook's regression testing and all existing test cases. In one instance, before migration, the overhead involved with storage descriptors and columns was ~11 GB. After migration, the overhead was ~1.5 GB. Thanks, Sohan Dedupe tables' column schemas from partitions in the metastore db - Key: HIVE-2246 URL: https://issues.apache.org/jira/browse/HIVE-2246 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2246.2.patch, HIVE-2246.3.patch, HIVE-2246.4.patch Note: this patch proposes a schema change, and is therefore incompatible with the current metastore. We can re-organize the JDO models to reduce space usage to keep the metastore scalable for the future. Currently, partitions are the fastest growing objects in the metastore, and the metastore keeps a separate copy of the columns list for each partition. We can normalize the metastore db by decoupling Columns from Storage Descriptors and not storing duplicate lists of the columns for each partition. An idea is to create an additional level of indirection with a Column Descriptor that has a list of columns. A table has a reference to its latest Column Descriptor (note: a table may have more than one Column Descriptor in the case of schema evolution). Partitions and Indexes can reference the same Column Descriptors as their parent table. Currently, the COLUMNS table in the metastore has roughly (number of partitions + number of tables) * (average number of columns pertable) rows. We can reduce this to (number of tables) * (average number of columns
[jira] [Commented] (HIVE-2246) Dedupe tables' column schemas from partitions in the metastore db
[ https://issues.apache.org/jira/browse/HIVE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13080328#comment-13080328 ] jirapos...@reviews.apache.org commented on HIVE-2246: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1183/#review1313 --- trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java https://reviews.apache.org/r/1183/#comment2984 should read 1-N actually - Sohan On 2011-08-06 01:40:49, Sohan Jain wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1183/ bq. --- bq. bq. (Updated 2011-08-06 01:40:49) bq. bq. bq. Review request for hive, Ning Zhang and Paul Yang. bq. bq. bq. Summary bq. --- bq. bq. This patch tries to make minimal changes to the API while keeping migration short and somewhat easy to revert. bq. bq. The new schema can be described as follows: bq. - CDS is a table corresponding to Column Descriptor objects. Currently, it only stores a CD_ID. bq. - COLUMNS_V2 is a table corresponding to MFieldSchema objects, or columns. A Column Descriptor holds a list of columns. COLUMNS_V2 has a foreign key to the CD_ID to which it belongs. bq. - SDS was modified to reference a Column Descriptor. So SDS now has a foreign key to a CD_ID which describes its columns. bq. bq. During migration, we create Column Descriptors for tables in a straightforward manner: their columns are now just wrapped inside a column descriptor. The SDS of partitions use their parent table's column descriptor, since currently a partition and its table share the same list of columns. bq. bq. When altering or adding a partition, give it it's parent table's column descriptor IF the columns they describe are the same. Otherwise, create a new column descriptor for its columns. bq. bq. When adding or altering a table, create a new column descriptor every time. bq. bq. Whenever you drop a storage descriptor (e.g, when dropping tables or partitions), check to see if the related column descriptor has any other references in the table. That is, check to see if any other storage descriptors point to that column descriptor. If none do, then delete that column descriptor. This check is in place so we don't have unreferenced column descriptors and columns hanging around after schema evolution for tables. bq. bq. bq. This addresses bug HIVE-2246. bq. https://issues.apache.org/jira/browse/HIVE-2246 bq. bq. bq. Diffs bq. - bq. bq.trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql PRE-CREATION bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1153927 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1153927 bq. trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MColumnDescriptor.java PRE-CREATION bq. trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java 1153927 bq.trunk/metastore/src/model/package.jdo 1153927 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1153927 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/MetaDataFormatUtils.java 1153927 bq. bq. Diff: https://reviews.apache.org/r/1183/diff bq. bq. bq. Testing bq. --- bq. bq. Passes facebook's regression testing and all existing test cases. In one instance, before migration, the overhead involved with storage descriptors and columns was ~11 GB. After migration, the overhead was ~1.5 GB. bq. bq. bq. Thanks, bq. bq. Sohan bq. bq. Dedupe tables' column schemas from partitions in the metastore db - Key: HIVE-2246 URL: https://issues.apache.org/jira/browse/HIVE-2246 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2246.2.patch, HIVE-2246.3.patch, HIVE-2246.4.patch Note: this patch proposes a schema change, and is therefore incompatible with the current metastore. We can re-organize the JDO models to reduce space usage to keep the metastore scalable for the future. Currently, partitions are the fastest growing objects in the metastore, and the metastore keeps a separate copy of the columns list for each partition. We can normalize the metastore db by decoupling Columns from Storage Descriptors and not storing duplicate lists of the columns for each partition. An
[jira] [Commented] (HIVE-2319) Calling alter_table after changing partition comment throws an exception
[ https://issues.apache.org/jira/browse/HIVE-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079470#comment-13079470 ] jirapos...@reviews.apache.org commented on HIVE-2319: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1213/ --- (Updated 2011-08-04 17:05:08.413032) Review request for hive and Paul Yang. Changes --- Removed a change in TestMetaStoreEventListener.java that somehow snuck in this diff. Summary --- Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. This addresses bug HIVE-2319. https://issues.apache.org/jira/browse/HIVE-2319 Diffs (updated) - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1153927 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1153927 Diff: https://reviews.apache.org/r/1213/diff Testing --- Added some test cases to HiveMetaStore that pass. Thanks, Sohan Calling alter_table after changing partition comment throws an exception Key: HIVE-2319 URL: https://issues.apache.org/jira/browse/HIVE-2319 Project: Hive Issue Type: Bug Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2319.2.patch, HIVE-2319.3.patch Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2346) Add hooks to run when execution fails.
[ https://issues.apache.org/jira/browse/HIVE-2346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079534#comment-13079534 ] jirapos...@reviews.apache.org commented on HIVE-2346: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1295/ --- Review request for hive and Ning Zhang. Summary --- I added a new type of hook, which will be run when a job fails. This addresses bug HIVE-2346. https://issues.apache.org/jira/browse/HIVE-2346 Diffs - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/hooks/HookContext.java 1153966 Diff: https://reviews.apache.org/r/1295/diff Testing --- I ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. In addition, I created a sample hook, which simply logged that it was run. I verified it was run on a failure, but not when a job succeeded. Thanks, Kevin Add hooks to run when execution fails. -- Key: HIVE-2346 URL: https://issues.apache.org/jira/browse/HIVE-2346 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong Attachments: HIVE-2346.1.patch.txt Currently, when a query fails, the Post Execution Hooks are not run. Adding hooks to be run when a query fails could allow for better logging etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2347) Make Hadoop Job ID available after task finishes executing
[ https://issues.apache.org/jira/browse/HIVE-2347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079549#comment-13079549 ] jirapos...@reviews.apache.org commented on HIVE-2347: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1296/ --- Review request for hive and Ning Zhang. Summary --- I added a field for the Hadoop Job ID to the Task class. This will make it accessible to the Driver and hence to the hooks for logging/debugging purposes. By including it in the Task, we only need to check that the type of the task is MAPRED, before getting the job ID. I considered adding it to several places: as separate fields in ExecDriver and BlockMergeTask: this would require duplicating code, require conditions to determine the type of a task and casting to either ExecDriver or BlockMergeTask in order to get the JobID from them in the MapRedWork: this would require modifying a field in MapRedWork in the execute function, and I could not find a precedent for this This addresses bug HIVE-2347. https://issues.apache.org/jira/browse/HIVE-2347 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1153966 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1153966 Diff: https://reviews.apache.org/r/1296/diff Testing --- Ran the TestCliDriver and TestNegativeCliDriver test suites and verified they passed. Also, created a sample post exec hook which simply logged the JobID for every map reduce task, and verified it. Thanks, Kevin Make Hadoop Job ID available after task finishes executing -- Key: HIVE-2347 URL: https://issues.apache.org/jira/browse/HIVE-2347 Project: Hive Issue Type: Improvement Reporter: Kevin Wilfong Assignee: Kevin Wilfong After Map Reduce tasks finish the execute method (ExecDriver and BlockMergeTask) the Hadoop Job ID is inaccessible to the Driver, and hence the hooks it runs. Expose this information could help to improve logging, debugging, etc. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2138) Exception when no splits returned from index
[
https://issues.apache.org/jira/browse/HIVE-2138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079630#comment-13079630
]
jirapos...@reviews.apache.org commented on HIVE-2138:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1255/
---
(Updated 2011-08-04 21:56:31.633797)
Review request for hive, John Sichi and Yongqiang He.
Changes
---
removed stale testcase
Summary
---
return an empty array of input splits when we get back no results from the
hiveIndexResult
This addresses bug HIVE-2138.
https://issues.apache.org/jira/browse/HIVE-2138
Diffs (updated)
-
ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java
1a1ecd7
ql/src/test/queries/clientpositive/index_auto_empty.q PRE-CREATION
ql/src/test/results/clientpositive/index_auto_empty.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/1255/diff
Testing
---
added index_auto_test_if_used.q to the test suite
Thanks,
Syed
Exception when no splits returned from index
Key: HIVE-2138
URL: https://issues.apache.org/jira/browse/HIVE-2138
Project: Hive
Issue Type: Bug
Components: Indexing
Affects Versions: 0.8.0
Reporter: Russell Melick
Assignee: Syed S. Albiz
Attachments: HIVE-2138.1.patch, HIVE-2138.2.patch, HIVE-2138.3.patch,
HIVE-2138.4.patch, index_auto_test_if_used.q
Running a query that uses indexing but doesn't return any results give an
exception.
{code} java.lang.IllegalArgumentException: Can not create a Path from an
empty string
at org.apache.hadoop.fs.Path.checkPathArg(Path.java:82)
at org.apache.hadoop.fs.Path.init(Path.java:90)
at org.apache.hadoop.util.StringUtils.stringToPath(StringUtils.java:224)
at
org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:282)
at
org.apache.hadoop.hive.ql.index.HiveIndexedInputFormat.getSplits(HiveIndexedInputFormat.java:123)
{code}
This could potentially be fixed by creating a new empty file to use for the
splits.
Once this is fixed, the index_auto_test_if_used.q can be used.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2110) Hive Client is indefenitely waiting for reading from Socket
[
https://issues.apache.org/jira/browse/HIVE-2110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079706#comment-13079706
]
jirapos...@reviews.apache.org commented on HIVE-2110:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1299/
---
Review request for hive and Carl Steinbach.
Summary
---
Pass the Login timeout to thrift/socket layer when connecting to the hive
server.
This addresses bug HIVE-2110.
https://issues.apache.org/jira/browse/HIVE-2110
Diffs
-
jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveConnection.java 7c5df83
jdbc/src/java/org/apache/hadoop/hive/jdbc/HiveDriver.java c61425f
Diff: https://reviews.apache.org/r/1299/diff
Testing
---
simple repro to verify the timeout. JDBC tests.
Thanks,
Prasad
Hive Client is indefenitely waiting for reading from Socket
---
Key: HIVE-2110
URL: https://issues.apache.org/jira/browse/HIVE-2110
Project: Hive
Issue Type: Bug
Components: JDBC
Affects Versions: 0.5.0
Environment: Hadoop 0.20.1, Hive0.5.0 and SUSE Linux Enterprise
Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5).
Reporter: Chinna Rao Lalam
Assignee: Prasad Mujumdar
Hive Client is indefenitely waiting for reading from Socket. Thread dump i
added below.
Cause is:
In the HiveClient, when client socket is created, the read timeout is
mentioned is 0. So the socket will indefinetly wait when the machine where
Hive Server is running is shutdown or network is unplugged. The same may
not happen if the HiveServer alone is killed or gracefully shutdown. At this
time, client will get connection reset exception.
Code in HiveConnection
---
{noformat}
transport = new TSocket(host, port);
TProtocol protocol = new TBinaryProtocol(transport);
client = new HiveClient(protocol);
{noformat}
In the Client side, they send the query and wait for the response
send_execute(query,id); recv_execute(); // place where client waiting is
initiated
Thread dump:
{noformat}
main prio=10 tid=0x40111000 nid=0x3641 runnable [0x7f0d73f29000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:258)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
locked 0x7f0d5d3f0828 (a java.io.BufferedInputStream)
at
org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:125)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
at
org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:314)
at
org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:262)
at
org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:192)
at
org.apache.hadoop.hive.service.ThriftHive$Client.recv_execute(ThriftHive.java:130)
at
org.apache.hadoop.hive.service.ThriftHive$Client.execute(ThriftHive.java:109)
locked 0x7f0d5d3f0878 (a org.apache.thrift.transport.TSocket)
at
org.apache.hadoop.hive.jdbc.HiveStatement.executeQuery(HiveStatement.java:218)
at
org.apache.hadoop.hive.jdbc.HiveStatement.execute(HiveStatement.java:154)
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2343) stats not updated for non load table desc operations
[ https://issues.apache.org/jira/browse/HIVE-2343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079004#comment-13079004 ] jirapos...@reviews.apache.org commented on HIVE-2343: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1279/ --- Review request for Ning Zhang. Summary --- table stats were not being updated for non load table desc plans due to NPE. This addresses bug hive-2343. https://issues.apache.org/jira/browse/hive-2343 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 1153601 trunk/ql/src/test/queries/clientpositive/stats16.q PRE-CREATION trunk/ql/src/test/results/clientpositive/stats16.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1279/diff Testing --- all clientpositive/stats* unit tests pass Thanks, Franklin stats not updated for non load table desc operations -- Key: HIVE-2343 URL: https://issues.apache.org/jira/browse/HIVE-2343 Project: Hive Issue Type: Bug Components: Query Processor Reporter: Franklin Hu Assignee: Franklin Hu Fix For: 0.8.0 Attachments: hive-2343.1.patch Bug introduced in HIVE-306 so that stats are updated only for LoadTableDesc operations. For other operations (analyze table), null ptr is thrown and stats are not updated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2337) Predicate pushdown erroneously conservative with outer joins
[
https://issues.apache.org/jira/browse/HIVE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079024#comment-13079024
]
jirapos...@reviews.apache.org commented on HIVE-2337:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1275/
---
(Updated 2011-08-03 21:02:11.625203)
Review request for hive.
Changes
---
Added test cases, fixed comment
Summary
---
Initial patch
This addresses bug HIVE-2337.
https://issues.apache.org/jira/browse/HIVE-2337
Diffs (updated)
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1153598
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join5.q.out
PRE-CREATION
Diff: https://reviews.apache.org/r/1275/diff
Testing
---
Thanks,
Charles
Predicate pushdown erroneously conservative with outer joins
Key: HIVE-2337
URL: https://issues.apache.org/jira/browse/HIVE-2337
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Charles Chen
Assignee: Charles Chen
Attachments: HIVE-2337v1.patch, HIVE-2337v2.patch
The predicate pushdown filter is not applying left associativity of joins
correctly in determining possible aliases for pushing predicates.
In hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases, the criteria for
pushing aliases is specified as:
{noformat}
/**
* Figures out the aliases for whom it is safe to push predicates based on
* ANSI SQL semantics For inner join, all predicates for all aliases can
be
* pushed For full outer join, none of the predicates can be pushed as
that
* would limit the number of rows for join For left outer join, all the
* predicates on the left side aliases can be pushed up For right outer
* join, all the predicates on the right side aliases can be pushed up
Joins
* chain containing both left and right outer joins are treated as full
* outer join. [...]
*
* @param op
* Join Operator
* @param rr
* Row resolver
* @return set of qualified aliases
*/
{noformat}
Since hive joins are left associative, something like a RIGHT OUTER JOIN b
LEFT OUTER JOIN c INNER JOIN d should be interpreted as ((a RIGHT OUTER
JOIN b) LEFT OUTER JOIN c) INNER JOIN d, so there would be cases where joins
with both left and right outer joins can have aliases that can be pushed.
Here, aliases b and d are eligible to be pushed up while the current criteria
provide that none are eligible.
Using:
{noformat}
create table t1 (id int, key string, value string);
create table t2 (id int, key string, value string);
create table t3 (id int, key string, value string);
create table t4 (id int, key string, value string);
{noformat}
For example, the query
{noformat}
explain select * from t1 full outer join t2 on t1.id=t2.id join t3 on
t2.id=t3.id where t3.id=20;
{noformat}
currently gives
{noformat}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
t1
TableScan
alias: t1
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 0
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t2
TableScan
alias: t2
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 1
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t3
TableScan
alias: t3
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
[jira] [Commented] (HIVE-2337) Predicate pushdown erroneously conservative with outer joins
[
https://issues.apache.org/jira/browse/HIVE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079023#comment-13079023
]
jirapos...@reviews.apache.org commented on HIVE-2337:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1275/
---
(Updated 2011-08-03 21:03:23.393902)
Review request for hive.
Summary (updated)
---
https://issues.apache.org/jira/browse/HIVE-2337
This addresses bug HIVE-2337.
https://issues.apache.org/jira/browse/HIVE-2337
Diffs
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1153598
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join5.q.out
PRE-CREATION
Diff: https://reviews.apache.org/r/1275/diff
Testing
---
Thanks,
Charles
Predicate pushdown erroneously conservative with outer joins
Key: HIVE-2337
URL: https://issues.apache.org/jira/browse/HIVE-2337
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Charles Chen
Assignee: Charles Chen
Attachments: HIVE-2337v1.patch, HIVE-2337v2.patch
The predicate pushdown filter is not applying left associativity of joins
correctly in determining possible aliases for pushing predicates.
In hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases, the criteria for
pushing aliases is specified as:
{noformat}
/**
* Figures out the aliases for whom it is safe to push predicates based on
* ANSI SQL semantics For inner join, all predicates for all aliases can
be
* pushed For full outer join, none of the predicates can be pushed as
that
* would limit the number of rows for join For left outer join, all the
* predicates on the left side aliases can be pushed up For right outer
* join, all the predicates on the right side aliases can be pushed up
Joins
* chain containing both left and right outer joins are treated as full
* outer join. [...]
*
* @param op
* Join Operator
* @param rr
* Row resolver
* @return set of qualified aliases
*/
{noformat}
Since hive joins are left associative, something like a RIGHT OUTER JOIN b
LEFT OUTER JOIN c INNER JOIN d should be interpreted as ((a RIGHT OUTER
JOIN b) LEFT OUTER JOIN c) INNER JOIN d, so there would be cases where joins
with both left and right outer joins can have aliases that can be pushed.
Here, aliases b and d are eligible to be pushed up while the current criteria
provide that none are eligible.
Using:
{noformat}
create table t1 (id int, key string, value string);
create table t2 (id int, key string, value string);
create table t3 (id int, key string, value string);
create table t4 (id int, key string, value string);
{noformat}
For example, the query
{noformat}
explain select * from t1 full outer join t2 on t1.id=t2.id join t3 on
t2.id=t3.id where t3.id=20;
{noformat}
currently gives
{noformat}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
t1
TableScan
alias: t1
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 0
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t2
TableScan
alias: t2
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 1
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t3
TableScan
alias: t3
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
[jira] [Commented] (HIVE-2337) Predicate pushdown erroneously conservative with outer joins
[
https://issues.apache.org/jira/browse/HIVE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079030#comment-13079030
]
jirapos...@reviews.apache.org commented on HIVE-2337:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1275/
---
(Updated 2011-08-03 21:13:25.076077)
Review request for hive.
Changes
---
Fix testcase
Summary
---
https://issues.apache.org/jira/browse/HIVE-2337
This addresses bug HIVE-2337.
https://issues.apache.org/jira/browse/HIVE-2337
Diffs (updated)
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1153598
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/ppd_outer_join5.q
PRE-CREATION
http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/ppd_outer_join5.q.out
PRE-CREATION
Diff: https://reviews.apache.org/r/1275/diff
Testing
---
Thanks,
Charles
Predicate pushdown erroneously conservative with outer joins
Key: HIVE-2337
URL: https://issues.apache.org/jira/browse/HIVE-2337
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Charles Chen
Assignee: Charles Chen
Attachments: HIVE-2337v1.patch, HIVE-2337v2.patch
The predicate pushdown filter is not applying left associativity of joins
correctly in determining possible aliases for pushing predicates.
In hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases, the criteria for
pushing aliases is specified as:
{noformat}
/**
* Figures out the aliases for whom it is safe to push predicates based on
* ANSI SQL semantics For inner join, all predicates for all aliases can
be
* pushed For full outer join, none of the predicates can be pushed as
that
* would limit the number of rows for join For left outer join, all the
* predicates on the left side aliases can be pushed up For right outer
* join, all the predicates on the right side aliases can be pushed up
Joins
* chain containing both left and right outer joins are treated as full
* outer join. [...]
*
* @param op
* Join Operator
* @param rr
* Row resolver
* @return set of qualified aliases
*/
{noformat}
Since hive joins are left associative, something like a RIGHT OUTER JOIN b
LEFT OUTER JOIN c INNER JOIN d should be interpreted as ((a RIGHT OUTER
JOIN b) LEFT OUTER JOIN c) INNER JOIN d, so there would be cases where joins
with both left and right outer joins can have aliases that can be pushed.
Here, aliases b and d are eligible to be pushed up while the current criteria
provide that none are eligible.
Using:
{noformat}
create table t1 (id int, key string, value string);
create table t2 (id int, key string, value string);
create table t3 (id int, key string, value string);
create table t4 (id int, key string, value string);
{noformat}
For example, the query
{noformat}
explain select * from t1 full outer join t2 on t1.id=t2.id join t3 on
t2.id=t3.id where t3.id=20;
{noformat}
currently gives
{noformat}
STAGE DEPENDENCIES:
Stage-1 is a root stage
Stage-0 is a root stage
STAGE PLANS:
Stage: Stage-1
Map Reduce
Alias - Map Operator Tree:
t1
TableScan
alias: t1
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 0
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t2
TableScan
alias: t2
Reduce Output Operator
key expressions:
expr: id
type: int
sort order: +
Map-reduce partition columns:
expr: id
type: int
tag: 1
value expressions:
expr: id
type: int
expr: key
type: string
expr: value
type: string
t3
TableScan
alias: t3
Reduce Output Operator
key expressions:
expr: id
[jira] [Commented] (HIVE-2342) mirror.facebook.net is 404ing
[ https://issues.apache.org/jira/browse/HIVE-2342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079047#comment-13079047 ] jirapos...@reviews.apache.org commented on HIVE-2342: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1280/ --- Review request for hive and John Sichi. Summary --- Review for HIVE-2342. This addresses bug HIVE-2342. https://issues.apache.org/jira/browse/HIVE-2342 Diffs - build.properties f85221f ivy/ivysettings.xml 9e8dfe6 Diff: https://reviews.apache.org/r/1280/diff Testing --- Thanks, Carl mirror.facebook.net is 404ing - Key: HIVE-2342 URL: https://issues.apache.org/jira/browse/HIVE-2342 Project: Hive Issue Type: Bug Components: Build Infrastructure Reporter: Andrew Bayer Attachments: HIVE-2342.1.patch.txt http://mirror.facebook.net/ and everything under it is 404ing, which is blocking any attempt to build Hive from working. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1734) Implement map_keys() and map_values() UDFs
[ https://issues.apache.org/jira/browse/HIVE-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079060#comment-13079060 ] jirapos...@reviews.apache.org commented on HIVE-1734: - bq. On 2011-08-01 10:03:11, Amareshwari Sriramadasu wrote: bq. ql/src/test/results/clientpositive/udf_map_keys.q.out, line 40 bq. https://reviews.apache.org/r/1082/diff/1/?file=22333#file22333line40 bq. bq. Shouldn't the output be [a, b, c] here? The set of key/value pairs in a Map is unordered, and the same property also applies if you look only at the set of keys, or the set of values. Unfortunately, Hive's type system does not have a 'set' type, so we're forced to return the result of map_keys() and map_values() as an array, which implies that the set is ordered when it really isn't. If you want to apply a specific ordering property to the output of map_keys() or map_values(), then you would have to do this with a sort(array) UDF (which is not yet implemented). bq. On 2011-08-01 10:03:11, Amareshwari Sriramadasu wrote: bq. ql/src/test/results/clientpositive/udf_map_values.q.out, line 40 bq. https://reviews.apache.org/r/1082/diff/1/?file=22334#file22334line40 bq. bq. Shouldn't the output be [1,2,3] instead of [2,1,3]? No. See previous comment. - Carl --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1082/#review1247 --- On 2011-07-12 00:40:11, Carl Steinbach wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1082/ bq. --- bq. bq. (Updated 2011-07-12 00:40:11) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. This patch adds map_keys(map) and map_values(map) UDFs. bq. bq. bq. This addresses bug HIVE-1734. bq. https://issues.apache.org/jira/browse/HIVE-1734 bq. bq. bq. Diffs bq. - bq. bq.ql/src/test/results/clientpositive/udf_map_values.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/udf_map_keys.q.out PRE-CREATION bq.ql/src/test/queries/clientpositive/udf_map_values.q PRE-CREATION bq.ql/src/test/results/clientnegative/udf_map_keys_arg_num.q.out PRE-CREATION bq.ql/src/test/results/clientnegative/udf_map_keys_arg_type.q.out PRE-CREATION bq.ql/src/test/results/clientnegative/udf_map_values_arg_num.q.out PRE-CREATION bq.ql/src/test/results/clientnegative/udf_map_values_arg_type.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/show_functions.q.out d4f6c98 bq.ql/src/test/queries/clientnegative/udf_map_values_arg_num.q PRE-CREATION bq.ql/src/test/queries/clientnegative/udf_map_values_arg_type.q PRE-CREATION bq.ql/src/test/queries/clientpositive/udf_map_keys.q PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 16a207e bq.ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapValues.java PRE-CREATION bq.ql/src/test/queries/clientnegative/udf_map_keys_arg_num.q PRE-CREATION bq.ql/src/test/queries/clientnegative/udf_map_keys_arg_type.q PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1082/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Carl bq. bq. Implement map_keys() and map_values() UDFs -- Key: HIVE-1734 URL: https://issues.apache.org/jira/browse/HIVE-1734 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.6.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-1734-mapudf.1.patch.txt, MapKeys.java, MapValues.java Implement the following UDFs: array map_keys(map) and array map_values(map) map_keys() takes a map as input and returns an array consisting of the key values in the supplied map. Similarly, map_values() takes a map as input and returns an array containing the map value fields. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2338) Alter table always throws an unhelpful error on failure
[ https://issues.apache.org/jira/browse/HIVE-2338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079130#comment-13079130 ] jirapos...@reviews.apache.org commented on HIVE-2338: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1276/ --- (Updated 2011-08-03 23:59:06.774216) Review request for hive and Paul Yang. Changes --- -remove unecessary comment Summary --- Every failure in an alter table function always return a MetaException. When altering tables and catching exceptions, we throw a MetaException in the finally part of a try-catch-finally block, which overrides any other exceptions thrown. This patch moves the code to throw an error on a commit failure outside of the try-catch-finally block. So any errors within the try block are thrown properly. Only when success == false will we throw a MetaException. This addresses bug HIVE-2338. https://issues.apache.org/jira/browse/HIVE-2338 Diffs (updated) - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1152020 Diff: https://reviews.apache.org/r/1276/diff Testing --- Tested failing alter table metastore commands; they threw InvalidOperationExceptions as they should, instead of meaningless MetaExceptions. Thanks, Sohan Alter table always throws an unhelpful error on failure --- Key: HIVE-2338 URL: https://issues.apache.org/jira/browse/HIVE-2338 Project: Hive Issue Type: Bug Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Priority: Minor Attachments: HIVE-2338.1.patch, HIVE-2338.2.patch Every failure in an alter table function always return a MetaException. When altering tables and catching exceptions, we throw a MetaException in the finally part of a try-catch-finally block, which overrides any other exceptions thrown. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2171) Allow custom serdes to set field comments
[
https://issues.apache.org/jira/browse/HIVE-2171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079136#comment-13079136
]
jirapos...@reviews.apache.org commented on HIVE-2171:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/790/
---
(Updated 2011-08-04 00:12:21.542815)
Review request for hive.
Changes
---
Updated patch based on review comments.
Summary
---
HIVE-2171: Allow custom serdes to set field comments
This addresses bug HIVE-2171.
https://issues.apache.org/jira/browse/HIVE-2171
Diffs (updated)
-
ivy/libraries.properties af856bd
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
c1fa4e5
serde/ivy.xml d6c836a
serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazyObjectInspectorFactory.java
4850601
serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/LazySimpleStructObjectInspector.java
e2fa9db
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryObjectInspectorFactory.java
2947e49
serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/objectinspector/LazyBinaryStructObjectInspector.java
3d5408f
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ColumnarStructObjectInspector.java
881c3c1
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/MetadataListStructObjectInspector.java
bd42a0c
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorFactory.java
0c8cc42
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ReflectionStructObjectInspector.java
4a934c5
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StandardStructObjectInspector.java
3b26e45
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/StructField.java
62c3017
serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/UnionStructObjectInspector.java
76ff736
serde/src/test/org/apache/hadoop/hive/serde2/TestSerdeWithFieldComments.java
PRE-CREATION
serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestStandardObjectInspectors.java
f139ea5
Diff: https://reviews.apache.org/r/790/diff
Testing
---
New unit test and refactor existing unit test.
Thanks,
Jakob
Allow custom serdes to set field comments
-
Key: HIVE-2171
URL: https://issues.apache.org/jira/browse/HIVE-2171
Project: Hive
Issue Type: Improvement
Affects Versions: 0.7.0
Reporter: Jakob Homan
Assignee: Jakob Homan
Attachments: HIVE-2171-2.patch, HIVE-2171.patch
Currently, while serde implementations can set a field's name, they can't set
its comment. These are set in the metastore utils to {{(from
deserializer)}}. For those serdes that can provide meaningful comments for a
field, they should be propagated to the table description. These
serde-provided comments could be prepended to (from deserializer) if others
feel that's a meaningful distinction. This change involves updating
{{StructField}} to support a (possibly null) comment field and then
propagating this change out to the myriad places {{StructField}} is thrown
around.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1735) Extend Explode UDTF to handle Maps
[ https://issues.apache.org/jira/browse/HIVE-1735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079200#comment-13079200 ] jirapos...@reviews.apache.org commented on HIVE-1735: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1256/#review1288 --- Ship it! +1. Will commit if tests pass. - Carl On 2011-08-03 04:45:10, Amareshwari Sriramadasu wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1256/ bq. --- bq. bq. (Updated 2011-08-03 04:45:10) bq. bq. bq. Review request for hive and Carl Steinbach. bq. bq. bq. Summary bq. --- bq. bq. Changes from last patch: bq. bq. Re-based to trunk. bq. Removed assert not null statements for list and map. bq. Updated explode_null.q to test NULL values for map also. bq. bq. bq. This addresses bug HIVE-1735. bq. https://issues.apache.org/jira/browse/HIVE-1735 bq. bq. bq. Diffs bq. - bq. bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFExplode.java 1151047 bq.trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported1.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported2.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported3.q PRE-CREATION bq.trunk/ql/src/test/queries/clientnegative/udtf_explode_not_supported4.q PRE-CREATION bq.trunk/ql/src/test/queries/clientpositive/explode_null.q 1151047 bq.trunk/ql/src/test/queries/clientpositive/udf_explode.q 1151047 bq.trunk/ql/src/test/queries/clientpositive/udtf_explode.q 1151047 bq. trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported1.q.out PRE-CREATION bq. trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported2.q.out PRE-CREATION bq. trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported3.q.out PRE-CREATION bq. trunk/ql/src/test/results/clientnegative/udtf_explode_not_supported4.q.out PRE-CREATION bq.trunk/ql/src/test/results/clientpositive/explode_null.q.out 1151047 bq.trunk/ql/src/test/results/clientpositive/udf_explode.q.out 1151047 bq.trunk/ql/src/test/results/clientpositive/udtf_explode.q.out 1151047 bq. bq. Diff: https://reviews.apache.org/r/1256/diff bq. bq. bq. Testing bq. --- bq. bq. All tests passed with the patch. bq. bq. bq. Thanks, bq. bq. Amareshwari bq. bq. Extend Explode UDTF to handle Maps -- Key: HIVE-1735 URL: https://issues.apache.org/jira/browse/HIVE-1735 Project: Hive Issue Type: New Feature Components: UDF Reporter: Carl Steinbach Assignee: Jonathan Natkins Fix For: 0.8.0 Attachments: HIVE-1735.1.patch, HIVE-1735.2.patch.txt, HIVE-1735.2.patch.txt, HIVE-1735.3.patch.txt The explode() UDTF currently only accepts arrays as input. We should modify it so that it can also handle map inputs, in which case it will output two columns corresponding to the key and value fields. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2331) Turn off compression when generating index intermediate results
[ https://issues.apache.org/jira/browse/HIVE-2331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13078366#comment-13078366 ] jirapos...@reviews.apache.org commented on HIVE-2331: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1254/ --- (Updated 2011-08-02 18:58:32.937434) Review request for hive and John Sichi. Changes --- Regenerated the testcase outputs since they were mixed up previously. Summary --- add a parameter to the hiveConf when generating the index query to ensure that the output is uncompressed This addresses bug HIVE-2331. https://issues.apache.org/jira/browse/HIVE-2331 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java c93c852 ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java f4b7c15 ql/src/test/queries/clientpositive/index_bitmap_compression.q PRE-CREATION ql/src/test/queries/clientpositive/index_compression.q PRE-CREATION ql/src/test/results/clientpositive/index_bitmap_compression.q.out PRE-CREATION ql/src/test/results/clientpositive/index_compression.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1254/diff Testing --- added index_compression.q and index_bitmap_compression.q to test both index handlers for this case. Thanks, Syed Turn off compression when generating index intermediate results --- Key: HIVE-2331 URL: https://issues.apache.org/jira/browse/HIVE-2331 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2331.1.patch, HIVE-2331.2.patch HiveIndexResult is not compression-aware, so for any index to work (regardless of compact/bitmap) we need to not compress the index intermediate file when we generate it. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2286) ClassCastException when building index with security.authorization turned on
[ https://issues.apache.org/jira/browse/HIVE-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13078382#comment-13078382 ] jirapos...@reviews.apache.org commented on HIVE-2286: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1137/ --- (Updated 2011-08-02 19:21:24.017675) Review request for hive, John Sichi and Ning Zhang. Changes --- fixed merge conflicts on correct branch Summary --- Save the original HiveOperation/commandType when we generate the index builder task and restore it after we're done generating the task so that the authorization checks make the right decision when deciding what to do. This addresses bug HIVE-2286. https://issues.apache.org/jira/browse/HIVE-2286 Diffs (updated) - cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 9fa7bc6 ql/src/java/org/apache/hadoop/hive/ql/Driver.java 602626f ql/src/test/queries/clientpositive/index_auth.q PRE-CREATION ql/src/test/results/clientnegative/alter_view_failure2.q.out 5915b4f ql/src/test/results/clientnegative/alter_view_failure4.q.out 97d6b18 ql/src/test/results/clientnegative/create_or_replace_view1.q.out 97bfa21 ql/src/test/results/clientnegative/create_or_replace_view2.q.out 8edac34 ql/src/test/results/clientnegative/index_compact_entry_limit.q.out fcb2673 ql/src/test/results/clientnegative/index_compact_size_limit.q.out fcb2673 ql/src/test/results/clientpositive/create_or_replace_view.q.out fb72971 ql/src/test/results/clientpositive/create_view_partitioned.q.out 3b4dd10 ql/src/test/results/clientpositive/index_auth.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto.q.out 8d65f98 ql/src/test/results/clientpositive/index_auto_file_format.q.out 194b35e ql/src/test/results/clientpositive/index_auto_multiple.q.out 6b81fc3 ql/src/test/results/clientpositive/index_auto_partitioned.q.out b0635db ql/src/test/results/clientpositive/index_auto_unused.q.out 3631bbc ql/src/test/results/clientpositive/index_bitmap.q.out 8f41ce3 ql/src/test/results/clientpositive/index_bitmap1.q.out 9f638f5 ql/src/test/results/clientpositive/index_bitmap2.q.out e901477 ql/src/test/results/clientpositive/index_bitmap3.q.out 116c973 ql/src/test/results/clientpositive/index_bitmap_auto.q.out cc9d91e ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out aa1f60c ql/src/test/results/clientpositive/index_bitmap_rc.q.out 9bd3c98 ql/src/test/results/clientpositive/index_compact.q.out c339ec9 ql/src/test/results/clientpositive/index_compact_1.q.out 34ba3ca ql/src/test/results/clientpositive/index_compact_2.q.out e8ce238 ql/src/test/results/clientpositive/index_compact_3.q.out d39556d ql/src/test/results/clientpositive/index_creation.q.out 532f07e ql/src/test/results/clientpositive/show_indexes_edge_cases.q.out 709bdc0 ql/src/test/results/clientpositive/show_indexes_syntax.q.out 2c2a4af Diff: https://reviews.apache.org/r/1137/diff Testing --- Added new testcase to TestCliDriver: index_auth.q Thanks, Syed ClassCastException when building index with security.authorization turned on Key: HIVE-2286 URL: https://issues.apache.org/jira/browse/HIVE-2286 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2286.1.patch, HIVE-2286.2.patch, HIVE-2286.6.patch, HIVE-2286.7.patch, HIVE-2286.8.patch, HIVE-2286.9.patch When trying to build an index with authorization checks turned on, hive issues the following ClassCastException: org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer cannot be cast to org.apache.hadoop.hive.ql.parse.SemanticAnalyzer at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:540) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:848) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:293) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:385) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:392) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:567) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
[jira] [Commented] (HIVE-2020) Create a separate namespace for Hive variables
[
https://issues.apache.org/jira/browse/HIVE-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13078474#comment-13078474
]
jirapos...@reviews.apache.org commented on HIVE-2020:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1204/#review1273
---
This patch doesn't appear to address most of the issues described in HIVE-2020,
which covers adding a separate namespace for variables that is distinct from
hiveconf and environment/system properties. This patch also needs a testcase.
cli/src/java/org/apache/hadoop/hive/cli/DefaultPreprocessor.java
https://reviews.apache.org/r/1204/#comment2906
Please extend/modify/use the code in
org.apache.hadoop.hive.ql.parse.VariableSubstitution instead of adding this
class.
cli/src/java/org/apache/hadoop/hive/cli/OptionsProcessor.java
https://reviews.apache.org/r/1204/#comment2905
I think these two option definitions can be combined, e.g:
.withArgName(key=value)
.withLongOpt(define)
.create(d);
- Carl
On 2011-07-27 21:07:40, Vaibhav Aggarwal wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1204/
bq. ---
bq.
bq. (Updated 2011-07-27 21:07:40)
bq.
bq.
bq. Review request for hive.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Create a separate option for Hive variable
bq.
bq.
bq. This addresses bug HIVE-2020.
bq. https://issues.apache.org/jira/browse/HIVE-2020
bq.
bq.
bq. Diffs
bq. -
bq.
bq.cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 9fa7bc6
bq.cli/src/java/org/apache/hadoop/hive/cli/DefaultPreprocessor.java
PRE-CREATION
bq.cli/src/java/org/apache/hadoop/hive/cli/KeyValue.java PRE-CREATION
bq.cli/src/java/org/apache/hadoop/hive/cli/OptionsProcessor.java 90084ed
bq.cli/src/java/org/apache/hadoop/hive/cli/Preprocessor.java PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/1204/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq.
bq. Thanks,
bq.
bq. Vaibhav
bq.
bq.
Create a separate namespace for Hive variables
--
Key: HIVE-2020
URL: https://issues.apache.org/jira/browse/HIVE-2020
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Carl Steinbach
Assignee: Vaibhav Aggarwal
Attachments: HIVE-2020.patch
Support for variable substitution was added in HIVE-1096. However, variable
substitution was implemented by reusing the HiveConf namespace, so there is
no separation between Hive configuration properties and Hive variables.
This ticket encompasses the following enhancements:
* Create a separate namespace for managing Hive variables.
* Add support for setting variables on the command line via '-hivevar x=y'
* Add support for setting variables through the CLI via 'var x=y'
* Add support for referencing variables in statements using either
'${hivevar:var_name}' or '${var_name}'
* Provide a means for differentiating between hiveconf, hivevar, system, and
environment properties in the output of 'set -v'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2335) Indexes are still automatically queried when out of sync with their source tables
[ https://issues.apache.org/jira/browse/HIVE-2335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13078497#comment-13078497 ] jirapos...@reviews.apache.org commented on HIVE-2335: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1260/ --- Review request for hive, John Sichi, Yongqiang He, and Ning Zhang. Summary --- Check the timestamps of the tables/partitions to ensure that they were not written to since the index was last generated This addresses bug HIVE-2335. https://issues.apache.org/jira/browse/HIVE-2335 Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java 8295687 ql/src/test/queries/clientpositive/index_stale.q PRE-CREATION ql/src/test/queries/clientpositive/index_stale_partitioned.q PRE-CREATION ql/src/test/results/clientpositive/index_stale.q.out PRE-CREATION ql/src/test/results/clientpositive/index_stale_partitioned.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1260/diff Testing --- added index_stale.q and index_stale_partitioned.q to the testcase suite Thanks, Syed Indexes are still automatically queried when out of sync with their source tables - Key: HIVE-2335 URL: https://issues.apache.org/jira/browse/HIVE-2335 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2335.1.patch The automatic index usage does not check whether or not the indexes are still up-to-date when generating the index queries. This can be addressed in two stages, the first is to add a check before generating the index query to ensure that the index is still valid. The next stage may be to add some sort of mode where indexes are automatically updated on table writes. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2337) Predicate pushdown erroneously conservative with outer joins
[
https://issues.apache.org/jira/browse/HIVE-2337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13078551#comment-13078551
]
jirapos...@reviews.apache.org commented on HIVE-2337:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1275/
---
Review request for hive.
Summary
---
Initial patch
This addresses bug HIVE-2337.
https://issues.apache.org/jira/browse/HIVE-2337
Diffs
-
http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/ppd/OpProcFactory.java
1153293
Diff: https://reviews.apache.org/r/1275/diff
Testing
---
Thanks,
Charles
Predicate pushdown erroneously conservative with outer joins
Key: HIVE-2337
URL: https://issues.apache.org/jira/browse/HIVE-2337
Project: Hive
Issue Type: Bug
Components: Query Processor
Reporter: Charles Chen
Assignee: Charles Chen
Attachments: HIVE-2337v1.patch
The predicate pushdown filter is not applying left associativity of joins
correctly in determining possible aliases for pushing predicates.
In hive.ql.ppd.OpProcFactory.JoinPPD.getQualifiedAliases, the criteria for
pushing aliases is specified as:
{noformat}
/**
* Figures out the aliases for whom it is safe to push predicates based on
* ANSI SQL semantics For inner join, all predicates for all aliases can
be
* pushed For full outer join, none of the predicates can be pushed as
that
* would limit the number of rows for join For left outer join, all the
* predicates on the left side aliases can be pushed up For right outer
* join, all the predicates on the right side aliases can be pushed up
Joins
* chain containing both left and right outer joins are treated as full
* outer join. [...]
*
* @param op
* Join Operator
* @param rr
* Row resolver
* @return set of qualified aliases
*/
{noformat}
Since hive joins are left associative, something like a RIGHT OUTER JOIN b
LEFT OUTER JOIN c INNER JOIN d should be interpreted as ((a RIGHT OUTER
JOIN b) LEFT OUTER JOIN c) INNER JOIN d, so there would be cases where joins
with both left and right outer joins can have aliases that can be pushed.
Here, aliases a and d are eligible to be pushed up while the current criteria
provide that none are eligible.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1734) Implement map_keys() and map_values() UDFs
[ https://issues.apache.org/jira/browse/HIVE-1734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073480#comment-13073480 ] jirapos...@reviews.apache.org commented on HIVE-1734: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1082/#review1247 --- ql/src/test/results/clientpositive/udf_map_keys.q.out https://reviews.apache.org/r/1082/#comment2850 Shouldn't the output be [a, b, c] here? ql/src/test/results/clientpositive/udf_map_values.q.out https://reviews.apache.org/r/1082/#comment2851 Shouldn't the output be [1,2,3] instead of [2,1,3]? - Amareshwari On 2011-07-12 00:40:11, Carl Steinbach wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1082/ bq. --- bq. bq. (Updated 2011-07-12 00:40:11) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. This patch adds map_keys(map) and map_values(map) UDFs. bq. bq. bq. This addresses bug HIVE-1734. bq. https://issues.apache.org/jira/browse/HIVE-1734 bq. bq. bq. Diffs bq. - bq. bq.ql/src/test/results/clientpositive/udf_map_values.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/udf_map_keys.q.out PRE-CREATION bq.ql/src/test/queries/clientpositive/udf_map_values.q PRE-CREATION bq.ql/src/test/results/clientnegative/udf_map_keys_arg_num.q.out PRE-CREATION bq.ql/src/test/results/clientnegative/udf_map_keys_arg_type.q.out PRE-CREATION bq.ql/src/test/results/clientnegative/udf_map_values_arg_num.q.out PRE-CREATION bq.ql/src/test/results/clientnegative/udf_map_values_arg_type.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/show_functions.q.out d4f6c98 bq.ql/src/test/queries/clientnegative/udf_map_values_arg_num.q PRE-CREATION bq.ql/src/test/queries/clientnegative/udf_map_values_arg_type.q PRE-CREATION bq.ql/src/test/queries/clientpositive/udf_map_keys.q PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 16a207e bq.ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapKeys.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFMapValues.java PRE-CREATION bq.ql/src/test/queries/clientnegative/udf_map_keys_arg_num.q PRE-CREATION bq.ql/src/test/queries/clientnegative/udf_map_keys_arg_type.q PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1082/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Carl bq. bq. Implement map_keys() and map_values() UDFs -- Key: HIVE-1734 URL: https://issues.apache.org/jira/browse/HIVE-1734 Project: Hive Issue Type: New Feature Components: UDF Affects Versions: 0.6.0 Reporter: Carl Steinbach Assignee: Carl Steinbach Attachments: HIVE-1734-mapudf.1.patch.txt, MapKeys.java, MapValues.java Implement the following UDFs: array map_keys(map) and array map_values(map) map_keys() takes a map as input and returns an array consisting of the key values in the supplied map. Similarly, map_values() takes a map as input and returns an array containing the map value fields. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2326) Turn off bitmap indexing when map-side aggregation is turned off
[ https://issues.apache.org/jira/browse/HIVE-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073669#comment-13073669 ] jirapos...@reviews.apache.org commented on HIVE-2326: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1232/#review1251 --- ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java https://reviews.apache.org/r/1232/#comment2863 Typo: wihtout ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java https://reviews.apache.org/r/1232/#comment2864 Throw the exception here instead of returning null. - John On 2011-07-31 00:26:44, Syed Albiz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1232/ bq. --- bq. bq. (Updated 2011-07-31 00:26:44) bq. bq. bq. Review request for hive, John Sichi and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. sets the correct object inspector for PARTIAL2, COMPLETE UDAF modes bq. bq. bq. This addresses bug HIVE-2326. bq. https://issues.apache.org/jira/browse/HIVE-2326 bq. bq. bq. Diffs bq. - bq. bq.ql/src/java/org/apache/hadoop/hive/ql/index/TableBasedIndexHandler.java 02ab78c bq. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5 bq. ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java 0a3df09 bq. bq. Diff: https://reviews.apache.org/r/1232/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Syed bq. bq. Turn off bitmap indexing when map-side aggregation is turned off Key: HIVE-2326 URL: https://issues.apache.org/jira/browse/HIVE-2326 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2326.1.patch Simply adding the CLUSTER BY clause on the ROW_OFFSET does not work with a GROUP BY clause, causing a SemanticException when trying to compile the the index builder task. Based on conversation with John Sichi, for now we will just turn off this feature. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2326) Turn off bitmap indexing when map-side aggregation is turned off
[ https://issues.apache.org/jira/browse/HIVE-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073796#comment-13073796 ] jirapos...@reviews.apache.org commented on HIVE-2326: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1232/ --- (Updated 2011-08-01 21:27:24.460533) Review request for hive, John Sichi and Ning Zhang. Changes --- removed index_bitmap_auto_no_map_aggr.q from the clientpositive suite Summary --- sets the correct object inspector for PARTIAL2, COMPLETE UDAF modes This addresses bug HIVE-2326. https://issues.apache.org/jira/browse/HIVE-2326 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/index/TableBasedIndexHandler.java 02ab78c ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5 ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7c91946 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java 0a3df09 ql/src/test/queries/clientnegative/index_bitmap_no_map_aggr.q PRE-CREATION ql/src/test/results/clientnegative/index_bitmap_no_map_aggr.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1232/diff Testing --- Thanks, Syed Turn off bitmap indexing when map-side aggregation is turned off Key: HIVE-2326 URL: https://issues.apache.org/jira/browse/HIVE-2326 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2326.1.patch, HIVE-2326.2.patch, HIVE-2326.3.patch Simply adding the CLUSTER BY clause on the ROW_OFFSET does not work with a GROUP BY clause, causing a SemanticException when trying to compile the the index builder task. Based on conversation with John Sichi, for now we will just turn off this feature. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2286) ClassCastException when building index with security.authorization turned on
[ https://issues.apache.org/jira/browse/HIVE-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073987#comment-13073987 ] jirapos...@reviews.apache.org commented on HIVE-2286: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1137/ --- (Updated 2011-08-01 23:23:46.966525) Review request for hive, John Sichi and Ning Zhang. Changes --- regenerated patch to avoid noise/clutter in testcase output Summary --- Save the original HiveOperation/commandType when we generate the index builder task and restore it after we're done generating the task so that the authorization checks make the right decision when deciding what to do. This addresses bug HIVE-2286. https://issues.apache.org/jira/browse/HIVE-2286 Diffs (updated) - cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 9fa7bc6 ql/src/java/org/apache/hadoop/hive/ql/Driver.java b278ffe ql/src/test/queries/clientpositive/index_auth.q PRE-CREATION ql/src/test/results/clientnegative/alter_view_failure2.q.out 5915b4f ql/src/test/results/clientnegative/alter_view_failure4.q.out 97d6b18 ql/src/test/results/clientnegative/create_or_replace_view1.q.out 97bfa21 ql/src/test/results/clientnegative/create_or_replace_view2.q.out 8edac34 ql/src/test/results/clientnegative/index_compact_entry_limit.q.out fcb2673 ql/src/test/results/clientnegative/index_compact_size_limit.q.out fcb2673 ql/src/test/results/clientpositive/create_or_replace_view.q.out fb72971 ql/src/test/results/clientpositive/create_view_partitioned.q.out 3b4dd10 ql/src/test/results/clientpositive/index_auth.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto.q.out 8d65f98 ql/src/test/results/clientpositive/index_auto_file_format.q.out 194b35e ql/src/test/results/clientpositive/index_auto_multiple.q.out 6b81fc3 ql/src/test/results/clientpositive/index_auto_partitioned.q.out b0635db ql/src/test/results/clientpositive/index_auto_unused.q.out 3631bbc ql/src/test/results/clientpositive/index_bitmap.q.out 8f41ce3 ql/src/test/results/clientpositive/index_bitmap1.q.out 9f638f5 ql/src/test/results/clientpositive/index_bitmap2.q.out e901477 ql/src/test/results/clientpositive/index_bitmap3.q.out 116c973 ql/src/test/results/clientpositive/index_bitmap_auto.q.out cc9d91e ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out 9003eb4 ql/src/test/results/clientpositive/index_bitmap_rc.q.out 9bd3c98 ql/src/test/results/clientpositive/index_compact.q.out c339ec9 ql/src/test/results/clientpositive/index_compact_1.q.out 34ba3ca ql/src/test/results/clientpositive/index_compact_2.q.out e8ce238 ql/src/test/results/clientpositive/index_compact_3.q.out d39556d ql/src/test/results/clientpositive/index_creation.q.out 532f07e ql/src/test/results/clientpositive/show_indexes_edge_cases.q.out 709bdc0 ql/src/test/results/clientpositive/show_indexes_syntax.q.out 2c2a4af Diff: https://reviews.apache.org/r/1137/diff Testing --- Added new testcase to TestCliDriver: index_auth.q Thanks, Syed ClassCastException when building index with security.authorization turned on Key: HIVE-2286 URL: https://issues.apache.org/jira/browse/HIVE-2286 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2286.1.patch, HIVE-2286.2.patch, HIVE-2286.6.patch When trying to build an index with authorization checks turned on, hive issues the following ClassCastException: org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer cannot be cast to org.apache.hadoop.hive.ql.parse.SemanticAnalyzer at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:540) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:848) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:293) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:385) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:392) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:567) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) at
[jira] [Commented] (HIVE-1694) Accelerate GROUP BY execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13075964#comment-13075964 ] jirapos...@reviews.apache.org commented on HIVE-1694: - bq. On 2011-07-28 21:40:30, John Sichi wrote: bq. ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java, line 61 bq. https://reviews.apache.org/r/1194/diff/1/?file=27052#file27052line61 bq. bq. Please run ant checkstyle and fix all the formatting discrepancies it reports for your new files. bq. bq. bq. Prajakta Kalmegh wrote: bq. Done! The code is still having checkstyle formatting errors only for places where we have used LinkedHashMap, HashMap and ArrayList. The error states Declaring variables, return values or parameters of type 'HashMap' is not allowed. Best practice is to only use interfaces (Map/List) except at the point of instantiation where you select a concrete class. Hive violates this in a number of places, and sometimes that forces you to violate it in new code too; but otherwise, please follow this one. bq. On 2011-07-28 21:40:30, John Sichi wrote: bq. ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java, line 603 bq. https://reviews.apache.org/r/1194/diff/1/?file=27062#file27062line603 bq. bq. Not sure why this new constructor is needed...after using it, all you do is get the table out of it. bq. bq. Prajakta Kalmegh wrote: bq. The only other constructor option for tableSpec needs the ASTNode as one of its parameters. Since we need to construct a new tableSpec using only the index table name, and we do not have a ASTNode for this, I need this constructor. If you have any other way in mind, please let me know. That would be helpful. I'm asking why you even need to construct a new tableSpec instance. All you do with it is reference ts.tableHandle. And to create that tableHandle, you can just do db.getTable(tableName). So I don't see the purpose of the tableSpec instance. - John --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1194/#review1212 --- On 2011-07-26 14:44:01, Prajakta Kalmegh wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1194/ bq. --- bq. bq. (Updated 2011-07-26 14:44:01) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. This patch has defined a new AggregateIndexHandler which is used to optimize the query plan for groupby queries. bq. bq. bq. This addresses bug HIVE-1694. bq. https://issues.apache.org/jira/browse/HIVE-1694 bq. bq. bq. Diffs bq. - bq. bq.common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b46976f bq.ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndex.java 591c9ff bq.ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 2ca63b3 bq.ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 590d69a bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 77a6dc6 bq.ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx.q PRE-CREATION bq.ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1194/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Prajakta bq. bq. Accelerate GROUP BY execution using indexes --- Key: HIVE-1694 URL: https://issues.apache.org/jira/browse/HIVE-1694 Project: Hive Issue Type: New Feature Components: Indexing, Query Processor Affects Versions: 0.7.0 Reporter: Nikhil Deshpande Assignee: Prajakta Kalmegh Attachments: HIVE-1694.1.patch.txt, HIVE-1694.2.patch.txt, HIVE-1694.3.patch.txt,
[jira] [Commented] (HIVE-1850) alter table set serdeproperties bypasses regexps checks (leaves table in a non-recoverable state?)
[
https://issues.apache.org/jira/browse/HIVE-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073434#comment-13073434
]
jirapos...@reviews.apache.org commented on HIVE-1850:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1233/
---
Review request for hive.
Summary
---
alter table set serdeproperties bypasses regexps checks (leaves table in a
non-recoverable state?)
This addresses bug HIVE-1850.
https://issues.apache.org/jira/browse/HIVE-1850
Diffs
-
trunk/ql/src/test/results/clientnegative/create_table_wrong_regex.q.out
PRE-CREATION
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java
1151047
trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java 1151047
trunk/ql/src/test/queries/clientnegative/alter_table_wrong_regex.q
PRE-CREATION
trunk/ql/src/test/queries/clientnegative/create_table_wrong_regex.q
PRE-CREATION
trunk/ql/src/test/results/clientnegative/alter_table_wrong_regex.q.out
PRE-CREATION
Diff: https://reviews.apache.org/r/1233/diff
Testing
---
All tests pass with the patch.
Thanks,
Amareshwari
alter table set serdeproperties bypasses regexps checks (leaves table in a
non-recoverable state?)
--
Key: HIVE-1850
URL: https://issues.apache.org/jira/browse/HIVE-1850
Project: Hive
Issue Type: Bug
Components: Serializers/Deserializers
Affects Versions: 0.7.0
Environment: Trunk build from a few days ago, but seen once before
with older version as well.
Reporter: Terje Marthinussen
Assignee: Amareshwari Sriramadasu
Fix For: 0.8.0
Attachments: patch-1850.txt
{code}
create table aa ( test STRING )
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (input.regex = [^\\](.*), output.format.string =
$1s);
{code}
This will fail. Great!
{code}
create table aa ( test STRING )
ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.RegexSerDe'
WITH SERDEPROPERTIES (input.regex = (.*), output.format.string =
$1s);
{code}
Works, no problem there.
{code}
alter table aa set serdeproperties (input.regex = [^\\](.*),
output.format.string = $1s);
{code}
Wups... I can set that without any problems!
{code}
alter table aa set serdeproperties (input.regex = (.*),
output.format.string = $1s);
FAILED: Hive Internal Error: java.util.regex.PatternSyntaxException(Unclosed
character class near index 7
[^\](.*)
^)
java.util.regex.PatternSyntaxException: Unclosed character class near index 7
[^\](.*)
^
at java.util.regex.Pattern.error(Pattern.java:1713)
at java.util.regex.Pattern.clazz(Pattern.java:2254)
at java.util.regex.Pattern.sequence(Pattern.java:1818)
at java.util.regex.Pattern.expr(Pattern.java:1752)
at java.util.regex.Pattern.compile(Pattern.java:1460)
at java.util.regex.Pattern.init(Pattern.java:1133)
at java.util.regex.Pattern.compile(Pattern.java:847)
at
org.apache.hadoop.hive.contrib.serde2.RegexSerDe.initialize(RegexSerDe.java:101)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:199)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:253)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:484)
at
org.apache.hadoop.hive.ql.metadata.Table.checkValidity(Table.java:161)
at org.apache.hadoop.hive.ql.metadata.Hive.getTable(Hive.java:803)
at
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeAlterTableSerdeProps(DDLSemanticAnalyzer.java:558)
at
org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer.analyzeInternal(DDLSemanticAnalyzer.java:232)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:238)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:686)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:142)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:370)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
[jira] [Commented] (HIVE-2326) Turn off bitmap indexing when map-side aggregation is turned off
[ https://issues.apache.org/jira/browse/HIVE-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073271#comment-13073271 ] jirapos...@reviews.apache.org commented on HIVE-2326: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1232/ --- Review request for hive, John Sichi and Ning Zhang. Summary --- sets the correct object inspector for PARTIAL2, COMPLETE UDAF modes This addresses bug HIVE-2326. https://issues.apache.org/jira/browse/HIVE-2326 Diffs - ql/src/java/org/apache/hadoop/hive/ql/index/TableBasedIndexHandler.java 02ab78c ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5 ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFEWAHBitmap.java 0a3df09 Diff: https://reviews.apache.org/r/1232/diff Testing --- Thanks, Syed Turn off bitmap indexing when map-side aggregation is turned off Key: HIVE-2326 URL: https://issues.apache.org/jira/browse/HIVE-2326 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2326.1.patch Simply adding the CLUSTER BY clause on the ROW_OFFSET does not work with a GROUP BY clause, causing a SemanticException when trying to compile the the index builder task. Based on conversation with John Sichi, for now we will just turn off this feature. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2303) files with control-A,B are not delimited correctly.
[ https://issues.apache.org/jira/browse/HIVE-2303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072785#comment-13072785 ] jirapos...@reviews.apache.org commented on HIVE-2303: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1219/ --- Review request for hive. Summary --- files with control-A,B are not delimited correctly. This addresses bug HIVE-2303. https://issues.apache.org/jira/browse/HIVE-2303 Diffs - trunk/data/files/in7.txt PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/PlanUtils.java 1151047 trunk/ql/src/test/queries/clientpositive/delimiter.q PRE-CREATION trunk/ql/src/test/results/clientpositive/combine2.q.out 1151047 trunk/ql/src/test/results/clientpositive/delimiter.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/filter_join_breaktask.q.out 1151047 trunk/ql/src/test/results/clientpositive/input23.q.out 1151047 trunk/ql/src/test/results/clientpositive/input42.q.out 1151047 trunk/ql/src/test/results/clientpositive/input_part7.q.out 1151047 trunk/ql/src/test/results/clientpositive/input_part9.q.out 1151047 trunk/ql/src/test/results/clientpositive/louter_join_ppr.q.out 1151047 trunk/ql/src/test/results/clientpositive/outer_join_ppr.q.out 1151047 trunk/ql/src/test/results/clientpositive/pcr.q.out 1151047 trunk/ql/src/test/results/clientpositive/rand_partitionpruner1.q.out 1151047 trunk/ql/src/test/results/clientpositive/rand_partitionpruner3.q.out 1151047 trunk/ql/src/test/results/clientpositive/regexp_extract.q.out 1151047 trunk/ql/src/test/results/clientpositive/router_join_ppr.q.out 1151047 trunk/ql/src/test/results/clientpositive/sample10.q.out 1151047 trunk/ql/src/test/results/clientpositive/sample6.q.out 1151047 trunk/ql/src/test/results/clientpositive/sample8.q.out 1151047 trunk/ql/src/test/results/clientpositive/sample9.q.out 1151047 trunk/ql/src/test/results/clientpositive/transform_ppr1.q.out 1151047 trunk/ql/src/test/results/clientpositive/transform_ppr2.q.out 1151047 trunk/ql/src/test/results/clientpositive/udf_explode.q.out 1151047 trunk/ql/src/test/results/clientpositive/udf_reflect.q.out 1151047 trunk/ql/src/test/results/clientpositive/udtf_explode.q.out 1151047 trunk/ql/src/test/results/clientpositive/union_ppr.q.out 1151047 trunk/ql/src/test/results/compiler/plan/cast1.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/groupby2.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/groupby3.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/groupby4.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/groupby5.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/groupby6.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/input20.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/input8.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/input_part1.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/input_testxpath.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/input_testxpath2.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/join4.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/join5.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/join6.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/join7.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/join8.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/sample1.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/udf1.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/udf4.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/udf6.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/udf_case.q.xml 1151047 trunk/ql/src/test/results/compiler/plan/udf_when.q.xml 1151047 Diff: https://reviews.apache.org/r/1219/diff Testing --- All tests passed with patch Thanks, Amareshwari files with control-A,B are not delimited correctly. --- Key: HIVE-2303 URL: https://issues.apache.org/jira/browse/HIVE-2303 Project: Hive Issue Type: Bug Reporter: Amareshwari Sriramadasu Assignee: Amareshwari Sriramadasu Attachments: patch-2303.txt The following is from one of our users: create external table impressions (imp string, msg string) row format delimited fields terminated by '\t' lines terminated by '\n' stored as textfile location '/xxx'; Some strings in my data contains Control-A, Control-B etc as internal delimiters. If I do a Select * from impressions limit 10; All fields were able to print correctly. However if I do a Select * from
[jira] [Commented] (HIVE-2305) UNION ALL on different types throws runtime exception
[ https://issues.apache.org/jira/browse/HIVE-2305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072961#comment-13072961 ] jirapos...@reviews.apache.org commented on HIVE-2305: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1223/ --- Review request for hive. Summary --- UNION ALL on columns of different types is semantically allowed but throws a runtime exception due to bugs in getting the correct object inspector after the parent MR input file is change and in setting the TypeInfo for the union row schema. This addresses bug hive-2305. https://issues.apache.org/jira/browse/hive-2305 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ColumnInfo.java 1151937 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1151937 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapOperator.java 1151937 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 1151937 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1151937 trunk/ql/src/test/queries/clientpositive/union24.q PRE-CREATION trunk/ql/src/test/results/clientpositive/union24.q.out PRE-CREATION trunk/ql/src/test/results/compiler/plan/union.q.xml 1151937 Diff: https://reviews.apache.org/r/1223/diff Testing --- unit tests pass Thanks, Franklin UNION ALL on different types throws runtime exception - Key: HIVE-2305 URL: https://issues.apache.org/jira/browse/HIVE-2305 Project: Hive Issue Type: Bug Affects Versions: 0.7.1 Reporter: Franklin Hu Assignee: Franklin Hu Attachments: hive-2305.1.patch, hive-2305.2.patch, hive-2305.3.patch Ex: SELECT * (SELECT 123 FROM ... UNION ALL SELECT '123' FROM ..) t; Unioning columns of different types currently throws runtime exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2080) Few code improvements in the ql and serde packages.
[ https://issues.apache.org/jira/browse/HIVE-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072196#comment-13072196 ] jirapos...@reviews.apache.org commented on HIVE-2080: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1144/ --- (Updated 2011-07-28 06:11:18.101615) Review request for hive. Changes --- Patch rebased Summary --- Few code improvements in the ql and serde packages. 1) Little performance Improvements 2) Null checks to avoid NPEs 3) Effective varaible management. This addresses bug HIVE-2080. https://issues.apache.org/jira/browse/HIVE-2080 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 1151731 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1151731 trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDeField.java 1151731 trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDeFieldType.java 1151731 trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDeFunction.java 1151731 Diff: https://reviews.apache.org/r/1144/diff Testing --- All unit test passed Thanks, chinna Few code improvements in the ql and serde packages. --- Key: HIVE-2080 URL: https://issues.apache.org/jira/browse/HIVE-2080 Project: Hive Issue Type: Bug Components: Query Processor, Serializers/Deserializers Affects Versions: 0.7.0 Environment: Hadoop 0.20.1, Hive0.7.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5). Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2080.1.Patch, HIVE-2080.2.Patch, HIVE-2080.Patch Few code improvements in the ql and serde packages. 1) Little performance Improvements 2) Null checks to avoid NPEs 3) Effective varaible management. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2319) Calling alter_table after changing partition comment throws an exception
[ https://issues.apache.org/jira/browse/HIVE-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072215#comment-13072215 ] jirapos...@reviews.apache.org commented on HIVE-2319: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1213/ --- Review request for hive and Paul Yang. Summary --- Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. This addresses bug HIVE-2319. https://issues.apache.org/jira/browse/HIVE-2319 Diffs - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1151219 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1151219 Diff: https://reviews.apache.org/r/1213/diff Testing --- Added some test cases to HiveMetaStore that pass. Thanks, Sohan Calling alter_table after changing partition comment throws an exception Key: HIVE-2319 URL: https://issues.apache.org/jira/browse/HIVE-2319 Project: Hive Issue Type: Bug Components: Metastore Reporter: Sohan Jain Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2319) Calling alter_table after changing partition comment throws an exception
[ https://issues.apache.org/jira/browse/HIVE-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072219#comment-13072219 ] jirapos...@reviews.apache.org commented on HIVE-2319: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1213/ --- (Updated 2011-07-28 07:06:24.236980) Review request for hive and Paul Yang. Changes --- Fixed some of the test case logic. Summary --- Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. This addresses bug HIVE-2319. https://issues.apache.org/jira/browse/HIVE-2319 Diffs (updated) - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1151219 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1151219 Diff: https://reviews.apache.org/r/1213/diff Testing --- Added some test cases to HiveMetaStore that pass. Thanks, Sohan Calling alter_table after changing partition comment throws an exception Key: HIVE-2319 URL: https://issues.apache.org/jira/browse/HIVE-2319 Project: Hive Issue Type: Bug Components: Metastore Reporter: Sohan Jain Altering a table's partition key comments raises an InvalidOperationException. The partition key name and type should not be mutable, but the comment should be able to get changed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2272) add TIMESTAMP data type
[ https://issues.apache.org/jira/browse/HIVE-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072494#comment-13072494 ] jirapos...@reviews.apache.org commented on HIVE-2272: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1135/ --- (Updated 2011-07-28 19:17:59.868991) Review request for hive. Changes --- Decided to store timestamps as UTC on disk and add offsets (localize) if user needs through UDF. This simplifies use cases (compared to MySQL) where users from different time zones querying the same data would receive different results. Will file a follow up JIRA ticket to add table level timezone defaults. This would allow a user to import existing data that is already offset from UTC and write new data to the table with that offset. Summary: -Added config var for default timezone -Added GenericUDF for converting from UTC to localized timezone (either default, or supplied from list of Java timezones) Ex: utc_to_localized_timestamp(timestamp) OR utc_to_localized_timestamp(timestamp, America/Chicago) -Fixed DateDiff bug I introduced Summary --- Adds TIMESTAMP type to serde2 with both string (LazySimple) and binary (LazyBinary) serialization. Supports SQL style jdbc timestamps of the format with nanosecond precision -MM-DD HH:MM:SS[.fff...] This addresses bug HIVE-2272. https://issues.apache.org/jira/browse/HIVE-2272 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1151571 trunk/conf/hive-default.xml 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDate.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateAdd.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateDiff.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateSub.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDayOfMonth.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHour.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMinute.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMonth.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSecond.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFWeekOfYear.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFYear.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovarianceSample.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStd.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStdSample.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVarianceSample.java 1151571 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java 1151571
[jira] [Commented] (HIVE-1694) Accelerate GROUP BY execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072550#comment-13072550 ] jirapos...@reviews.apache.org commented on HIVE-1694: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1194/#review1212 --- ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java https://reviews.apache.org/r/1194/#comment2711 Please run ant checkstyle and fix all the formatting discrepancies it reports for your new files. ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java https://reviews.apache.org/r/1194/#comment2695 Don't you need to reuse the compact implementation here so that the index can be used for WHERE (not just GROUP BY)? ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java https://reviews.apache.org/r/1194/#comment2696 This method is redundant now. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java https://reviews.apache.org/r/1194/#comment2698 I can't think of a case where it would be worse. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java https://reviews.apache.org/r/1194/#comment2699 Actually group-by is now preserved in all cases. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java https://reviews.apache.org/r/1194/#comment2700 Please use HTML bullet syntax for Javadoc (otherwise it all gets run together into one line when rendered). ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java https://reviews.apache.org/r/1194/#comment2701 indentation ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java https://reviews.apache.org/r/1194/#comment2703 Shouldn't this be BIGINT? Also, I think you're supposed to use a TypeInfoFactory for this purpose. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java https://reviews.apache.org/r/1194/#comment2702 indentation ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java https://reviews.apache.org/r/1194/#comment2704 typo: Repace ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java https://reviews.apache.org/r/1194/#comment2707 Not sure why this new constructor is needed...after using it, all you do is get the table out of it. ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx.q https://reviews.apache.org/r/1194/#comment2709 This should *not* be using the index, since the index is built on count(l_shipdate), and l_shipdate may contain nulls, whereas the query is referencing count(1), which is insensitive to nulls. ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx.q https://reviews.apache.org/r/1194/#comment2710 Need additional tests to verify all the cases where the optimization should *not* be used: * when configuration disables it * when index partitions do not cover table partitions (I still don't see the code for this case) * ... all the other conditions checked for in the code ... - John On 2011-07-26 14:44:01, Prajakta Kalmegh wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1194/ bq. --- bq. bq. (Updated 2011-07-26 14:44:01) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. This patch has defined a new AggregateIndexHandler which is used to optimize the query plan for groupby queries. bq. bq. bq. This addresses bug HIVE-1694. bq. https://issues.apache.org/jira/browse/HIVE-1694 bq. bq. bq. Diffs bq. - bq. bq.common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b46976f bq.ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndex.java 591c9ff bq.ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 2ca63b3 bq.ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 590d69a bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java PRE-CREATION bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java PRE-CREATION bq.
[jira] [Commented] (HIVE-2272) add TIMESTAMP data type
[ https://issues.apache.org/jira/browse/HIVE-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072553#comment-13072553 ] jirapos...@reviews.apache.org commented on HIVE-2272: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1135/ --- (Updated 2011-07-28 21:59:38.146393) Review request for hive, Yongqiang He, Ning Zhang, and Siying Dong. Changes --- Add corresponding UDF for converting to UTC from a localized timestamp Summary --- Adds TIMESTAMP type to serde2 with both string (LazySimple) and binary (LazyBinary) serialization. Supports SQL style jdbc timestamps of the format with nanosecond precision -MM-DD HH:MM:SS[.fff...] This addresses bug HIVE-2272. https://issues.apache.org/jira/browse/HIVE-2272 Diffs (updated) - trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1152003 trunk/conf/hive-default.xml 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDate.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateAdd.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateDiff.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateSub.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDayOfMonth.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHour.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMinute.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMonth.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSecond.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFWeekOfYear.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFYear.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovarianceSample.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStd.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStdSample.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVarianceSample.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java 1152003 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFFromUtcTimestamp.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFToUtcTimestamp.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_t_create3.q 1152003 trunk/ql/src/test/queries/clientpositive/timestamp_1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_comparison.q PRE-CREATION
[jira] [Commented] (HIVE-2020) Create a separate namespace for Hive variables
[
https://issues.apache.org/jira/browse/HIVE-2020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072011#comment-13072011
]
jirapos...@reviews.apache.org commented on HIVE-2020:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1204/
---
Review request for hive.
Summary
---
Create a separate option for Hive variable
This addresses bug HIVE-2020.
https://issues.apache.org/jira/browse/HIVE-2020
Diffs
-
cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 9fa7bc6
cli/src/java/org/apache/hadoop/hive/cli/DefaultPreprocessor.java PRE-CREATION
cli/src/java/org/apache/hadoop/hive/cli/KeyValue.java PRE-CREATION
cli/src/java/org/apache/hadoop/hive/cli/OptionsProcessor.java 90084ed
cli/src/java/org/apache/hadoop/hive/cli/Preprocessor.java PRE-CREATION
Diff: https://reviews.apache.org/r/1204/diff
Testing
---
Thanks,
Vaibhav
Create a separate namespace for Hive variables
--
Key: HIVE-2020
URL: https://issues.apache.org/jira/browse/HIVE-2020
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Carl Steinbach
Assignee: Vaibhav Aggarwal
Support for variable substitution was added in HIVE-1096. However, variable
substitution was implemented by reusing the HiveConf namespace, so there is
no separation between Hive configuration properties and Hive variables.
This ticket encompasses the following enhancements:
* Create a separate namespace for managing Hive variables.
* Add support for setting variables on the command line via '-hivevar x=y'
* Add support for setting variables through the CLI via 'var x=y'
* Add support for referencing variables in statements using either
'${hivevar:var_name}' or '${var_name}'
* Provide a means for differentiating between hiveconf, hivevar, system, and
environment properties in the output of 'set -v'
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2318) Support multiple file systems
[ https://issues.apache.org/jira/browse/HIVE-2318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13072036#comment-13072036 ] jirapos...@reviews.apache.org commented on HIVE-2318: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1210/ --- Review request for hive. Summary --- Support multiple file systems This addresses bug HIVE-2318. https://issues.apache.org/jira/browse/HIVE-2318 Diffs - ql/src/java/org/apache/hadoop/hive/ql/Context.java 1a1ffc5 ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java c031f40 ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java c9e61a1 ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java c6f6755 ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java a57f9cf ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 092a5cd ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 337a8eb ql/src/java/org/apache/hadoop/hive/ql/parse/ImportSemanticAnalyzer.java 3001575 ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java 6f311c6 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java e1febc3 ql/src/java/org/apache/hadoop/hive/ql/plan/ConditionalResolverMergeFiles.java 5c09789 ql/src/java/org/apache/hadoop/hive/ql/plan/DynamicPartitionCtx.java 29cfe9d Diff: https://reviews.apache.org/r/1210/diff Testing --- Thanks, Vaibhav Support multiple file systems - Key: HIVE-2318 URL: https://issues.apache.org/jira/browse/HIVE-2318 Project: Hive Issue Type: New Feature Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Currently some of the Hive tasks like MoveTask, ConditionalMergeResolver assume that the data is being copied or moved on the same file system. These operators file if the source table is in one filesystem (like HDFS) and destination table is in another file system (like s3). This patch aims at: 1. Support moving data between different file systems. 2. Add support for file systems which do not support 'move' operation like s3. 3. Remove redundant operations like moving data from and to the same location. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1694) Accelerate GROUP BY execution using indexes
[ https://issues.apache.org/jira/browse/HIVE-1694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13071130#comment-13071130 ] jirapos...@reviews.apache.org commented on HIVE-1694: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1194/ --- Review request for hive and John Sichi. Summary --- This patch has defined a new AggregateIndexHandler which is used to optimize the query plan for groupby queries. This addresses bug HIVE-1694. https://issues.apache.org/jira/browse/HIVE-1694 Diffs - common/src/java/org/apache/hadoop/hive/conf/HiveConf.java b46976f ql/src/java/org/apache/hadoop/hive/ql/index/AggregateIndexHandler.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndex.java 591c9ff ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 2ca63b3 ql/src/java/org/apache/hadoop/hive/ql/optimizer/Optimizer.java 590d69a ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyCtx.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteCanApplyProcFactory.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteGBUsingIndex.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteParseContextGenerator.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndex.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/optimizer/index/RewriteQueryUsingAggregateIndexCtx.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 77a6dc6 ql/src/test/queries/clientpositive/ql_rewrite_gbtoidx.q PRE-CREATION ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1194/diff Testing --- Thanks, Prajakta Accelerate GROUP BY execution using indexes --- Key: HIVE-1694 URL: https://issues.apache.org/jira/browse/HIVE-1694 Project: Hive Issue Type: New Feature Components: Indexing, Query Processor Affects Versions: 0.7.0 Reporter: Nikhil Deshpande Assignee: Prajakta Kalmegh Attachments: HIVE-1694.1.patch.txt, HIVE-1694.2.patch.txt, HIVE-1694.3.patch.txt, HIVE-1694_2010-10-28.diff, demo_q1.hql, demo_q2.hql The index building patch (Hive-417) is checked into trunk, this JIRA issue tracks supporting indexes in Hive compiler execution engine for SELECT queries. This is in ref. to John's comment at https://issues.apache.org/jira/browse/HIVE-417?focusedCommentId=12884869page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_12884869 on creating separate JIRA issue for tracking index usage in optimizer query execution. The aim of this effort is to use indexes to accelerate query execution (for certain class of queries). E.g. - Filters and range scans (already being worked on by He Yongqiang as part of HIVE-417?) - Joins (index based joins) - Group By, Order By and other misc cases The proposal is multi-step: 1. Building index based operators, compiler and execution engine changes 2. Optimizer enhancements (e.g. cost-based optimizer to compare and choose between index scans, full table scans etc.) This JIRA initially focuses on the first step. This JIRA is expected to hold the information about index based plans operator implementations for above mentioned cases. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2286) ClassCastException when building index with security.authorization turned on
[ https://issues.apache.org/jira/browse/HIVE-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13071315#comment-13071315 ] jirapos...@reviews.apache.org commented on HIVE-2286: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1137/#review1188 --- ql/src/java/org/apache/hadoop/hive/ql/Driver.java https://reviews.apache.org/r/1137/#comment2597 java.util.Stack is deprecated since it adds unnecessary synchronization. We don't have a replacement yet (HIVE-1626) so we've just been using ArrayList. Also, instead of typecasting to/from Object, use a static inner class for holding the record of state variables. - John On 2011-07-25 23:03:22, Syed Albiz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1137/ bq. --- bq. bq. (Updated 2011-07-25 23:03:22) bq. bq. bq. Review request for hive, John Sichi and Ning Zhang. bq. bq. bq. Summary bq. --- bq. bq. Save the original HiveOperation/commandType when we generate the index builder task and restore it after we're done generating the task so that the authorization checks make the right decision when deciding what to do. bq. bq. bq. This addresses bug HIVE-2286. bq. https://issues.apache.org/jira/browse/HIVE-2286 bq. bq. bq. Diffs bq. - bq. bq.ql/src/java/org/apache/hadoop/hive/ql/Driver.java b278ffe bq.ql/src/test/queries/clientpositive/index_auth.q PRE-CREATION bq.ql/src/test/results/clientnegative/index_compact_entry_limit.q.out fcb2673 bq.ql/src/test/results/clientnegative/index_compact_size_limit.q.out fcb2673 bq.ql/src/test/results/clientpositive/index_auth.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto.q.out 8d65f98 bq.ql/src/test/results/clientpositive/index_auto_file_format.q.out 194b35e bq.ql/src/test/results/clientpositive/index_auto_multiple.q.out 6b81fc3 bq.ql/src/test/results/clientpositive/index_auto_partitioned.q.out b0635db bq.ql/src/test/results/clientpositive/index_auto_unused.q.out 3631bbc bq.ql/src/test/results/clientpositive/index_bitmap.q.out 8f41ce3 bq.ql/src/test/results/clientpositive/index_bitmap1.q.out 9f638f5 bq.ql/src/test/results/clientpositive/index_bitmap2.q.out e901477 bq.ql/src/test/results/clientpositive/index_bitmap3.q.out 116c973 bq.ql/src/test/results/clientpositive/index_bitmap_auto.q.out cc9d91e bq.ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out 9003eb4 bq.ql/src/test/results/clientpositive/index_bitmap_rc.q.out 9bd3c98 bq.ql/src/test/results/clientpositive/index_compact.q.out c339ec9 bq.ql/src/test/results/clientpositive/index_compact_1.q.out 34ba3ca bq.ql/src/test/results/clientpositive/index_compact_2.q.out e8ce238 bq.ql/src/test/results/clientpositive/index_compact_3.q.out d39556d bq.ql/src/test/results/clientpositive/index_creation.q.out 532f07e bq. bq. Diff: https://reviews.apache.org/r/1137/diff bq. bq. bq. Testing bq. --- bq. bq. Added new testcase to TestCliDriver: index_auth.q bq. bq. bq. Thanks, bq. bq. Syed bq. bq. ClassCastException when building index with security.authorization turned on Key: HIVE-2286 URL: https://issues.apache.org/jira/browse/HIVE-2286 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2286.1.patch, HIVE-2286.2.patch When trying to build an index with authorization checks turned on, hive issues the following ClassCastException: org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer cannot be cast to org.apache.hadoop.hive.ql.parse.SemanticAnalyzer at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:540) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:848) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:293) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:385) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:392) at
[jira] [Commented] (HIVE-2272) add TIMESTAMP data type
[ https://issues.apache.org/jira/browse/HIVE-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13071368#comment-13071368 ] jirapos...@reviews.apache.org commented on HIVE-2272: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1135/ --- (Updated 2011-07-26 21:11:35.218104) Review request for hive. Changes --- Rebase Summary --- Adds TIMESTAMP type to serde2 with both string (LazySimple) and binary (LazyBinary) serialization. Supports SQL style jdbc timestamps of the format with nanosecond precision -MM-DD HH:MM:SS[.fff...] This addresses bug HIVE-2272. https://issues.apache.org/jira/browse/HIVE-2272 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ErrorMsg.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/TypeCheckProcFactory.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDate.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateAdd.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateDiff.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDateSub.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFDayOfMonth.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFHour.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMinute.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFMonth.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFSecond.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToBoolean.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToByte.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToDouble.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFWeekOfYear.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFYear.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovarianceSample.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStd.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStdSample.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVarianceSample.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFnGrams.java 1151189 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFTimestamp.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/invalid_t_create3.q 1151189 trunk/ql/src/test/queries/clientpositive/timestamp_1.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_comparison.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_udf.q PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_create_tbl1.q.out 1151189 trunk/ql/src/test/results/clientnegative/invalid_t_alter1.q.out 1151189 trunk/ql/src/test/results/clientnegative/invalid_t_alter2.q.out 1151189 trunk/ql/src/test/results/clientnegative/invalid_t_create1.q.out 1151189 trunk/ql/src/test/results/clientnegative/invalid_t_create2.q.out 1151189 trunk/ql/src/test/results/clientnegative/invalid_t_transform.q.out
[jira] [Commented] (HIVE-2286) ClassCastException when building index with security.authorization turned on
[ https://issues.apache.org/jira/browse/HIVE-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13071428#comment-13071428 ] jirapos...@reviews.apache.org commented on HIVE-2286: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1137/ --- (Updated 2011-07-26 23:28:13.279889) Review request for hive, John Sichi and Ning Zhang. Changes --- refactor patch to dump query state into an inner class rather than a Stack. Summary --- Save the original HiveOperation/commandType when we generate the index builder task and restore it after we're done generating the task so that the authorization checks make the right decision when deciding what to do. This addresses bug HIVE-2286. https://issues.apache.org/jira/browse/HIVE-2286 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/Driver.java b278ffe ql/src/test/queries/clientpositive/index_auth.q PRE-CREATION ql/src/test/results/clientnegative/addpart1.q.out f4da8f1 ql/src/test/results/clientnegative/alter_concatenate_indexed_table.q.out 8ae1f9d ql/src/test/results/clientnegative/alter_non_native.q.out 8be2c3b ql/src/test/results/clientnegative/alter_view_failure.q.out 9954b66 ql/src/test/results/clientnegative/alter_view_failure2.q.out 5915b4f ql/src/test/results/clientnegative/alter_view_failure4.q.out 97d6b18 ql/src/test/results/clientnegative/alter_view_failure5.q.out 2291ca6 ql/src/test/results/clientnegative/alter_view_failure6.q.out 03b2bc3 ql/src/test/results/clientnegative/alter_view_failure7.q.out d0f958c ql/src/test/results/clientnegative/alter_view_failure8.q.out 4420c57 ql/src/test/results/clientnegative/alter_view_failure9.q.out 67306d3 ql/src/test/results/clientnegative/altern1.q.out c52ca04 ql/src/test/results/clientnegative/analyze_view.q.out 99def40 ql/src/test/results/clientnegative/archive1.q.out 0927686 ql/src/test/results/clientnegative/archive2.q.out 25baefa ql/src/test/results/clientnegative/authorization_fail_1.q.out ab1abe2 ql/src/test/results/clientnegative/authorization_fail_3.q.out cd7ceb1 ql/src/test/results/clientnegative/authorization_fail_4.q.out b05f9b7 ql/src/test/results/clientnegative/authorization_fail_5.q.out f5bdc6a ql/src/test/results/clientnegative/authorization_fail_7.q.out a52fd1c ql/src/test/results/clientnegative/authorization_part.q.out 625d60c ql/src/test/results/clientnegative/column_rename1.q.out 7c30e4e ql/src/test/results/clientnegative/column_rename2.q.out 0ca78f9 ql/src/test/results/clientnegative/column_rename4.q.out f14fd48 ql/src/test/results/clientnegative/create_or_replace_view1.q.out 97bfa21 ql/src/test/results/clientnegative/create_or_replace_view2.q.out 8edac34 ql/src/test/results/clientnegative/create_or_replace_view4.q.out 89dd5f5 ql/src/test/results/clientnegative/create_or_replace_view5.q.out a0aed59 ql/src/test/results/clientnegative/create_or_replace_view6.q.out df44e33 ql/src/test/results/clientnegative/create_or_replace_view7.q.out 9356dcc ql/src/test/results/clientnegative/create_or_replace_view8.q.out 4161659 ql/src/test/results/clientnegative/create_view_failure1.q.out 43cded4 ql/src/test/results/clientnegative/create_view_failure2.q.out a038067 ql/src/test/results/clientnegative/create_view_failure4.q.out f968569 ql/src/test/results/clientnegative/database_create_already_exists.q.out 08c04f9 ql/src/test/results/clientnegative/database_create_invalid_name.q.out 1e58089 ql/src/test/results/clientnegative/database_drop_does_not_exist.q.out 80c00cd ql/src/test/results/clientnegative/database_drop_not_empty.q.out baa8f37 ql/src/test/results/clientnegative/database_drop_not_empty_restrict.q.out b297a99 ql/src/test/results/clientnegative/database_switch_does_not_exist.q.out 8b5674d ql/src/test/results/clientnegative/drop_partition_failure.q.out 8a7c63d ql/src/test/results/clientnegative/drop_table_failure2.q.out 9b63102 ql/src/test/results/clientnegative/drop_view_failure1.q.out 61ec927 ql/src/test/results/clientnegative/dyn_part3.q.out 5f4df65 ql/src/test/results/clientnegative/exim_00_unsupported_schema.q.out 814b742 ql/src/test/results/clientnegative/exim_01_nonpart_over_loaded.q.out 0351bc1 ql/src/test/results/clientnegative/exim_02_all_part_over_overlap.q.out d40ff27 ql/src/test/results/clientnegative/exim_03_nonpart_noncompat_colschema.q.out adff0f8 ql/src/test/results/clientnegative/exim_04_nonpart_noncompat_colnumber.q.out b84e954 ql/src/test/results/clientnegative/exim_05_nonpart_noncompat_coltype.q.out 96f8452 ql/src/test/results/clientnegative/exim_06_nonpart_noncompat_storage.q.out 25deaa3
[jira] [Commented] (HIVE-2246) Dedupe tables' column schemas from partitions in the metastore db
[ https://issues.apache.org/jira/browse/HIVE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070337#comment-13070337 ] jirapos...@reviews.apache.org commented on HIVE-2246: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1183/#review1176 --- trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql https://reviews.apache.org/r/1183/#comment2467 is the CHARSET (latin1) the same as SDS? This will require the user's comments to be in latin1 which prevents UTF chars. trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql https://reviews.apache.org/r/1183/#comment2466 can you also add migration script for derby? we support derby as a default metastore RDBMS as well. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java https://reviews.apache.org/r/1183/#comment2468 here do you check if the 'alter table' command changes the schema (columns definition)? If it just set a table property, then you don't need to create a new ColumnDescriptor right? Also if a table's schema got changed, a new CD will be created, but the old partition will still have the old CDs. When we query the old partition, do we use the old partitons's CD or the table's CD? Also in the above case, when you run 'desc table partition old_partition', do you return the old partition's CD or the table's CD? - Ning On 2011-07-22 05:30:29, Sohan Jain wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1183/ bq. --- bq. bq. (Updated 2011-07-22 05:30:29) bq. bq. bq. Review request for hive, Ning Zhang and Paul Yang. bq. bq. bq. Summary bq. --- bq. bq. This patch tries to make minimal changes to the API while keeping migration short and somewhat easy to revert. bq. bq. The new schema can be described as follows: bq. - CDS is a table corresponding to Column Descriptor objects. Currently, it only stores a CD_ID. bq. - COLUMNS_V2 is a table corresponding to MFieldSchema objects, or columns. A Column Descriptor holds a list of columns. COLUMNS_V2 has a foreign key to the CD_ID to which it belongs. bq. - SDS was modified to reference a Column Descriptor. So SDS now has a foreign key to a CD_ID which describes its columns. bq. bq. During migration, we create Column Descriptors for tables in a straightforward manner: their columns are now just wrapped inside a column descriptor. The SDS of partitions use their parent table's column descriptor, since currently a partition and its table share the same list of columns. bq. bq. When altering or adding a partition, give it it's parent table's column descriptor IF the columns they describe are the same. Otherwise, create a new column descriptor for its columns. bq. bq. When adding or altering a table, create a new column descriptor every time. bq. bq. Whenever you drop a storage descriptor (e.g, when dropping tables or partitions), check to see if the related column descriptor has any other references in the table. That is, check to see if any other storage descriptors point to that column descriptor. If none do, then delete that column descriptor. This check is in place so we don't have unreferenced column descriptors and columns hanging around after schema evolution for tables. bq. bq. bq. This addresses bug HIVE-2246. bq. https://issues.apache.org/jira/browse/HIVE-2246 bq. bq. bq. Diffs bq. - bq. bq.trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql PRE-CREATION bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1148945 bq. trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MColumnDescriptor.java PRE-CREATION bq. trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java 1148945 bq.trunk/metastore/src/model/package.jdo 1148945 bq. bq. Diff: https://reviews.apache.org/r/1183/diff bq. bq. bq. Testing bq. --- bq. bq. Passes facebook's regression testing and all existing test cases. In one instance, before migration, the overhead involved with storage descriptors and columns was ~11 GB. After migration, the overhead was ~1.5 GB. bq. bq. bq. Thanks, bq. bq. Sohan bq. bq. Dedupe tables' column schemas from partitions in the metastore db - Key: HIVE-2246 URL: https://issues.apache.org/jira/browse/HIVE-2246 Project: Hive Issue Type: Improvement
[jira] [Commented] (HIVE-2299) Optimize Hive query startup time for multiple partitions
[ https://issues.apache.org/jira/browse/HIVE-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070768#comment-13070768 ] jirapos...@reviews.apache.org commented on HIVE-2299: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1188/ --- Review request for hive. Summary --- Review request for HIVE-2299. This addresses bug HIVE-2299. https://issues.apache.org/jira/browse/HIVE-2299 Diffs - ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 421140f Diff: https://reviews.apache.org/r/1188/diff Testing --- Thanks, Carl Optimize Hive query startup time for multiple partitions Key: HIVE-2299 URL: https://issues.apache.org/jira/browse/HIVE-2299 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Attachments: HIVE-2299.patch Added an optimization to the way input splits are computed. Reduced an O(n^2) operation to O n operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2299) Optimize Hive query startup time for multiple partitions
[ https://issues.apache.org/jira/browse/HIVE-2299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070777#comment-13070777 ] jirapos...@reviews.apache.org commented on HIVE-2299: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1188/#review1180 --- Ship it! +1. Will commit if tests pass. - Carl On 2011-07-25 21:22:09, Carl Steinbach wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1188/ bq. --- bq. bq. (Updated 2011-07-25 21:22:09) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. Review request for HIVE-2299. bq. bq. bq. This addresses bug HIVE-2299. bq. https://issues.apache.org/jira/browse/HIVE-2299 bq. bq. bq. Diffs bq. - bq. bq.ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java 421140f bq. bq. Diff: https://reviews.apache.org/r/1188/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Carl bq. bq. Optimize Hive query startup time for multiple partitions Key: HIVE-2299 URL: https://issues.apache.org/jira/browse/HIVE-2299 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Attachments: HIVE-2299.patch Added an optimization to the way input splits are computed. Reduced an O(n^2) operation to O n operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2298) Fix UDAFPercentile to tolerate null percentiles
[ https://issues.apache.org/jira/browse/HIVE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070803#comment-13070803 ] jirapos...@reviews.apache.org commented on HIVE-2298: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1189/ --- Review request for hive. Summary --- Review request for HIVE-2298. This addresses bug HIVE-2298. https://issues.apache.org/jira/browse/HIVE-2298 Diffs - ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.java 92db544 Diff: https://reviews.apache.org/r/1189/diff Testing --- Thanks, Carl Fix UDAFPercentile to tolerate null percentiles --- Key: HIVE-2298 URL: https://issues.apache.org/jira/browse/HIVE-2298 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.7.0 Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Attachments: HIVE-2298.patch UDAFPercentile when passed null percentile list will throw a null pointer exception. Submitting a small fix for that. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2298) Fix UDAFPercentile to tolerate null percentiles
[ https://issues.apache.org/jira/browse/HIVE-2298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070807#comment-13070807 ] jirapos...@reviews.apache.org commented on HIVE-2298: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1189/#review1181 --- ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.java https://reviews.apache.org/r/1189/#comment2483 Please fix the following checkstyle violations: Line 238: File contains tab characters (this is the first instance). Line 240: is longer than 100 characters. Line 245: '}' should be on the same line. - Carl On 2011-07-25 21:53:12, Carl Steinbach wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1189/ bq. --- bq. bq. (Updated 2011-07-25 21:53:12) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. Review request for HIVE-2298. bq. bq. bq. This addresses bug HIVE-2298. bq. https://issues.apache.org/jira/browse/HIVE-2298 bq. bq. bq. Diffs bq. - bq. bq.ql/src/java/org/apache/hadoop/hive/ql/udf/UDAFPercentile.java 92db544 bq. bq. Diff: https://reviews.apache.org/r/1189/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Carl bq. bq. Fix UDAFPercentile to tolerate null percentiles --- Key: HIVE-2298 URL: https://issues.apache.org/jira/browse/HIVE-2298 Project: Hive Issue Type: Bug Components: UDF Affects Versions: 0.7.0 Reporter: Vaibhav Aggarwal Assignee: Vaibhav Aggarwal Attachments: HIVE-2298.patch UDAFPercentile when passed null percentile list will throw a null pointer exception. Submitting a small fix for that. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2286) ClassCastException when building index with security.authorization turned on
[ https://issues.apache.org/jira/browse/HIVE-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13070847#comment-13070847 ] jirapos...@reviews.apache.org commented on HIVE-2286: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1137/ --- (Updated 2011-07-25 23:03:22.871042) Review request for hive, John Sichi and Ning Zhang. Changes --- Addressed comments, still need to regenerate a lot of testcase output since this will change the prehook/posthook messages for a lot of testcases. Summary --- Save the original HiveOperation/commandType when we generate the index builder task and restore it after we're done generating the task so that the authorization checks make the right decision when deciding what to do. This addresses bug HIVE-2286. https://issues.apache.org/jira/browse/HIVE-2286 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/Driver.java b278ffe ql/src/test/queries/clientpositive/index_auth.q PRE-CREATION ql/src/test/results/clientnegative/index_compact_entry_limit.q.out fcb2673 ql/src/test/results/clientnegative/index_compact_size_limit.q.out fcb2673 ql/src/test/results/clientpositive/index_auth.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto.q.out 8d65f98 ql/src/test/results/clientpositive/index_auto_file_format.q.out 194b35e ql/src/test/results/clientpositive/index_auto_multiple.q.out 6b81fc3 ql/src/test/results/clientpositive/index_auto_partitioned.q.out b0635db ql/src/test/results/clientpositive/index_auto_unused.q.out 3631bbc ql/src/test/results/clientpositive/index_bitmap.q.out 8f41ce3 ql/src/test/results/clientpositive/index_bitmap1.q.out 9f638f5 ql/src/test/results/clientpositive/index_bitmap2.q.out e901477 ql/src/test/results/clientpositive/index_bitmap3.q.out 116c973 ql/src/test/results/clientpositive/index_bitmap_auto.q.out cc9d91e ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out 9003eb4 ql/src/test/results/clientpositive/index_bitmap_rc.q.out 9bd3c98 ql/src/test/results/clientpositive/index_compact.q.out c339ec9 ql/src/test/results/clientpositive/index_compact_1.q.out 34ba3ca ql/src/test/results/clientpositive/index_compact_2.q.out e8ce238 ql/src/test/results/clientpositive/index_compact_3.q.out d39556d ql/src/test/results/clientpositive/index_creation.q.out 532f07e Diff: https://reviews.apache.org/r/1137/diff Testing --- Added new testcase to TestCliDriver: index_auth.q Thanks, Syed ClassCastException when building index with security.authorization turned on Key: HIVE-2286 URL: https://issues.apache.org/jira/browse/HIVE-2286 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2286.1.patch, HIVE-2286.2.patch When trying to build an index with authorization checks turned on, hive issues the following ClassCastException: org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer cannot be cast to org.apache.hadoop.hive.ql.parse.SemanticAnalyzer at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:540) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:848) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:293) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:385) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:392) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:567) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:156) -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069465#comment-13069465 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/ --- (Updated 2011-07-22 09:36:21.858818) Review request for hive. Changes --- lol Summary --- https://issues.apache.org/jira/browse/HIVE-1078 This addresses bug HIVE-1078. https://issues.apache.org/jira/browse/HIVE-1078 Diffs - http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view5.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view6.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view7.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view8.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/recursive_view.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view5.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view6.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view7.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view8.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/recursive_view.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 Diff: https://reviews.apache.org/r/1058/diff Testing (updated) --- Unit tests passed for r10 (HIVE-1078v9.patch)!!! Thanks, Charles CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch, HIVE-1078v6.patch, HIVE-1078v7.patch, HIVE-1078v8.patch, HIVE-1078v9.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information on JIRA, see:
[jira] [Commented] (HIVE-2282) Local mode needs to work well with block sampling
[ https://issues.apache.org/jira/browse/HIVE-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069635#comment-13069635 ] jirapos...@reviews.apache.org commented on HIVE-2282: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1132/ --- (Updated 2011-07-22 17:40:44.736466) Review request for hive and Siying Dong. Changes --- I added the q.out file which I had forgotten for the new q file. I also modified the test queries to select count(1) instead of selecting keys and values. Summary --- A query should run in local mode when block sampling is used and the sample is small enough. The size of the sample is currently being estimated, as it is done to estimate the number of reducers. This addresses bug HIVE-2282. https://issues.apache.org/jira/browse/HIVE-2282 Diffs (updated) - ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 53769a0 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cd3de76 ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyIsLocalModeHook.java PRE-CREATION ql/src/test/queries/clientpositive/sample_islocalmode_hook.q PRE-CREATION ql/src/test/results/clientpositive/sample_islocalmode_hook.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1132/diff Testing --- TestCliDriver TestNegativeCliDriver, manually tested Thanks, Kevin Local mode needs to work well with block sampling - Key: HIVE-2282 URL: https://issues.apache.org/jira/browse/HIVE-2282 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Kevin Wilfong Attachments: HIVE-2282.1.patch.txt, HIVE-2282.2.patch.txt, HIVE-2282.3.patch.txt Currently, if block sampling is enabled and large set of data are sampled to a small set, local mode needs to be kicked in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2292) Comment clause should immediately follow identifier field in CREATE DATABASE statement
[ https://issues.apache.org/jira/browse/HIVE-2292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069675#comment-13069675 ] jirapos...@reviews.apache.org commented on HIVE-2292: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1186/ --- Review request for hive. Summary --- Review for HIVE-2292. This addresses bug HIVE-2292. https://issues.apache.org/jira/browse/HIVE-2292 Diffs - ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 3a6d0a6 ql/src/test/queries/clientpositive/database_location.q ac8687f ql/src/test/results/clientpositive/database_location.q.out a09f5a2 Diff: https://reviews.apache.org/r/1186/diff Testing --- Thanks, Carl Comment clause should immediately follow identifier field in CREATE DATABASE statement -- Key: HIVE-2292 URL: https://issues.apache.org/jira/browse/HIVE-2292 Project: Hive Issue Type: Bug Components: SQL Reporter: Carl Steinbach Assignee: Carl Steinbach Fix For: 0.8.0 Attachments: HIVE-2292.1.patch.txt -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2183) In Task class and its subclasses logger is initialized in constructor
[
https://issues.apache.org/jira/browse/HIVE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069854#comment-13069854
]
jirapos...@reviews.apache.org commented on HIVE-2183:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1146/#review1174
---
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java
https://reviews.apache.org/r/1146/#comment2463
Why is this NO_OF_THREADS_CAN_BE_LAUNCHED for sleeptime?
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
https://reviews.apache.org/r/1146/#comment2462
I don't think this is an improvement; it's better to see the rule
definition inline. If this was suggested by an automatic code scanner,
probably tell it to ignore this instance.
- John
On 2011-07-20 14:36:59, chinna wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1146/
bq. ---
bq.
bq. (Updated 2011-07-20 14:36:59)
bq.
bq.
bq. Review request for hive.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. In Task class and its subclasses logger is initialized in constructor. Log
object no need to initialize every time in the constructor, Log object can make
it as static object.
bq.
bq.
bq. This addresses bug HIVE-2183.
bq. https://issues.apache.org/jira/browse/HIVE-2183
bq.
bq.
bq. Diffs
bq. -
bq.
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1145025
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 1145025
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1145025
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1145025
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
1145025
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java
1145025
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java
1145025
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 1145025
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 1145025
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1145025
bq.
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
1145025
bq.
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
1145025
bq.
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java
1145025
bq.
bq. Diff: https://reviews.apache.org/r/1146/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. All unit tests passed
bq.
bq.
bq. Thanks,
bq.
bq. chinna
bq.
bq.
In Task class and its subclasses logger is initialized in constructor
-
Key: HIVE-2183
URL: https://issues.apache.org/jira/browse/HIVE-2183
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.5.0, 0.8.0
Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise
Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5)
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Priority: Minor
Attachments: HIVE-2183.1.patch, HIVE-2183.patch
In Task class and its subclasses logger is initialized in constructor. Log
object no need to initialize every time in the constructor, Log object can
make it as static object.
{noformat}
Ex:
public ExecDriver() {
super();
LOG = LogFactory.getLog(this.getClass().getName());
console = new LogHelper(LOG);
this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
}
{noformat}
Need to change like this
{noformat}
private static final Log LOG = LogFactory.getLog(ExecDriver.class);
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2236) Cli: Print Hadoop's CPU milliseconds
[ https://issues.apache.org/jira/browse/HIVE-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069094#comment-13069094 ] jirapos...@reviews.apache.org commented on HIVE-2236: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/948/ --- (Updated 2011-07-21 17:30:55.228025) Review request for hive, Yongqiang He, Ning Zhang, and namit jain. Changes --- fix a bug Summary --- In hive CLI, print out CPU msec from Hadoop MapReduce coutners. This addresses bug HIVE-2236. https://issues.apache.org/jira/browse/HIVE-2236 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1148623 Diff: https://reviews.apache.org/r/948/diff Testing --- run the updated codes against real clusters and make sure it printing is correct. Thanks, Siying Cli: Print Hadoop's CPU milliseconds Key: HIVE-2236 URL: https://issues.apache.org/jira/browse/HIVE-2236 Project: Hive Issue Type: New Feature Components: CLI Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-2236.1.patch, HIVE-2236.2.patch, HIVE-2236.3.patch CPU Milliseonds information is available from Hadoop's framework. Printing it out to Hive CLI when executing a job will help users to know more about their jobs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2247) ALTER TABLE RENAME PARTITION
[ https://issues.apache.org/jira/browse/HIVE-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069146#comment-13069146 ] jirapos...@reviews.apache.org commented on HIVE-2247: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1105/#review1156 --- Please try to add the new column in the middle first. If that works, we should do that way to make it consistent with alter_table() call. If that doesn't work, it's OK to add it to the end now. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java https://reviews.apache.org/r/1105/#comment2385 Why we still need another function call rename_partition_core()? Can't we just modify alter_partition_core() to always use the same logic? - Siying On 2011-07-21 01:20:25, Weiyan Wang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1105/ bq. --- bq. bq. (Updated 2011-07-21 01:20:25) bq. bq. bq. Review request for Siying Dong. bq. bq. bq. Summary bq. --- bq. bq. Implement ALTER TABLE PARTITION RENAME function to rename a partition. bq. Add HiveQL syntax ALTER TABLE bar PARTITION (k1='v1', k2='v2') RENAME TO PARTITION (k1='v3', k2='v4'); bq. This is my first Hive diff, I just learn everything from existing codebase and may not have a good understanding on it. bq. Feel free to inform me if I make something wrong. Thanks bq. bq. bq. This addresses bug HIVE-2247. bq. https://issues.apache.org/jira/browse/HIVE-2247 bq. bq. bq. Diffs bq. - bq. bq.trunk/metastore/if/hive_metastore.thrift 1145366 bq.trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1145366 bq.trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1145366 bq. trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1145366 bq. trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1145366 bq. trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1145366 bq. trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1145366 bq. trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1145366 bq.trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1145366 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1145366 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1145366 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1145366 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1145366 bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1145366 bq.trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1145366 bq. trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1145366 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1145366 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1145366 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1145366 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1145366 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1145366 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java 1145366 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1145366 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1145366 bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RenamePartitionDesc.java PRE-CREATION bq. trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure.q PRE-CREATION bq. trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure2.q PRE-CREATION bq. trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure3.q PRE-CREATION bq.trunk/ql/src/test/queries/clientpositive/alter_rename_partition.q PRE-CREATION bq. trunk/ql/src/test/queries/clientpositive/alter_rename_partition_authorization.q PRE-CREATION bq. trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure.q.out PRE-CREATION bq. trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure2.q.out PRE-CREATION bq. trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure3.q.out PRE-CREATION bq.
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069240#comment-13069240 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/ --- (Updated 2011-07-21 22:07:29.150219) Review request for hive. Changes --- Fix failure of create_or_replace_view.q Summary --- https://issues.apache.org/jira/browse/HIVE-1078 This addresses bug HIVE-1078. https://issues.apache.org/jira/browse/HIVE-1078 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view5.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view6.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view7.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view8.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/recursive_view.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view5.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view6.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view7.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view8.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/recursive_view.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 Diff: https://reviews.apache.org/r/1058/diff Testing --- Passes unit tests Thanks, Charles CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch, HIVE-1078v6.patch, HIVE-1078v7.patch, HIVE-1078v8.patch, HIVE-1078v9.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information on JIRA, see:
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple tables
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069299#comment-13069299 ] jirapos...@reviews.apache.org commented on HIVE-2128: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1010/ --- (Updated 2011-07-21 23:52:23.929900) Review request for hive and John Sichi. Changes --- Added order by to testcases. This revealed an existing bug where we would walk the entire operator tree for each task in the task tree in IndexWhereTaskDispatcher. I amended this to only walk the subset of the operator tree in the current task. Summary --- Grab the indexed tables during optimized query generation, grab the associated path URIs, and keep those around in the Configuration object. When the job is passed to ExecDriver, this data is extracted and used in HiveIndexedInputFormat to decide whether to use the index file or delegate to the parent (HiveInputFormat) class. Not sure if this is robust. This addresses bug HIVE-2128. https://issues.apache.org/jira/browse/HIVE-2128 Diffs (updated) - ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out 4c9efd1 ql/src/test/results/clientpositive/index_auto_self_join.q.out PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java b9b586e ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java f1ee95d ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5 ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7c91946 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java dbc489f ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereTaskDispatcher.java da084f6 ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java a03a9a6 ql/src/test/queries/clientpositive/index_auto_mult_tables.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_mult_tables_compact.q PRE-CREATION ql/src/test/queries/clientpositive/index_auto_self_join.q PRE-CREATION ql/src/test/results/clientpositive/index_auto_mult_tables.q.out PRE-CREATION ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1010/diff Testing --- added new testcase index_auto_mult_tables.q Thanks, Syed Automatic Indexing with multiple tables --- Key: HIVE-2128 URL: https://issues.apache.org/jira/browse/HIVE-2128 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: Russell Melick Assignee: Syed S. Albiz Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch, HIVE-2128.6.patch Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2246) Dedupe tables' column schemas from partitions in the metastore db
[ https://issues.apache.org/jira/browse/HIVE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069389#comment-13069389 ] jirapos...@reviews.apache.org commented on HIVE-2246: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1183/ --- Review request for hive, Ning Zhang and Paul Yang. Summary --- This patch tries to make minimal changes to the API while keeping migration short and somewhat easy to revert. The new schema can be described as follows: - CDS is a table corresponding to Column Descriptor objects. Currently, it only stores a CD_ID. - COLUMNS_V2 is a table corresponding to MFieldSchema objects, or columns. A Column Descriptor holds a list of columns. COLUMNS_V2 has a foreign key to the CD_ID to which it belongs. - SDS was modified to reference a Column Descriptor. So SDS now has a foreign key to a CD_ID which describes its columns. During migration, we create Column Descriptors for tables in a straightforward manner: their columns are now just wrapped inside a column descriptor. The SDS of partitions use their parent table's column descriptor, since currently a partition and its table share the same list of columns. When altering or adding a partition, give it it's parent table's column descriptor IF the columns they describe are the same. Otherwise, create a new column descriptor for its columns. When adding or altering a table, create a new column descriptor every time. Whenever you drop a storage descriptor (e.g, when dropping tables or partitions), check to see if the related column descriptor has any other references in the table. That is, check to see if any other storage descriptors point to that column descriptor. If none do, then delete that column descriptor. This check is in place so we don't have unreferenced column descriptors and columns hanging around after schema evolution for tables. This addresses bug HIVE-2246. https://issues.apache.org/jira/browse/HIVE-2246 Diffs - trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1148945 trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java 1148945 trunk/metastore/src/model/package.jdo 1148945 Diff: https://reviews.apache.org/r/1183/diff Testing --- Passes facebook's regression testing and all existing test cases. In one instance, before migration, the overhead involved with storage descriptors and columns was ~11 GB. After migration, the overhead was ~1.5 GB. Thanks, Sohan Dedupe tables' column schemas from partitions in the metastore db - Key: HIVE-2246 URL: https://issues.apache.org/jira/browse/HIVE-2246 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2246.2.patch We can re-organize the JDO models to reduce space usage to keep the metastore scalable for the future. Currently, partitions are the fastest growing objects in the metastore, and the metastore keeps a separate copy of the columns list for each partition. We can normalize the metastore db by decoupling Columns from Storage Descriptors and not storing duplicate lists of the columns for each partition. An idea is to create an additional level of indirection with a Column Descriptor that has a list of columns. A table has a reference to its latest Column Descriptor (note: a table may have more than one Column Descriptor in the case of schema evolution). Partitions and Indexes can reference the same Column Descriptors as their parent table. Currently, the COLUMNS table in the metastore has roughly (number of partitions + number of tables) * (average number of columns pertable) rows. We can reduce this to (number of tables) * (average number of columns per table) rows, while incurring a small cost proportional to the number of tables to store the Column Descriptors. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2246) Dedupe tables' column schemas from partitions in the metastore db
[ https://issues.apache.org/jira/browse/HIVE-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13069392#comment-13069392 ] jirapos...@reviews.apache.org commented on HIVE-2246: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1183/ --- (Updated 2011-07-22 05:30:29.026246) Review request for hive, Ning Zhang and Paul Yang. Changes --- Adding some files I missed in the last diff. Summary --- This patch tries to make minimal changes to the API while keeping migration short and somewhat easy to revert. The new schema can be described as follows: - CDS is a table corresponding to Column Descriptor objects. Currently, it only stores a CD_ID. - COLUMNS_V2 is a table corresponding to MFieldSchema objects, or columns. A Column Descriptor holds a list of columns. COLUMNS_V2 has a foreign key to the CD_ID to which it belongs. - SDS was modified to reference a Column Descriptor. So SDS now has a foreign key to a CD_ID which describes its columns. During migration, we create Column Descriptors for tables in a straightforward manner: their columns are now just wrapped inside a column descriptor. The SDS of partitions use their parent table's column descriptor, since currently a partition and its table share the same list of columns. When altering or adding a partition, give it it's parent table's column descriptor IF the columns they describe are the same. Otherwise, create a new column descriptor for its columns. When adding or altering a table, create a new column descriptor every time. Whenever you drop a storage descriptor (e.g, when dropping tables or partitions), check to see if the related column descriptor has any other references in the table. That is, check to see if any other storage descriptors point to that column descriptor. If none do, then delete that column descriptor. This check is in place so we don't have unreferenced column descriptors and columns hanging around after schema evolution for tables. This addresses bug HIVE-2246. https://issues.apache.org/jira/browse/HIVE-2246 Diffs (updated) - trunk/metastore/scripts/upgrade/mysql/008-HIVE-2246.mysql.sql PRE-CREATION trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1148945 trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MColumnDescriptor.java PRE-CREATION trunk/metastore/src/model/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java 1148945 trunk/metastore/src/model/package.jdo 1148945 Diff: https://reviews.apache.org/r/1183/diff Testing --- Passes facebook's regression testing and all existing test cases. In one instance, before migration, the overhead involved with storage descriptors and columns was ~11 GB. After migration, the overhead was ~1.5 GB. Thanks, Sohan Dedupe tables' column schemas from partitions in the metastore db - Key: HIVE-2246 URL: https://issues.apache.org/jira/browse/HIVE-2246 Project: Hive Issue Type: Improvement Components: Metastore Reporter: Sohan Jain Assignee: Sohan Jain Attachments: HIVE-2246.2.patch, HIVE-2246.3.patch Note: this patch proposes a schema change, and is therefore incompatible with the current metastore. We can re-organize the JDO models to reduce space usage to keep the metastore scalable for the future. Currently, partitions are the fastest growing objects in the metastore, and the metastore keeps a separate copy of the columns list for each partition. We can normalize the metastore db by decoupling Columns from Storage Descriptors and not storing duplicate lists of the columns for each partition. An idea is to create an additional level of indirection with a Column Descriptor that has a list of columns. A table has a reference to its latest Column Descriptor (note: a table may have more than one Column Descriptor in the case of schema evolution). Partitions and Indexes can reference the same Column Descriptors as their parent table. Currently, the COLUMNS table in the metastore has roughly (number of partitions + number of tables) * (average number of columns pertable) rows. We can reduce this to (number of tables) * (average number of columns per table) rows, while incurring a small cost proportional to the number of tables to store the Column Descriptors. Please see the latest review board for additional implementation details. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068165#comment-13068165 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/#review1128 --- http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/1058/#comment2356 Defer the db.getPartitions (which could be expensive) so that we don't do it unless we're sure that the partition keys are actually changing. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java https://reviews.apache.org/r/1058/#comment2357 Avoid usage of java.util.Stack. Some old Hive code uses it but it's deprecated because it's synchronized for no good reason. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java https://reviews.apache.org/r/1058/#comment2358 add spaces around operators such as = - John On 2011-07-20 01:01:53, Charles Chen wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1058/ bq. --- bq. bq. (Updated 2011-07-20 01:01:53) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. This addresses bug HIVE-1078. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 bq. bq. Diff: https://reviews.apache.org/r/1058/diff bq. bq. bq. Testing bq. --- bq. bq. Passes unit tests bq. bq. bq. Thanks, bq. bq. Charles bq. bq. CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch, HIVE-1078v6.patch, HIVE-1078v7.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information on JIRA, see:
[jira] [Commented] (HIVE-2236) Cli: Print Hadoop's CPU milliseconds
[ https://issues.apache.org/jira/browse/HIVE-2236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068169#comment-13068169 ] jirapos...@reviews.apache.org commented on HIVE-2236: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/948/ --- (Updated 2011-07-20 06:27:19.820431) Review request for hive, Yongqiang He, Ning Zhang, and namit jain. Changes --- remove MapRedStats from DriverContext and add more counters to it. Summary --- In hive CLI, print out CPU msec from Hadoop MapReduce coutners. This addresses bug HIVE-2236. https://issues.apache.org/jira/browse/HIVE-2236 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/MapRedStats.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1148623 trunk/ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1148623 Diff: https://reviews.apache.org/r/948/diff Testing --- run the updated codes against real clusters and make sure it printing is correct. Thanks, Siying Cli: Print Hadoop's CPU milliseconds Key: HIVE-2236 URL: https://issues.apache.org/jira/browse/HIVE-2236 Project: Hive Issue Type: New Feature Components: CLI Reporter: Siying Dong Assignee: Siying Dong Priority: Minor Attachments: HIVE-2236.1.patch, HIVE-2236.2.patch CPU Milliseonds information is available from Hadoop's framework. Printing it out to Hive CLI when executing a job will help users to know more about their jobs. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2080) Few code improvements in the ql and serde packages.
[ https://issues.apache.org/jira/browse/HIVE-2080?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068404#comment-13068404 ] jirapos...@reviews.apache.org commented on HIVE-2080: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1144/ --- Review request for hive. Summary --- Few code improvements in the ql and serde packages. 1) Little performance Improvements 2) Null checks to avoid NPEs 3) Effective varaible management. This addresses bug HIVE-2080. https://issues.apache.org/jira/browse/HIVE-2080 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CommonJoinOperator.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FilterOperator.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/GroupByOperator.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Operator.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ReduceSinkOperator.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/SelectOperator.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/TaskFactory.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/UnionOperator.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ASTNode.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/ParseContext.java 1148179 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1148179 trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDeField.java 1148179 trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDeFieldType.java 1148179 trunk/serde/src/java/org/apache/hadoop/hive/serde2/dynamic_type/DynamicSerDeFunction.java 1148179 Diff: https://reviews.apache.org/r/1144/diff Testing --- All unit test passed Thanks, chinna Few code improvements in the ql and serde packages. --- Key: HIVE-2080 URL: https://issues.apache.org/jira/browse/HIVE-2080 Project: Hive Issue Type: Bug Components: Query Processor, Serializers/Deserializers Affects Versions: 0.7.0 Environment: Hadoop 0.20.1, Hive0.7.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5). Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2080.1.Patch, HIVE-2080.Patch Few code improvements in the ql and serde packages. 1) Little performance Improvements 2) Null checks to avoid NPEs 3) Effective varaible management. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2183) In Task class and its subclasses logger is initialized in constructor
[
https://issues.apache.org/jira/browse/HIVE-2183?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068408#comment-13068408
]
jirapos...@reviews.apache.org commented on HIVE-2183:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1146/
---
Review request for hive.
Summary
---
In Task class and its subclasses logger is initialized in constructor. Log
object no need to initialize every time in the constructor, Log object can make
it as static object.
This addresses bug HIVE-2183.
https://issues.apache.org/jira/browse/HIVE-2183
Diffs
-
trunk/ql/src/java/org/apache/hadoop/hive/ql/Driver.java 1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/CopyTask.java 1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FetchTask.java 1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java
1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionTask.java 1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java 1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java 1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/StatsTask.java 1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Task.java 1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/MapJoinProcessor.java
1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
1145025
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/FunctionSemanticAnalyzer.java
1145025
Diff: https://reviews.apache.org/r/1146/diff
Testing
---
All unit tests passed
Thanks,
chinna
In Task class and its subclasses logger is initialized in constructor
-
Key: HIVE-2183
URL: https://issues.apache.org/jira/browse/HIVE-2183
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.5.0, 0.8.0
Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise
Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5)
Reporter: Chinna Rao Lalam
Assignee: Chinna Rao Lalam
Priority: Minor
Attachments: HIVE-2183.1.patch, HIVE-2183.patch
In Task class and its subclasses logger is initialized in constructor. Log
object no need to initialize every time in the constructor, Log object can
make it as static object.
{noformat}
Ex:
public ExecDriver() {
super();
LOG = LogFactory.getLog(this.getClass().getName());
console = new LogHelper(LOG);
this.jobExecHelper = new HadoopJobExecHelper(job, console, this, this);
}
{noformat}
Need to change like this
{noformat}
private static final Log LOG = LogFactory.getLog(ExecDriver.class);
{noformat}
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068519#comment-13068519 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/ --- (Updated 2011-07-20 18:03:04.848086) Review request for hive. Changes --- Add testcases, fixed issues in comments above (btw the previous revision passed unit tests) Summary --- https://issues.apache.org/jira/browse/HIVE-1078 This addresses bug HIVE-1078. https://issues.apache.org/jira/browse/HIVE-1078 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view5.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view6.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view7.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view8.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/recursive_view.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view5.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view6.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view7.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view8.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/recursive_view.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 Diff: https://reviews.apache.org/r/1058/diff Testing --- Passes unit tests Thanks, Charles CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch, HIVE-1078v6.patch, HIVE-1078v7.patch, HIVE-1078v8.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information
[jira] [Commented] (HIVE-2201) reduce name node calls in hive by creating temporary directories
[ https://issues.apache.org/jira/browse/HIVE-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068715#comment-13068715 ] jirapos...@reviews.apache.org commented on HIVE-2201: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/952/ --- (Updated 2011-07-20 23:31:54.007436) Review request for hive, Yongqiang He, Ning Zhang, and namit jain. Changes --- 1. change block merge task too 2. change the capital file name Summary --- reduce name node calls in hive by creating temporary directories This addresses bug HIVE-2201. https://issues.apache.org/jira/browse/HIVE-2201 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 1148905 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 1148905 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java 1148905 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFileOutputFormat.java 1148905 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/BlockMergeTask.java 1148905 trunk/ql/src/java/org/apache/hadoop/hive/ql/io/rcfile/merge/RCFileMergeMapper.java 1148905 Diff: https://reviews.apache.org/r/952/diff Testing --- Thanks, Siying reduce name node calls in hive by creating temporary directories Key: HIVE-2201 URL: https://issues.apache.org/jira/browse/HIVE-2201 Project: Hive Issue Type: Improvement Reporter: Namit Jain Assignee: Siying Dong Attachments: HIVE-2201.1.patch, HIVE-2201.2.patch, HIVE-2201.3.patch, HIVE-2201.4.patch Currently, in Hive, when a file gets written by a FileSinkOperator, the sequence of operations is as follows: 1. In tmp directory tmp1, create a tmp file _tmp_1 2. At the end of the operator, move /tmp1/_tmp_1 to /tmp1/1 3. Move directory /tmp1 to /tmp2 4. For all files in /tmp2, remove all files starting with _tmp and duplicate files. Due to speculative execution, a lot of temporary files are created in /tmp1 (or /tmp2). This leads to a lot of name node calls, specially for large queries. The protocol above can be modified slightly: 1. In tmp directory tmp1, create a tmp file _tmp_1 2. At the end of the operator, move /tmp1/_tmp_1 to /tmp2/1 3. Move directory /tmp2 to /tmp3 4. For all files in /tmp3, remove all duplicate files. This should reduce the number of tmp files. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2247) ALTER TABLE RENAME PARTITION
[ https://issues.apache.org/jira/browse/HIVE-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068753#comment-13068753 ] jirapos...@reviews.apache.org commented on HIVE-2247: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1105/ --- (Updated 2011-07-21 01:20:25.242756) Review request for Siying Dong. Changes --- Refactor the code, rename_partition shares the same thrift API as alter_partition, we do alter_partition when part_vals is empty, we do rename_partition when part_vals is given Summary --- Implement ALTER TABLE PARTITION RENAME function to rename a partition. Add HiveQL syntax ALTER TABLE bar PARTITION (k1='v1', k2='v2') RENAME TO PARTITION (k1='v3', k2='v4'); This is my first Hive diff, I just learn everything from existing codebase and may not have a good understanding on it. Feel free to inform me if I make something wrong. Thanks This addresses bug HIVE-2247. https://issues.apache.org/jira/browse/HIVE-2247 Diffs (updated) - trunk/metastore/if/hive_metastore.thrift 1145366 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1145366 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1145366 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1145366 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1145366 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1145366 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1145366 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1145366 trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1145366 trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RenamePartitionDesc.java PRE-CREATION trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/alter_rename_partition.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/alter_rename_partition_authorization.q PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/alter_rename_partition.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/alter_rename_partition_authorization.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1105/diff Testing --- Add a partition A in the table Rename partition A to partition B Show the partitions in the table, it returns partition B. SELECT the data from partition A, it returns no results SELECT the data from partition B, it returns the data originally stored in partition A Thanks, Weiyan ALTER TABLE RENAME PARTITION Key: HIVE-2247 URL: https://issues.apache.org/jira/browse/HIVE-2247 Project: Hive Issue Type: New Feature Reporter: Siying Dong Assignee: Weiyan Wang Attachments: HIVE-2247.3.patch.txt,
[jira] [Commented] (HIVE-2296) bad compressed file names from insert into
[ https://issues.apache.org/jira/browse/HIVE-2296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068794#comment-13068794 ] jirapos...@reviews.apache.org commented on HIVE-2296: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1155/ --- Review request for hive and Siying Dong. Summary --- Fixes problem of bad compressed file names by stripping off the file format (ex .gz) and reappending it to the path later. This addresses bug HIVE-2296. https://issues.apache.org/jira/browse/HIVE-2296 Diffs - trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1148973 trunk/ql/src/test/queries/clientpositive/insert_compressed.q PRE-CREATION trunk/ql/src/test/results/clientpositive/insert_compressed.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1155/diff Testing --- Unit tests pass Thanks, Franklin bad compressed file names from insert into -- Key: HIVE-2296 URL: https://issues.apache.org/jira/browse/HIVE-2296 Project: Hive Issue Type: Bug Affects Versions: 0.8.0 Reporter: Franklin Hu Assignee: Franklin Hu Attachments: hive-2296.1.patch, hive-2296.2.patch When INSERT INTO is run on a table with compressed output (hive.exec.compress.output=true) and existing files in the table, it may copy the new files in bad file names: Before INSERT INTO: 00_0.gz After INSERT INTO: 00_0.gz 00_0.gz_copy_1 This causes corrupted output when doing a SELECT * on the table. Correct behavior should be to pick a valid filename such as: 00_0_copy_1.gz -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1884) Potential risk of resource leaks in Hive
[
https://issues.apache.org/jira/browse/HIVE-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067638#comment-13067638
]
jirapos...@reviews.apache.org commented on HIVE-1884:
-
bq. On 2011-07-18 20:45:08, John Sichi wrote:
bq. trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java, line 392
bq. https://reviews.apache.org/r/1098/diff/1/?file=25035#file25035line392
bq.
bq. Rename this to rc to avoid confusion with the processReader method
name.
processReader variable name renamed to rc
bq. On 2011-07-18 20:45:08, John Sichi wrote:
bq.
trunk/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesWritableInput.java,
line 385
bq. https://reviews.apache.org/r/1098/diff/1/?file=25036#file25036line385
bq.
bq. Fix indentation for the try block.
properly formatted the try block
- chinna
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1098/#review1101
---
On 2011-07-13 14:28:47, chinna wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1098/
bq. ---
bq.
bq. (Updated 2011-07-13 14:28:47)
bq.
bq.
bq. Review request for hive and John Sichi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. There are couple of resource leaks.
bq. For example,
bq. In CliDriver.java, Method :- processReader() the buffered reader is not
closed.
bq.
bq. Also there are risk(s) of resource(s) getting leaked , in such cases
refactored the code to move closing of resources in finally block.
bq.
bq.
bq. This addresses bug HIVE-1884.
bq. https://issues.apache.org/jira/browse/HIVE-1884
bq.
bq.
bq. Diffs
bq. -
bq.
bq.trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1145950
bq.
trunk/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesWritableInput.java
1145950
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1145950
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFileInputFormat.java
1145950
bq.
bq. Diff: https://reviews.apache.org/r/1098/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. All unit tests passed
bq.
bq.
bq. Thanks,
bq.
bq. chinna
bq.
bq.
Potential risk of resource leaks in Hive
Key: HIVE-1884
URL: https://issues.apache.org/jira/browse/HIVE-1884
Project: Hive
Issue Type: Bug
Components: CLI, Metastore, Query Processor, Server Infrastructure
Affects Versions: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0
Environment: Hive 0.6.0, Hadoop 0.20.1
SUSE Linux Enterprise Server 11 (i586)
Reporter: Mohit Sikri
Assignee: Chinna Rao Lalam
Attachments: HIVE-1884.1.PATCH, HIVE-1884.2.patch, HIVE-1884.3.patch,
HIVE-1884.4.patch, HIVE-1884.5.patch
h3.There are couple of resource leaks.
h4.For example,
In CliDriver.java, Method :- processReader() the buffered reader is not
closed.
h3.Also there are risk(s) of resource(s) getting leaked , in such cases we
need to re factor the code to move closing of resources in finally block.
h4. For Example :-
In Throttle.java Method:- checkJobTracker() , the following code snippet
might cause resource leak.
{code}
InputStream in = url.openStream();
in.read(buffer);
in.close();
{code}
Ideally and as per the best coding practices it should be like below
{code}
InputStream in=null;
try {
in = url.openStream();
int numRead = in.read(buffer);
}
finally {
IOUtils.closeStream(in);
}
{code}
Similar cases, were found in ExplainTask.java, DDLTask.java etc.Need to re
factor all such occurrences.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2086) Data loss with external table
[ https://issues.apache.org/jira/browse/HIVE-2086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067833#comment-13067833 ] jirapos...@reviews.apache.org commented on HIVE-2086: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/604/ --- (Updated 2011-07-19 16:50:07.742470) Review request for hive. Changes --- The bug was fixed as a part of HIVE-1218, and this diff adds some regression tests for this particular issue. Summary --- Review request for HIVE-2086. The external parameters were being set, but then being zeroed out by a call to params.clear(), resulting in an external table that wasn't marked as external. This addresses bug HIVE-2086. https://issues.apache.org/jira/browse/HIVE-2086 Diffs (updated) - build-common.xml b6807fa data/files/ext_test/test.dat PRE-CREATION ql/src/test/org/apache/hadoop/hive/ql/QTestUtil.java 6d742e5 ql/src/test/queries/clientpositive/create_like.q 3f8e58d ql/src/test/results/clientpositive/create_like.q.out 118235e Diff: https://reviews.apache.org/r/604/diff Testing --- Automated tests added Thanks, Jonathan Data loss with external table - Key: HIVE-2086 URL: https://issues.apache.org/jira/browse/HIVE-2086 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.7.0 Environment: Amazon elastics mapreduce cluster Reporter: Q Long Assignee: Jonathan Natkins Attachments: HIVE-2086.1.patch, HIVE-2086.2.patch, create_like.q.out Data loss when using create external table like statement. 1) Set up an external table S, point to location L. Populate data in S. 2) Create another external table T, using statement like this: create external table T like S location L Make sure table T point to the same location as the original table S. 3) Query table T, see the same set of data in S. 4) drop table T. 5) Query table S will return nothing, and location L is deleted. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2286) ClassCastException when building index with security.authorization turned on
[ https://issues.apache.org/jira/browse/HIVE-2286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067856#comment-13067856 ] jirapos...@reviews.apache.org commented on HIVE-2286: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1137/ --- (Updated 2011-07-19 17:37:34.434426) Review request for hive, John Sichi and Ning Zhang. Changes --- Add a set of helpers to save/restore query specific session state. As far as I know, there isn't really anything else command specific that needs to be done. The other change here is to add the resetTaskIds flag = false which I think should have been the case all along if we wanted these queries to be properly re-entrant. I guess in the case of ALTER INDEX, where nesting was not a big issue this was less likely to break? Summary --- Save the original HiveOperation/commandType when we generate the index builder task and restore it after we're done generating the task so that the authorization checks make the right decision when deciding what to do. This addresses bug HIVE-2286. https://issues.apache.org/jira/browse/HIVE-2286 Diffs (updated) - ql/src/test/results/clientpositive/index_bitmap.q.out 8f41ce3 ql/src/test/results/clientpositive/index_auto_unused.q.out 8fe3ac9 ql/src/test/results/clientpositive/index_auto_multiple.q.out 28add9f ql/src/test/results/clientpositive/index_auto_partitioned.q.out b332a8b ql/src/test/results/clientpositive/index_auto_file_format.q.out 8430296 ql/src/test/results/clientpositive/index_auto.q.out 222db34 ql/src/test/results/clientpositive/index_auth.q.out PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/Driver.java b278ffe ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5 ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7c91946 ql/src/test/queries/clientpositive/index_auth.q PRE-CREATION ql/src/test/results/clientnegative/index_compact_entry_limit.q.out fcb2673 ql/src/test/results/clientnegative/index_compact_size_limit.q.out fcb2673 ql/src/test/results/clientpositive/index_bitmap1.q.out 9f638f5 ql/src/test/results/clientpositive/index_bitmap2.q.out e901477 ql/src/test/results/clientpositive/index_bitmap3.q.out 8091359 ql/src/test/results/clientpositive/index_bitmap_auto.q.out e73877a ql/src/test/results/clientpositive/index_bitmap_auto_partitioned.q.out 4c9efd1 ql/src/test/results/clientpositive/index_bitmap_rc.q.out 9bd3c98 ql/src/test/results/clientpositive/index_compact.q.out c339ec9 ql/src/test/results/clientpositive/index_compact_1.q.out 34ba3ca ql/src/test/results/clientpositive/index_compact_2.q.out e8ce238 ql/src/test/results/clientpositive/index_compact_3.q.out d39556d ql/src/test/results/clientpositive/index_creation.q.out 532f07e Diff: https://reviews.apache.org/r/1137/diff Testing --- Added new testcase to TestCliDriver: index_auth.q Thanks, Syed ClassCastException when building index with security.authorization turned on Key: HIVE-2286 URL: https://issues.apache.org/jira/browse/HIVE-2286 Project: Hive Issue Type: Bug Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2286.1.patch, HIVE-2286.2.patch When trying to build an index with authorization checks turned on, hive issues the following ClassCastException: org.apache.hadoop.hive.ql.parse.DDLSemanticAnalyzer cannot be cast to org.apache.hadoop.hive.ql.parse.SemanticAnalyzer at org.apache.hadoop.hive.ql.Driver.doAuthorization(Driver.java:540) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:431) at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:335) at org.apache.hadoop.hive.ql.Driver.run(Driver.java:848) at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:224) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:358) at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:293) at org.apache.hadoop.hive.cli.CliDriver.processReader(CliDriver.java:385) at org.apache.hadoop.hive.cli.CliDriver.processFile(CliDriver.java:392) at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:567) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.jav a:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessor Impl.java:25) at
[jira] [Commented] (HIVE-2156) Improve error messages emitted during task execution
[ https://issues.apache.org/jira/browse/HIVE-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067858#comment-13067858 ] jirapos...@reviews.apache.org commented on HIVE-2156: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/777/ --- (Updated 2011-07-19 17:42:08.908937) Review request for hive and John Sichi. Changes --- Rebased patch against latest trunk Summary --- - Add local error messages to point to job logs and provide TaskIDs - Add a timeout to the fetching of task logs and errors This addresses bug HIVE-2156. https://issues.apache.org/jira/browse/HIVE-2156 Diffs (updated) - build-common.xml b6807fa common/src/java/org/apache/hadoop/hive/conf/HiveConf.java d29cc77 conf/hive-default.xml 0ca7085 ql/build.xml 449b47a ql/src/java/org/apache/hadoop/hive/ql/exec/HadoopJobExecHelper.java 40d2644 ql/src/java/org/apache/hadoop/hive/ql/exec/JobDebugger.java PRE-CREATION ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 53769a0 ql/src/java/org/apache/hadoop/hive/ql/exec/MapredLocalTask.java 691f038 ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cd3de76 ql/src/test/queries/clientnegative/minimr_broken_pipe.q PRE-CREATION ql/src/test/results/clientnegative/dyn_part3.q.out 5f4df65 ql/src/test/results/clientnegative/index_compact_entry_limit.q.out fcb2673 ql/src/test/results/clientnegative/index_compact_size_limit.q.out fcb2673 ql/src/test/results/clientnegative/minimr_broken_pipe.q.out PRE-CREATION ql/src/test/results/clientnegative/script_broken_pipe1.q.out d33d2cc ql/src/test/results/clientnegative/script_broken_pipe2.q.out afbaa44 ql/src/test/results/clientnegative/script_broken_pipe3.q.out fe8f757 ql/src/test/results/clientnegative/script_error.q.out c72d780 ql/src/test/results/clientnegative/udf_reflect_neg.q.out f2082a3 ql/src/test/results/clientnegative/udf_test_error.q.out 5fd9a00 ql/src/test/results/clientnegative/udf_test_error_reduce.q.out ddc5e5b ql/src/test/templates/TestNegativeCliDriver.vm ec13f79 Diff: https://reviews.apache.org/r/777/diff Testing --- Tested TestNegativeCliDriver in both local and miniMR mode Thanks, Syed Improve error messages emitted during task execution Key: HIVE-2156 URL: https://issues.apache.org/jira/browse/HIVE-2156 Project: Hive Issue Type: Improvement Reporter: Syed S. Albiz Assignee: Syed S. Albiz Attachments: HIVE-2156.1.patch, HIVE-2156.2.patch, HIVE-2156.4.patch Follow-up to HIVE-1731 A number of issues were related to reporting errors from task execution and surfacing these in a more useful form. Currently a cryptic message with Execution Error and a return code and class name of the task is emitted. The most useful log messages here are emitted to the local logs, which can be found through jobtracker. Having either a pointer to these logs as part of the error message or the actual content would improve the usefulness substantially. It may also warrant looking into how the underlying error reporting through Hadoop is done and if more information can be propagated up from there. Specific issues raised in HIVE-1731: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.MapRedTask * issue was in regexp_extract syntax FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask * tried: desc table_does_not_exist; -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2128) Automatic Indexing with multiple tables
[ https://issues.apache.org/jira/browse/HIVE-2128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067878#comment-13067878 ] jirapos...@reviews.apache.org commented on HIVE-2128: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1010/#review1112 --- ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java https://reviews.apache.org/r/1010/#comment2271 Why was this comment truncated? ql/src/test/queries/clientpositive/index_auto_mult_tables.q https://reviews.apache.org/r/1010/#comment2273 All of these SELECT statements need ORDER BY for determinism. - John On 2011-07-19 03:15:17, Syed Albiz wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1010/ bq. --- bq. bq. (Updated 2011-07-19 03:15:17) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. Grab the indexed tables during optimized query generation, grab the associated path URIs, and keep those around in the Configuration object. When the job is passed to ExecDriver, this data is extracted and used in HiveIndexedInputFormat to decide whether to use the index file or delegate to the parent (HiveInputFormat) class. Not sure if this is robust. bq. bq. bq. This addresses bug HIVE-2128. bq. https://issues.apache.org/jira/browse/HIVE-2128 bq. bq. bq. Diffs bq. - bq. bq.ql/src/test/results/clientpositive/index_auto_self_join.q.out PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto_mult_tables_compact.q.out PRE-CREATION bq.ql/src/test/queries/clientpositive/index_auto_self_join.q PRE-CREATION bq.ql/src/test/results/clientpositive/index_auto_mult_tables.q.out PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/plan/MapredWork.java a03a9a6 bq.ql/src/test/queries/clientpositive/index_auto_mult_tables.q PRE-CREATION bq.ql/src/test/queries/clientpositive/index_auto_mult_tables_compact.q PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexResult.java b9b586e bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexedInputFormat.java f1ee95d bq. ql/src/java/org/apache/hadoop/hive/ql/index/bitmap/BitmapIndexHandler.java 61bbbf5 bq. ql/src/java/org/apache/hadoop/hive/ql/index/compact/CompactIndexHandler.java 7c91946 bq. ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/index/IndexWhereProcessor.java dbc489f bq.ql/src/java/org/apache/hadoop/hive/ql/index/HiveIndexQueryContext.java 617723e bq. bq. Diff: https://reviews.apache.org/r/1010/diff bq. bq. bq. Testing bq. --- bq. bq. added new testcase index_auto_mult_tables.q bq. bq. bq. Thanks, bq. bq. Syed bq. bq. Automatic Indexing with multiple tables --- Key: HIVE-2128 URL: https://issues.apache.org/jira/browse/HIVE-2128 Project: Hive Issue Type: Improvement Components: Indexing Affects Versions: 0.8.0 Reporter: Russell Melick Assignee: Syed S. Albiz Attachments: HIVE-2128.1.patch, HIVE-2128.1.patch, HIVE-2128.2.patch, HIVE-2128.4.patch, HIVE-2128.5.patch Make automatic indexing work with jobs which access multiple tables. We'll probably need to modify the way that the index input format works in order to associate index formats/files with specific tables. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2139) Enables HiveServer to accept -hiveconf option
[
https://issues.apache.org/jira/browse/HIVE-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067933#comment-13067933
]
jirapos...@reviews.apache.org commented on HIVE-2139:
-
bq. On 2011-07-19 00:08:19, Carl Steinbach wrote:
bq. metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java,
line 3382
bq. https://reviews.apache.org/r/958/diff/2/?file=26007#file26007line3382
bq.
bq. This should be $METASTORE_PORT instead of HIVE_PORT (see
metastore.sh).
I missed that, thanks.
bq. On 2011-07-19 00:08:19, Carl Steinbach wrote:
bq. common/build.xml, line 30
bq. https://reviews.apache.org/r/958/diff/2/?file=26001#file26001line30
bq.
bq. This target overrides the definition of 'compile' in
build-common.xml, but neglects to include this action which copies over
hive-log4j.properties:
bq.
bq. copy todir=${build.classes} failonerror=false
bq. fileset dir=${src.dir}/conf/
bq. /copy
bq.
bq. If this is added in I think the missing hive-log4j.properties issue
will be resolved.
Ok -- FYI I originally copied this section from cli/build.xml, not
build-common.xml. (it's the same except for this missing copy)
- Patrick
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/958/#review1106
---
On 2011-07-15 23:51:55, Patrick Hunt wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/958/
bq. ---
bq.
bq. (Updated 2011-07-15 23:51:55)
bq.
bq.
bq. Review request for hive and Carl Steinbach.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. This patch updates HiveServer and HiveMetastore to add proper cli handling
- similar to that used in CliDriver (ie GnuParser).
bq.
bq. There's a common HiveCli class that's used by both main classes.
bq.
bq. I've attempted to make the cli's backward compatible with the prior
command line processing. Notice I've deprecated (via warnings, but the code
still runs) if the old style CLI usage is used.
bq.
bq. commands such as the following now work as expected:
bq.
bq. bin/hive --service hiveserver -t 200 -p 12000 --hiveconf
hive.root.logger=DEBUG,console
bq.
bq. as does the following which generates usage information:
bq.
bq. bin/hive --service hiveserver -h
bq.
bq. Note: HiveMetastore as not initializing log4j, I updated the code to do
similar to HiveServer (otw the hiveconf hive.root.logger option above didn't
work).
bq.
bq.
bq. This addresses bug HIVE-2139.
bq. https://issues.apache.org/jira/browse/HIVE-2139
bq.
bq.
bq. Diffs
bq. -
bq.
bq.ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
728958d
bq.service/src/java/org/apache/hadoop/hive/service/HiveServer.java ea04be9
bq.ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 090ecfc
bq.ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 61393c8
bq.bin/ext/hiveserver.sh b5edce4
bq.bin/ext/metastore.sh db15f6e
bq.cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java bd9f6b1
bq.common/build.xml 0969e16
bq.common/ivy.xml bee3bf0
bq.common/src/java/org/apache/hadoop/hive/common/LogUtils.java PRE-CREATION
bq.common/src/java/org/apache/hadoop/hive/common/cli/CommonCliOptions.java
PRE-CREATION
bq.hwi/src/java/org/apache/hadoop/hive/hwi/HWISessionItem.java 1bf8c97
bq.metastore/ivy.xml 83218fc
bq.metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
e540c63
bq.
bq. Diff: https://reviews.apache.org/r/958/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. I couldn't find any tests for these changes, so I verfied the changes
manually.
bq.
bq.
bq. Thanks,
bq.
bq. Patrick
bq.
bq.
Enables HiveServer to accept -hiveconf option
-
Key: HIVE-2139
URL: https://issues.apache.org/jira/browse/HIVE-2139
Project: Hive
Issue Type: Improvement
Components: CLI
Environment: Linux + CDH3u0 (Hive 0.7.0+27.1-2~lucid-cdh3)
Reporter: Kazuki Ohta
Assignee: Patrick Hunt
Attachments: HIVE-2139.patch, HIVE-2139.patch
Currently, I'm trying to test HiveHBaseIntegration on HiveServer. But it
doesn't seem to accept -hiveconf command.
{code}
hive --service hiveserver -hiveconf hbase.zookeeper.quorum=hdp0,hdp1,hdp2
Starting Hive Thrift Server
java.lang.NumberFormatException: For input string: -hiveconf
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
[jira] [Commented] (HIVE-2139) Enables HiveServer to accept -hiveconf option
[
https://issues.apache.org/jira/browse/HIVE-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067934#comment-13067934
]
jirapos...@reviews.apache.org commented on HIVE-2139:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/958/
---
(Updated 2011-07-19 20:15:07.278811)
Review request for hive and Carl Steinbach.
Changes
---
Updated with Carl's comments.
Summary
---
This patch updates HiveServer and HiveMetastore to add proper cli handling -
similar to that used in CliDriver (ie GnuParser).
There's a common HiveCli class that's used by both main classes.
I've attempted to make the cli's backward compatible with the prior command
line processing. Notice I've deprecated (via warnings, but the code still
runs) if the old style CLI usage is used.
commands such as the following now work as expected:
bin/hive --service hiveserver -t 200 -p 12000 --hiveconf
hive.root.logger=DEBUG,console
as does the following which generates usage information:
bin/hive --service hiveserver -h
Note: HiveMetastore as not initializing log4j, I updated the code to do similar
to HiveServer (otw the hiveconf hive.root.logger option above didn't work).
This addresses bug HIVE-2139.
https://issues.apache.org/jira/browse/HIVE-2139
Diffs (updated)
-
common/build.xml 0969e16
common/ivy.xml bee3bf0
common/src/java/org/apache/hadoop/hive/common/LogUtils.java PRE-CREATION
common/src/java/org/apache/hadoop/hive/common/cli/CommonCliOptions.java
PRE-CREATION
hwi/src/java/org/apache/hadoop/hive/hwi/HWISessionItem.java 1bf8c97
metastore/ivy.xml 83218fc
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
e540c63
ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 090ecfc
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 61393c8
ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java 728958d
service/src/java/org/apache/hadoop/hive/service/HiveServer.java ea04be9
cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java bd9f6b1
bin/ext/hiveserver.sh b5edce4
bin/ext/metastore.sh db15f6e
Diff: https://reviews.apache.org/r/958/diff
Testing
---
I couldn't find any tests for these changes, so I verfied the changes manually.
Thanks,
Patrick
Enables HiveServer to accept -hiveconf option
-
Key: HIVE-2139
URL: https://issues.apache.org/jira/browse/HIVE-2139
Project: Hive
Issue Type: Improvement
Components: CLI
Environment: Linux + CDH3u0 (Hive 0.7.0+27.1-2~lucid-cdh3)
Reporter: Kazuki Ohta
Assignee: Patrick Hunt
Attachments: HIVE-2139.patch, HIVE-2139.patch
Currently, I'm trying to test HiveHBaseIntegration on HiveServer. But it
doesn't seem to accept -hiveconf command.
{code}
hive --service hiveserver -hiveconf hbase.zookeeper.quorum=hdp0,hdp1,hdp2
Starting Hive Thrift Server
java.lang.NumberFormatException: For input string: -hiveconf
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:449)
at java.lang.Integer.parseInt(Integer.java:499)
at org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:382)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:186)
{code}
Therefore, you need to throw the query like set
hbase.zookeeper.quorum=hdp0,hdp1,hdp2 everytime. It's not convenient for
separating the configuration between server-side and client-side.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068111#comment-13068111 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/ --- (Updated 2011-07-20 01:01:53.872666) Review request for hive. Changes --- Fixed issues in above comment Summary --- https://issues.apache.org/jira/browse/HIVE-1078 This addresses bug HIVE-1078. https://issues.apache.org/jira/browse/HIVE-1078 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 Diff: https://reviews.apache.org/r/1058/diff Testing --- Passes unit tests Thanks, Charles CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch, HIVE-1078v6.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2209) Provide a way by which ObjectInspectorUtils.compare can be extended by the caller for comparing maps which are part of the object
[ https://issues.apache.org/jira/browse/HIVE-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13068128#comment-13068128 ] jirapos...@reviews.apache.org commented on HIVE-2209: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/879/ --- (Updated 2011-07-20 02:25:36.169590) Review request for hive and Yongqiang He. Summary --- Patch for HIVE-2209 This addresses bug HIVE-2209. https://issues.apache.org/jira/browse/HIVE-2209 Diffs - serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/CrossMapEqualComparer.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/FullMapEqualComparer.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/MapEqualComparer.java PRE-CREATION serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 2b77072 serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/SimpleMapEqualComparer.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestCrossMapEqualComparer.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestFullMapEqualComparer.java PRE-CREATION serde/src/test/org/apache/hadoop/hive/serde2/objectinspector/TestSimpleMapEqualComparer.java PRE-CREATION Diff: https://reviews.apache.org/r/879/diff Testing --- Tests added Thanks, Krishna Provide a way by which ObjectInspectorUtils.compare can be extended by the caller for comparing maps which are part of the object - Key: HIVE-2209 URL: https://issues.apache.org/jira/browse/HIVE-2209 Project: Hive Issue Type: Improvement Reporter: Krishna Kumar Assignee: Krishna Kumar Priority: Minor Attachments: HIVE-2209v0.patch, HIVE-2209v2.patch, HIVE2209v1.patch Now ObjectInspectorUtils.compare throws an exception if a map is contained (recursively) within the objects being compared. Two obvious implementations are - a simple map comparer which assumes keys of the first map can be used to fetch values from the second - a 'cross-product' comparer which compares every pair of key-value pairs in the two maps, and calls a match if and only if all pairs are matched Note that it would be difficult to provide a transitive greater-than/less-than indication with maps so that is not in scope. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066820#comment-13066820 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/ --- (Updated 2011-07-18 07:01:02.376155) Review request for hive. Changes --- Added back check: New view must contain all old partition columns Summary --- https://issues.apache.org/jira/browse/HIVE-1078 This addresses bug HIVE-1078. https://issues.apache.org/jira/browse/HIVE-1078 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1058/diff Testing --- Passes unit tests Thanks, Charles CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066818#comment-13066818 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/#review1089 --- http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/1058/#comment2223 Oops the comment should be here - Charles On 2011-07-18 07:01:02, Charles Chen wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1058/ bq. --- bq. bq. (Updated 2011-07-18 07:01:02) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. This addresses bug HIVE-1078. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1058/diff bq. bq. bq. Testing bq. --- bq. bq. Passes unit tests bq. bq. bq. Thanks, bq. bq. Charles bq. bq. CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066822#comment-13066822 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/#review1091 --- http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/1058/#comment2225 Here *** - Charles On 2011-07-18 07:01:02, Charles Chen wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1058/ bq. --- bq. bq. (Updated 2011-07-18 07:01:02) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. This addresses bug HIVE-1078. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1058/diff bq. bq. bq. Testing bq. --- bq. bq. Passes unit tests bq. bq. bq. Thanks, bq. bq. Charles bq. bq. CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066819#comment-13066819 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/#review1088 --- http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/1058/#comment Haha ok so now I remember why we need this: (unit test clientnegative/create_or_replace_view1.q): -- New view must contain all old partition columns. drop view v; create view v partitioned on (ds, hr) as select * from srcpart; create or replace view v as select key, hr from srcpart; Here, we don't explicitly change the partition columns, so the new view inherits the old columns, but those columns better still exist. - Charles On 2011-07-18 07:01:02, Charles Chen wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1058/ bq. --- bq. bq. (Updated 2011-07-18 07:01:02) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. This addresses bug HIVE-1078. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1058/diff bq. bq. bq. Testing bq. --- bq. bq. Passes unit tests bq. bq. bq. Thanks, bq. bq. Charles bq. bq. CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1218) CREATE TABLE t LIKE some_view should create a new empty base table, but instead creates a copy of view
[ https://issues.apache.org/jira/browse/HIVE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067159#comment-13067159 ] jirapos...@reviews.apache.org commented on HIVE-1218: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1057/#review1096 --- r9 passes unit tests - Charles On 2011-07-16 22:51:22, Charles Chen wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1057/ bq. --- bq. bq. (Updated 2011-07-16 22:51:22) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. https://issues.apache.org/jira/browse/HIVE-1218 bq. bq. bq. This addresses bug HIVE-1218. bq. https://issues.apache.org/jira/browse/HIVE-1218 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146883 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146883 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java 1146883 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_like.q 1146883 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_like_view.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_like.q.out 1146883 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_like_view.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1057/diff bq. bq. bq. Testing bq. --- bq. bq. Unit tests passed bq. bq. bq. Thanks, bq. bq. Charles bq. bq. CREATE TABLE t LIKE some_view should create a new empty base table, but instead creates a copy of view -- Key: HIVE-1218 URL: https://issues.apache.org/jira/browse/HIVE-1218 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Fix For: 0.8.0 Attachments: HIVE-1218v0.patch, HIVE-1218v2.patch, HIVE-1218v3.patch, HIVE-1218v4.patch, HIVE-1218v5.patch, HIVE-1218v6.patch, HIVE-1218v7.patch, HIVE-1218v8.patch I think it should copy only the column definitions from the view metadata. Currently it is copying the entire descriptor, resulting in a new view instead of a new base table. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067160#comment-13067160 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/#review1097 --- r7 passes unit tests - Charles On 2011-07-18 07:01:02, Charles Chen wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1058/ bq. --- bq. bq. (Updated 2011-07-18 07:01:02) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. This addresses bug HIVE-1078. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1058/diff bq. bq. bq. Testing bq. --- bq. bq. Passes unit tests bq. bq. bq. Thanks, bq. bq. Charles bq. bq. CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch, HIVE-1078v6.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2272) add TIMESTAMP data type
[ https://issues.apache.org/jira/browse/HIVE-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067222#comment-13067222 ] jirapos...@reviews.apache.org commented on HIVE-2272: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1135/ --- (Updated 2011-07-18 19:12:08.745755) Review request for hive. Summary (updated) --- Adds TIMESTAMP type to serde2 with both string (LazySimple) and binary (LazyBinary) serialization. Supports SQL style jdbc timestamps of the format with nanosecond precision -MM-DD HH:MM:SS[.fff...] This addresses bug HIVE-2272. https://issues.apache.org/jira/browse/HIVE-2272 Diffs - trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestamp.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestamp.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1146895 trunk/ql/src/test/results/clientpositive/timestamp_3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/timestamp_comparison.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/timestamp_udf.q.out PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java PRE-CREATION trunk/ql/src/test/results/clientpositive/timestamp_2.q.out PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToFloat.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToInteger.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToLong.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToShort.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFToString.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFUnixTimeStamp.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFWeekOfYear.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/UDFYear.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFAverage.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFContextNGrams.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCorrelation.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovariance.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFCovarianceSample.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFHistogramNumeric.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFPercentileApprox.java 1146895
[jira] [Commented] (HIVE-2272) add TIMESTAMP data type
[ https://issues.apache.org/jira/browse/HIVE-2272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067223#comment-13067223 ] jirapos...@reviews.apache.org commented on HIVE-2272: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1135/ --- (Updated 2011-07-18 19:13:31.962359) Review request for hive. Summary --- Adds TIMESTAMP type to serde2 with both string (LazySimple) and binary (LazyBinary) serialization. Supports SQL style jdbc timestamps of the format with nanosecond precision -MM-DD HH:MM:SS[.fff...] This addresses bug HIVE-2272. https://issues.apache.org/jira/browse/HIVE-2272 Diffs (updated) - trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/PrimitiveObjectInspector.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/JavaTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorConverter.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorFactory.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/PrimitiveObjectInspectorUtils.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/SettableTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/TimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/primitive/WritableTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinarySerDe.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryTimestamp.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryUtils.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorConverters.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/objectinspector/ObjectInspectorUtils.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazybinary/LazyBinaryFactory.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyTimestampObjectInspector.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/objectinspector/primitive/LazyPrimitiveObjectInspectorFactory.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyFactory.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyTimestamp.java PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyUtils.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/binarysortable/BinarySortableSerDe.java 1146895 trunk/serde/src/java/org/apache/hadoop/hive/serde2/io/TimestampWritable.java PRE-CREATION trunk/ql/src/test/results/clientpositive/timestamp_comparison.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/timestamp_udf.q.out PRE-CREATION trunk/serde/src/java/org/apache/hadoop/hive/serde2/SerDeUtils.java 1146895 trunk/ql/src/test/results/clientpositive/timestamp_3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/timestamp_2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/wrong_column_type.q.out 1146895 trunk/ql/src/test/results/clientpositive/show_functions.q.out 1146895 trunk/ql/src/test/results/clientpositive/timestamp_1.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_t_create2.q.out 1146895 trunk/ql/src/test/results/clientnegative/invalid_t_transform.q.out 1146895 trunk/ql/src/test/results/clientnegative/invalid_t_alter2.q.out 1146895 trunk/ql/src/test/results/clientnegative/invalid_t_create1.q.out 1146895 trunk/ql/src/test/queries/clientpositive/timestamp_2.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_comparison.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/timestamp_udf.q PRE-CREATION trunk/ql/src/test/results/clientnegative/invalid_create_tbl1.q.out 1146895 trunk/ql/src/test/results/clientnegative/invalid_t_alter1.q.out 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFStdSample.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFSum.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVariance.java 1146895 trunk/ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDAFVarianceSample.java 1146895
[jira] [Commented] (HIVE-2184) Few improvements in org.apache.hadoop.hive.ql.metadata.Hive.close()
[ https://issues.apache.org/jira/browse/HIVE-2184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067285#comment-13067285 ] jirapos...@reviews.apache.org commented on HIVE-2184: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1088/#review1099 --- trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java https://reviews.apache.org/r/1088/#comment2230 We should get rid of the member variable standAloneClient too since it is part of the dead code. Now that we'll start calling shutdown (which we were skipping before), what will be the effect? - John On 2011-07-12 14:06:28, chinna wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1088/ bq. --- bq. bq. (Updated 2011-07-12 14:06:28) bq. bq. bq. Review request for hive and John Sichi. bq. bq. bq. Summary bq. --- bq. bq. Few improvements in org.apache.hadoop.hive.ql.metadata.Hive.close() bq. bq. bq. This addresses bug HIVE-2184. bq. https://issues.apache.org/jira/browse/HIVE-2184 bq. bq. bq. Diffs bq. - bq. bq. trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1144190 bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1144190 bq. bq. Diff: https://reviews.apache.org/r/1088/diff bq. bq. bq. Testing bq. --- bq. bq. Ran all the testcases bq. bq. bq. Thanks, bq. bq. chinna bq. bq. Few improvements in org.apache.hadoop.hive.ql.metadata.Hive.close() --- Key: HIVE-2184 URL: https://issues.apache.org/jira/browse/HIVE-2184 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.5.0, 0.8.0 Environment: Hadoop 0.20.1, Hive0.8.0 and SUSE Linux Enterprise Server 10 SP2 (i586) - Kernel 2.6.16.60-0.21-smp (5) Reporter: Chinna Rao Lalam Assignee: Chinna Rao Lalam Attachments: HIVE-2184.1.patch, HIVE-2184.1.patch, HIVE-2184.patch 1)Hive.close() will call HiveMetaStoreClient.close() in this method the variable standAloneClient is never become true then client.shutdown() never call. 2)Hive.close() After calling metaStoreClient.close() need to make metaStoreClient=null -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1884) Potential risk of resource leaks in Hive
[
https://issues.apache.org/jira/browse/HIVE-1884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067286#comment-13067286
]
jirapos...@reviews.apache.org commented on HIVE-1884:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1098/#review1101
---
trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java
https://reviews.apache.org/r/1098/#comment2233
Rename this to rc to avoid confusion with the processReader method name.
trunk/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesWritableInput.java
https://reviews.apache.org/r/1098/#comment2234
Fix indentation for the try block.
- John
On 2011-07-13 14:28:47, chinna wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1098/
bq. ---
bq.
bq. (Updated 2011-07-13 14:28:47)
bq.
bq.
bq. Review request for hive and John Sichi.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. There are couple of resource leaks.
bq. For example,
bq. In CliDriver.java, Method :- processReader() the buffered reader is not
closed.
bq.
bq. Also there are risk(s) of resource(s) getting leaked , in such cases
refactored the code to move closing of resources in finally block.
bq.
bq.
bq. This addresses bug HIVE-1884.
bq. https://issues.apache.org/jira/browse/HIVE-1884
bq.
bq.
bq. Diffs
bq. -
bq.
bq.trunk/cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java 1145950
bq.
trunk/contrib/src/java/org/apache/hadoop/hive/contrib/util/typedbytes/TypedBytesWritableInput.java
1145950
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1145950
bq.trunk/ql/src/java/org/apache/hadoop/hive/ql/io/RCFileInputFormat.java
1145950
bq.
bq. Diff: https://reviews.apache.org/r/1098/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. All unit tests passed
bq.
bq.
bq. Thanks,
bq.
bq. chinna
bq.
bq.
Potential risk of resource leaks in Hive
Key: HIVE-1884
URL: https://issues.apache.org/jira/browse/HIVE-1884
Project: Hive
Issue Type: Bug
Components: CLI, Metastore, Query Processor, Server Infrastructure
Affects Versions: 0.3.0, 0.4.0, 0.4.1, 0.5.0, 0.6.0
Environment: Hive 0.6.0, Hadoop 0.20.1
SUSE Linux Enterprise Server 11 (i586)
Reporter: Mohit Sikri
Assignee: Chinna Rao Lalam
Attachments: HIVE-1884.1.PATCH, HIVE-1884.2.patch, HIVE-1884.3.patch,
HIVE-1884.4.patch
h3.There are couple of resource leaks.
h4.For example,
In CliDriver.java, Method :- processReader() the buffered reader is not
closed.
h3.Also there are risk(s) of resource(s) getting leaked , in such cases we
need to re factor the code to move closing of resources in finally block.
h4. For Example :-
In Throttle.java Method:- checkJobTracker() , the following code snippet
might cause resource leak.
{code}
InputStream in = url.openStream();
in.read(buffer);
in.close();
{code}
Ideally and as per the best coding practices it should be like below
{code}
InputStream in=null;
try {
in = url.openStream();
int numRead = in.read(buffer);
}
finally {
IOUtils.closeStream(in);
}
{code}
Similar cases, were found in ExplainTask.java, DDLTask.java etc.Need to re
factor all such occurrences.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067334#comment-13067334 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/#review1102 --- http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/1058/#comment2236 We'll allow this if no partitions currently exist. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java https://reviews.apache.org/r/1058/#comment2237 Shouldn't we allow for changes to the view comment? Also, any TBLPROPERTIES specified here should be applied to the view (but we shouldn't delete old ones, only overwrite them if they already exist). http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q https://reviews.apache.org/r/1058/#comment2235 These need explicit PARTITIONED ON clauses to indicate that the existing partitioning scheme is being preserved. Also, need test case for modifying the definition of an unpartitioned view. Plus new cases we discussed for allowing change in partitioning when no partitions currently exist. Also, negative case for combining IF NOT EXISTS with OR REPLACE. Also, negative case for updating the view to have an invalid definition. Also, positive case for using this to fix an invalid view. Have you tried creating a view cycle? - John On 2011-07-18 07:01:02, Charles Chen wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1058/ bq. --- bq. bq. (Updated 2011-07-18 07:01:02) bq. bq. bq. Review request for hive. bq. bq. bq. Summary bq. --- bq. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. This addresses bug HIVE-1078. bq. https://issues.apache.org/jira/browse/HIVE-1078 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION bq. http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1058/diff bq. bq. bq. Testing bq. --- bq. bq. Passes unit tests bq. bq. bq. Thanks, bq. bq. Charles bq. bq. CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi
[jira] [Commented] (HIVE-2139) Enables HiveServer to accept -hiveconf option
[
https://issues.apache.org/jira/browse/HIVE-2139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13067411#comment-13067411
]
jirapos...@reviews.apache.org commented on HIVE-2139:
-
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/958/#review1106
---
common/build.xml
https://reviews.apache.org/r/958/#comment2245
This target overrides the definition of 'compile' in build-common.xml, but
neglects to include this action which copies over hive-log4j.properties:
copy todir=${build.classes} failonerror=false
fileset dir=${src.dir}/conf/
/copy
If this is added in I think the missing hive-log4j.properties issue will be
resolved.
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
https://reviews.apache.org/r/958/#comment2244
This should be $METASTORE_PORT instead of HIVE_PORT (see metastore.sh).
- Carl
On 2011-07-15 23:51:55, Patrick Hunt wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/958/
bq. ---
bq.
bq. (Updated 2011-07-15 23:51:55)
bq.
bq.
bq. Review request for hive and Carl Steinbach.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. This patch updates HiveServer and HiveMetastore to add proper cli handling
- similar to that used in CliDriver (ie GnuParser).
bq.
bq. There's a common HiveCli class that's used by both main classes.
bq.
bq. I've attempted to make the cli's backward compatible with the prior
command line processing. Notice I've deprecated (via warnings, but the code
still runs) if the old style CLI usage is used.
bq.
bq. commands such as the following now work as expected:
bq.
bq. bin/hive --service hiveserver -t 200 -p 12000 --hiveconf
hive.root.logger=DEBUG,console
bq.
bq. as does the following which generates usage information:
bq.
bq. bin/hive --service hiveserver -h
bq.
bq. Note: HiveMetastore as not initializing log4j, I updated the code to do
similar to HiveServer (otw the hiveconf hive.root.logger option above didn't
work).
bq.
bq.
bq. This addresses bug HIVE-2139.
bq. https://issues.apache.org/jira/browse/HIVE-2139
bq.
bq.
bq. Diffs
bq. -
bq.
bq.ql/src/test/org/apache/hadoop/hive/ql/history/TestHiveHistory.java
728958d
bq.service/src/java/org/apache/hadoop/hive/service/HiveServer.java ea04be9
bq.ql/src/java/org/apache/hadoop/hive/ql/exec/ExecDriver.java 090ecfc
bq.ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 61393c8
bq.bin/ext/hiveserver.sh b5edce4
bq.bin/ext/metastore.sh db15f6e
bq.cli/src/java/org/apache/hadoop/hive/cli/CliDriver.java bd9f6b1
bq.common/build.xml 0969e16
bq.common/ivy.xml bee3bf0
bq.common/src/java/org/apache/hadoop/hive/common/LogUtils.java PRE-CREATION
bq.common/src/java/org/apache/hadoop/hive/common/cli/CommonCliOptions.java
PRE-CREATION
bq.hwi/src/java/org/apache/hadoop/hive/hwi/HWISessionItem.java 1bf8c97
bq.metastore/ivy.xml 83218fc
bq.metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
e540c63
bq.
bq. Diff: https://reviews.apache.org/r/958/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. I couldn't find any tests for these changes, so I verfied the changes
manually.
bq.
bq.
bq. Thanks,
bq.
bq. Patrick
bq.
bq.
Enables HiveServer to accept -hiveconf option
-
Key: HIVE-2139
URL: https://issues.apache.org/jira/browse/HIVE-2139
Project: Hive
Issue Type: Improvement
Components: CLI
Environment: Linux + CDH3u0 (Hive 0.7.0+27.1-2~lucid-cdh3)
Reporter: Kazuki Ohta
Assignee: Patrick Hunt
Attachments: HIVE-2139.patch, HIVE-2139.patch
Currently, I'm trying to test HiveHBaseIntegration on HiveServer. But it
doesn't seem to accept -hiveconf command.
{code}
hive --service hiveserver -hiveconf hbase.zookeeper.quorum=hdp0,hdp1,hdp2
Starting Hive Thrift Server
java.lang.NumberFormatException: For input string: -hiveconf
at
java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
at java.lang.Integer.parseInt(Integer.java:449)
at java.lang.Integer.parseInt(Integer.java:499)
at org.apache.hadoop.hive.service.HiveServer.main(HiveServer.java:382)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
[jira] [Commented] (HIVE-2247) ALTER TABLE RENAME PARTITION
[ https://issues.apache.org/jira/browse/HIVE-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066756#comment-13066756 ] jirapos...@reviews.apache.org commented on HIVE-2247: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1105/ --- (Updated 2011-07-18 00:10:53.957462) Review request for Siying Dong. Changes --- Change thrift interface to void rename_partition(1:string dbname, 2:string name, 3:liststring part_vals, 4:liststring newpart_vals) Rename the data directory of old partition to that of new partition and change location/name parameter of partition metadata use Alter_partition to update the partition metadata Summary --- Implement ALTER TABLE PARTITION RENAME function to rename a partition. Add HiveQL syntax ALTER TABLE bar PARTITION (k1='v1', k2='v2') RENAME TO PARTITION (k1='v3', k2='v4'); This is my first Hive diff, I just learn everything from existing codebase and may not have a good understanding on it. Feel free to inform me if I make something wrong. Thanks This addresses bug HIVE-2247. https://issues.apache.org/jira/browse/HIVE-2247 Diffs (updated) - trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 1145366 trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java 1145366 trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1145366 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote 1145366 trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py 1145366 trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure2.q PRE-CREATION trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure3.q PRE-CREATION trunk/ql/src/test/queries/clientpositive/alter_rename_partition.q PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure2.q.out PRE-CREATION trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure3.q.out PRE-CREATION trunk/ql/src/test/results/clientpositive/alter_rename_partition.q.out PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RenamePartitionDesc.java PRE-CREATION trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzerFactory.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/DDLWork.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 1145366 trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1145366 trunk/metastore/if/hive_metastore.thrift 1145366 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h 1145366 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp 1145366 trunk/metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp 1145366 trunk/metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/ThriftHiveMetastore.java 1145366 trunk/metastore/src/gen/thrift/gen-php/hive_metastore/ThriftHiveMetastore.php 1145366 Diff: https://reviews.apache.org/r/1105/diff Testing --- Add a partition A in the table Rename partition A to partition B Show the partitions in the table, it returns partition B. SELECT the data from partition A, it returns no results SELECT the data from partition B, it returns the data originally stored in partition A Thanks, Weiyan ALTER TABLE RENAME PARTITION Key: HIVE-2247 URL: https://issues.apache.org/jira/browse/HIVE-2247 Project: Hive Issue Type: New Feature Reporter: Siying Dong Assignee: Weiyan Wang Attachments: HIVE-2247.3.patch.txt, HIVE-2247.4.patch.txt We need a ALTER TABLE TABLE RENAME PARTITIONfunction that
[jira] [Commented] (HIVE-2247) ALTER TABLE RENAME PARTITION
[
https://issues.apache.org/jira/browse/HIVE-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066757#comment-13066757
]
jirapos...@reviews.apache.org commented on HIVE-2247:
-
bq. On 2011-07-13 23:58:21, Siying Dong wrote:
bq. trunk/metastore/if/hive_metastore.thrift, line 272
bq. https://reviews.apache.org/r/1105/diff/1/?file=25058#file25058line272
bq.
bq. Do we really need a new metastore function? Can we just use
alter_partition()?
alter_partition() just updates the partition metadata and neither check whether
the new partition exists nor do physical rename of data directory. It's
basically used by alter_table when there is no data in the old partition.
bq. On 2011-07-13 23:58:21, Siying Dong wrote:
bq.
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java,
line 1704
bq. https://reviews.apache.org/r/1105/diff/1/?file=25075#file25075line1704
bq.
bq. why we still continue when tab is null?
Fixed, throw an exception when tab is null.
bq. On 2011-07-13 23:58:21, Siying Dong wrote:
bq. trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/HiveOperation.java,
line 39
bq. https://reviews.apache.org/r/1105/diff/1/?file=25080#file25080line39
bq.
bq. Privilege.ALTER_METADATA doesn't sound right to me here. Can you use
Privilede.DROP and Priviledge.CREATE?
Change it to ALTERTABLE_RENAMEPART(ALTERTABLE_RENAMEPART, new
Privilege[]{Privilege.DROP}, new Privilege[]{Privilege.CREATE})
Not sure if it sounds right to you.
bq. On 2011-07-13 23:58:21, Siying Dong wrote:
bq.
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RenamePartitionDesc.java, line
1
bq. https://reviews.apache.org/r/1105/diff/1/?file=25081#file25081line1
bq.
bq. Looks like you copied from the file AddPartitionDesc, but do you
need all those class members for renaming partition?
Only keep necessary functions
- Weiyan
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1105/#review1052
---
On 2011-07-18 00:10:53, Weiyan Wang wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/1105/
bq. ---
bq.
bq. (Updated 2011-07-18 00:10:53)
bq.
bq.
bq. Review request for Siying Dong.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Implement ALTER TABLE PARTITION RENAME function to rename a partition.
bq. Add HiveQL syntax ALTER TABLE bar PARTITION (k1='v1', k2='v2') RENAME TO
PARTITION (k1='v3', k2='v4');
bq. This is my first Hive diff, I just learn everything from existing codebase
and may not have a good understanding on it.
bq. Feel free to inform me if I make something wrong. Thanks
bq.
bq.
bq. This addresses bug HIVE-2247.
bq. https://issues.apache.org/jira/browse/HIVE-2247
bq.
bq.
bq. Diffs
bq. -
bq.
bq.
trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
1145366
bq.
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
1145366
bq.trunk/metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java
1145366
bq.
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/IMetaStoreClient.java
1145366
bq.
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStoreClient.java
1145366
bq.
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java
1145366
bq.
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
1145366
bq.trunk/metastore/src/gen/thrift/gen-rb/thrift_hive_metastore.rb 1145366
bq.
trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore-remote
1145366
bq.
trunk/metastore/src/gen/thrift/gen-py/hive_metastore/ThriftHiveMetastore.py
1145366
bq.
trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure.q
PRE-CREATION
bq.
trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure2.q
PRE-CREATION
bq.
trunk/ql/src/test/queries/clientnegative/alter_rename_partition_failure3.q
PRE-CREATION
bq.trunk/ql/src/test/queries/clientpositive/alter_rename_partition.q
PRE-CREATION
bq.
trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure.q.out
PRE-CREATION
bq.
trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure2.q.out
PRE-CREATION
bq.
trunk/ql/src/test/results/clientnegative/alter_rename_partition_failure3.q.out
PRE-CREATION
bq.trunk/ql/src/test/results/clientpositive/alter_rename_partition.q.out
PRE-CREATION
bq.
trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/RenamePartitionDesc.java
PRE-CREATION
bq.
[jira] [Commented] (HIVE-1218) CREATE TABLE t LIKE some_view should create a new empty base table, but instead creates a copy of view
[ https://issues.apache.org/jira/browse/HIVE-1218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066550#comment-13066550 ] jirapos...@reviews.apache.org commented on HIVE-1218: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1057/ --- (Updated 2011-07-16 22:51:22.078393) Review request for hive. Changes --- Regenerated unit tests that had EXTERNAL : FALSE Summary --- https://issues.apache.org/jira/browse/HIVE-1218 This addresses bug HIVE-1218. https://issues.apache.org/jira/browse/HIVE-1218 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146883 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146883 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateTableLikeDesc.java 1146883 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_like.q 1146883 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_like_view.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_like.q.out 1146883 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_like_view.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1057/diff Testing --- Unit tests passed Thanks, Charles CREATE TABLE t LIKE some_view should create a new empty base table, but instead creates a copy of view -- Key: HIVE-1218 URL: https://issues.apache.org/jira/browse/HIVE-1218 Project: Hive Issue Type: Bug Components: Metastore Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Fix For: 0.8.0 Attachments: HIVE-1218v0.patch, HIVE-1218v2.patch, HIVE-1218v3.patch, HIVE-1218v4.patch, HIVE-1218v5.patch, HIVE-1218v6.patch, HIVE-1218v7.patch, HIVE-1218v8.patch I think it should copy only the column definitions from the view metadata. Currently it is copying the entire descriptor, resulting in a new view instead of a new base table. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2284) bucketized map join should allow join key as a superset of bucketized columns
[ https://issues.apache.org/jira/browse/HIVE-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066080#comment-13066080 ] jirapos...@reviews.apache.org commented on HIVE-2284: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1136/ --- Review request for hive and namit jain. Summary --- Allow bucketed mapjoin if join key is a superset of bucket columns. This addresses bug HIVE-2284. https://issues.apache.org/jira/browse/HIVE-2284 Diffs - trunk/contrib/build.xml 1146922 trunk/eclipse-templates/.classpath 1146922 trunk/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketMapJoinOptimizer.java 1146922 trunk/ql/src/test/queries/clientpositive/smb_mapjoin_10.q PRE-CREATION trunk/ql/src/test/results/clientpositive/smb_mapjoin_10.q.out PRE-CREATION Diff: https://reviews.apache.org/r/1136/diff Testing --- passed all unit tests. Thanks, Ning bucketized map join should allow join key as a superset of bucketized columns - Key: HIVE-2284 URL: https://issues.apache.org/jira/browse/HIVE-2284 Project: Hive Issue Type: Improvement Reporter: Ning Zhang Assignee: Ning Zhang Fix For: 0.8.0 Attachments: HIVE-2284.patch Currently bucketized mapjoin only allow the join keys being exactly the same as bucketized columns. This is too restrictive and is missing some optimization opportunities. If tables S and T are both bucketized on column A with the same # of buckets, and the query is something like: code SELECT /*+ MAPJOIN (S) */ ... FROM S join T ON (S.A = T.A AND S.B = T.B) code We should allow bucketized mapjoin since it's straightforward that bucket 1 from S join with bucket 2 from T on such join condition must be empty. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2282) Local mode needs to work well with block sampling
[ https://issues.apache.org/jira/browse/HIVE-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066083#comment-13066083 ] jirapos...@reviews.apache.org commented on HIVE-2282: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1132/#review1080 --- ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java https://reviews.apache.org/r/1132/#comment2209 This function name seems to be confusing. Looks like the input size is set even if there is no sampling, right? Also, can you add comments to this function? Other than that, the patch looks OK. - Siying On 2011-07-15 02:16:34, Kevin Wilfong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1132/ bq. --- bq. bq. (Updated 2011-07-15 02:16:34) bq. bq. bq. Review request for hive and Siying Dong. bq. bq. bq. Summary bq. --- bq. bq. A query should run in local mode when block sampling is used and the sample is small enough. The size of the sample is currently being estimated, as it is done to estimate the number of reducers. bq. bq. bq. This addresses bug HIVE-2282. bq. https://issues.apache.org/jira/browse/HIVE-2282 bq. bq. bq. Diffs bq. - bq. bq.ql/src/test/queries/clientpositive/sample_islocalmode_hook.q PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 53769a0 bq.ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cd3de76 bq.ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyIsLocalModeHook.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1132/diff bq. bq. bq. Testing bq. --- bq. bq. TestCliDriver TestNegativeCliDriver, manually tested bq. bq. bq. Thanks, bq. bq. Kevin bq. bq. Local mode needs to work well with block sampling - Key: HIVE-2282 URL: https://issues.apache.org/jira/browse/HIVE-2282 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Kevin Wilfong Attachments: HIVE-2282.1.patch.txt Currently, if block sampling is enabled and large set of data are sampled to a small set, local mode needs to be kicked in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2282) Local mode needs to work well with block sampling
[ https://issues.apache.org/jira/browse/HIVE-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066086#comment-13066086 ] jirapos...@reviews.apache.org commented on HIVE-2282: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1132/#review1081 --- ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyIsLocalModeHook.java https://reviews.apache.org/r/1132/#comment2210 We need a header for licensing. - Siying On 2011-07-15 02:16:34, Kevin Wilfong wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/1132/ bq. --- bq. bq. (Updated 2011-07-15 02:16:34) bq. bq. bq. Review request for hive and Siying Dong. bq. bq. bq. Summary bq. --- bq. bq. A query should run in local mode when block sampling is used and the sample is small enough. The size of the sample is currently being estimated, as it is done to estimate the number of reducers. bq. bq. bq. This addresses bug HIVE-2282. bq. https://issues.apache.org/jira/browse/HIVE-2282 bq. bq. bq. Diffs bq. - bq. bq.ql/src/test/queries/clientpositive/sample_islocalmode_hook.q PRE-CREATION bq.ql/src/java/org/apache/hadoop/hive/ql/exec/MapRedTask.java 53769a0 bq.ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java cd3de76 bq.ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyIsLocalModeHook.java PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/1132/diff bq. bq. bq. Testing bq. --- bq. bq. TestCliDriver TestNegativeCliDriver, manually tested bq. bq. bq. Thanks, bq. bq. Kevin bq. bq. Local mode needs to work well with block sampling - Key: HIVE-2282 URL: https://issues.apache.org/jira/browse/HIVE-2282 Project: Hive Issue Type: Improvement Reporter: Siying Dong Assignee: Kevin Wilfong Attachments: HIVE-2282.1.patch.txt Currently, if block sampling is enabled and large set of data are sampled to a small set, local mode needs to be kicked in. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2252) Display a sample of partitions created when Fatal Error occurred due to too many partitioned created
[ https://issues.apache.org/jira/browse/HIVE-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066108#comment-13066108 ] jirapos...@reviews.apache.org commented on HIVE-2252: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1013/ --- (Updated 2011-07-15 18:20:41.419263) Review request for hive. Changes --- rebased on the current trunk Summary --- In dynamic partition inserts, if a mapper created too many partitions, a fatal error is raised and the job got killed. Sometimes the error is caused by data error and it will be helpful for users to debug if we display a sample of dynamic partitions generated. This addresses bug HIVE-2252. https://issues.apache.org/jira/browse/HIVE-2252 Diffs (updated) - trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java 1147261 Diff: https://reviews.apache.org/r/1013/diff Testing --- passed all unit tests. Thanks, Ning Display a sample of partitions created when Fatal Error occurred due to too many partitioned created Key: HIVE-2252 URL: https://issues.apache.org/jira/browse/HIVE-2252 Project: Hive Issue Type: Improvement Reporter: Ning Zhang Assignee: Ning Zhang Attachments: HIVE-2252.patch In dynamic partition inserts, if a mapper created too many partitions, a fatal error is raised and the job got killed. Sometimes the error is caused by data error and it will be helpful for users to debug if we display a sample of dynamic partitions generated. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-1078) CREATE VIEW followup: CREATE OR REPLACE
[ https://issues.apache.org/jira/browse/HIVE-1078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13066161#comment-13066161 ] jirapos...@reviews.apache.org commented on HIVE-1078: - --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/1058/ --- (Updated 2011-07-15 19:58:33.134187) Review request for hive. Changes --- Alter view in place Summary --- https://issues.apache.org/jira/browse/HIVE-1078 This addresses bug HIVE-1078. https://issues.apache.org/jira/browse/HIVE-1078 Diffs (updated) - http://svn.apache.org/repos/asf/hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/Hive.g 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java 1146902 http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view1.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view2.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view3.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientnegative/create_or_replace_view4.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/queries/clientpositive/create_or_replace_view.q PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view1.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view2.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view3.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientnegative/create_or_replace_view4.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_or_replace_view.q.out PRE-CREATION http://svn.apache.org/repos/asf/hive/trunk/ql/src/test/results/clientpositive/create_view.q.out 1146902 Diff: https://reviews.apache.org/r/1058/diff Testing --- Passes unit tests Thanks, Charles CREATE VIEW followup: CREATE OR REPLACE Key: HIVE-1078 URL: https://issues.apache.org/jira/browse/HIVE-1078 Project: Hive Issue Type: Improvement Components: Query Processor Affects Versions: 0.6.0 Reporter: John Sichi Assignee: Charles Chen Attachments: HIVE-1078v3.patch, HIVE-1078v4.patch, HIVE-1078v5.patch Currently, replacing a view requires DROP VIEW v; CREATE VIEW v AS new-definition; CREATE OR REPLACE would allow these to be combined into a single operation. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
